Phaser and Asset Revving

[davyboy]

Our company has developed, and is continuing to modify, a container console for wrapping around html5 games for mobile and desktop distribution, and one of the concerns we face, is to minimise the amount of time and traffic it takes for an end user to download the latest version of games, especially when it comes to bug-fixing of games. We are already working with a number of development houses to write exclusive content before we start opening this out to the wider audience (i.e. we're ironing out the bugs before we let others break it :-) ).

 

As part of this, we are looking at asset-revving. We find that between incremental fixes of games, the majority of the spritesheets, background images, and audio doesn't change, just the game logic in the JS files themselves. Some of these games can stretch to 10s of megabytes in size (especially when delivering for high definition devices like Retina's or Desktop machines), and having a user download all of these assets again, for a bug fix is not in their best interest, or ours as a distributor. Where filenames are being explicitly declared in javascript files, this is a relatively trivial task to conduct a replacement in the code prior to deployment onto webservers, but in the case of game engines like Phaser, this code is currently squirrelled away deep in the guts.

 

Where this becomes even more of a problem for us, is when the javascript does device detection, and dynamically generates the assets path based upon the device support (i.e. ogg vs m4a, or high definition images instead of low definition images). At this point, it becomes incumbent upon the developer to consciously to have to handle asset revving, by applying a mapping of their desired file name and path, to the actual file, if indeed that file has been revved (or if the revving has been conducted). I would like to get to the point where the game developers do not have to concern themselves directly with this, except for some preparation at the start of their game, to pass in any externally defined rev map to the game engine, and have the game engine itself conduct mapping of the content under the hood.

 

For example:

 

myGame.load.audio('default-background-music', 'audio/music/Default.mp3');

myGame.load.atlasJSONArray('texts', 'img/' + resolution +'/dialogues/texts.png', 'img/' + resolution + '/dialogues/texts.json');

myGame.load.atlasJSONArray('hint', 'img/' + resolution +'/animations/hints/tile-hint.png', 'img/' + resolution +'/animations/hints/tile-hint.json');

 

The first asset here is easy enough to rev out as it is explicitly defined, but the second and third assets are dynamically chosen by the variable resolution, and the files contained within each will have very different revved filenames, because their content will be different (for example texts.png for a hd resolution will be bigger and therefore have a different MD5 to texts.png for an sd resolution).

 

I could resolve this with a call to a rev mapping function, but this would be for every single line in the code where assets are obtained.

myGame.load.audio('default-background-music', getRevMapping('audio/music/Default.mp3'));

myGame.load.atlasJSONArray('texts', getRevMapping('img/' + resolution +'/dialogues/texts.png'), getRevMapping('img/' + resolution + '/dialogues/texts.json'));

myGame.load.atlasJSONArray('hint', getRevMapping('img/' + resolution +'/animations/hints/tile-hint.png'), 'img/' + resolution +'/animations/hints/tile-hint.json');

 

But this requires a lot more to be concerned with by the game developer, and is prone to error (note how in the third example I have accidentally not added the getRevMapping to the json file). Further, as a developer, I would have to write a rev mapping function, repeatedly for each game I develop. What would be better is if the game developer could continue to request the names of the files they are expecting, and have a small initial configuration prior to loading assets for each distributor as is required.

 

We have looked at this, and one method of approaching this is inside our library we expose to the developer, we can dynamically alter the low level Phaser.Loader.load function, so that we can re-map the files on the fly:

phaserRevMapReplace:function(game){

    var loader = game.load;

    var originalLoadFile = loader.loadFile;

    loader.loadFile = function() {

        var file = loader._fileList[loader._fileIndex];

        if (file) {

            file.url = map(file.url);

        }

        originalLoadFile.call(this);

    }

},

map:function(url){

    if (ourLibrary.revMap) {

        return ourLibrary.revMap;

    } else {

        return url;

    }

}

...

phaserRevMapReplace(myGame);

myGame.load.audio('default-background-music', 'audio/music/Default.mp3');

myGame.load.atlasJSONArray('texts', 'img/' + resolution +'/dialogues/texts.png', 'img/' + resolution + '/dialogues/texts.json');

myGame.load.atlasJSONArray('hint', 'img/' + resolution +'/animations/hints/tile-hint.png', 'img/' + resolution +'/animations/hints/tile-hint.json');

 

But the problem this makes, is that we end up tying our library code to a very specific set of versions of phaser, and any changes to phaser's loading mechanism will likely break our code.

 

Instead, I am going to fork Phaser, and suggest to merge changes back in to Phaser if approved, whereby the game developer will only need to pass any rev mapping object (a map object like revmap['audio/music/Default.mp3']='revved/audio/music/Default.abc123ef.mp3';) to the Phaser.Loader, and this will be used by the load method internally if defined, or silently work if not.

[davyboy]

Created a pull request:

 

https://github.com/photonstorm/phaser/pull/904

 

Assets are loaded like:

 

myGame.load.audio('default-background-music', 'audio/music/Default.mp3');
myGame.load.atlasJSONArray('texts', 'img/' + resolution +'/dialogues/texts.png', 'img/' + resolution + '/dialogues/texts.json');
myGame.load.atlasJSONArray('hint', 'img/' + resolution +'/animations/hints/tile-hint.png', 'img/' + resolution +'/animations/hints/tile-hint.json');

 

And supporting revving would be to take a mapping of files, and set the loader's revvingMapping to this mapping:

 

myGame.load.revvingMapping = JSON.parse('{"audio/music/Default.mp3":"revved/audio/music/Default.abc123ef.mp3","img/sd/dialogues/texts.png":"revved/img/sd/dialogues/texts.fed456ba.png","img/sd/dialogues/texts.json":"revved/img/sd/dialogues/texts.123bdc59.json"}');
myGame.load.audio('default-background-music', 'audio/music/Default.mp3');
myGame.load.atlasJSONArray('texts', 'img/' + resolution +'/dialogues/texts.png', 'img/' + resolution + '/dialogues/texts.json');
myGame.load.atlasJSONArray('hint', 'img/' + resolution +'/animations/hints/tile-hint.png', 'img/' + resolution +'/animations/hints/tile-hint.json');

 

Or alternatively, the revving mapping could have been provided by the distributor, through an API, and this is used here instead.

 

myGame.load.revvingMapping = myDistributorLibrary.getRevvingMapping();

myGame.load.audio('default-background-music', 'audio/music/Default.mp3');

myGame.load.atlasJSONArray('texts', 'img/' + resolution +'/dialogues/texts.png', 'img/' + resolution + '/dialogues/texts.json');

myGame.load.atlasJSONArray('hint', 'img/' + resolution +'/animations/hints/tile-hint.png', 'img/' + resolution +'/animations/hints/tile-hint.json');

[davyboy]

PR#904 was from the wrong branch. Created https://github.com/photonstorm/phaser/pull/905 from and to the dev branch.

[rich]

Thanks for updating the PR. I'll have a look at this more closely.

 

Can anyone here (other than davyboy obviously ) confirm if this is an actual requirement you've ever had from a site / client / sponsor or not please? It's certainly not one I've come across (I had never heard of 'revving' until reading this post to be honest).

[davyboy]

Adding information here (not to force a decision but to provide further understanding for the requirements).

 

We have three sets of people who have differing requirements:
* Game Developers - Care about developing great addictive content, care about users, don't need to care about how it's distributed

* Users - Care about great addictive games, want games to be delivered fast and free of bugs, but don't care about how it's distributed

* Distributors - Care about distributed content to users as fast as possible for the least amount of money.

 

As far as I can see, lots of public development is catered for game developers (hence the forum ), and how to make great games for users. The problem comes with how we go about distributing these games to users. If your game contains say 50Mb of assets at different resolutions and formats, and you have a bug in game content, you will want to patch the game's javascript files, send that out to the distributers and they will upload to their sites and the content is pushed to the users. However, what has actually changed is maybe 500kb of javascript, and the remaining 50mb remains the same.

 

Best practices in web delivery suggest setting cache expiry times on all content to be as long as possible, and use other methods of cache-busting when new content is required. This means that once the user's browser downloads content, any further requests needed for that same content should be cached locally and no download required. From a distributors point of view, this means we transmit less traffic (costs us less), and the user has a faster experience. Also from a distributors point of view, we want to deliver updated games as quickly to the user as possible. If a distributor doesn't provide a long cache, then they can always have one central point of truth for the user to download the latest game, but the user will be forced to download all the assets afresh.

Asset revving is being more heavily used by HTML applications (ruby applications for example) for this exact reason, incremental updates require incremental downloads by the user, not full content downloads.

 

I doubt that any game developer is currently facing this issue:

* it is seen as a distributors problem ("I wrote it, you deliver it")

* no game libraries yet support asset revving, so game developers (may be) unaware of the needs of the distributor

 

This PR provides a transparent solution insomuchas every single developer may never use this, and those that don't are unaffected by this change. Those that may require to consider asset revving will now have an injection to provide this mapping at the low level, without having to write boilerplate code around every single request. The desire here is to provide a way for game developers to have to concern themselves with asset revving as a throwaway concern. "Please Mr.Developer, can you add support to your game so we can revise the asset names to allow us to provide better more cost effective caching to the users" - "Sure, give me a map of my file names, to your filenames, and you'll have it in 5 minutes of coding"

 

Phaser isn't being picked on, I am looking at how we can achieve asset revving in other game engines as well, but Phaser is the first game engine we've had development with, so it's the first to face this issue.

 

In terms of real numbers, the following are tests we've carried out with revved assets inside the game (by hard-coding the hashed assets). The first column is the version of the game we are loading, the second column is how long it took to download the game to an initial state, from a completely clean cache, the third column is how long it takes to load that version when you have loaded the previous version of the game in the browser (and caching hashed assets), and the final column is how long it takes to reload that version (and should use browser cache as much as possible).

 

version | Initial Load (clean cache) | Initial Load (existing cache) | Reload

========+============================+===============================+=========

v_69    | 14.2s                      | n/a                           | 2.8s

v_70    | 11.2s                      | 4.2s                          | 3.4s

v_71    | 11.3s                      | 3.8s                          | 3.0s

v_72    | 11.1s                      | 6.9s                          | 2.0s

v_73    | 15.5s                      | 4.5s                          | 2.8s

v_74    | 11.2s                      | 3.7s                          | 4.6s

v_75    | 13.8s                      | 5.4s                          | 5.3s

 

We are seeing here 1/2 - 2/3 speed improvement to end users (and obviously a drop in traffic hitting our servers which costs us less). It should be noted that when asset revving is disabled, whenever we release a new version of a game, the second column is how long it takes a user to download a new version of the game as they don't have assets cached for that version of the game.

[rich]

Now that the Phaser.Loader supports json file based asset lists (rather than lots of separate calls) I'm pretty sure you can now achieve what you want by just dynamically creating that json object and passing it to the Loader.

 

It's not that I've anything against your PR, I really don't - and I understand the concept completely, it's something we used extensively in the Flash days - it's just I've never heard of the term before, and a Google search doesn't bring up anything significant either. As I reckon this can be done easily with the new Loader updates I'm tempted to let this one pass as-is.

[jouniii]

In fact I had something like this in discussion, not directly but talk about distribution and making game loads faster (especially on mobile). Right now the production games are just put into different sub-directory which makes all files different in browser's point of view. This is also to prevent browser issues with caching, which I noticed in development server. Sometimes browser would load old texture atlas image from cache instead of the updated. However game code references to updated json data and Phaser/PIXI fails to load the frames from the texture (errors like not texture not fitting, etc).