Limits of HTML5 developing a Hidden Object Game for Learning Languages

As production of "Pavel Piezo - Trip to the Kite Festival" draws to a close later this year I reviewed the material I collected for the Postmortem and found it too much and too diverse to put in one huge article. So, I identified the topics that can stand very well on their own, which were not limited to this specific game or production, and decided to write three short(er) posts in advance to the Postmortem.

[This post is cross posted from gamedev.net]

Setting Up Production

Earlier this year, after the game design and concept for "Pavel Piezo - Trip to the Kite Festival" were almost done and funding was secured, the crucial question had to be answered: Which game engine or programming system should be used for production?

Within our company, intolabs GmbH, the core team for this production consists only of two people, Sven Toepke as core programmer and me as game designer, producer, additional programmer, auxiliary art director and everything else. Sure, we would have external, outsourced help with artwork, audio, marketing and so on, but the core development and production would be split up between us two.

The game is to be released for tablets with iOS, later for tablets with Android and Windows and after that for Windows and Mac desktop.

A specific game engine springs to mind? Yes, it does.

But as we had virtually no budget we chose a different solution. We both had successfully done various projects with HTML5, Javascript, jQuery, Cordova / Phonegap etc. and earlier this year Adobe committed to the CreateJS suite to deal with canvas, sound, tweens, preloading and such. Since we had done some prototyping with this combination already, the decision was made to use this as the "game engine".

After all, it's just big static pictures with some sprites for items, a few animations and sounds, right? Well, yes and no.

Although the game does run well, even on our minimum-specification test device, we did hit some limits and road bumps that are worth noting for anyone who plans to delve into creating games with HTML5/JS/canvas.

Missing or Insufficient Libraries or Functions

This one's the easiest to convey as you may be already aware if you dabble in HTML5/JS. In particular, we missed particle effects and a more elaborate animation library. Sure, you can solve the problems yourself or use additional third-party libraries but although you can overcome all problems described in this article, it adds to the complexity, memory usage, programming effort and production time.

Particle Effects

Since we only needed one nice effect, instead of using an additional library for particle effects we opted to use sprite-animations with semi-transparent sprites. This is a valid solution and works well, but adds to memory management and overdraw (see below). No biggie, but using a particle system would have been easier.

Animations

Again, we opted to work with what is provided within the described combination of systems. In our case we were missing circular animations (clarification: I meant automatic animations along a curved path, we needed to animate along a circle). Sven programmed these himself, which is very doable with Javascript. However having this function within, say, CreateJS/TweenJS would have saved time. (You may find more elaborate functions for animations in libraries like melonJS, CAAT, Canvas Engine or impactJS.)

2D Overdraw in canvas

While this one wasn't bothering us too much, we could see the effect in early performance tests and countered its drag on frame rate at the very beginning. In case you are not familiar with the problem of overdraw, you can do a quick search here on Gamasutra, on gamedev.net, stackoverflow or simply by searchengine, but overdraw in 2D sprite-based games is quickly explained:

If the graphics of your sprites (objects) are not a perfectly filled rectangle, which they seldom are, you'll have transparent pixels around your sprite's graphic that fill up into the closest rectangle. This rectangle is your sprite (object) that you place and move around. Now, if you have overlapping objects which do overlap in the transparent areas, the renderer still has to calculate the visibility for every transparent pixel. If you have multiple overlapping objects/sprites this has to be calculated for every pixel in the Z (depth) order individually. This will add up, especially when animating. CreateJS/EaselJS allows you to group objects together in a container-object, which in turn can be cached, but again, this adds to complexity and programming effort.

Right after seeing the issues in our first tests we decided to preemptively cut background-layers with huge transparent areas into smaller objects and arrange them together on stage. Additionally, we changed sprites that were originally positioned partially behind a "background" object to be a rectangle that includes the pixels of the "background". That then allowed the sprite to be placed in front of the background layer, switching the sprite in question to a version with transparency only for animation.

Again, extra work.

Advanced 2D game engines provide additional, better techniques, such as putting sprites on non-rectangular meshes or using specific, well adapted renderers. These were not available in any HTML5/JS/canvas library we looked at. That may change with time but I found it's due to constraints in canvas and its implementations in different browsers itself.

Sound and FX

"Pavel Piezo - Trip to the Kite Festival" is relying heavily on sound. Each level has two different background loops, which are played out of alignment, sound effects for the GUI and game-events, plus there are a few different short spoken phrases for every item and active area in each level / picture. 

For instance, if you see in the GUI that you are to look for sunglasses, tapping on those in the GUI will play one of two or three short sentences, like "You need to find the sunglasses". Tapping on them in the picture (finding them), will again play one of two or three sentences, like "Nice! You found the sunglasses." Tapping on an item, which you are not to find (yet), in the picture will also play one of two or three sentences, like "Very useful, a toothbrush." Remember, Pavel Piezo is a game for getting to know foreign languages, so we maximize exposition to the vocabulary and phrases as well as to the association between the picture of an item and the spoken word.

The biggest problem we had with handling all these sound files is described in the passage about memory, see below, but there were additional quirks and annoyances. In the end our finding was that we had to cheat our way around obvious negligence in the implementations of sound in different versions of browsers, webkit or even the version of the underlying OS or hardware.

When preloading all sounds for a level proved to be too much for the memory, we switched to streaming the bigger and less frequently used files. However, depending on the system, that added up to 500ms latency before the sound started playing. This was not (only) limited to the least powerful system we tested with. The latency varied (seemingly random) between OS versions, etc.

The most bizarre glitch we found was that one target system didn't seem to like some of our MP3s and would cut them a few hundred milliseconds short upon playing. We tried re-encoding, checking the files meta-data as well as several other methods to identify the culprit. It wasn't the longest sounds, it wasn't ones with a specific length, it wasn't something we could identify in the meta-data; we didn't find the reason within our limited time. We could however reproduce the glitch, it was always the same ones that got cut short.

In the end we just checked the playback of all audio files and extended the ones that got cut with half a second of silence... more work.

[Addendum: User EDI asked why I didn't mention audio-sprites. I simply forgot, because we could see that memory constraints didn't allow for that solution very early in the process. It is a good solution, though, that might fit your production.]

Memory Management

"But Carsten, you are developing for mobile devices, you have to be aware of memory constraints right from the start!" you say and prepare to skip this paragraph.

“Yes”, I say, “we know”, I say, “we were”.

What we were not completely prepared for is how much developing a webkit-based application tightens the corset even further, despite having developed similar applications in the past. The webkit on our minimum-spec test device left us with little more than 200 MB of usable memory, of which Cordova and the Javascript-libraries ate a good 100 MB from the start.

We spent a good deal of time and effort optimizing the graphics, handling preloading and streaming, finding the highest compression for sounds while preserving the desired quality, cutting graphics to save on transparent areas (which eat up memory when displayed on stage) and utilizing other tricks mobile developers are well acquainted with.

While time consuming, it was still very doable, of course, as you should optimize the heck out of your application for mobile devices, regardless. But even with our min-spec system, using a game engine that does not run on top of webkit would have granted us more than double the memory to use.

Conclusion

All the small problems aside, HTML5/JS/canvas can be a very viable combination for your development and with Cordova/Phonegap, there are very few other ways to have your application cross-platform capable with that little effort.

Just be aware of the constraints that are still in place today.

Until full-fledged game engines like Unreal Engine or Unity3D become available to run on top of canvas there'll be a bunch of extra work and there's still the additional memory constraints to keep in mind. We feel that we have reached some limits of what is possible with "Pavel Piezo", especially on older devices that are still widely used. It's clear that we could still optimize further with much more tinkering. From the perspective of production, though, it's just more feasible to use a full-fledged system for cross-platform game development.

Still, in our opinion the combination HTML5/JS/jQuery/createJS/Cordova/Phonegap/... is the choice to make nifty, good looking cross-platform apps in record time. Just like modern HTML/CSS, the application doesn't have to look "html-y" and if the application consists mostly of logic, "screens" and some slick animations for transitions, popups, slide-ins etc. you can't beat the speed and ease of cross-platform development for a production of a certain scope. As we relied heavily on sound and tried to use as much animation as possible for "Pavel Piezo - Trip to the Kite Festival" we did hit a point where it became clear that pushing further with coming releases, the effort for optimization would be too high, compared to using a full-fledged game system.

Maybe that will change (again) in the future but for now we have a very good idea of the boundaries that exist, when to use HTML5 and when to use purpose-built tools and engines.