In the upcoming release of Making History II: The War of the World, Muzzy Lane Software is charting a new course for the company by bringing its 3D strategy games into the web browser, and surrounding them with features that make for a more engaging multiplayer experience.

MHII is being built on Sandstone, Muzzy Lane's new platform for the creation of fully web-integrated, 3D browser-based games with social networking features.

The same tech will also power Muzzy Lane's serious games projects, including the Clearlab Project, an initiative to develop games for middle-school science learning.

In building MHII, Muzzy Lane encountered a variety of engineering challenges. How best to render true 3D games in a web browser? How to approach backend services? How to handle content distribution?

This article will explore and discuss the challenges Muzzy Lane faced in implementing them. It will illuminate several of the key design decisions that constitute the Sandstone backbone, and which enable Making History and other Muzzy Lane games to provide dynamic player experiences.

Rendering 3D Games in a Browser

One of the first engineering challenges was determining the best way to run our 3D games in the browser. Truthfully, even though we were firmly committed to building web-based games, it did take some time to fully commit to the idea of games in the browser.

Originally, we had planned to build games that would just launch from a browser, and then run separately. In part, this was because it was easier to implement, and in part because that's just how 3D games have always been done -- full screen, as a separate application. Flash games are played in the browser, sure, but true hardcore 3D games?

However, the more we looked at web content like YouTube videos, podcasts, and e-book readers, the less sense that approach made. 3D games playable in the browser, with an option to go full-screen, just made sense.

Making History II: The War of the World, a browser-based, fully web-integrated WWII Strategy Game, built on Muzzy Lane Software's Sandstone platform. 

As it turned out, it was not all that difficult to render hardware accelerated 3D graphics in a browser window. To do so, the basic challenge (in Windows) that needs to be overcome is getting a handle to a window. (This is similar in Mac OS X or in Linux.)

Most 3D-in-the-browser solutions currently solve this issue by writing a browser plug-in. However, there are two drawbacks to this. First, many people distrust custom browser plug-ins (Flash and Java plug-ins being the exception.) Second, each browser plug-in is very browser specific. Even the Mozilla plug-in architecture, that many browsers support, still often needs to be slightly tweaked for each individual browser. This eventually becomes a maintenance nightmare.

Our solution has been to instead create a Java extension. In general, Java applets are run in a very secure sandbox that allows little access to the client machine. This often does not allow running native code, which is necessary for hardware acceleration.

The way around this is to create a Java extension which is installed on the client machine. The classes in the Java extension are assumed to be trusted because the end user installed them. These classes can then load and run native code without issue.

Once the Sandstone Player is installed, including the Java extension, an applet class from this extension is placed on a page. This applet can download whatever content it needs and save it to disk.

Once this content is downloaded, a call is made to C++ from Java which loads C++ code that was just downloaded, essentially loading the game engine. The engine is told which downloaded package to run, whether to connect to a server, etc. Most importantly, it is given a handle to the parent window which was created by the applet in Java and accessed from C++. This gives the engine a place to render, which is actually on the web page in the browser.

This solution, using Java as a shim between the browser and C++, has its own drawbacks. It is dependent on Java being installed on the client machine. While this is usually the case, when it is not, the end user has to install not just the Sandstone Player, but also the Java plug-in.

The Java extension mechanism ostensibly supports installing the extension on the fly if it does not exist, allowing it to be installed without restarting the browser. This always seemed like a great feature, allowing someone to go to a web page, and assuming they already have Java installed, be able to install the Sandstone Player, download the game, and play the game, all without having to restart the browser. This was one of the main reasons we chose this path in the first place.

In practice, it doesn't always work that way. The extension mechanism gives little or no feedback to the user about what it is doing when downloading an extension. To just get a progress bar on the screen quickly, you have to use a very small installer that then downloads the rest of what is being installed.

In addition, the Java plug-in has trouble in some environments running extension installers as administrator, even when all the hoops of UAC are jumped through properly. The result is that for the time being, the Sandstone Player is a manual installer that requires the browser to be restarted as part of installation.

On the plus side, though, the Java plug-in is dealing with all of the browser specific issues for us, so we end up only having to build and maintain something on a per-platform basis rather than on a per-browser basis. As a result, we've had the player running in many different browsers on Windows. IE and Firefox, even Chrome and Opera, have all worked without any special code or extra support.

The Sandstone platform supports true hardcore 3D games, like Making History II in the browser.

Running Backend Services

Early on, we realized that there would need to be a number of services running on the backend to support Sandstone. Each of these services potentially needs to talk to other services, and some of them need to be accessible by web servers that allow people to play our games. We chose to have most of the API for these services be HTTP requests.

This makes it easy for any web framework to talk to the Sandstone service, no matter what language it is written in. As long as it can make HTTP requests (which is a fundamental ability of most web frameworks), it can interact with the services.

Initially, most data was passed back and forth as XML. This proved to be fairly unwieldy, as most data was simple data, the most complicated parts being name-value pairs and lists.

Eventually, we changed over to use JSON for this data, which was a lot smaller than the equivalent XML and was much easier to parse in the languages we used.

Since these service APIs are implemented as HTTP requests, we needed to use a language that makes it easy to implement HTTP requests.

We chose to use Python instead of PHP or Ruby, because it is a very self-consistent language, has a lot of support for web services, and is easy to bridge to C++ code (which is very useful for running game servers).

Content Distribution

When it comes to running a game in a web page, rendering 3D graphics in a browser is not the real problem. Rather, a large part of the problem is getting the 3D art assets to the client machine in the first place. The art assets for 3D games are generally much larger than the art assets for more traditional browser-based 2D games.

From the beginning, we recognized that it would not be acceptable to download assets as needed, stored in memory. It has gotten to a point where waiting for some of the more complex Flash games to load is becoming unwieldy, and most 3D games would likely be larger than this.

Rather than come up with a way of compressing art assets to the point where the download and extract times would be acceptable (an extremely hard, if not impossible problem), we instead decided to try to build the Sandstone Player so that it made it possible to create better user experiences while waiting for the download.

We chose to write our downloader in Java, since we were already using Java in the Sandstone Player to bridge the gap between the browser and our C++ game engine. This also turned out to be easier than C++, because Java has much better library support for making HTTP requests, which is how the content is downloaded.

This downloader can be run as a separate applet from the applet that runs the game, and it can communicate with the page around it, so the page can do whatever it wants with this information.

It can put up a progress bar, it can let you chat with other people also waiting for the game to download, and it could even give you a small 2D minigame to play while you're waiting for the download to complete. The bottom line, it becomes more of a user experience question than a technical question at that point.

We made two other important decisions aimed at improving content distribution. First, we chose to break up our content in small, reusable chunks. And second, we chose to cache these chunks on the local disk of the client machine.

This means that if two scenarios use 99% of the same content, if you've already downloaded all the content to play scenario A, then you will already have 99% of the content it needed for scenario B cached on the local disk. Therefore, the scenario B download will be very small.

Next, we gave our content system two versioning systems. The first system simply changes the identifier of a content package, and is used for major upgrades of a package, when a change is made that will "break" other existing content.

For example, when code is changed that won't work with other modules calling it, or a 3D model is changed and is a completely different size so that it won't fit where it used to. When this is done, any other package that uses this package can manually change to use the new version.

This is useful because the new and old versions are identified differently, and they can coexist on a client machine at the same time. So if Scenario A uses Version 1 and Scenario B uses Version 2, a client machine can have both Version 1 and Version 2 on disk at the same time to play both Scenario A and Scenario B.

The other versioning system changes the revision of a package in place. This is used for non-breaking changes of a package, things like bug fixes. For example, a memory leak in code is fixed, or a 3D model is changed to fix a mesh that sometimes renders incorrectly. Updates of this type happen automatically when getting content, so the game you play stays up-to-date all the time.

The Making History Gaming Headquarters (GHQ) facilitates the sharing of user created content.

Game to Web Server Communication

As we started building services that talked to each other, we realized we needed to have some way of keeping track of what servers were up and available. For example, when a game is started, there needs to be one specific game server it is started on. What if there are several game servers running? How do we know what game servers are running to choose between them?

That was the original mandate of MISSIVE, our inter-service communication system. We needed a way to have services talk to each other, be aware that these services were out there, and send simple messages to each other synchronously.

A predecessor to MISSIVE was set up to use HTTP requests just like all of our other APIs, but we quickly realized that this didn't work very well -- HTTP requests only support a poll model, where information can only be learned if you choose to ask for it.

When servers are started or stopped, we needed to distribute this information throughout the existing servers, ideally immediately. Thus, MISSIVE uses a simple TCP/IP connection and a simple message format to pass messages. The primary implementation of this is done in Python, allowing our Python web services to communicate with each other.

After we built this system, we realized that something similar could be extremely useful within games. If a game can send a message whenever something important happens, this message could be received by another server and data could be recorded in a database.

Once data is in a database, it is instantly accessible to whatever web site wants to display the information. To this end, we've implemented a MISSIVE client for our game engine, so that data can be sent from within the game to whatever external server wants this data, and messages can be sent into the game from externally.

Choice of Game Engine/Modularity

We tried to make as few assumptions about the game engine being used as possible. We started with a C++ interface that must be implemented by a game engine to be plugged into Sandstone.

Whenever Sandstone starts up a game, whether it is starting a game server or running a local game in the browser, it calls through this interface to start up the game. In fact, the implementations of this interface are themselves content packages.

So the entire game engine that is used is not pre-installed, it is downloaded and updated just like any other content. This allows us to easily push bug fixes to our engine, as well as have several games that run on different major versions of the engine all work at the same time.

The question, though, is why did we choose to build our own LOCUST game engine? Why not use one of the many existing 3D game engines there to build our games and then use our new technology to run them differently? The truth is that no engine really matched the requirements we had for a game engine. These requirements are:

It's surprising that most game engines don't work for the first two points. Most game engines assume that the working directory of the process is the location of the .exe file of the game and look up content from there. This simply does not work if the process is the browser process, and it could have done any number of things to the working directory. And this becomes a basic assumption of the entire game engine, meaning that #1 is not feasible.

The second point is often not the case either. Many game engines assume that the game is running as a top level window. Asking it to render under a parent window is not supported at all. Despite these issues, we have hacked a couple game engines to actually render in the browser using Sandstone. It was not elegant, we would never want to ship such a game, but it worked enough to show what was possible.

However, even if a game engine could render within a window, most game engines aren't built for #3 or #4. Most game content is built on a scenario or level basis, where an entire level is built as one optimized set of stuff. This does not make for any reusable chunks, and every level ends up needing to be downloaded separately. Finally, it becomes very useful to communicate data out of the game into the web services, and we needed a game engine that could do this easily.

In addition, we wanted to improve our game-building and modding capabilities, both to support Making History's lively modding community, and to provide greater development speed and power in our serious games work. We built on architecture we developed for the first Making History game, The Calm and the Storm, which allows game data to be set up from XML.

We have expanded this engine to make it possible to build complete levels entirely from XML, using embedded JavaScript for scripting. We are seeing improvements in productivity and flexibility in our work on Making History II and our other projects.

Our path to web-integrated, browser-based games has been challenging, and we are excited about the results: the upcoming release of Making History II, and the possibilities the platform provides to create and deploy new kinds of engaging, web-integrated 3D games.