Distributing Your Game via CDN - Not Just for Megacorps Anymore

Originally from onemanmmo.com

When I started developing The Imperial Realm :: Miranda, Content Delivery Networks (CDNs) were big and fancy and expensive and only used by giant megacorps, so I never imagined I'd be able to afford to use one.

With me posting more and more big screenshots on my websites like this one:

[Random screenshot of the week.]

Last weekend I decided it was time to research solutions to improve my websites' performance.

Initially I was thinking I needed to rent a web server, but then I came across CloudFlare which for $20 a month offers to speed up your website by cacheing static website content on nice big fast servers. My non-static data is pretty small, I have a lot of big bitmaps I'm serving, I want that to be fast, so a solution that didn't need a new web server sounded pretty attractive. (I've rented web servers before, and they have all been problematic. Sometimes another website on the server is using all the CPU, or somebody misconfigures their CMS resulting in a breakin, or the thing is just down and nobody knows why or can tell you when it will be fixed.) Doing my due-diligence on CloudFlare, I found a few worrying stories about downtime, but even more worrying was a benchmark that showed CloudFlare sometimes having several hundred milliseconds of latency serving requests. Amazon CloudFront was on the same graph with very low latency across the board.

While reading Amazon's website, I came across information on their S3 service. S3 is basically a really stupid web server. What interested me about S3 though, is that it has smart tools for uploading files.

At a guesstimate, it looks like Miranda is going to clock in at around 100GB of data. Luckily, most people will only need to download the client which is pretty small and a small portion of that 100GB. But I still have to manage the full 100GB which is a very large and unwieldy chunk of data. At that size, just getting the data to the web server is a big problem. My reasonably fast internet connection will take about 100 hours to upload that (I'll likely upgrade to internet that can do that in about twelve and a half hours.) Until fibre appeared in my front yard earlier this year, I figured I'd have to take the data to some data centre on a hard drive whenever I wanted to do an update.

But S3 has a better way. They have a tool which will compare your local files with hashes of the files in the server bucket so you can upload only the files that have changed. The terrain data in Miranda is pretty stable, so it will be rare that I need to upload all 100GB of Miranda.

The problem then is that S3 isn't designed to support lots of people downloading lots of files. Luckily, you can point CloudFront at an S3 bucket to get fast performance and smart file management all in one package!

So to learn how Amazon's services work, I signed up for CloudFront and set up theimperialrealm.com, onemanmmo.com, and secretlairgames.com to use it. That was surprisingly easy, all I needed to do was add a CNAME record for each domain and point that at a CloudFront URL then change all the static data links to point to the new subdomain. Easy peasy. onemanmmo.com went from about 7 seconds to load to under 1 on the website speed test I used.

At current levels, all three domains will be free for the first year.

The cacheing which makes CloudFront fast, makes distributing Miranda a little more complicated. If I simply update a file in the game, it may take a while for the changed file to be distributed to all the CDN servers, so for a time, it could be serving both the old and the new versions of that file, which doesn't really work for Miranda. CDN's do have a feature to invalidate files so that you can be sure everybody is on the same version, but it takes minutes not seconds, and it is a cost and management headache. Fortunately there's a better option: versioning.

If you put a version number as part of the URL, then as soon as clients start requesting that version, it will be instantly and automatically propagated to all the CDN servers.

When I designed Miranda's HTTP file request system I built it with the idea that I'd need to have a number of web servers to support the demand of thousands of players. To do that, the client can pull files from several web servers simultaneously and distribute the load. Using a CDN, as far as the client is concerned, there's only one server so all of that isn't strictly necessary anymore. The change I did need was to add support for inserting a version number into the URL's that the game requests.

With distribution at long last figured out, next up is to figure out what to do about the game server.

By the way, I also did due diligence on Amazon as well. Whatever you do, don't upload your Amazon credentials to Github.