Sponsored: Leveraging the cloud to make an MMO with a tiny team, in Crime Coast

Author: by Lee Stott, Technical Evangelist, Microsoft UK

This article is brought to you by Microsoft

How many times have you sat and watched a film, been thrilled by a TV show, or constantly enthralled by a book and thought…this would make an awesome game. If you’re anything like me, you have this feeling all the time.

This is a blog about UK Indie videogame company Pixel Squad. Pixel Squad has no mainstream backing, but they managed to overcome a raft of challenges to create Crime Coast, a game featured by both Microsoft and Apple, with a team a fraction of the size of their competitors.

Crime Coast was inspired by the love of the Breaking Bad TV series, and it began life as nothing more than a hobby, with initially just one person Will Mallouk, writing the code and a small team of remote contractors to create the artwork.  From the developer’s perspective, it became clear that it was going to be a very big challenge to reach a high quality bar without the full attention of someone talented and experienced on the art side.

 Before and after Krzysztof and Paul Joined the Company

Krzysztof joined Pixel Squad in August 2014 and they expected to release the game in October of that year, talk about optimism! For indie developers release dates are like ex­girlfriends; you might cry when you miss them, but that’s basically it. Despite working crazy hours they missed deadline after deadline that they had set themselves. It didn’t matter how much they wanted the game to be ready now, they had to come to the realisation that with such a small team, this was going to take some time.

Things really started to pick up when they were awarded funding from Creative England’s & Microsoft’s Greenshoots programme in November 2014. I was super impressed with their Greenshoots application and the proposal received great feedback from all the members of the judging panel, and this cash injection gave the team a huge boost in every area of the game. It also meant their backup plan of turning to actual crime to fund the game was shelved. :)

The next member of their gradually expanding team was Paul, who joined in March 2015. They found it really difficult to find a contractor who could do high quality visual effects, and even though Paul’s specialty is in environment art, they dragged Paul back to his roots and made him a full­time member of Pixel Squad.

After a whole lot of pixelated blood, sweat and tears, they finally soft launched the game in March 2015 on Google Play and iOS. With a shoestring soft­launch marketing budget, the small volume of users playing Crime Coast didn’t give them as much data as they had hoped for. It was definitely a soft launch with a lot of ups and downs. If we they had followed the conventional Lean Startup wisdom and paid attention to the naysayers they would have never completed and launched the game. With lessons learnt, they set themselves a strict, no turning back deadline for our global launch: 11^th of June 2015.

We have all experienced crunch time when working towards a gaming release. One of Will’s most entertaining quotes to me was ‘I’ve never been in a war, but I imagine it’s a lot like that except with more caffeine and less chance of seeing daylight.’

The sprint they did for Crime Coast’s finish line was on a whole new level to anything they had collectively experienced before. It was an endurance challenge for several reasons. And contractors aside, no one in the team was getting paid for this. Yes, I know you’re thinking they must have been insane!

Also, since we were committing to a sizeable marketing investment for their company’s size, we decided that it would be better to release the game on as many platforms as they could right at launch. They went for iOS, Windows Phone, and Google Play from the start.

Something familiar?

Having to support so many platforms also meant that they were working mostly on the client side up to the last minute. Subsequently, their backend improvements became as neglected as their partners and dietary requirements, and they ended up with only five days left to do stress tests and server tweaks.

Out of the five precious days they had left before launch, two were gobbled up straight away just to setup the stress tests. In the remaining three days they tested with up to 2000 simultaneous players on this simple architecture:

Front End VM: A single IIS + Redis running on the same virtual machine

Database: Managed SQL Server Azure database.

Unfortunately, due to time and resource constraints, they only designed an architecture that could scale vertically. But without time for much else their only option was to wing it, with the hope they could serve any amount of traffic they received and work on things after the game went live.

They did have a backup plan, but it was on a par with “Hey if this plane crashes, I’ll just jump out at the very last minute”. It wasn’t really an option. 

On launch day they didn’t know if the game was going to be featured by Apple, but as the team was part of Microsoft’s Greenshoots program, they did know that they were going to be featured on the Windows Phone Store. But they had no idea what kind of traffic to expect.

On the 11th of June at 9AM, UK time, Crime Coast went live across the globe. Considering PixelSquad’s (lack of) budget, they made an impressive marketing effort, having coordinated with as many actions and events they could find to coincide with that date. The result was a decent amount of traffic throughout the day, which they were very happy with. But at 5PM something amazing started happening. The traffic on their servers gradually increased, and they began receiving messages from their friends all over the world: South Africa, Australia, Brazil, The United States. They checked the Apple Store. They checked the Windows Store. The game was featured…globally. By June 12^th they had hit 75000 downloads on that day alone.

After surviving the difficult build up to launch day they suddenly had a more urgent and difficult problem to solve than any they had before. Fail this; and everything they had strived to achieve over the past year would end up straight in the bin. To keep their game alive, they would need to keep their servers alive.

It doesn’t matter how powerful a database server is, if your application only scales vertically then you can find yourself in a lot of trouble. On the morning of the 11^th of June they started with a 50 DTU capacity Azure SQL database. They scaled once, twice, three times, and by the end of the day they were on the top tier database server, using a total of 800 DTU. The easy part was the transition to a higher capacity DB, as on Azure it is totally seamless, with zero lost connections and downtime. The actual database is moved to totally different hardware and swapped live depending on the requested change, requiring no work at all on their end.

Their stress tests with 2000 simultaneous users were done using a database with 100 DTU capacity, which they passed with flying colours. At the end of day one though, they were already using the maximum capacity of 800 DTU, and they needed more.

Having run out of options to scale up they knew they needed to optimise the database with the game live and running. Everyone tells you this is a great problem to have, but at the end of a long slog with no in-house SQL Server expert, they did feel like rabbits in the headlights of cars driven by psychotics. Eventually Crime Coast prevailed, and by the end of the first week they managed to provide all their players with quality service, registering only 15 minutes of downtime.

Will stated ‘In this frantic first week the Azure support team saved us a hell of lot of time identifying and solving our issues. We could call them half-awake at 3am and set them off investigating issues for us whenever we needed. They became almost like an extra member of our team, helping investigate problems, teaching us the inner workings of things we weren’t familiar with, and preparing us so we could identify and resolve issues if the problems returned. I dread to think what we’d have gone through without purchasing the top level of support from Azure. In those critical first 30 days of our operation, it was like having Gandalf, Master Chief and Optimus Prime on your side.’

At the end of their first month Pixel Squads Azure bill was over £3500 pounds, and by far the most expensive item in the breakdown was the SQL server. This was one more convincing reason to streamline their backend as that would also allow them to operate much more efficiently.

Party Time

So they dedicated the next two weeks to optimising the servers and rearranging the architecture to allow it to scale horizontally.

Here is the latest setup of their system:

●An array of load balanced IIS VMs handling the client requests

●Managed Azure Redis Cache

●Azure Blob Storage for any object greater than 5 kilobytes

●Managed SQL Server strictly for data that needs to be relational

They knew early on that this was a good setup to use, but with such a small team and so much to do they just didn’t have the time to implement it before launch. Since they weren’t sure if they were going to have such a volume of players, it was natural they ended up focusing more on other areas of the game.

The silver bullet in this setup is the Azure Blob Storage. One of the most satisfying moments for Pixel Squad was when they offloaded parts of the data that were handled by the SQL database onto the Blob storage. After this solution went live, they were able to magically scale down from the expensive 800 DTU database to the inexpensive 50 DTU option. The performance is so good that they were able to move things that were on the Redis cache onto the Blob storage. Needless to say, it’s a much more efficient, scalable, and cost effective option, and it gives them no work to maintain. I imagine distributed file systems like the Blob storage are the solution for many problems that developers of online games have.

My favorite features of the Redis cache are the sorted lists and sets. It’s a very efficient way of managing things like leaderboards and player lists. As the name suggests, it’s a cache, so it’s great for data that needs to be accessed very fast and frequently. However, it isn’t as easy to scale up and not as inexpensive as the Blob storage, so it’s better to limit the amount of things you use it for.

From a code perspective you can store things in the Blob Storage in a similar way to how it’s done on the Redis cache, making it potentially an easy transition.

Compared to all other storage options, the SQL Server is definitely the slowest, costliest, and most challenging to scale horizontally. However, the power of SQL queries is unparalleled, and it’s difficult to imagine how developers can operate without the kinds of answers they get from it.

For a massive multiplayer online game, I would recommend it to be used only where you need such important and complex queries.

When you consider Crime Coast was developed with such a small team, its simple to realise the opportunity is only there thanks to the technology that is available to developers today.

Today, modern programming languages (mostly) insulate us from time-consuming problems such as memory overwrites, we have incredibly powerful and refined tools that allow distributed teams to create cross platform games efficiently, and we can harness an almost limitless amount of computing power on the cloud. That’s why Crime Coast is possible, and that’s why it’s not difficult to foresee that we will keep seeing a stream of awesome indie games coming out with an ever increasing quality level in the coming years.

As Mark Pincus once said, every time you scale up you discover a new set of challenges.

That is what happened with Pixel Squad and I hope you have enjoyed the story of how they overcame everything hurled in their path to emerge with a game they are extremely proud of.

Everyone at Pixel Squad knows this is just the beginning for Crime Coast; they will continue working to deliver a beautiful, unique, and engaging experience to their players and hopefully, in the not too distant future.

If you want to try out Crime Coast, you can download it here on your favourite platform:

And if you are developing your next game, please check out our free Azure trial or – if you are a startup – check out the Azure benefits in our BizSpark program.