Systems Admin for the Indie Dev

xkcd: Devotion to Duty

Hello again! It’s been a long time since I’ve posted anything, but I’d like to let you know that Jake and I are both still alive and well, heads down on development of our upcoming RPG, Ethereal Legends. We’ve still got a good amount of work to do on that before it’s released, but I recently took some time off from game dev to reconfigure our network infrastructure, and I felt as though my experience could be helpful for any other indie devs looking to start their own game companies.  This post isn’t aimed at being a nitty-gritty tutorial on exactly how to configure your network, but is more directed at being a high-level overview of some of the choices we made when building our network, and why. So here we go!

I have a bit of prior experience with this subject, as systems administration for small business is my day job, outside of game development. I think systems administration is something a lot of new indie devs simply don’t think about. Chances are that a lot of starting indie devs are sitting at home, where they don’t worry about static IP addressing. They don’t have physical servers or a proper source control setup. They’ve never had a reason to connect two machines together with anything other than Dropbox or Google Drive. And while this is a perfectly fine way to do things when you’re working by yourself, it becomes extremely cumbersome when you start scaling and working with other people. Eventually, you need some way to centralize, organize, and secure your files, in a way that is easily accessible to the people you work with.

Even though Systems Administration is my day job, and I love doing what I do, SysAdmin for me has always been a “means to an end.” I don’t really build networks or servers because I enjoy it (even though I truly do); instead I taught myself how to do all these things out of necessity. I neeeded a website for the games I was building. I needed a shared file server with source control, in order to more easily perform backups. I needed a local domain to securely authenticate users in order to work with others on those same files. I needed an FTP server to deliver the game download to players over the internet. These were all things that I felt we absolutely needed at Soverance, and while there are many ways to accomplish these tasks, I wanted something that was robust, fully featured, and able to scale, in the event Soverance was able to grow from the tiny operation it is today and into the true gamedev company I want it to become.

Our main Workstation setup – Jake’s desk and our MediaServer. That’s my office on the right, and our VR room on the left.

To accomplish these goals, we turned to Microsoft and the Windows platform. In addition to my previous experience with Microsoft, the main reason we chose to go with MS is because of their BizSpark program. If you’ve not heard of the program before, it’s basically like a “hands-off” startup accelerator – they give you three years to basically go hog-wild with Microsoft software and services, free of charge. That includes a significant amount of credits for Azure cloud, as well as license keys for numerous versions of Windows, Server, Office, SQL Server, and whatever else MS makes that you might find useful. As an fledgling indie game company, this type of support is invaluable. Yes, you can put a monetary cost on what they give you, and it’d be into the tens of thousands of dollars… but if you put it to use, you can learn a lot of valuable knowledge about how your company will operate in the future.  Either way, if you’re starting a game studio (or any company, really), I highly recommend checking out the MS BizSpark program.

After we gained access to the software, we had to plan out our network and determine our requirements. How many users would we have? What services did we need? What hardware did we need to deliver those services with high availability and decent performance? What was our budget? Our requirements were essentially as follows:

The last requirement was the most difficult, but luckily BizSpark has provided us with everything we needed. Now let’s jump down into some of the basics of Windows Server administration.

First, to get the user administration we want, we must run a Windows Server role known as Active Directory. Without going real in depth, Active Directory (AD) is basically the equivalent of a master control room for your domain network; you’ll use AD to create and manage users, permissions, and security policies for every device connected to your network. It’s also an integral part of your internal DNS infrastructure, allowing you to refer to computers with their names instead of their IP addresses. Microsoft recommends as best practice that you run Active Directory using two servers; Primary and Secondary Domain Controllers. They’ll replicate data between themselves, as well as help spread the load from queries to the AD and DNS services so you’re not overloading one machine. Setting up two servers as Domain Controllers also takes care of the file sharing requirement, as you can have one of the servers pull double duty as the file server.

You’ll need to budget for a third server to run Windows Server Update Services (WSUS). This box will generally be authoritative over every other Windows machine on your local network, in regards to updates. A vital part of any MS network, once it’s set up it’ll just run in the background and update your machines when appropriate. I set the machines on my network to update once per week, at an early morning hour when I know we’re not using the network.

Mmm servers….

For anything public-facing, like the website you’re reading this on or the game’s FTP server, we needed a different solution if we were going to maintain high-availability and security. I wasn’t exactly keen on letting people hit our office’s public IP when visiting the website, as it opens us up to some security concerns, and I didn’t want to have to worry about the website or game being down because our power or internet went out here in the building. That’s where Microsoft’s Azure cloud platform comes in. Once again, Microsoft recommends that you run not just one, but two virtual machines in what they call an “Availability Set”, which helps to limit the effects of hardware failures, network outages, and power interruptions inside the Azure datacenter. Through the BizSpark program, you receive $150 per month in Azure credits, which is more than enough to run two VMs full time all month.

The real trick came when it was time to connect the Azure VMs to the local domain. Since they’re not physically in the building, creating a secure connection requires a VPN device, to create an always-on VPN connection between your Azure virtual network and your private local network. We’ve fortunately got one, an old Netgear FVS124G, which you can probably find somewhere for less than $100. I was surprised that this particular VPN device worked with Azure, but sure enough, it did. There’s documentation for some of the more common firewalls (like Cisco/Juniper/FortiGate stuff), but basically nothing on Netgear. Trial and error did the trick, but I lost about a day messing with it. Either way, once you’ve established the VPN connection between your local domain and Azure, you can join them to your domain just as any other locally connected machine. This obfuscates the fact that they’re run at all in the cloud, allowing me to access them from my dev workstation just as I would any of the local machines.

Finally, we needed stable email, and for that we really only had two choices:  Google for Work or Office 365.  We tried Office 365 first, mostly because I wanted to stay within the Microsoft ecosystem. Unfortunately, one of the biggest features for me was the Single Sign On feature, which would have allowed us to manage the Office 365 accounts using our local Active Directory, and users could sign in using their AD credentials. Unfortunately, I ran into some issues setting this up due to my Azure and Office 365 accounts not wanting to play nicely with each other, and after many hours on the line with O365 support, I decided to just drop it and move to Google. We’ve been on Google for Work for a number of years before all this, so we were already familiar with it’s management. Unfortunately, it’s now the only service we use that isn’t under the same roof, which kind of annoys me but isn’t a problem.

When all said and done, our network diagram looks like this:

As you can see, we’re pretty simple; we’ve honestly got more horsepower than necessary for a company of our size. Let’s go over the purpose of each device in the diagram, above.

As you can see, we’ve got a pretty solid network. We found a great deal on the servers themselves, getting the used 2950 donated to us by a friend, and finding the used 1950’s on CraigsList for $50 each. Spent a few hundred bucks putting the server bank together, as far as buying the rack, rails, and associated items (like the PSU and monitor tray). I was actually amazed that we spent more money on the rack itself than we spent on the servers… but hey, what are you gonna do?

More than enough space left in our rack, if we need to expand. Yes, that’s a Kinect v2 on the tripod.

Altogether I think we came out with something pretty nice. There’s still a few upgrades I’d like to make… for example, I’d like to get a large NAS to use as our backup device, instead of the crappy little external drive I’m using now, which could store regular and complete images of each machine’s system state. Definitely want to get a better VPN firewall, preferably one that has wifi. Our local servers all have 15k rpm SAS drives as their OS drives, and while that’s decent enough for our purposes, I wouldn’t mind replacing them with SSDs like our workstations have. We’ve also got enough drive slots left over in the 2950 that we could add more storage if necessary.

Probably the next upgrade we’ll do, though, is buying a portable air conditioning unit. The room we put the servers in easily rose in temperature anywhere from 10 to 15 degrees, maybe more. Not a big deal when we turn the air on… there’s a lot of ventilation and the A/C gets cold. But we don’t always leave the A/C running, and one day we plan to move the servers into a smaller closet, so we’ll need the cooling.

In the end, I’m super happy with how our network has matured here at Soverance. We’ve got the infrastructure in place now to build some really cool things. Hopefully this post has helped inspire some other indie devs that know that they too can build great networks to help facilitate the success of their teams.

Now it’s time to get back to game development!

Hey it’s me!