Optimizing Project Hospital

Project Hospital is a building-management game, with all the typical aspects of the genre - dynamic scenes created by the player, a lot of active characters and objects and an extensive UI system. Getting it to run well on different hardware took quite a bit of effort and also was a great example of the infamous case of 'a death by a thousand cuts', so a lot of small steps, solving a lot of specific problems and a lot of time spent in profiler.

Performance targets - what we actually aimed to achieve

Early in development we set the main goals of how big scenes we want to support and what the performance goals and hardware requirements should look like.
We aimed for at least a hundred active fully animated characters displayed on screen at one time, three hundred active characters altogether and roughly 100x100 tile maps with up to four floors.
We definitely wanted the game to be able to run at 1080p at decent frame rates even on integrated graphics cards, which on its own wasn't too hard to achieve, as the CPU seems to be the main factor, especially as the hospitals grow. Modern integrated graphics cards only start to struggle at higher resolutions around 2560 x 1440.

For easier mod support, most of the data in the game are open, which means sacrificing some performance compared to packed files, but this doesn't seem to have a big impact, apart from slightly longer loading times.

Graphics

As Project Hospital is a 'classical' 2D isometric game, you can imagine that everything is rendered from back to front - in Unity this is represented by setting correct Z values (or the distance from the camera) of individual graphical objects. Where possible, objects that don't interact with each other are organized to layers, for example floors are independent from objects and characters.



All geometry in the isometrically rendered scene is created dynamically in C#, so for graphics performance the frequency of how often the geometry needs to be rebuilt is one of two most important parts, the second one being the number of draw calls.

Draw calls

The number of individual objects drawn in one frame, regardless of how simple they are, is a major limitation especially on low-end hardware (and Unity itself also has some extra overhead). The obvious solution is to batch more graphical objects into a single draw call wherever possible. This has some interesting results, for example the objects that can be batched are the objects at the same distance from the camera, so other graphics get correctly rendered in front or behind.



Just some numbers: on a 96 x 96 map you can theoretically place 9216 objects, which would be 9216 draw calls - after batching, the numbers goes down to 192.
In real life it gets a bit more complicated though, as only objects with the same texture can be batched, so the results are a bit less optimal - still, this works really well.



Most of the batching is done manually to have control over the results - we also use Unity's dynamic batching as a 'last resort' solution, but it is a double-edged sword - it can indeed help reduce draw calls, but has performance overhead every frame and can be unpredictable in some cases. For example two overlapping sprites in the same distance from the camera will get rendered in different order in different frames, causing flickering - something that doesn't happen with handmade batches.

Multiple floors

Allowing players to construct buildings with several floors adds a lot of complexity, but can surprisingly help with performance. Only characters and objects on the active floor and outdoors need to get animated and rendered, everything inside the hospital on floors below and above the active floor can be hidden.

Shaders

Project Hospital uses relatively simple custom shaders with a few tricks like color replacement. For example the character shader can substitute up to five colors (using conditions in shader code) and is relatively expensive, but this doesn't seem to be a concern as characters rarely take up a lot of space on screen. This was definitely worth it as having unlimited colors of clothing is great way to add a lot of variety to the characters and environment.

We also learned quite quickly to avoid setting shader parameters and using vertex colors instead wherever possible.

Texture quality

One interesting bit of information is that we don't use any texture compression in Project Hospital - with the vector-style graphics it looks really bad with certain textures.
To save GPU memory on systems with less than 1 GB, we automatically downscale the ingame textures to half resolution (apart from textures in the user interface) - you can tell this ingame as "texture quality : low" in options. UI textures keep the original resolution.

Optimizing CPU performance - multithreading

While Unity script logic is basically single threaded, you always have the option to run more threads in C# directly. This might not be a plausible approach for game logic, but there are often some non-time-critical tasks that can benefit from running on separate threads in some form of a job system - in our case we used threads for two features:

Pathfinding jobs, especially on big maps and with bad layouts can take up to some hundreds of milliseconds, so this was an ideal candidate to be moved away from the main thread. The number of parallel jobs takes in account the number of hardware threads on the machine.

Lightmaps are also updated on a separate thread, but only one floor at a time - this is not a critical system and automatic lights in rooms fade out at a rate that works well with a slower update.

Animations

We decided quite early in the development process to go with a 2D skeletal animation system. After considering different animation software available at the time, we ended up modifying a simple system I made a few years ago (basically as a hobby project) to suit the specific use in Project Hospital - you can imagine a simpler Spine with direct support for creating character variations. Similar to Spine it uses a C# runtime which is obviously more expensive than native code, so there were a couple of rounds of optimizations done during development. Luckily the rigs are very simple, only about 20 bones per character.

Random fact: the most important bit turned out to be switching from a map lookup to simple indexing in an array when accessing transforms of individual bones.



One more trick related to animations apart from not animating characters outside of the camera view is that characters hidden behind the main UI windows also don't need to get animated - unfortunately switching to semi-transparent UI prevented us from using this in the final version.

Caching

Wherever possible, we try to run some more demanding computations only when there's a change that affects the values - the best example is probably rooms and elevators - when the player places an elevator or builds walls, we run a flood-fill algorithm that marks which tiles the elevators and rooms are accessible from - this then speeds up pathfinding and can be used to show the player which rooms are currently inaccessible.

Scattered and delayed updates

In some cases where it makes sense we run certain updates only once in a while and there are a few approaches we use:

Some updates can be run only on a part of the characters every frame, so for example the behavior scripts of half of the patients get only updated on odd frames, the second half on even frames (while the animations and movement run smoothly).

In specific states, especially when characters are idle but calling some expensive bits of code (for example employees checking which needs to fill and looking for free equipment), this is only done once in a certain period of time, for example once per second.

One of the most expensive and at the same time one of the most common calls is the evaluation of which examinations are available for each patient. There's a lot of factors that need to be evaluated - for example which staff of a department is currently busy and which equipment is currently reserved. This information is also not common to all patients, as their assigned doctor and for example their ability to speak also have an effect. There can be dozens of available examinations that need to be checked, so the update only goes through a few every frame and continues on the next frame.



Conclusions / lessons learned?

Optimizing a tycoon game with a lot of different interacting parts turned out (not surprisingly) to be a continuous process, working with the profiler in Unity and taking down the worst offenders became a regular part of the development process on my side.

While there's always room for improvement, we're pretty happy with the results, the game runs according to the original targets and players are regularly modding the game to significantly exceed the original character limit.

Probably worth mentioning that even compared to some of the AAA games I worked on, I consider Project Hospital to have the most complicated gameplay logic I've ever seen, so a lot of the issues were really project-specific. Still, reserving enough time for optimizations to match the complexity of the game is definitely recommended on any project.