In-House Engine Development: Process Tips

All tips in my previous post were rather technical, but in the end creating software is not really that much about technology. The human side to it - the developers writing, architecting, fixing and using the code - is infinitely more complex than the programs we write. It might seem shocking, but the image of a programmer being a simple machine that converts coffee to code is not really true. That is why I want to deal with the human side of software creation this time.

During the work on our in-house ABYSS Engine at Deep Silver FISHLABS, I have come across a workflow that helps us ensure product quality by incorporating feedback and knowledge sharing as well as automation into our daily work.

These ideas probably work for most software projects, but as my main expertise is in C++ engine development, I am going to focus on that.

1. Talk About Code

The team should constantly talk about the code of all individual team members. Talk about code you are going to write, are writing and have written. This has multiple uses:

There are multiple steps to this process and the first one starts before writing a single line of code. The team gets a set of requirements for a new system from somewhere (does not matter from where here). Then, the person mainly responsible for implementing that system does some research and comes up with an initial design.

This design is presented during a first kick-off meeting, where the requirements are summarized once more. The design is evaluated to figure out if it fulfills the requirements and also if it conforms with the general design of the project. The team may have to iterate on the design, but once a consensus has been reached, the writing of the code may begin.

After the code has been written, tested, and found to be good enough by the implementer(s), it is time to integrate the changes into the project's mainline.

Before doing so, however, a complete code review should be done. Generally, this review can be done by any other team member. You should make sure that this team member also feels, to some degree, responsible for the new code. Code reviews fulfill all three advantages listed above.

2. Make Releases

Even though, as an in-house engine, we have a rather small team and only very few "customers", we still make sure to create releases of our software. Do not make the mistake of just allowing the game to use the latest engine version. Make sure that, instead, you have control over what version gets used.

We do this by first planning ahead what releases we may have. This is more of a project management task. Eventually you should have a number of releases with a date and a list of included features. If you are not that organized - and we are not always like that - just implement any required features and/or fixes. When the time is right, create a release from that current state of the engine.

When you create a release, you first make sure you have all changes that should go into it. Then you need to test the new version as well as you can. At least for bigger changes it also makes sense to involve the QA team and have them do a complete check of the game with the engine release candidate.

Once the release is ready, choose a good time for the actual release. Normally, it is not a good idea to update the engine right before an important milestone or when a new build of the game is due. Depending on your workflow, inform the proper stakeholders and either supply them with the new release or integrate the release into the game's mainline yourself.

After the integration, you have to do some advertising. Let the team know a new version is available and what changes it brings. Possibly assist with making any necessary changes on the game's side and also help with adopting new features. If no one knows of your new features, they may never be used.

This way, you have complete control over which engine version the game uses. You can also ensure that this version has thoroughly been tested and fulfills the necessary requirements.

3. Let the Machines help

While coming up with a good design, writing the code, planning releases, and so on (still) have to be done by a human, there are quite a few other steps we can and should automate.

First, what I almost do not dare mentioning, but have to say: You should, of course, use a version control system. We use Git which supports our workflow very well. We do a lot of branching and merging. For every feature and bug fix, a branch is created. Any changes are implemented on that branch and then, once the change is complete, the branch is merged back into the main development branch. You should have a branching model that everyone knows and follows.

So far, Git Flow has worked very well for us. The model in short:

Before merging anything into the develop branch, do code reviews as explained above. As also stated previously, we as game or engine developers often have to cover lots of different platforms and configurations. Rather quickly, this becomes impossible to do locally on our machines. Thus, you need a build server for this that builds any new commits on the develop and master branches for all target platforms and informs the responsible developers if something does not work.

With the basis of a build server building the engine, you should go one step further and introduce automatic testing of the builds. You may come up with sophisticated ways to test a game without human interaction, but I think you should start with good old unit tests. Games engines are often already designed in a way that makes unit testing very easy. Do it. Choose a test framework like Google Test or Catch and just do it. Even if your tests are not very elaborate, it is already very helpful to just have some code use the engine code. That is especially true for C++ template code, which does not even get instantiated before it is used by other code.

With proper branching, you can make sure your main branches (such as develop and master) are more often in a working state than not. Also, employing build servers and automatic testing helps with catching bugs earlier than is otherwise possible.

4. Do Not Let Everyone Do Everything

For the actions explained above, like creating releases or even merging into the "develop" branch, it may be beneficial to assign responsibilities to specific team members. That way you can make sure that your quality goals and the process are respected.

If you want to ensure that no mistakes are done by accident, version control systems (or their management software, such as GitHub and GitLab often allow to restrict access to branches. That way only developers who have a specific role or belong to a specific group can commit/merge to these branches.

While it is a privilege to touch these branches, it also comes with the responsibility to manage them. For the develop branch, this means doing code reviews. For the master branch, release management has to be done as described above.

Summary

Having a roadmap and clear branching model, makes it easy to determine what a given release’s feature set is. Share any findings with as many developers as possible and make sure bugs as well as compilation errors are caught as early as possible. Also, in order to ensure that the process is followed, only empower those that have internalized the process to make releases.