Opinion: Know Your Middleware... Source Code Please!

[In this reprinted #altdevblogaday-opinion piece, Yager Development's Andre Dittrich explains why knowing your middleware's source code is a must, and why middleware developers need to give their clients the source code.] It sounds so nice and easy: instead of spending time and money on developing features that basically every game needs anyway, we license the fitting middleware. Just plug it in and you get awesome physics, astounding AI and the best graphics … and we do not need nearly as many programmers! Well … What Does Middleware Mean For Us? Middleware is an "easy" way for us to add features to our game we do not want and/or have the resources or time for to develop on our own. It allows us to concentrate our resources on the things that make our games great and special. And how many new rigid body physics simulation implementation does this world need anyway? So I guess pretty much every game production of a certain scale uses middleware to some extend. What Does Middleware Mean For A Development Team? When I started at Yaager, we still had our own engine. The engine was used to build our first game (Yager) and for some of the demos we did afterwards. While working on new pitches, we soon realized that we needed to add middleware to keep up with the development in the industry. Animation and physics were among the first things to add. At some point, we moved to a complete engine (Unreal Engine) because we did not want to spend so many resources in developing standard tools like a level editor or a build system. That decision was not taken lightly as we loved to work with our own technology. After that, the programming team grew mainly in three areas: gameplay, AI, consoles/performance. The team was a lot smaller then it would have been if we would have continued with our own tech. It was a team that was focused on getting our demos and then the production of our game project done. Some publishers even seem to see it as a big plus if your team uses proven middleware. In short I think middleware is the right way to go if you want to focus your team on doing games instead of tech. And there is always enough tech to develop in addition to the middleware anyway. But enough introduction – I would like to share the most important lesson we learned while using very different types of middleware. Know your middleware! Currently we use the Unreal Engine 3, an AI middleware for pathbuilding, a sound middleware, and a few additional middleware packages that are part of the Unreal Engine 3. Like my last, this article I will mainly focus on the high level issues. If you have more specific questions, just comment on this article or contact me via email. You Have To Know Your Middleware! One of the main mistakes I have seen when using middleware is treating it like a black box that does magic things for you that are described in some documentation. This almost always leads to a lot of bugs caused by using it wrong or a bit different then intended by the developers of the middleware. For whole engines this is especially true. The Unreal Engine is a very powerful beast, but if you use it wrong, it can become very slow, need a lot of memory, or have all sorts of "under the hood" problems. The reason for this is pretty simply the fact that the engine is developed for the games Epic is doing. For that, Epic is implementing certain features or content in a certain way, and the engine works very well and is pretty easy to handle when you do your stuff in a similar fashion. Of course, you can do things different, but the amount of work it takes to get that running smoothly is significantly higher and you loose the fact that "Epic's way" is proven and bugfixed by multiple game productions. So the key here is to actually know how your middleware works and how it is supposed to be used in as much detail as you possibly can. Whenever you implement something in your game based on or using a middleware, you should take the time to find out how exactly the middleware works in that area. If you have the source code, read it! Do easy experiments! Try to find out where the limits are by testing extreme cases. Imagine you are planning on having ragdolls in your game. Who would not want that? Find out how complex it is to set them up? How many of them you could have in a level? How fast do they go to sleep? … We found the answers to questions like that way too often by having the feature in the game and then seeing the problems. This may sound obvious, but way too often middleware is just explored on the surface and by reading the documentation alone. This is even worse when you use a whole engine. In that case, it is even more important to really understand how the things work you base your game on. The worst of it all is, that you are usually hit by problems with your middleware close to an important milestone (vertical slices, demos …). Why? Because it is exactly at these times when your team tries to get the most out of the middleware and tries to push it to its limits. So if you run into an issue at that point you face the following situation: You try to fix a problem in or with a system you do not know well enough, and if you are really in a bad spot you do not even have the source code for it. You rely on trial and error and on the support of your middleware provider, and that might be hard to get on a Saturday night 20 hours before your delivery date. If you would have studied the middleware in depth before, it is very likely that at least some of the extreme cases your team is trying now were already covered or ruled out by your early experiments, and even if not, you should be a lot better prepared for fixing issues because you know the system. An example: As explained above, we are using a middleware to generate our pathdata for the AI. As long as the environments we were doing this for were nice, flat, and blocky-like in our early test levels we did not encounter any issues. Problem is: our game is not like that. The levels have a lot of uneven ground and have totally not blocky geometry. Even then in 90% of the cases the pathdata generation still worked fine. But again and again we encountered situations where the pathbuilding algorithm would produce very strange results and the documentation was not able to explain why. The best step now would have been to check the source code to find out how exactly the parameters we could tweak would influence the result but we did not have that. So we needed to use trial and error and the support to find out how to handle these situations. Of course, we were hit by these issues closely to delivering an important press demo. So having licensed a middleware that saves you time and development risks is not the end of the story. It still means you need to know it deeply. Getting to know it will cost a lot of time, but it is time you have to spend or you will very likely regret it at some point. The example above brings me to my next point: Source Code Please! Ask any programmer on the planet for the best source of documentation of a software system, and you will have a high chance of getting this answer: the source code. I am not saying that we do not need well written documentation. On the contrary! The better the documentation, the less dependent we are on the source code and non-programmers can usually only use the documentation and/or examples anyway. But when things get complicated and you need to know exactly how a system behaves, there is no better place to look then the source code. We have both situations in our project: middleware with and without source code. Unreal Engine comes with full source code and you even compile it for your game. This is really the best situation and I guess for full engines the only way usable. I cannot count the number of times we needed that source access to understand and debug and sometimes even fix issues we had. Another example: We had issues with the texture streaming on the PS3. The textures seemed to stream in a totally random fashion. It sometimes even looked like they would stream in the background before they would stream for foreground objects. As we have the source code for the game engine, we could debug the issue and found a bug in the engine code that we could easily fix and report to Epic. If I remember correctly, they had also found that issue shortly before that and had fixed it already. When reading the Epic mailing list I can see that this happens again and again, that licensees find bugs, fix them, and report that fix. And another example: Our game would crash very infrequently somewhere in the physics middleware code and we had no idea why. For this, we had no source code access. In the end, we found out that under certain very rare conditions, our gameplay code would send NaNs into the middleware and that made it crash somewhere pretty deep. It took some time to track down that problem, and once we knew, it was pretty easy to fix. It was not even a bug in the middleware but our own fault. With source code access, we would have been a lot faster with identifying that issue. So this to all middleware providers: Give Us The Source Code! We license middleware because we need somebody else to write and maintain the code for the features we do not want to write and maintain on our own. As you can see above, having the source makes your middleware a lot more valuable to us. We can understand your middleware a lot better and we can debug issues that are caused by misusing it a lot easier, we can identify problems in your code and by that improve your product, and if really necessary we can fix a bug ourselves and then simply give the fix to you. I can perfectly understand the fear that somebody might steal your technology, but I guess everybody that works in the software development industry knows: Writing code (or as you might fear – stealing it) is one thing – maintaining it usually is much more work, and that is what we license. When we consider middleware these days, not getting full source code access is a pretty big minus, and for some middleware like AI, we would not accept that again. The risk is simply too high. [This piece was reprinted from #AltDevBlogADay, a shared blog initiative started by @mike_acton devoted to giving game developers of all disciplines a place to motivate each other to write regularly about their personal game development passions.]