The Engine Survey: Technology Results

Recently I surveyed a large group of industry executives, seeking information on their thoughts and perceptions of game engine middleware. Last week I shared the general results of that survey with you; this week we’ll talk about some of the technology-related results.

The survey was split into two sections, one for production-oriented folks, and one for technologists. This blog post will describe the results from the technologists, who were largely CTOs, VP Techs, Tech Directors, Engineering Managers, etc. The questions were focused to get their opinions about game engines in general, not feedback on particular ones: what features do they look for in a game engine, what tools do they expect, what engines are they currently using, that sort of thing. We could have delved much deeper into this, but perhaps that’s something to address later.

As discussed last week, the responders to this survey leaned heavily toward what is considered the “core” game industry, with their current games being designed for these platforms:

As a result, feedback on engines which run primarily on Mac, Linux, or iPhone did not accumulate enough information to make the results statistically significant.

Engine Strengths

55% of the responders stated that they are using a middleware game engine on their current project. Of those people, 39% are using Unreal, and 22% are using Gamebryo, with other engines landing significantly smaller percentages. But our survey showed that only 9.3% would actually choose to use a licensed game engine if they felt they were able to easily do otherwise. If that’s the case, why are all these people using game engine middleware?

These were the only responses that achieved over a 50% “yes” rate, and they echo some of the things we discovered last week: one of the primary reasons for using a game engine is to give the content creators more time to work on the title, especially during the prototyping and early concept stage. When a tech team needs to create an engine and tools from scratch, it’s harder to give the artists and designers something useful to work with right away. Giving the content creators valuable prototyping tools should conceivably result in more refined or unique gameplay. Similarly, using game engine middleware should allow the programmers to focus on creating technology that distinguishes the game from others of a similar genre. One commenter mentioned that they like using a licensed game engine because it’s easier to hire people who are already familiar with it, as opposed to bringing someone up to speed on a custom engine!

As echoed in the results from last week, we again see the idea that perhaps a game engine can help us develop a game more quickly, though one of the lowest valued benefits in this section was using an engine to reduce engineering staff headcount – the technologists by and large don’t believe that using an engine will allow for smaller tech teams, rather the team will need to remain the same size but the extra resources will be used to more effectively distinguish the game from others.

Using game engine middleware to bolster technology sharing across multiple titles at a studio was low value, as was the idea of avoiding rewriting general purpose systems. The very lowest rated reason to use game engine middleware was the idea that purchasing a game engine would give you state-of-the-art technology from game engine specialists! Clearly, this is not an idea that should wind up in any engine’s marketing material.

Engine Concerns

On the other side of the coin, the technologists expressed substantial concerns about using middleware game engines in general. The results echoed back some of these same themes. What concerns do they have?

The strongest concern is that a particular game engine may not work well for your game’s genre. One of the things game engine manufacturers should be doing to aid technology decision makers is highlighting games of various genres that use their tech, in order to demonstrate that the engine works well for those genres. Most of the engine manufacturing companies at this point also make games themselves – certainly this is a great way to highlight that an engine excels at one particular genre, but it can also reinforce the belief that it is only good for that one. There’s also a potential unfortunate side effect if that game fails in the marketplace or plays poorly. It could spell doom for sales of the game engine!

A strong theme coming out of the technologists’ comments in particular is the difficulty of working with and modifying an unfamiliar code base. It is stated repeatedly in multiple sections of the survey – it is the most important thing next to genre appropriateness. Having access to engine source code helps out immensely, but being able to evaluate the engine to make sure that it integrates well with existing tools/tech is vital, as is the level of support. Commenters pointed out that they have spent more time debugging poorly-crafted middleware than they would have spent writing it from scratch themselves. And what if the middleware company were to go out of business, or be purchased? Clearly having source code helps immensely in these circumstances, but it’s a frightening prospect that should also be covered in any legal agreements.

Platform feature parity also came up repeatedly in the comments, especially around platform launches. Having an engine that runs well on one platform but poorly on another makes the engine as a whole only as useful as the lowest-performing platform. Chasing feature development in a piece of middleware by continuously re-integrating version after version is time consuming and aggravating for all involved, but sometimes difficult to avoid. Platform manufacturers would do well by the developer community to bring middleware engine and library providers into the development process of new platforms early.

Valuable Practices

The practices by engine manufacturers vary significantly, in terms of support, source availability, delivering and sticking to a roadmap, etc. What do technologists really want from their engine provider? We asked about a few important practices to see which would stand out. Here’s what stood out as being most important:

Clearly having source code access and easy integration with other middleware libraries is most important, even more important than details of the engine itself, such as having the ability to modify memory allocators. Some of the engine providers give continual access to new engine builds, and this is valuable indeed but it seems less valuable than the currently working feature set. The fact that having a clear engine roadmap is judged as having the same value as ongoing access seems to imply a desire to lock in on an engine version and not worry about upgrades during the development process (something mirrored in the way that many studios lock in on a version of their third-party tools during the span of a game’s development).

One very important comment about having an engine’s development roadmap was made by someone who said “I would not purchase nor use any middleware where I am dependent on a future feature – tying my studio to the whims of development of a third party is not good business.” I think that says it all.

Engine Capabilities

Looking at the various game engine options available, it’s clear that different engine manufacturers have different definitions of what “game engine middleware” should provide. What systems should be included in an engine? Should the engine be a completely custom and thorough system, or simply a thin base layer into which middleware libraries from other vendors can be easily integrated? Here’s how the responders replied to the importance of various engine subsystems or features:

This desire for a multi-threaded or multiprocessor-capable is almost certainly new within the past five years, but of utmost importance now. What rated lowest? AI routines were the absolute lowest, with a camera system and user interface library following close behind. These systems are all very game-dependent and require a lot of tweaking no matter what. However, there are also strong middleware libraries for fundamental AI and user interface design, so these features don’t necessarily need to be provided by the engine. The importance of ease of integration again reared its head in the comments, “I prefer to see an engine have hooks for these systems, than to attempt to provide everything.”

Given that technologists are very interested in being able to plug in other middleware libraries, I asked what libraries they’re using in their current projects. 89.7% of responders indicated that they are currently using some kind of run-time middleware library! How times have changed.

The most popular middleware libraries were:

One of the most interesting results here is the strong preference for Havok (40.6%) over PhysX (15.6%). What makes the difference? Is it the platforms it is available for? The feature set? Havok won a Game Developer Front Line Award in 2008, and in the article describing why they won the award, Jeremy Gordon and Michael Boccieri of Sega Studios pointed to not just a solid codebase, but the tight connection Havok has with its developers, providing strong support and continuing to update the product with features requested by the community.

Tools

A middleware game engine is not complete without a suite of tools that make the engine simpler to use. What tools are most important for a game engine to provide? Does an engine need to provide a full content pipeline for artists, for designers? Or is something thinner more interesting? In the last blog entry, we noted that developers are very interested in rapid prototyping and rapid iteration; will the responses to the value of tools bear that out?

It is debatable whether profiling capability should be considered a system or tool, but clearly it is incredibly important. If I’m making a game on your engine and can’t easily tell why I’m getting 10 fps instead of my target of 30, I’m very likely to make some angry phone calls and figure it is the engine’s fault instead of my own. As we talked about last week, having live preview on the target so the content creators can see what they’re creating properly and iterate on it quickly, is also very important.

The three tools which really popped in these results were: a world builder, a particle system editor, and a scripting system. The tools which rated lowest included a shader editor and music/sfx editor, two things for which there are already very good solutions on the market, for free or for purchase. One of the commenters noted “the items of lesser importance are simply ones that I know I would have a high probability of using my own game-specific solution for anyway.”

A standalone world builder is something most of the game engines seem to provide now. However, I’ve seen many younger game studios that use their 3D art package as the world building tool, through the use of custom extensions. But as one responder said, when it comes to licensing a game engine, “a game editor is perceived as a must have, as building one from scratch is still a heavy duty.” Asked in a separate question if people are using their 3D art package or a custom application for world building, 73.2% pointed to using a custom application. Some replies remarked that while using the art package is sometimes the shortest path, it is not necessarily a good way to go. In fact, “...for PS3/X360, the tools don’t handle dense scenes well, so the art package doesn’t work terribly well as a world building tool.”

What 3D art packages are people using? It is conventional wisdom that in the U.S. most people are using 3ds Max or Maya, while in other regions packages like Softimage are more popular. Our responders replied that 65.9% of them are using Maya, while 53.7% are using 3ds Max (some are using both in their pipeline)... and one person was using Blender.

Conclusion

The technologists in general echo common themes. In addition to a desire for rapid prototyping and iteration capability they say: please, make it easy for me to understand your game engine, and understand whether it will work well for my game. Give me a standalone world builder and live preview on the target platforms, and give me the option of easily integrating third-party middleware libraries if I so choose. Give me source code, good documentation and great support, and the ability to profile the engine so I can find my performance bottlenecks.

Games have differing technology requirements, and the engines that will work best vary depending on the situation. But on these fundamental requirements we agree.