Opinion: Why On Earth Would We Write Our Own Game Engine?

[In this reprinted #altdevblogaday opinion piece, Blackbird Interactive CTO Yossarian King looks at the various reasons why you might want to (or might not want to) build your own game engine.] I'm addicted to building technology. I can't help myself, it's just so fun and shiny. Tools, games, and especially engines. Mmmm, game engines. Sound familiar? Why do we feel so compelled to write game engines? Shouldn't we be building games? In 15 years with Electronic Arts I used and worked on several proprietary game engines. This seemed to make sense at the time, but when I left EA to help start Blackbird Interactive our technology choices were very different. As the lead tech guy, I had to get a foundation in place to support our development. I quickly went from "should we write our own engine?" to "could we write our own engine?" to "why would we write our own engine??" Faced with the buy versus build dilemma, we chose to buy. This article explains why. By game engine, I mean the core technology foundation that underlies a game. An engine abstracts game code from the specific hardware platform, provides key functionality such as rendering, animation, physics, and networking, and usually includes a tool chain and content pipeline. High quality game engines and integrated middleware are readily available. Engines like Unreal, Unity, Torque, or CryEngine could do our heavy lifting so we can focus on our game. Engine prices are flexible, from free to cheap to a share of our eventual revenue. As I started investigating the alternatives, I soon found myself asking why on earth would we write our own game engine, when we could leverage existing technology and immediately get to work on our game? On the other hand, many game companies, even small ones, choose to build their own proprietary engines. The decision whether to build or buy engine technology would have a huge influence on all our future development. The choice could not be taken lightly. When I worked at EA on FIFA Soccer, I really didn't have time to look at other technology. Or at least that's how it felt. With iterative products, six-to-twelve month development cycles, and ship dates that didn't move we had to focus on the task at hand, not gratuitous experiments with novel technology. Electronic Arts was also rather insular. There were so many smart people and so much strong technology within the company that if I needed to reach out, I would contact someone I knew in another department, or at another EA studio. Since going indie in late 2010, I've been reaching out like crazy. A dozen guys in a garage cannot stay internally focused. I have benefited from the wisdom and experience of many different communities, former co-workers, and free or cheap technology. Game engines are a case in point — some are free, most at least provide trial versions. Mark DeLoura's 2009 Engine Survey and GDC 2010 followup was a great place to start as we began our evaluation. We Chose To Buy In the end, we chose to buy, not build. Here are our reasons... The Features We Need Are Mostly Supported Scott Greig, BioWare director of programming, gave a great talk at the Vancouver International Game Summit in 2008. To create the BioWare technology roadmap he started with a poll of key staff on which features needed to be world class in order for the company to hit its goals, which had to be competitive with the rest of the industry, which just needed to be functional, and what was unimportant. With this in mind, I made a long list of key features and technology areas, and asked everyone in the company (all eight of us at that point) to rate our need for each feature on a scale from world class to irrelevant. For us large environments, vehicles, heads-up display, and audio landed at the top of the list, with characters and MMO networking somewhere down the bottom. The survey didn't produce any great surprises, but it was good to see that we had a consensus, and it made everyone feel involved in the technology evaluation process. As we evaluated different engines, we used the feature set as a checklist to see how well each engine covered our requirements. Most engines were capable of doing the things we needed, so this wasn't a critical factor, but it definitely played into the decision. When I was lead programmer on FIFA Soccer for the PC, around about the year 2000, we rewrote the graphics engine three years running. The PC team worked alongside the Xbox and PS2 teams, each of which used different engines. Down the hall, other large teams worked on basketball, hockey and baseball games — you guessed it, each with their own unique engines. Apparently when a bunch of human figures in sports uniforms run around inside a virtual stadium, the size and shape of the object they hurl around, and whether they do so with their feet, hands, or a stick, makes a big difference to technology requirements. Er, not Each development group evolved their own technology over time, and arrived at unique solutions to similar problems. If we had shared our feature requirements from day one things might have worked out much differently. All Our Target Platforms Are Supported When an opportunity comes up to put your game on a new platform, the right engine can make this almost trivial, while the wrong one can make it well nigh impossible. Building your own technology gives you more flexibility, but at the cost of doing the grunt work of porting yourself. Choose the right engine, and the grunt work is done for you. In addition, developing for multiple platforms simultaneously is much easier, since the bulk of the platform-specific code has already been written. For us at Blackbird, platform requirements were nebulous. We wanted to launch on social networks like Facebook, and then move onto mobile platforms including iPhone and Android, but tablets looked pretty good too, and a standalone PC build for Steam distribution also made sense. Consoles were a possibility in the future, depending on the success of the game. With platform needs shifting as we analyzed the market and considered different partners to work with, flexibility was key. High end web graphics were not broadly supported, and mobile was hit and miss. Unity had done well on both fronts, and their console support would no doubt mature by the time we needed it. Platforms are more stable now than they were 10 or even 5 years ago. Back in the late 90′s and early 2000′s, 3D graphics on the PC was a hotbed of activity. Hardware acceleration was a new thing. Custom API's like 3dfx Glide and PowerVR were competing with the nascent D3D. At EA, we needed FIFA to look great on hot new hardware, but still run well on older computers with software rasterization and 256 colors. A custom, rapidly evolving, hybrid engine was almost inevitable. On the consoles, Xbox was like a clean, simple, fast PC, but there was no point burdening it with all the legacy rendering code, and unfortunately our engine wasn't tidy enough to simply take what we wanted. So we started from scratch. As for the PlayStation 2, Japanese trade officials had placed restrictions on export of PS2 dev kits for fear they might be used for military purposes, so our PS2 guy got shipped off to Japan for several months to work in isolation with a snapshot of the PC codebase. Needless to say, the code diverged. New platforms still exist, but they come in large equivalence groups ("consoles and PC", "mobile", "online"), each of which can be served by a single engine. Some engines even span all the groups. Rumor has it that Crytek and Unreal are both considering Flash support. You could argue that HTML5 is a new platform that requires a custom engine, but spend an evening trying to find and run HTML5 demos on different browsers and (cool as they are) you'll see HTML5 is not quite ready for prime time. Besides, Adobe's already way ahead of us. Engine developers today are well-organized, well-funded, and well aware of emerging opportunities — it's unlikely we'll beat them to the punch. If you work on truly bleeding edge platforms (iPad 3? PlayStation 4?) you may not get the support you need from a third party engine, as happened to Silicon Knights when they tried to use Unreal Engine 3 on the Xbox 360 before it was ready. On the other hand, perhaps there is an opportunity to partner with the engine developer and accelerate both your progress. Good Workflows Are Available Immediately Never underestimate the importance of good tools for your artists and designers. And never overestimate your ability and desire to deliver them — especially when push comes to shove and you have to decide whether to add a new feature to your game or a new feature to a tool. Chances are high that the game will get your attention, at the expense of the tool chain. A good engine should enable good workflows out of the box. As we evaluated engines, we looked not just at runtime features, but also at the development environment, the tools provided, and the content pipeline. Visual concepts can quickly be tested with an art pipeline that quickly takes models, textures and animation from our art package into the game. Workflows are easy to overlook or get wrong. Being at a small company like Blackbird I have direct hands-on experience with every aspect of the game — I know how everything works and what everyone does. This gives me great insight into the workflow requirements across the team, but time is limited and it is hard to do much more than meet the minimum needs. With a working foundation, I can focus on adding small improvements rather than building complete pipelines. At EA, the opposite was often true — large teams and specialization act to limit interaction between different disciplines, so the programmers are much less likely to know how the artists and designers work. The result is that workflow improvements frequently fall through the cracks or get implemented poorly, because the person implementing the tool feature didn't really understand how it would be used in practice. Most of the third party engines we evaluated had better workflows than the proprietary engines I've worked with in the past. While it is certainly possible to build good workflows internally, all too often they fall by the wayside as the schedule slips and all engineering resources are put onto game features. Anything That's Missing, We Can Add Extensibility is vital. No engine will do everything you want out of the box, you will need to customize, tweak, and extend it to suit the needs of your game. Custom content pipelines, new editor windows, and of course unique features for your game itself. Clearly building your own engine gives you the ultimate in control and customization — just remember that you have to build it before you can extend it. With a third party engine you can jump straight to the extensions. Source code access is a hot topic when it comes to engines. When you write it yourself, you can see and modify any source code at any time. Many people feel that using an engine without source code access is asking for trouble. "What if there's a showstopper bug?" My experience has been that showstopper bugs are much more common in my own code than in commercial software, and I'm willing to give up source code access in return for well-supported, high quality products. We Need To Focus On Our Game, Not Core Tech Our game is completely new — new IP, an unusual camera, a curious mix of client and server requirements — it will take a ton of experimentation to prove out gameplay ideas and confirm and refine our design. We can't afford months of tech development before we start testing the limits of our game design. With a third party engine, core systems like the event loop, game object model, input handling, mesh rendering, particle effects and animation just work, so we can jump immediately to prototypes. We decided that the opportunity cost of building an engine was too great — those months of programmer effort are much better spent on the product itself. We have no illusions about licensing the tech we develop, we have no plan to produce the Next Great Engine as a side effect of our product development. The market for middleware is active and healthy — but very competitive, and the leaders are extending their lead on a daily basis. If anything, we might build a few special-purpose tools that integrate with major third party engines that we could sell to the people who use those engines. A good example of this approach is Autodesk's integration of their Scaleform UI library with Unity. And with over 6,000 employees and annual revenue approaching $2 billion, Autodesk's involvement in game engine middleware is also a good example of the depth of competition in the middleware market. At EA we focused plenty on core tech, but we still never had any plan to sell it. (Besides, who would license a render engine with functions like RENDER_ball and RENDER_stadium?) In fact we thought our internal technology represented a significant competitive advantage that we couldn't possibly let other have. Though there was an embarrassing incident in which a significant chunk of the FIFA PC source code from 1997 was accidentally included on a demo CD. I suspect that if this fell into Konami's hands, it may have set International Superstar Soccer back by several years as they tried to figure out just what we were up to. We Save Money We did the math, and developing our own engine would be way too expensive. Sure, we could whip up a quick graphics demo in OpenGL or Direct3D in no time. Add some audio, google up some physics code, and in a couple days you'll feel like you're halfway there. But what about memory management? user input? animation? the file system? a content pipeline? tools for artists? cross-platform compatibility? Suppose it would take four programmers three months to get a decent engine in place, supporting a subset of features sufficient to create a particular game, with tools and workflows good enough to get the content team started. That's one staff year, worth say $100,000, give or take. For that money, you can buy over 60 Unity Pro licenses, or over 500 copies of Torque 3D. Granted, Unreal and CryEngine will set you back significantly more than that, and probably want a hefty royalty as well. But remember, you're licensing a complete product — for $100,000 you're only getting the prototype of a game engine, just enough to get started on, with likely lots of bugs and a pile of missing features. Also keep in mind that most of these engines have free versions you can get started with — just be careful what you get hooked on! For us, we were able to run with the free version of Unity for over six months before we decided to upgrade to Pro. Don't underestimate the time it will take to write an engine. Why not invest that effort into the game itself, and not into the foundation required to support the game? There is a lot to a modern game engine, and it is a huge effort to build one. Look at the number of people in the Unity team photo, or the number of programmer job openings at Crytek. Sure, you don't need to write a complete commercial product — but why not use one that already exists? At EA, money was rarely an issue. At Blackbird, it's a huge issue. It would have been irresponsible to spend our investors money building another game engine when there were several engines to choose from that would serve our needs and let us focus on developing our game. You May Choose To Build Our situation is not unique, but it's not universal either. Just because we decided to buy an engine doesn't mean you can't build your own. Here are my thoughts on when it makes sense to build. You Already Did If you have already written a game engine that you use in your games, switching to a new engine is a huge challenge. If you are happy with what you have, more power to you — if not, well, I can sympathize. From the late 1990′s I worked on eight consecutive iterations of FIFA, and there had been at least three products before that and at least eight since. The very first version was written for the Sega Genesis in late 1993. I'd put the odds about 8 to 1 that none of that Genesis code runs on the Playstation 3 today — unlikely, but not impossible. The codebase evolved incrementally over time, with systems added, removed and refactored by generations of programmers. When you have a stable, mature engine, it is extremely difficult to stop using it. You need to tear down your game and build it back up on a new foundation. When I was a kid, my dad did just that with our house. He poured a new foundation and built a floor on it, cut the house off at the ground floor with a chainsaw, and used hydraulic jacks laid horizontally to push the house across greased skids onto the new foundation. Mum took us kids away camping for several weeks. You can rebuild, but it is time-consuming, messy, and extremely disruptive. The opportunity to change comes when you start a new project. If you begin a new game, at least consider alternative engines. Take a couple weeks and prototype something in Unreal, or compare the Crytek physics solution with your own, or look at the editor extension capabilities of Unity. Analyse the costs and benefits of your own technology versus someone else's. Even if you decide to keep using your own engine, you will get ideas from others that you can use to improve your own systems. If you decide to switch, be very careful. EA Black Box recently shipped Need For Speed: The Run, during the development of which they switched from a custom engine to Frostbite 2 (an EA-developed engine, but external to the Black Box team): "According to designer Alex Grimbley, it has apparently taken a year to re-purpose the tech for driving rather than shooting." (Wikipedia) The switch was intended to help "really blow the doors off the category," but despite a three year development window, the game is currently sitting at a Metacritic rating of 64. Granted it was an ambitious project and many other factors are at play, but a disruptive technology shift can't have helped. Your Boss Told You To Doing what management wants is typically a good idea, at least if you want to keep getting a paycheque. You can certainly question their decisions (if you can't, perhaps it's time to move on), but at the end of the day they make the call. At EA, in about 2004, I was briefly the tech lead for the EA Graphics Library (EAGL) team, a central tech group that had developed graphics technology that was used in a large and growing number of teams across the company. EAGL was what finally brought the PC, Xbox and PlayStation 2 versions of FIFA onto a single graphics solution (not to mention hockey and basketball). I joined the EAGL team at a time when EA decided it wasn't in the business of building technology, it was in the business of building games. Management chose to acquire Criterion and its successful Renderware graphics library and have it used by as many teams and games as possible. This made some logical sense, but the strategy backfired. Renderware was acquired on the strength of Renderware 3.7 on the PlayStation 2, but then Renderware 4.0 was adopted for the PlayStation 3. Renderware 4.0 was a ground-up rewrite that was not ready for prime time. To make it worse, we all grossly under-estimated the cost of transitioning to a new render engine. Ironically, Renderware 3.7 the strong third party solution became Renderware 4 the failed EA proprietary technology. In attempting the break the cycle of over-investing in internal technology, we instead repeated it. Renderware is no longer a commercial product. Whether your boss tells you to build an engine or buy one, make sure there are solid business and technical reasons for doing so, and a clear execution plan. It's Fun And You'll Learn Something This is a great reason to write your own engine. Building all the systems required to make a game will be a lot of fun, and hugely educational — you will definitely become a better low-level developer. Reinventing wheels is a great way to learn about wheel-building, but if you want to race cars it may not be the best place to start. Before you jump into that engine rewrite, consider using an existing engine and focusing on AI and gameplay. And unless you are independently wealthy and can afford years to build your game, writing from scratch may not make business sense. After 15 years making games with proprietary engines, I've finally learned my lesson. I'm still addicted to building technology, but now I'm building games, not engines. You may not agree with me, but at least consider the alternatives and make your engine decision a conscious one. If you have ideas and experiences on the build versus buy game engine debate, please join the discussion, I'd love to hear from you. [This piece was reprinted from #AltDevBlogADay, a shared blog initiative started by @mike_acton devoted to giving game developers of all disciplines a place to motivate each other to write regularly about their personal game development passions.]