Interview: Midway's Palsson On Sharing Core Tech Worldwide

Publisher Midway has invested heavily into its central technology group, which aims to create a company-wide development backbone that can help streamline game production across numerous genres. The company is using Epic's Unreal Engine 3 as its starting point, adapting it to fit the needs of titles that stray considerably from its shooter roots. The Wheelman producer Pall Palsson is involved in central technology out of Midway's Chicago studio, until recently having worked at the game's primary developer Midway Newcastle. The open-world, driving-focused game represents a particularly shart departure from the traditional usage of Unreal Engine 3, making the game a key early player in Midway's gamble on the investment. Gamasutra caught up with Palsson to discuss the thinking behind the central technology group, how Midway manages the worldwide logistics inherent to it -- and how the team is preparing for the inevitable console generation shift down the line. What's the goal with the central technology effort? PP: We have all of Midway's games running on the Unreal Engine 3. To take the Unreal Engine 3 and to do a game like Wheelman -- which is an open world, streaming game -- takes a lot of engineering effort, because that's not what the engine is built for. So there was a lot of collaboration between a bunch of studios, to get that thing working, which led me to move more into a central role in technology. It's been interesting, in terms of this move to centralized tech, because I don't think any other studio is really trying to move onto one engine platform and make it work on a wide variety of genres. PP: Everything. No, I think you're absolutely right, and it certainly hasn't been easy; it's been a challenge, but right now we're getting to the point where we're really going to start reaping some of the benefits. Wheelman for example: We've obviously got it running as a fully open, streaming world. In addition to that, we in Newcastle, because the pedigree of the studio is racing games, we've worked extensively on the vehicle AI, the handling of the cars, the physics, and all that stuff, to make a great racing game -- but we also wanted to have some on-foot shooting. Now, that's not necessarily something that our engineers know how to do, or our designers. But we have shipped a game like Stranglehold, which is a great run-and-shoot type game, and because of that, because of the shared technology effort, we could use Stranglehold designers to make our game, because they know all the tools, and then we could also just grab some of their technology. If they have an auto-aim system, we can grab their auto-aim system, and that's incredibly helpful. It's my understanding that This is Vegas and Wheelman have been in development concurrently, and there have been efforts to have one team work on one technological feature, and have the other team work on another technological feature, shared back and forth, to an extent. PP: Absolutely. I mean, Vegas took the lead on the streaming technology, and we then took a drop of them, when they'd been working on it for a while. [We] modified it to fit our world, because we have a lot more high speed than they do; they grabbed our traffic system, and I think there are two other studios within Midway that have our HSVD -- which is the High Speed Vehicle Driving AI. So definitely -- there's no point in Newcastle spending a bunch of resources on something that another studio is also spending a bunch of resources doing, when you can share it. How modular can you make it when you're working on these solutions? PP: In the ideal world they would be completely modular. (laughs) But it often takes more time to build a really nice modular system, so in reality, when you're making a game, and you're crunched for time, you may have to write something in a way that's very specific to your game. But then we have a centralized tech group in Chicago that goes into game teams, looks at what they've done, and goes, "Yeah, that could be shared between multiple teams." Then they take that, refactor the code, and make it more modular and easily-applied to a bunch of our games. Also, at Newcastle, we might go directly into the Vegas codebase, and grab some of the stuff that they're doing that is applicable to us. Do you have any developers in the central team working on features like that, or is it all the developers who are working on those specific games that are working on those features? PP: No, the centralized group will work on features, definitely. The centralized group tends to do the more heavily technical stuff, like some of the serious SPU optimizations for doing things for the PS3, and moving systems onto that; that will be done by the centralized group. They're doing fantastic work with particles right now. Those engineers get to work on systems and features as well. It just depends on what their priority is. If we say to a team, "You can have four of the really good centralized tech guys," and they want to have them do gameplay tasks, they are perfectly capable of doing that. Do you just look at their work remotely? Or do you actually talk to people, hold video conferences? Because you have studios all the way from Newcastle to Chicago to California; that's a wide geographic spread. PP: It is, and it's often difficult to get them together, because between Newcastle and our California studios, there's an eight hour time difference. That's a working day, basically. So if we want to all get together, then either Newcastle stays late, or the other guys come in a bit early. We'll basically do both; we'll both visit studios. Also, we have a lot of initiatives that help designers share ideas; the designers will get together on vidcon and share tools. I mean, I'd say the vidcon is the primary source of our communication. Building For The Future Midway has spent a lot of money and effort on the core tech. How long do you think that effort is going to last? Inevitably, we're going to reach another generational leap. PP: Yeah. The effort that we put in to our current engine is not lost when we move to the next generation. We can still port a lot of that stuff over, even though we might have to rewrite the core renderer, or whatever, but you'll still be able to carry a lot of those core systems over. Where Midway is really going to start seeing the benefit is when they start doing Wheelman 2. When they do a second iteration of a lot of the games that they've already done, you start getting into that sort-of 2, 3, 4, and making the fourth iteration of a game is not nearly as hard as making the first, and you're building on top of all the other stuff, so you're really end up building a game that should kick ass, and be cheaper to make. That's another part of your mandate, right? It's not only to create technology that's portable amongst the studios, but also to create technology that can endure. PP: Absolutely. Midway takes painstaking effort to make sure that any code that gets integrated into the central team is well documented, and there's just a lot of information available about that. Because if we have a shared system that no-one knows about, it doesn't really do anyone any good. I've talked to Mark Rein about implementing features in the Unreal Engine, because everyone I talk to seems to work with the Unreal engine -- most of them seem to quite like the experience, but all of them do a heavy amount of customization to it, because it's not necessarily applicable to what they're trying to do. PP: Yeah, well, the Unreal Engine is an engine made to create first person shooters. You [have to] realize that when you buy the engine, and accept the fact that you have to modify it if you want to make something else. You know, BioShock was a very good game, in my opinion, based on Unreal, and they do little areas that they load in, and they have the little airlocks that hide the loads for another area. If you then want to take that engine, that obviously does that really well, and make it into an open world game, you have to be prepared to spend that engineering power. But what you get with Unreal is, you get an amazing toolset with it. You also get a base to start from, and the support from Epic. We were talking about pulling in some of the features that have been created by external teams into the engine, and he said that was something they couldn't do, because if they don't personally know about it, they can't support it. PP: But there's also the fact that if Midway pays Unreal to use their engine, then spends millions of dollars engineering something, we're not really going to give it back to Epic, so they can give it to other teams. The relationship would have to go both ways. PP: Yeah, absolutely. If Unreal Engine 3 was developed as a shareware effort, then yeah, absolutely. Do you see the potential of any sort of collaborative open-source engine tech? PP: Generally, if you're looking for a mainstream engine, I don't think there will ever be an "open source" engine. The actual amount of manpower that goes into making a good engine is phenomenal. And then you need the level of support, as well, because if something crashes very deep in the code, you need the guy who understands it; you need that support. You don't want to have a fifteen or twenty million dollar game sitting, waiting on a bug that happens somewhere in code you know nothing about. It doesn't seem feasible -- but it was an interesting idea! PP: It's a lovely idea, because I would love to see game development get more to sort-of its roots, where you can do a game at a lower cost, with a couple of guys, and we could start seeing games made in garages again. Distributed Testing I was talking to some of the Wheelman team, and one thing that I found that was interesting about the game was that they said there was a lot of testing -- every two weeks, testing it with focus groups and all that. PP: Yeah, absolutely. We have focus group tests as frequently as we have something to focus test. Developing a game is really, really, really expensive, and it really doesn't make any sense to develop a game, put it on the shelf, and then no one likes playing it. When you are developing a game yourself, you get blind to the little faults that it has, and something that is blatantly obvious to you is just not going to make sense to someone new, someone who has never seen that feature evolve. That's often what it is: A feature goes through multiple stages of iteration, and because you've seen it all the way through, you know exactly what it does, but someone who comes in fresh to it, doesn't so much. So that's why we spend a bit of money getting people in to do some focus testing -- get them some free games or pizza or whatever, monitor that, and then get feedback, then maybe get the same group in when we've done the fixes, and maybe see how they do. Do you feel that there's a regionalization? I got the impression that they did most of their focus testing locally, getting people from the Newcastle area; do you also want to have focus tests, say, in America, or in Asia, to understand the game's response? PP: Yes, absolutely. Most of the focus tests that we do, specifically for us, which are the frequent ones that we do, we do in Newcastle, because we need our designers to be able to basically be there, and look at it; and we can't really afford to lose them for the time that it would take to send them to the states. But, at the same time, once in a while -- probably one out of every three or four focus tests that we do -- we do out of San Diego, because Midway has a group there that has handled focus tests for all of Midway's games. They'll do a blind focus test for us, we basically give them the questions that we want asked, we give them the levels to play, and they give us a bunch of guys who'll sit down and play the game. And so, yeah, there's definitely a difference. One thing that I've been talking to people about is that things are taking more iterations to get right; once they get feedback from focus testers, they're taking more iterations than they expected. People are more confused than they expected, perhaps, and they have a tendency to correct that confusion, or make the game more accessible, or comprehensible. PP: It's difficult to say. I think it really depends on the feature. For example, for Wheelman, some of the things that we're doing aren't necessarily intuitive. I mean, we're doing stuff that has never been done before, and that is absolutely true. You'll see a problem, and there will be our airjack, which is the ability to jump from one moving car to another without having to stop. If your car is on fire, you can jump to another car, and that car now has full hit points, and you can continue the mission. That feature, to activate it, was fairly simple, but it caused a lot of confusion for players. We realized, "Oh, that's because they think that they need to be side-by-side to do it, and not behind, which is the way the move is triggered." Right. Logically, you would assume you'd smash out the passenger window, or something, and slide in. PP: Absolutely. But for us, because the way the feature was designed was always from the back, it never occurred to anyone in the development studio that that would be the most logical way to do it. Because the idea, when someone, had it, they had it from behind, and that's how they pitched to the studio, and that's what was ingrained in everyone's head. That's easier from a playing perspective; it's just not necessarily the most logical way from a "thinking about physically being in a car" perspective. PP: Absolutely, absolutely. Yeah, so when we fixed that, we made it clear to the player that they had to be behind, so we fixed what we thought was the major issue. And then we got into a timing thing. They didn't quite understand that they needed to hold the button, and then release it, and all of this kind of stuff. So we had to then change the mechanics, and because you have like two weeks between focus tests, it does take a while to get it right. But the really important thing to remember is that you're doing it so that the end user who actually gets the game in their hands actually enjoys it. And at least we're not releasing product where we have a really cool feature and they're just going, "I have no idea how to do that." I always think about Driv3r, when it came out, and every review said, "Why do I stop dead when I hit a light post?" If they'd just had some people play it, and listen to them complain about that, it wouldn't have been like that. PP: Yes -- but it also depends on your publishing partner, because I've worked on games in the past that weren't as good as they should have been, and that is mostly because there wasn't enough development time on the project. Pretty much always, the people who worked on the game know what the major issues are; they just don't have the time to fix them. Midway has been really good with going, "OK, you are making a really good game -- we want it to be great. We're going to give you more time." They want to release quality games.