Development Lessons From Killzone 2: An Interview

In February, Sony-owned Dutch developer Guerrilla Games' long-awaited Killzone 2 shipped for the PlayStation 3. Sony has claimed that the game was one of its fastest sellers ever. But the game, as with any long-awaited title, has not been without its controversies.

Of course, it was announced with a trailer that proved to be a target render -- something that you'll see, from the interview, still stings the developers four years later. Still, that's mostly washed away by the title's success, with an impressive 91 Metascore and positive fan response.

But even in that success, the learning process has not finished, reveals development director Arjan Brussee. In the following interview he, along with the studio's managing director Hermen Hulst, discuss the company's evolving understanding of its space in the marketplace and how development continues on the game.

In the following interview, conducted at the recent Game Developers Conference, Hulst and Brussee discuss the size and structure of the game's team -- which topped out at 190 people.

Multiplayer and single-player development and testing are contrasted, the process of patching the game is discussed, and even the rendering engine is stripped back for the world to see:

You guys finally shipped. I shouldn't say "finally". Every game is "finally", right? No matter how long or how short development is.

Hermen Hulst: It's fine. We've given birth, so we know what it feels like now.

How do you feel to have gotten through it, gotten it out?

HH: I think we're pretty ecstatic. It's great being out here at GDC. We've spoken to a lot of peers. One of them actually, earlier this morning over breakfast, said, "Well, it must feel for you guys like you've climbed a mountain and planted like a big fucking flag on top of the hill." And that's what it feels like after all those years. It's been four solid years, quite intense.

About how many people did you have on the team working behind the production?

HH: We peaked I believe for a very brief period at about 190, but much more sort of steady, we were at about 140.

Wow. That's still a huge number.

HH: It is a big team, yeah. It's a big single-player campaign and it's a big multiplayer online experience. And that was at the end. Of course, we started with about 55 people. We scaled it up over the years.

How did you structure the development of multiplayer and single-player? With Resistance, some of the content was related or repurposed, but ultimately, it was more like silos, I think. Did you take a similar approach?

HH: I think... And Arjan can tag onto that later, but I think looking back, it wouldn't have been quite healthy from a production management perspective had it been more integrated than it actually turned out to be, to be completely frank with you. We just have one game director that was technically overseeing both creations.

However, the use of the word "silos" may be a bit extreme. But they were actually two different tracks, particularly on the design side. Obviously, from a technology and an art perspective, there's a lot more integration there. That is definitely I think something that we are looking at going forward.

Arjan Brussee: I think the main difference is that with online, we started having playable in a good fashion so much earlier. I think we started three years ago with a thing that we called Friday Fragfest where we just had the whole team play through the game. At first, it was super laggy, no animations, mo-cap, etcetera, just trying out things.

That iterative approach has really, really worked well. That was in good shape real early on. I think that's required for online, so you can't really take the same approach as you do for single-player, because only at a very late moment of time, you start reaching the kind of quality that's required.

Obviously there are balance issues with multiplayer that are really important, which that really benefits. But also, is it because of content creation in a sense that single-player is waiting for assets and all kinds of things to come online?

AB: Well, they share a lot of assets, right? But I think online is getting fun real quick, even if you're in a mo-cap environment and you have a couple guys running around. It becomes really fun already, so you can test the [ideas] in a very base fashion really quickly, and really quickly iterate.

So you can see, you know -- the medic. Is it being played really well? We know -- that's a thing that we found out two years ago. We have three weeks, and every week, a different kind of medic implementation. The test is like that, to figure out what is the best way to get people involved in being a medic. And that kind of approach really worked well. You can do it really early. On that side.

HH: A single-player campaign is only fun once there are enemy types that each have their own sort of behavior, reasoning with the world. That's one of the lessons we learned from Killzone 1 -- AI needs to be a lot better. And we've spent very considerable time with a considerable amount of people on that, and I think that's what Arjan refers to do. You don't require it in multiplayer.

A lot of things are required for single-player campaigns. Design is obviously really important in single-player and multiplayer. Before things were nailed down, you could immediately start testing. Did you resist design documents for the multiplayer because you were being iterative?

AB: [laughs] I think I've made a couple of games in my life now, and I think the online component is one of the only games that I've ever worked on where the initial design is actually more than 90% translated into the final product. That really worked well. Our vision, we really kept true to that. But we did actually still iterate. We still throw away things, try things, etcetera.

We don't value design documents that much, but you have to have that vision. You have to have those poles in the ground. What is it going to be about? What are we going to do with squads and badges? And those kinds of notions were available really early.

You can iterate as you like. You don't need 300-page design bibles. But you know, that 30-page initial vision, and you have people that are really hands-on and driven towards chasing their ideas throughout the company. That's more valuable than any piece of paper.

How did you communicate? You say that 90% of the original design came through in the end, and there was a really long process and also a very large team. So how did you communicate that and keep it consistent over that? That sounds like a pretty impressive accomplishment to me.

HH: I think what Arjan just mentioned just now is a very, very critical part of our process. Every Friday, everybody in the company drops what they're doing and plays the game, so people at all times are aware of the state of the game. There's no better way than sharing where you're at and what you want to achieve than forcing people to play the bloody game. So that's probably the single most important thing, forcing that.

Yeah, so it has the benefit of everyone being on the same page essentially.

HH: It's got that, plus you get feedback from a hundred guys. A hundred guys that play a lot of games, and getting that feedback and doing something with it, never dismissing that, responding to it, week-by-week. That's very early user-testing.

AB: And for online, that works really well. Single-player, it's a bit less so. The pipelines are a bit deeper, so it takes a long time for people to see their stuff in the game, finally playing. But in online, because of the quick iteration and the kind of way that you structure it from mo-cap to final, the game was fun and completed before there was any art.

So, that's a really interesting approach. It worked wonders for that. And that meant we could also have other teams, even within Sony, work on the online levels to help us out, so we didn't have to touch it up ourselves anymore because we knew it played well. "Okay, sure. Now make it look nice," you know.

Totally changing track, relatively soon after the game shipped, you came out with a control patch. I just want to talk about that process. What spurred it? And what was that like? Because prior to this you guys have had PC development experience, but this is the first console game where you've been able to ship a patch.

AB: Well, we've done patches also for Killzone 1, right? Though the systems then were less meant to do these patches, but still, we were able to do that.

I was talking to some other guys out of Bungie at GDC on the idea that games don't ever finish anymore, right? So, you go into this kind of service model, and you keep on improving your game. We kind of found this issue, we investigated some things, and if we find issues that we think, "Yeah, that's a bug, we have not seen this" or "we've forgotten about this," then we can patch it, and we have that system.

We're thinking about continuous improvements to our game. If we find issues, we now have a million-plus people playing the game actively -- if you look at the Killzone.com site, [that's] how many people are playing it. So they're bound to find issues or cheat.

You know, we had an issue with auto-aim being a kind of cheaty solution sometimes, so we fixed that, just to make the experience nice for everybody. I think that will continue for the next half year.

You know, the Halo guys are working on Halo 3 still two years down the line. I think that's the kind of model that you're looking at. The Left 4 Dead guys, I think they did 70 patches or so? And they slip it under the curtain. You don't see it anymore, with Steam. That's the kind of way that you need to think about these kinds of things.

So, I think it's a strong thing. It's a strong thing that we listen to our customers and we fix these things when we find them, and just help improve the game continuously.

First of all, you must get a huge volume of data with a huge audience like that. But also is it things that people say? Do you also mine data based on server logs and stuff like that?

AB: So, data mining is a big thing for games, right? I've been attending all these lectures at GDC, and everybody is doing that. You look at Valve, the Steam stats, that Bungie thing, and we're doing a lot of stuff on Killzone that's going with the Battle Replay. It's really awesome, I think that's the next step in how you see that kind of stuff evolving.

And trying to get that data back in and anonymize it -- within the legal bounds of not dealing with personal information -- there's a lot of value to be had there. I think that you can also use it to keep on improving your game.

And even the single-player aspects. You know, if you find levels that have too many deaths in certain sections, you can kind of fix those during development but maybe also after development. We can also patch such things if we need to.

And just analyzing all that data, you know. 700,000 or more people, how long are they playing? Why are they leaving? What patch do they reach? Is it our ramp up of reaching patches? Is that good enough? Does that need to be steeper or not?

Those kinds of patterns, we're all analyzing and thinking about, "Do we want to fix this for Killzone 2 or for our next game?" Such things, you know, it's really important.

You are also getting data about the single-player game. Is that something you prioritize when releasing patches, or is it mainly multiplayer?

AB: So, the longevity of the game is certainly in multiplayer. People are spending way more hours in multiplayer. That's dynamic, you know. It's where people are talking about the cool games they're having and inviting friends over. It's the key to success, to selling really, really many millions of copies.

HH: Plus single-player, you can user-test it a lot better before release. So, you know, there's only so much you can do in terms of duration and in terms of amount of players in a beta. We try to do as much as we can. There's always stuff you're going to oversee, that slips internal QA as well as external user-testing, particularly on the multiplayer side.

It's also a complex game, right?

HH: It's rather complex. I mean, it's very feature-rich, and it's got a lot of stuff in it. A lot of stuff can be changed.

To change tracks, Alex pointed out that you had a real-time demo of the 4D bullet stuff, so I'm actually really curious... [note: the demo was released onto PSN April 2.]

HH: Couple opening comments on that. You've seen the trailer probably on the TV here, but we wanted to create an interactive version of it, kind of as a testament of our technology and kind of a statement that we as Guerilla Games no longer have to rely on pre-rendered stuff to communicate our vision.

I think this is a great way of doing that. Arjan is just demonstrating it now. He tracked the bullet across the battlefield, just like you are in the trailer, however...

Obviously, this is scripted and for show, but when you do something more experimental like this and more visual, does this give you a jumping off point where you can do something? Like give you ideas to experiment for gameplay.

AB: Yes, I think it also has that, but more importantly for us... So Martin came up with the idea of having another kind of trailer and let's render it in-engine. You know, "Let's do it in real time! Let's get away from that whole fricking trailer thing, ever."

[laughs]

AB: That was kind of scary, but we managed to do that.

But also, what you see is that you put in Hollywood quality, 100,000 polygon models into an engine, and to see how that pipeline works is really interesting. Sometimes in development, we think, "Well, this is so difficult. Nobody understands how we do this kind of content."

And we've got people from the special effects industry, that helped us with throwing some high-end models in. And that actually worked fine, so our engine can even handle that kind of stuff. And we had to fix a couple things, so we found a couple of things that actually we can improve. So with future games, we can do even more high-poly models, etcetera. So it's a really good learning experience for us.

That's actually really interesting. Because to a certain extent, you have expectations when you create technology, both based on the hardware and based on the application of the technology. But to find that when you're asked to exceed some of your expectations, that's what it creates.

AB: Yeah.

HH: We definitely added stuff to our technology base on the back of this project.

AB: So, for instance, I think that you can see here in this scene, featuring now, [indicates screen] is a system... This is actually a real-time reflection. So, that wasn't in Killzone 2. We developed this for this demo as a kind of test. So, you know, we'll use that in future games. We can now do good reflections, so it's really interesting.

But this is the Killzone engine. Is that was the engine is called? Or do you have an internal name for it?

AB: Oh, it's the Killzone engine, yeah.

It's something you would potentially use. It's your studio's tech.

HH: Yes. It's fully proprietary.

You will definitely leverage it for any project that your studio does. I'm assuming that's your goal.

HH: Yeah, a lot of it has come from previous projects. It's evolved, going back a long way.

AB: Another thing that this features that's kind of cool, is you can actually see these different buffers. Let me slow it down a bit -- because you can actually see it run in real time.

The things that I explained before in GameTrailers, the different depth buffers or lighting buffers and etcetera. So, it is kind of this tech demo in a sense that people can have a view about the underlying technologies that we used to make that.

HH: Are you somewhat familiar with what we're doing? We created this deferred rendering engine that allows us to composite this image by layering various special effects, lighting, various material buffers on top of it? Maybe take a step back, Arjan.

AB: Yeah, sorry. [laughs]

HH: You're assuming too much knowledge already. [laughs]

Actually, that sort of leads into my question, right. So, this is going to be on PSN, right?

HH: Yeah, that's right.

So, what do you think about pulling back the curtain for your audience, you know what I mean? Letting them see how these things are. I mean obviously, you know in film, a lot of making of features hit DVDs and whatever, but this is sort of on a different kind of level because, well, A, it's interactive, and B, it's so technical. What do you think about that?

HH: I think it's fantastic that end users get to understands how complex it actually is. It's just one of those quirky terms. It's a "deferred rendering engine". What the hell does it mean? What the hell does it allow you to create?

I think for us to visually present that, and that people toy around it any play with it and sort of experiment with it in their own time at their own pace is very helpful to making them aware of how complex it is. You started with the question, "You spent quite a number of years," and this is a way I think to express the complexities that went into this project.

AB: And we had talks on videos on GameTrailers, etcetera, and they were super highly ranked. And you look at the comments, people were like, "Whoa, we need more of this information!" So, you know, let's give this kind of stuff to them.

And to actually once more show this is real time. This is not fake, this is not a movie that we're playing in slow-mo or whatever, but you can actually see all that kind of stuff in the background.

And it's a unique thing that we developed for the Killzone engine. It's a completely different way how to render your screen, and we kind of explain how we did that. So, I think that's interesting for many people.

And another feature that it has, is there are director commentaries. So, we had our art director, lead technical artist, senior tech coder, and the director from Zoic who helped with this, took over what you're seeing on the screen, etcetera, so.

I think that people will find that pretty fascinating. It's interesting because among most people, even developers, very few proportionally are working on games of this stature. You know what I mean, triple-A, first party, single platform, 190 people at a top of the line team.

HH: It's kind of a developer's dream to get that kind of opportunity to not only make a game of that magnitude but also do it specifically for one platform and not have to sort of find the lowest common denominator.

It really is great. That's also very much the reason why Arjan and myself decided to sell the company to Sony, because we got that opportunity three years ago.