Halo: Reach - The Beta Story

As you read this, Bungie is readying Halo: Reach, the most significant new title in the Halo series since 2007's Halo 3. Last month, the Kirkland, Washington-based developer wrapped up the beta process for Reach, in which 2.7 million unique players gave the game a test on the Xbox Live service.

Gamasutra recently got a chance to speak to Bungie's community director Brian Jarrard and multiplayer design lead Chris Carney about the developer's approach to the beta process, what use it makes of the data it accrues, and about interfacing with the community during the beta process.

The team feels that multiplayer is an indispensable part of the gaming landscape in 2010, and here Jarrard and Carney share their thoughts on how Bungie -- widely recognized for spearheading multiplayer innovation with services like Forge, which allowed its community to interact with Halo 3 in ways unlike any other console shooter -- sees the evolution of the form.

You guys recently finished the beta, and it was tremendously larger than what you've done in the past.

Chris Carney: It was big.

Brian Jarrard: Yes, absolutely. It was probably several orders of magnitude larger. If you want the real nitty gritty, for perspective, the Halo 3 beta was approximately 800,000 people over its lifespan, and we surpassed that in the first day of Reach. We saw a total number of unique players at 2.7 million, which was pretty awesome for us.

Is having so many more people primarily to get a broader data set, or is it just to include more of your community? What's the thinking behind that?

CC: I think part of it is there was just more availability in this beta, because anyone who bought ODST could have access to the beta. We also had a lot of codes that we were giving out to a bunch of folks, so it was just a bigger population that had the ability to play the beta.

BJ: And we didn't really know what to expect. We just knew that we could conceivably have several million people if everybody with a copy of ODST decided to play, but I think our goal was just we had hoped to have as many people as possible, which could really allow us to put all the server backing stuff under real stress.

The first day of the beta, we actually had some hiccups with some of our servers that sit between the game and Xbox Live. That's the kind of thing we only could have discovered with a million people hammering it at the same time, so we were really fortunate to be able to put things through their paces at such a very large scale.

I suppose it's also much later in the generation compared to when you did the Halo 3 beta.

BJ: Oh, sure. I'm sure the install base alone has grown enormously since back in 2007 when we did that.

How has players' expectations of multiplayer evolved over the current generation? We're at a very different spot now, with many multiplayer hits on the market. First of all, did you expect the multiplayer to take on such a dominant role in establishing player base?

CC: Yeah. I mean, for us, we've always been focused on multiplayer in the studio. Single-player's absolutely a huge part of the game, but I think maybe, because we're competitive people, we always like playing multiplayer against each other. So as we developed Reach -- we'd all played a ton of Halo 3 -- we were asking ourselves, "What cool things do we want to do with multiplayer that we weren't able to do with Halo 3, or where do we want to take multiplayer?" That really helped form the framework around which creation of multiplayer is based.

BJ: I hear you saying, Christian, just regarding general multiplayer itself and how it's become more and more prominent and I guess necessarily required in this day and age to have a game that stands out. I think, for us, that really kicked into full swing with the release of Halo 2 and the advent of Xbox Live. I think ever since then it's become more and more necessary and expected by gamers to have these types of experiences.

Using Halo as an example, what we've seen is -- I think the universe and the story and the characters and the campaign experience really captivate and draw people in -- but, honestly, it is the multiplayer that really keeps Halo going strong. I think multiplayer gives all games that have had great success the long tail and the long lifespan that, years ago, you just wouldn't have because you'd have the campaign, you'd play mods, and generally that was it.

So I think the whole marketplace has shifted. I think, as a result of Halo and a lot of other games that have made interesting, cool headway into online experiences, you just have to do it now. Fortunately, like Chris said, even back from the beginning that's always been a real pillar of all Bungie games. I think the studio is always trying to find ways to keep pushing the bar and leading that charge versus trying to react and tack it onto a game.

Do you think that the longevity issue influences the design, or is it mainly just about doing exactly what you want to do?

CC: Probably the longevity of the game makes us think about it harder because we end up playing it more. There's more pressure not to fuck it up because people are really into it. But a lot of the design goals for Reach multiplayer are based on things that just we wanted out of the game; they're not a response to -- because people have been the game for so long, they're demanding x, y, or z; that's absolutely something we think about, but mainly it's driven by what we want to do with our next game.

That's why it's critical for us that Reach not feel like just the next version of Halo; it needs to be its own unique, exciting multiplayer experience.

BJ: You know, if you look back at Halo 3 specifically, there's two things that come to mind that I think inherently did stem from something internally the team always wished we had that we could do, but also definitely there were discussions around, for example, Forge -- that's kind of our light map editor.

That was really designed as a way to be a great tool for us to be able to stay in the game after launch and go in and, when we find problems with an objective or where the flags are located or where weapons are placed.

In Halo 2, that was a pretty massive undertaking. In fact, I think that might have even required a title update just to make that tweak. So we knew, for a game to be able to live and withstand three-plus years of hardcore play, we needed a tool to be able to do that, but we also thought about, "What would the fans do if they had those tools?"

I think seeing something like Forge is definitely a good example of how we want to be able to empower the audience and our player base to eventually sort of take the reigns of the game.

There comes a point in that life cycle when we kind of go from leading the charge to letting the community dictate what happens to the game. I think Halo 3, over its lifespan, has seen a lot of different gameplay styles that have been formed strictly by things fans have done with tools we've provided. Nowadays, on Xbox Live, some of the more popular playlists and game offerings have actually been created by fans outside of Bungie, so that's pretty awesome.

So that's definitely, again, stuff that I think has a dual purpose: It always originated from a practical or fun desire in the studio, but we very quickly have discussions about what our fans would do if we enabled them to do x, y, and z. I think in Reach you're going to see some similar themes carry forward.

When you saw the response in Halo 3, did it become increasingly relevant to you to empower that fan base?

BJ: Absolutely. Just in the beta alone, we had -- looking at my numbers here -- 1.4 million pieces of user-created content were uploaded. People can't make game types yet, but that's just screenshots. I think through Halo 3 we definitely learned that give people tools, and they will use them and do awesome things.

Then you start to discuss: Well, what additional tools might we be able to make available to them? How can we possibly improve usability? What can we do on our side socially to elevate these things that people are making to make them more accessible or discoverable?

In Reach, we add a really robust system where you can tag all this content now -- in the game and on the website. It almost simultaneously updates in both places. One of the bigger challenges that we learned from Halo 3 was that we almost had too much content to know what to do with, so at the beginning of Reach it was a very conscious decision of: "All right. It's no longer a question of will people use it; we know they will. How can we make the experience better and make sure that content is touching more people?"

I definitely want to ask about the continuity of the experience across the series. It sounds like you're very aware that, on one hand, the fans expect a continuity of play across the multiplayer modes, but at the same time you want to make sure that it's worth their time -- it's not just another iteration, slap another number on it. So how do you balance that from a design perspective?

CC: Again, I think what we're motivated by is: What do we find interesting? How, internally, do we evolve gameplay in a way that we all find satisfying? That's totally where armor abilities and loadouts come from. The initial conversations about loadouts were: You play Halo 3 in The Pit, fighting over the rocket launcher or the invis is fun, but fighting over the needler or the plasma pistol really isn't that fun.

There are certain weapons that control the map and then certain weapons that you just run into when you run around. So what are the ways that we could arm the player at the beginning with a set of simple choices that help guide the way they want to play that next chunk of the game?

And then armor abilities were a way to simplify equipment. Equipment was really cool in Halo 3, but even in the development process it came in very late. There are some great moments with it, and then there are some moments that are kind of terrible -- like you hold onto the armor ability to the perfect, absolute last moment that you can use it, and then you end up dying and watch the thing roll down the hill.

So what are ways that we can use that a little more intelligently or build that a little more intelligently into the gameplay where making these decisions about what piece of equipment you want to use a cool choice? That really helped guide these decisions about loadouts and armor abilities that ended up going into the game.

BJ: It's definitely been a real challenge; I think it's been a creative challenge that's been fun to embrace with the team, but also a slippery slope -- moreso for Reach than even Halo 3. Halo 1, 2, and 3, you have Master Chief all the way through; I think you saw different elements of additions to the game throughout, but you're still playing Master Chief all the way.

Reach, as a prequel -- we don't really have those sort of confines to work in anymore, which I think is empowering to the team and interesting to think about: what might I do in a new, standalone period of time when I don't have to worry about fitting into this really firm framework that, at the same time, still needs to feel like a Halo game?

I know that everyone, working on campaign and multiplayer, has been first and foremost making sure that we retain and really crystallize that essence at the center of what we think makes a Halo game a Halo game; but now we can sort of have creative fun and think about what happened at this point of time in the universe when there's no longer Master Chief involved.

We have a whole team of Spartans, and they're a little bit different; they have a little bit different tactic on the battlefield. Maybe they're a little bit weaker; maybe they run a little bit slower. That's an example right there of the movement speed as something that we actually intentionally changed in Reach to adapt a different style of gameplay, but that's probably one of the main things that we heard the most vocal feedback about in the beta.

A lot of people, after playing three years of Halo 3 and moving at Master Chief's speed and super jumps and floating in the air, now suddenly Reach didn't feel like Halo to them. That's a real discussion that we've had internally. I think I would say before the beta started, if you asked anybody here, "Hey, would you guys consider tweaking run speed and jump height?" I think the answer around would have been "no."

But it was such a dominant, consistent topic throughout the beta that, now, the design team has sat back down, and they're thinking about these things. They're evaluating potential tweaks to still keep the game a little fresh and different, which is the intent, but try to bring it a little bit back to what people expect to have in a Halo game.

How do you make decisions when you get feedback like that, about whether there's a greater purpose in terms of the decision you're making for the game design versus "is that what the community wants?"

CC: You have to all input from the community in the lens of "Is it better for the game?" We probably are too close to it at times to appreciate how radically it's changed because, with such things, like Brian was saying, like movement speed and jump height, we actually tweak a bunch and iterate on it and tweak it some more; so we don't get that perspective of "Hey! I just played a game of Halo 3 and now I'm playing a game of Reach beta, and it feels totally different!"

When all of these people start saying, "Hey, this does not feel like Halo to me!" then we talk about it and see other ways that we can adjust it but also still fit that in the framework of what we are trying to do with the game and then play that.

What we've found with movement speed is that it actually helped us solve a few other issues that we were struggling with. So you just have to tweak and figure out what's appropriate.

BJ: But really, at the end of the day, the team here is ultimately going to make the final call, whether it's Sage [Merrill] on the weapon and sandbox side or Chris and his team for the game types and the core multiplayer stuff; these generally are people that are building on the same foundation that was laid down ten years ago.

It's their jobs to make these gut decisions and gut calls, and I think ultimately this boils down to trying to wade through all the noise.

Realize that the people that you're probably hearing the most from really represent a very small subset of your overall population but are probably the most hardcore people that you actually care a lot about; so how do you balance their needs against the needs of the broader population who maybe doesn't care or might even feel the opposite?

In this example, talking about run speed, one of the easy things for us is that we offer a whole host of customizable game settings in the final game. So if you and your friends would prefer to play something that feels more like Halo 3, you'll have the ability to do that through custom game settings. So some of those things we can actually mitigate by, again, empowering players to custom-tailor their own game experience to their own liking.

Do you guys use a lot of the data you get from gameplay? I've talked to people who sift through server data and get reports, and it's my understanding that sometimes things are counter-intuitive in terms of seeing how players actually are reacting compared to what they're saying.

CC: Yeah, the big numbers that we were using from the beta were things like: Invasion Slayer, Elites are winning 60 percent of the time, and they're winning by actually a pretty substantial margin -- the final score of those games. So that's an example of something where we weren't sure if Elites were dominating so much in Invasion Slayer because of their sandbox, their access to the weapons, their spawn being closer to some of the territories than Spartans.

So, the second week of the beta, we actually came out with a 4v4 (4 Spartans versus 4 Elites) game type on Sword Base and Powerhouse that was just purely for science, is the best way to think about that, to see if those numbers were really consistent. Sure enough, after about 60,000 games, we saw that, again, Elites were winning about 60 percent of the time by a really high margin, so that's something we completely react to.

BJ: An example I can think of off the top of my head of something where the anecdotal feedback conflicted with the data that we had was that, at the outset, a lot of people were kind of vocally spread. They thought that the magnum, the pistol in the Reach beta, was fairly useless; it was inaccurate; "This thing is a piece of garbage. Why would I ever want to use it?"

We didn't really believe that. We heard people starting to say that -- yet, at the end of the beta, of all the new weapons, the pistol ended up being the weapon that generated the most kills globally, throughout the entire population.

So I think by week 2 most people ended up understanding that this is not the same as the Halo 3 pistol; you actually have to play with it differently. It requires a little different, nuanced approach to it in the rhythm of firing and how you waited how you choose to use it.

I think that, as a result, we tried to educate our player base on how they might want to think about it differently, and by the end of the beta it ended up being the weapon they got more kills with than anything else.

If we had reacted simply to forum posts, who knows? Maybe we would have done something bad and actually changed, but I think the team stayed true to their gut. We thought the pistol was exactly what it needed to be -- for the most part; we might give a minor tweak here or there -- but the data backed it up and showed us that the majority of people were in fact using the pistol to great effect and it was working as we had intended.

That's a good thing to talk about, because you guys have a strong community strategy. Is that something that you think people just shake out after they play for awhile and come to grips with the learning curve?

BJ: Yeah, I think the beta, for us... The way I like to think about is: Bungie shipped Halo 3 and immediately moved into development on ODST and Reach, so, really, for the last three years, our team has been spending time working on and playing ODST and Reach, while meanwhile the rest of the world has been playing Halo 3 and racking up literally thousands of games of Halo 3.

So I figure over the past two years, as we've been iterating on Reach internally and playing it in our test labs and our take-home tests, we've all just become acclimated and gradually moved into the Reach mindset of how the game's meant to be played.

I think maybe we're forgetting what that first experience is like, when you go right from Halo 3 to Reach. It's pretty jarring, and a lot of that is by design; it was definitely meant to be different, to take the core Halo gameplay and mix it up and offer some new experiences.

A lot of people definitely received that very well and were excited about it, but a lot of other people I think definitely took some time to warm up, to throw away their preconceived notions of how they played Halo 3 and logged thousands of hours with the battle rifle, and try to appreciate Reach as something different.

I really do see over the course of the beta, by the second week a lot of people started to change their tune about certain weapons and certain behaviors because they started to play the game differently.

What we tried to do was get out in front of that and set expectations for people; we talked about it in articles, we put together some materials -- we did our best to proactively get people in the right mindset and understand as much as they could about the game before it started and even during the beta to try and speed up that learning curve and that process. I definitely did see that happen over the course of the beta.

I think, initially, most people were definitely... [there was] just a lot to grasp. A lot of things were different; a lot of things were subtly changed. There were people flying through the air with jet packs. A lot of people didn't quite know how to get their head around it.

It's interesting because also, in the same sense, I'd imagine that just the beta process helps with communication inasmuch as there's a snapshot out there, and people won't be going through this in the same way when the game actually comes out.

BJ: Yeah, it's good and bad. I think the hard part of that, for us, is that we weren't able to actually apply bug fixes and patches to the beta while it was in progress. The best we can do now -- which I think is very important to us -- is to make sure people know, "Here's our top issues that we've gathered from the beta based on data, based on all of your feedback and forum posts; these are the areas that we definitely are looking into. We've heard you; we definitely agree that there is some room here to explore and do some tweaks and iterating."

At that point, it's basically just, "Hey, trust us. It's going to be better in the final game, but people have to wait." Ideally, it would have been awesome if we could have reacted during the beta, but it just wasn't something we could do.

The other thing which I think made it even more complicated is, when the beta went live, by that point where we were in the studio real-time, we were maybe six weeks ahead of that in terms of our core code base.

So we were already putting in an old build, basically, that included bugs that we'd already fixed internally that we had to release, unfortunately, just due to timing. So we had cases of people really reacting and responding and really having their whole perception of the game skewed by bugs that we had already addressed and knew about, but unfortunately...

An example of a bug with our melee system: It started to muddy together, and this one melee bug and the way melee works started making people question everything from the efficiency of weapons, how the health and shield system was working; and everything kind of bled over.

For all they knew, that was one of our new design goals for Reach, and for what we knew it was a real bug that, unfortunately, we couldn't fix before the beta launched. As a result, a lot of people had it kind of cloud their judgment. So definitely we learned a tremendous amount, and hopefully people will take it with a grain of salt, realizing it was a beta, and be happy with changes that we're going to make as a result of it.

You guys were at the forefront of online multiplayer on console for shooters, but clearly there's a lot more competition these days. How much of your energy goes towards the idea of staying ahead of that competition?

BJ: I can tell you honestly that it's not something the team really thinks about. Even going back to Combat Evolved, every Bungie game ultimately originates with a cool idea for a game in the universe that the people in the studio at that time can't find on the shelf and really want to play themselves. From there, it's on the Marathon universe; the Myth universe; the Halo universe.

So the Halo universe is really not much different. The studio definitely doesn't have this approach of trying to keep up with the competition or really spending too many cycles thinking about what other people are doing; instead, we try to think about ways to keep innovating and doing our own cool stuff, like Halo 2 basically invented matchmaking and the whole notion of a party system and how people now come to play online.

I think that, to this day, a lot of people tell us that they wish every game would employ a matchmaking system like Halo does. Some other games now obviously do, but certainly it was far from the norm.

I remember, years ago, the day Halo 2 launched and people found out there was no server browser, that was like the end of the world. We fell on that sword pretty hard, and it took about six months for people to finally realize that, actually, this new way is probably better for consoles; no one's looked back since.

I think Bungie as a team is always gonna keep trying to push forward and innovate and add new features -- not to say that we don't play a lot of games ourselves, too! I think we actually find stuff in other games that we generally think are cool or good ideas or, "Hey, I think they're on to something here, but I think we can do it better. What if we did this and this?"

Those kinds of discussions do happen, but there's never a competitive analysis: "Okay. We need to have x, y, and z in our game to be able to compete this holiday." That's just not how Bungie approaches development.

CC: Yeah. Even when Halo 3 came out, within about two or three months, internally a lot of us had a list of, like, "This is all of the things that are broken in this game" or "This is all the stuff that I want to change." Like Brian said, we were already working on ODST and thinking about Reach at that point, so that's where our energy goes to.

Yeah, we play other games, and there's a lot of other awesome multiplayer games out there; but in the end it's stuff like... I think the biggest compliment I can pay to everyone here is that, when Reach ships, we're all gonna be super excited to play the game online because it has all the stuff in it that all of us really want in a multiplayer experience.

You guys are inspired, but how do you sit down and address these things? It's intriguing to me that you don't do things like competitive analysis.

CC: Yeah. Like I was mentioning before about loadouts and armor abilities, there's definitely people in the studio who will be the voice of a feature and will prototype and try it and say, "Hey, that's really awesome" or "Hey, that kinda sucks, but maybe if we tweaked a few things it could be really awesome." There's some features that have grown out of totally crazy ideas that, I think, when the game ships, people are going to be pretty blown away by. And there are other features that we're cutting right now just because we don't have time to finish them -- that's just the nature of the beast.

It is kind of a combination of multiple people; there's not, at least in the world of multiplayer, one person who says, "I want x, y, and z in this game, and you have three years to build that." It's definitely much more collaborative, and the game evolves as all of us continue to work on it. I think that's the way we've sort of always done it. In the end, it usually turns out pretty well overall; we're relatively happy with it and totally don't want to kill each other.

BJ: I can think of one example that might be a little more specific that we can talk about. One of the new big systems in Reach is this whole notion of player investment and player rewards, right? I think most people would probably agree that, in terms of those types of elements being present in a shooter, I don't think anybody really saw that before Modern Warfare, so let's go ahead and just call that what it is. I think that was a really interesting element that they brought to the genre.

I think we, as much as anybody else, really enjoy that aspect of progression while playing a shooter. We also have a lot of people here that play World of Warcraft and lots of other types of games that are more traditionally based around that.

I think with Reach, knowing where the fiction was going and the fact that we have these Spartans that inherently are a little more scrappy and are a little more heavy on the battlefield customization and rag-tag for filling out their armor -- that was a really nice way for us to start to think about, well, we all like progression, too. Everyone likes to collect a reward while they're playing a game.

Those discussions turned into the feature that ultimately made it into Reach, but it's very different from us saying, "Well, Modern Warfare has player rewards and a deep progression system; we need to have that, too." Maybe deep down, subconsciously, some of us were thinking that, but it really was that we like this as gamers.

We think it's very cool to have stuff to do in the game; it gives you more stuff to look forward to, more reasons to play, more ways to reward people who play with different play styles. That's how that feature made it into Reach versus really a competitive analysis where somebody came back and said, "These are the three things that have to be in the game in order for it to compete in the holidays 2010."

CC: Even a little more about that: even at the very beginning of Reach, we had armor that was available in Halo 3; we had a couple pieces that you could get through skulls and through other things. But because the story is about what happened on Reach and what happened to the Spartans on Reach, we knew we were gonna have a ton of different Spartans in the game.

We also made a decision really early on that the Spartan that you play in single-player or in co-op or in multiplayer is always you; it's always this unique Spartan that you create. So we had this idea about all this armor that we wanted people to be able to change to really customize themselves to be their version of a Spartan, and then as we started talking about player progression those things just kind of naturally worked together and turned into the awesome armory that it is today.

I know this isn't Halo 4; they're not numbered right now, but into the online multiplayer this is a higher iteration of it. You've done it several times. Is there an optimal set of the way you look at the way maps need to ship in the box, or is it something that changes as players' tastes change, as player behavior changes? Is there a way you look at weapons like that? Do there become rules as you understand them, or does it always change?

BJ: I think it's something that evolves, again, as we continue development. As far as maps go, I think Halo 1 we had 13 maps; Halo 2 we had 12; and Halo 3 we had 11. (Laughs) It gets smaller and smaller because we try to focus more on quality now and less on quantity.

The awesome benefit we had with Halo 3 was Forge, and especially in Halo 3 DLC, releasing these different Forge maps where players could build all sorts of variants from the same basic environment created a lot of cool flexibility within the environment. That's absolutely some things we've been talking about internally for Reach in how we leverage all that for ship.

Yes, we need enough maps to cover the number of players that we want... and for all the different gameplay experiences we want to have in the game, but then we also want to give players a lot of tools that they can use to build cool worlds. So it's more evolution than really some hard number of "We must have 10 maps for ship."

And then, for weapons, with Reach we actually pulled back on the total number of weapons that we usually have in a game because, for Halo 3, honestly, I think we had too many weapons. A lot of them did not have very specific roles; so, again, we try to focus more on depth and quality for each weapon rather than a crazy number of weapons. That was just more of us sitting around and talking about, specifically, what did we want that sandbox experience to be for Reach.

You guys are lucky in the sense that other people who are working might get asked by their publisher, "Hey, if you guys don't have 25 weapons, we can't put 'Has 25 weapons!' on the back of the box."

BJ: Yeah, I think you're right. It's all us, since we've been here; but I think, with the success of Halo and the franchise and just Bungie's track record in general, it does definitely afford a lot of leeway -- sort of a lot of discussions about "Just trust us. This is gonna work. Trust us." Thankfully, it has.

CC: Yeah, and even like what Brian was saying earlier about the pistol, there's so much subtlety in the pistol -- so much time and effort has gone into that. We had the Halo 3 pistol, and it was good, but the pistol for Reach just -- once you get used to it and once you understand exactly how it works and exactly where it's effective -- it's just so damned fun to use that one gun. We're trying to take that philosophy and use it on every weapon in the game.

DLC has become increasingly relevant; you guys are, I'm sure, planning it -- thoughts on DLC strategy?

BJ: Well, we haven't announced anything, and I'm not going to be able to announce anything today. You are right that it is something that you have to think about even before the game is done. I think it's safe to say that we have a pretty long tradition of supporting our games through and beyond launch, and that will be the case for Reach, as well.

A lot of times, as you're working on the game, there are always more ideas than there is time to actually develop content, and that usually becomes a really nice spark to kick off downloadable content-type discussions -- when there's an opportunity in the development cycle to come up for air and actually look forward.

Looking back at what you've done in the past, did you feel like, again, is that something where, after you're done with it, you felt, "Oh, we had all of these cool ideas for what we could have done with DLC; we can worry about them in the future," or is it not the same as working on your main game content?

CC: The nice thing about DLC is exactly what Brian said, where we can push some ideas off to DLC if we do that. The other thing is that, in the past, working on DLC has been great for us because the engine is stable and the game is stable.

Right now, there's so many things changing that, even when I'm working on something specifically in multiplayer, there could be something else that's crashing the game. But for DLC, because the engine's already out there, it's a nice, stable development universe, so we can be a little more ambitious about some of the things that we try -- or at least we have in the past.