In-Depth: The Guide To Rescuing A Troubled Project

There's a lot of info about the "utopian dev cycle", says EA senior development director, Jeff Charvat, but what about saving a project that's going down the tubes? At the 2009 IGDA Leadership Forum, Charvat spoke about the most effective way to take charge of a troubled project. Charvat has had a lot of experience taking troubled projects and making them work -- some good, and some bad. At Broderbund, he was sent in to work on the company's Living Books project -- and learned that charging in with answers, rather than questions, isn't the way to go. Major problems, however, like senior staff changes or management-dictated direction shifts affect big projects all the time, and Charvat has been sent in to save troubled games with increasing frequency. At EALA, he was called on to work on Medal of Honor: Rising Sun; he later worked on Medal of Honor: Airborne. Currently, he's working on the more stable Command & Conquer 4. Each, however, taught him important techniques of coming into a project well past its inception. When it came to Medal of Honor: Rising Sun, says Charvat: "A big problem happened. This team was over 100 people, it was in complete disarray, and this is when they asked me to step over and take over that team." He recognizes that the project had problems, but the game shipped successfully. He ended up getting sent to save Airborne -- "luckily by this point in my career I was getting pretty good at this," says Charvat. What do all game devleopment teams all have in common? Says Charvat, "They all have really talented people, but they always have wackos, too, and you have to be prepared to deal with them. They all have good new processes, and you get to learn from it... But equally, you see some processes when you join these teams and you're like, 'Really, that's how you do that?'" "They all have a history that formed team culture, and you need to spend that first month learning that culture." And there's one more thing, he adds: "They all want more time. The truth is that the most of them are right. But you need to be careful when you play that card." Even if a project has a leader who departed on bad terms, says Charvat, "Be careful what you say about these people, because there is a lot of love and respect for these people." The First Five Days Charvat says that your actions in the first five days of being sent in to handle a problem project are crucial. "I like to ask a ton of questions but answer very few," says Charvat. "If you just tell people... 'I need a little more time to figure it out,' they'll understand." While you won't understand the dynamic straight away, "You want to leverage your fresh eyes and see things that a lot of the team members cannot see," says Charvat. "Every team has some sort of core leadership to it. They normally have some sort of daily sync and get to be a part of that as early as they can." But also do generic meet and greets, rather than just being involved in the nuts and bolts of the project. You also want to have a "goal-oriented meeting" with "just one of those leaders," says Charvat. "To set the tone that you are moving this thing forward. You'll have a lot of fluffy talk, but you want to set the tone that it's about goals and getting those goals achieved." However, you have to be careful not to meet with every leader on the project -- it affects morale, and people begin to worry that you're just "meeting guy," says Charvat. He also says that you don't need to finish all of these things, just get the processes going. Charvat thinks that it's crucial to get set up in the development environment immediately -- "nothing gives you a truer status of the project than the software itself," he says. It also allows him to "lead by example and let them know that I'm playing this game." "It always surprises me, especially on dysfunctional teams, that they do not play the game enough," says Charvat. And this process has an ulterior motive -- setting up the dev environment is complicated and the docs, he says, are usually out of date. "This gives you a chance to engage" with the team, he says. Make sure you get access to everything (i.e. servers) the developers do -- documented and undocumented. When it comes to taking over a project you're not intimately familiar with, says Charvat, it's not enough to just concentrate on the game itself -- "Play other games. All games have competition, so you need to go play the competitive games. Some games use even other, noncompetitive games as reference. You need to go play those." "Many games are sequels," he observes. You'll want to play the prior game for obvious reasons, but one that's less obvious is that the team you inherited probably made the prior game. "You need to spend a lot of time on that platform," even if it's not one you game on frequently, says Charvat. And if the genre is unfamiliar, study it. However, he says, "Playing these games all the way through to the end, I do not recommend." You can play a game for two hours "and learn 80% of what I'd learn if [you] played till the end." You also need to get deeply into the docs and plans for the game, and then talk to the sub-teams / cells / etc. that are executing these plans. It's worth noting, however, that "I don't want people to update plans for me.I'd rather wait till the end of the milestone" or other natural break point to update the docs. Charvat's technique for figuring out what's missing from the plans, and how the game's going to get made, is to take something incomplete "and try to build it. I don't build it to share it or because I want to lead." It's time to get your hands dirty and this leads to questions... These questions will just come ripping out and I will then go walk around and start getting answers to them," says Charvat. For writing documents, in general, says Charvat, "the main beneficiary is not the reader; it's the writer." 
 One-on-Ones Charvat schedules one-on-one meetings with as many of the team as he can -- all disciplines, all levels of hierarchy. "You get a ton of data in a short amount of time," he says. Make sure you "sample points from a different part of the project," and learn about team culture and function, as well as the project. You should speak to 25 percent of the team, or "until you dread the meetings", whichever comes first. "Find and make sure you schedule time with 'the really smart ones'," says Charvat. Usually, other staffers will tell you, casually, who's the most competent on the project -- just listen, and then make sure to speak with them. And if there's anybody you've worked with before, "make sure they get on your list. They're going to be more open with you." And don’t just limit it to your team; look outside. Speak to "either your QA lead, or your marketing manager, or your general manager... Someone who is not part of the development team, because you want to get an idea of people outside the project view the team. A lot of these dysfunctional teams are not self-aware," says Charvat. What to talk about? - The Person. Their career, what games they're playing, their significant other - Yourself. Give an introduction and brief, career history -- "What I've done, why I'm here, how I work" - The Project. Get to the project within 10 minutes -- in a 30 minute meeting. The questions you should concentrate on include, "What does the team do well?" and "What could the team improve at?" Says Charvat, "You don't want to change what the team does well. The second question is what's going to take up the entire meeting, because this is what people want to talk about, and they will probably have a whole list prepared already and they'll dump it on you."
 Close with "what should I do?" and
 "what shouldn't I do?", says Charvat. He notes that by the time you speak to most of the team members, they'll already have an idea what track you're on, and be prepared to answer this. Crafting A Report Once this process is over, Charvat crafts a report on the state of the project. "This is very straightforward, and I let everybody know I am doing it," he says. The report should cover four areas: People, Product, Process, and Tech. It should be written in a postmortem style with good/bad/ugly points, and there should be two versions, says Charvat. "There's the one you're going to share with the [management] guy you're going to link arms with. And there's the other one I share with the core leadership team," he says. "In both cases, you want both of these reports to sting a little bit." The external version will contain "sensitive data that may not be appropriate to share with the broader team" -- like info on personnel. The process for writing this report -- gathering the data, analyzing it, and assembling it, is not much fun, says Charvat. "It's very eye-opening and it will probably make you feel sad for yourself, as you hear about how fucked up the project is, but it's better to do it earlier than later." The Whore Charvat has another process called "being a whore", which to him means getting involved in as many pots as possible -- spreading himself around the team, in other words. Get involved in meetings and mailing lists and forums -- gather as much data as you can. Ask the leads what recurring meetings there are, and observe them -- both leaders and participants. "The key is you need to shut up," says Charvat. "The fact that you're there in the first place and it's going to change the meeting regardless... as much as possible you want to be in the background." "You're not there to change the meeting; you're there to observe the team," he adds. If you have different views on how things should be done, "Share those outside the meeting." When it comes to mailing lists, "review the list on the mail server, review a bunch of them, and then lurk." It's worth remembering that "you don't want to try to lead each individual effort on the team," so when it comes to ideas from meetings, grab the meeting leader and discuss your ideas, but let them handle it. When it comes to mailing lists and recurring meetings drop them as they become ineffective -- but make sure to let the groups know you're dropping. Make a Splash "Do something big that the entire team is going to notice. It's usually pretty obvious, but you have to make sure it benefits the project. You want to make sure the splash benefits the team, and with your fresh eyes, it should be easy to pick these things out," says Charvat. "This lets people know you mean business, that you're not there just to accept the status quo." When it comes to picking what to do, "you have to let people you're not afraid to make bold decisions." Of course, bold decisions are tough -- so "you need to not let yourself talk yourself out of it... And similarly you can't let someone else talk you out of it. You need to trust your instincts. At this early phase you are seeing things from a viewpoint few people can see from." For example, Charvat reorganized the Medal of Honor: Airborne team, which had been broken up by discipline, into feature-based groups: Gameplay, Shell, Single Player, and Multiplayer. The vision holders -- leads -- were in the middle, and producers were embedded in the subgroups. "This one definitely shakes up the boat a lot," he notes."When you make everybody pack their boxes and move, they will notice there's a new guy." Another splash you can make that helps a great deal is to stop a bad process or a recurring meeting that's unnecessary -- "you with your fresh see eyes that we don't need that meeting, so you stop it," says Charvat. For example, there was a daily status report on MOHA, which leads had to file every day but had not been used for any purpose on the project for a very long time -- so he killed it, to the immense relief of the team. However, says Charvat, you have to make sure to "not be grandstanding, or just make noise because you're there." Instead, you should "trust your instincts" when looking for the right splash to make. New Processes When it comes to introducing a new process to the team, says Charvat, "you have to make sure you're fixing a problem. You don't want to birng your favorite process because it's your favorite process." 
 "The first thijng I try to do is just find a process I can just tack a little more onto. Process is a really evil thing when it gets out of control," he says. Look for a process that involves an extant topic, fits with a group of attendees who are already meeting, etc. If you can't add onto a process, then replace a broken one. "You might need to expand your new process to fill a hole that the previous one was doing, but it's a good way to add a new process," says Charvat. "I find a lot of these dysfunctional teams I join keep adding process upon process upon process," he says, in an attempt to fix what's broken -- so make sure to spare attention for redundant or unnecessary ones. 
 When it comes to a new process, you have to clearly define it. "Define a goal. And define stakeholder roles and responsibilities," says Charvat. And define these by title, (i.e. "art director") rather than by name, so the developers will understand each role's place in the process. You need to communicate the cadence and chronology of the process -- possibly with a flowchart -- and provide detailed templates and examples. You also have to name it. Says Charvat, "Try to find something creative or interesting, give it some character or style. It will take on a life of its own and people will not be so hesitant to use it." And even if you run it at first, hand it off to someone else once it's in place: "Chances are they'll love to do it; chances are you'll be too busy to do it." Monitoring and refining any new process is just as crucial as introducing it properly, says Charvat. "People [who introduce new processes] just think that they just know it, and they think it's just going to work. It's a horrible mistake you can make." You need to send docs ahead of the adoption of the process, and schedule 30 minute meetings for walkthroughs and feedback. In the first month, especially, "you won't know the team as well as you think you do," says Charvat, so "expect a change and update your document." Launch the new process at this point, but expect to have to put a lot of effort into running it. Says Charvat, "people won't know what you epxect... You're going to have to do a lot of one on one feedback." Stick the process documents up on the walls, too. "I am a big fan that if you're launching something you get it up on the walls." And as a final note, it's tough to work with a dysfunctional team, says Charvat, because "they hate the leadership from the get-go and you're just some poindexter coming in trying to fix things." Teams whose projects are in trouble harbor "resentment towards leadership. There's no good way to get over that obstacle besides putting your nose to the grindstone, and getting thick skin," he warns.