Taming the Project with Process: Bending The Mythical Man-Month

(Taming the Project with Process)

Game development is a young industry. Beginning in garages and basements across the world during the late 70s, it has since evolved into a larger, more structured business. Single programmers making entire games have been replaced by dozens or even hundreds of employees across many disciplines working together. The functional categories are varied: engineers, artists, designers, marketers, producers, and testers; all with a common goal but often speaking different technical languages. Keeping these disparate groups in sync is the essence of management. When someone is referred to as a manager most people envision managing people, but I prefer to look at it as managing information. And while a programmer works with C++ and an artist with Max, managers work with schedule and process.

In this article's first part, I wrote about using schedule as the first step in effectively managing a project. The next step involves the daily process necessary to express that schedule across the team. The process's central purpose is to increase communication flow which naturally becomes a larger bottleneck as a project's size increases. In the classic book The Mythical Man-Month, the author describes how communication requirements for a team raise exponentially as it grows. There are essentially two ways to keep control of this growth: one is to reduce cross-person dependencies and thus reduce the communication requirements; the other is to increase communication bandwidth. These two different approaches together will allow a manager to, in effect, bend the mythical man-month as far as possible to get the most productivity from each member of the team.

The most basic rule of process is to first do no harm. Most people in our industry tend to fear rigorous process because they worry that it will stifle creativity and bury people under mounds of bureaucracy. Many people get into making games because they do not want an image of themselves developing database software in some Office Space-like cube farm filled with an endless stream of TPS reports. But process doesn't need to get in the way. It simply needs to exist enough to get around people's natural inclination to stay focused on their own goals instead of others'. Modern game projects are simply too large not to have the structured lines of communication that process creates.

Hierarchy

During my tenure at Microsoft, I oversaw the publishing of many external projects. My job was to verify that the engineering progress was on target for their milestones and that they were pushing to make the best game possible. With the many development teams I worked with, one common flaw I often identified was a flat hierarchy across their functional groups. They always had a lead programmer or artist, but seldom had sub-leads. As teams reach sizes of ten, twenty, and larger, one lead is simply not enough to manage the process. And to make matters worse, the lead always seemed to be responsible for at least a couple major pieces of the project. It's simply too much work for one person to do effectively.

One of the best ways for a lead to keep control of the development process is by appointing sub-leads. Much like a military organization, creating a triangular hierarchy of management is extremely helpful when keeping a large group of people focused on a common goal. The higher up the triangle you move, the less hands-on work you're actually doing and the more effort you're placing on managing information flow. I find that this is often a difficult concept for people to implement because of preconceived notions and their personal experience. When people work their way up the managerial ladder they often cannot keeping themselves from micromanaging the people below them--because their subordinates are doing the job that the manger once did. So the manager with a strong opinion on the correct way of doing things doesn't hand responsibility down the ladder as needed. He in essences does not trust his direct reports to take on the responsibility. This in turn keeps the manager from focusing on his new duties as someone tracking the product at a higher level.

I am a strong proponent of empowerment at work. I believe that a well-built team must be stacked with employees that can be proactive in solving their own problems. I call this "ownership" and it's a critical part of my management style. Delegating responsibility to the sub-leads is part of this process. With the size and complexity of today's game development, no one person can own all issues. You must at some point trust that the people you have hired are going to be able to drive components autonomously--otherwise you'll turn yourself into the information bottleneck that you should be striving to eliminate.

Once you agree that your team is large enough to need sub-leads, your next goal is to select them. Common art sub-leads are animation, background, texture, lighting, character, conceptual, and so on. Common engineering sub-leads are tools, graphics, AI, and gameplay. As an example, if you're a lead engineer and you have four people working on graphics look for one that will stand out as the natural lead due to his experience and/or communication skills. Speak with that person and see if he's interested in a position of increased responsibility. If he accepts, give him a public mandate along with the task of managing the graphic engine. Allow him to start looking at the system at a higher level. Instead of being focused on one particular piece such as bump mapping, his job is now to view the graphic engine as a whole and think about details such as frame render time and cross platform expandability. Make him feel responsible for the system and he'll become your point of contact for all issues graphical. Do the same for other systems and your team of twenty engineers will suddenly seem more like five. It's the only way to effectively manage dozens of people.

Information as a Resource

Once the hierarchy is in place, the next goal is to allow information to flow throughout the system. All communication can be described by two factors: ease-of-use vs. thoroughness. A quick E-mail is very simple but is often ignored by the recipients. It's ease-of-use translates into increased volume, which in turn raises the danger of it becoming background noise. On the other hand, a long meeting can certainly pound information into everyone's collective head, but the time drain is enormous. If you used meetings to convey all information, you would spend your entire day in a conference room. Effective information transportation includes many different methods each with a different ratio of easy verses thoroughness. Improving communications isn't about one solution that covers all cases--instead it's about many small systems that, when all added together, create a significant change in throughput.

When thinking about your team's communication, start by identifying the lines of communication that you use. Obviously in the modern age, E-mail is a major force for hourly communication. The recent rise of internet messaging such as Windows Messenger gives a more conversation-style approach and can be extremely useful for at-a-moment exchange of ideas. However, the most powerful form of communication is as it always has been: face-to-face. Electronic communication can easily be misconstrued due to the lack of emotion and the less than real-time speed. Nothing conveys information more exactly than a real meeting between people, and its importance should never be ignored. This is often a daily managerial challenge, particularly in our industry, which traditionally has quite a few introverts. Making sure that people on your team discuss issues in person will help in keeping people and tasks from falling off the radar.

Another information-dynamic to consider is push vs. pull. While pushing information in directed E-mails and meetings is one side; pulling information on demand is the other. Standard network repositories for weekly task status are helpful for disseminating information only when it's needed. Make sure that any person on the team can easily gain read access to the most current design documents for gameplay, art, and engineering. Use the versioning functionality built into Word for documents, and asset databases or revision control systems such as Perforce or Alien Brain to allow a person to track the evolution of the resource. This level of accessibility in your assets will allow people to work on their own timetables instead of always waiting for information to flow from requests to other people.

With these basic forms of communication identified, now the question becomes one of content. What do you need to talk about? How many times have you seen problems occur simply because someone didn't know what another person did just thirty minutes earlier? It's a classic example of the left hand not knowing what the right hand is doing, and with the number of people touching the asset database increasing, the chances of stepping on each others toes goes up considerably. This is why so many asset databases support automatic check-in E-mails. Every time a developer changes an asset and needs to check it back into the database, they must describe their change. Make sure your team is verbose when writing these descriptions. Nothing is less helpful than a check-in message that says, "Changed stuff". Mention what you've fixed and point out how possible flaws might express themselves in the build due to the change. Include a list of the changed files to catch people's eyes if they are working with the same system or asset. These check-in E-mails are then sent to the entire team automatically. Will everyone read and remember everything in these emails? Of course not. But if you're working on character animation and you receive a check-in email with the words "character animation" in the title, you'll be more likely to look at it and it just might prevent an ugly conflict later that day.

While check-in emails cover lots of information in a very passive way, daily morning meetings can help make the process more directed. Take ten to fifteen minutes every morning per subgroup to go over yesterday's progress and today's goals. Have each sub-lead run their own group, with the group lead overseeing the discussions. With the lead attending all of the mini-meetings, he will be able to convey information among them. This allows the sub-lead to stay focused on his goal while the lead focuses on the project as a whole. The lead then takes the progress from the morning meetings and resolves it against his schedule in order to keep an accurate view of the project. It is this daily push of fresh information into the schedule that allows it to survive as a living document. Without constant updates, a schedule can age into obsolesces in a matter of weeks.

Building a communication infrastructure in your project should also focus on improving the individual's ability to communicate. This is one of the reasons why I like using check-in buddies on my teams. A check-in buddy is simply a person who listens to you explain your changes before you check them in and will on occasion offer suggested improvements. Unfortunately, if you say the phrase "check-in buddy" most developers nearby will get a very annoyed look on their faces. Arguments that it takes too long or isn't really needed start getting tossed around at this point. However, I believe most people look at the process incorrectly, and that leads them to fight against it. On projects where I've used check-in buddies, the most important part of the process is the original developer saying out loud what they've done. Things inside your own head tend to make sense within the confines of your mind, but as soon as you say it to another person it often allows you to see it differently--perhaps for the first time. How many times have you seen someone explaining something only to stop and announce, "Well that was dumb of me." Check-in buddies are not so another developer can change your work, but instead to simply act as a sounding board for you to bounce your ideas off. And the fact that this process also keeps people on your team talking over the course of the day, and thus improving the individual's ability to converse in general is a positive side-effect.

______________________________________________________

Cross-Functional

Now that the internal workings of a given functional group are moving forward, it's time to look at cross-functional communication. This is often the trickiest part of process, because now people start talking different languages. The amount of frustration that I've seen as programmers and artists try to work together is mind numbing and almost always due to a basic difference in language. Most of us learned our particular crafts in college where we were segregated from other disciplines. We developed vocabulary specific to our job that was understood by our peers and no one else. And now we need to work with other groups that lack years of training in our niche. This is where the cross-functional breakdowns occur. Therefore, having technically bilingual employees on a team is a requirement. Technical artists and artistic programmers go a long way in merging disparate groups into a cohesive team.

Another valuable step in managing multiple team functions is to set up a minimum of one meeting per week for all group leads to gather and discuss past progress and current requirements. This is the setting where the engineering lead would let the art lead know that a new rendering effect had just come online and was ready for use, or where the test lead would inform the group that the latest milestone build passed successfully but still had a few issues. This is where high-level discussions take place without all the tiny details of "how" getting in the way. At this level, the leads should have enough years of experience that no one is blind to the needs of the other groups.

With these systems in place, the functional teams are now regularly exchanging information every week, but we're still a long way from true knowledge sharing. The largest step here is consolidating schedules across disciplines so that dependencies between art, design, and code are visible. It surprises me how many teams I have worked with over the years keep their departmental schedules separate with only occasionally discussions merging them in each lead's respective head. But until you actually see the dates line up on paper, you won't really know if a mistake is waiting for you up ahead because the art team needs a tool done in March but engineering isn't planning on working on it until May. Therefore, the leads must come together with their manager and create the single project schedule.

During the final stages of development (alpha and beta), information flow between the departments becomes even more critical, as fires are erupting within hours and minutes. A special-case, cross-functional process that I've used in the past is called the triumvirate. At Microsoft, we used a triumvirate of producer, lead engineer, and lead test to run the final bug push. Driven almost entirely by the bug database, we would all gather once a day, or even more often, to scrub all new bugs and assign them or resolve them as "no fix". All issues would be argued within the triumvirate, with the producer given final authority over all deadlocks. However with this power, the most effective producers used this final veto power very infrequently; instead trusting in the engineering and test representatives to guide them in the correct direction. This check-and-balance group becomes the center of all decisions related to the game and shipping on time.

Globalization

So far the processes I've written about assume that everyone is located between four walls. But what happens when you decide to use off-site support? As games become more advanced, companies are looking to off-site developers more and more. Whether it's buying shrink-wrapped software from Rad Game Tools, working with New Pencil for art asset creation, or using external programmers that are sharing your source base, a lead has to be able to deal with the difficulties of global management. During the past year I've worked on multiple projects, including Counter-Strike for Xbox at Microsoft, and my latest title for EA that both required some level of programmers sharing resource across time zones. This greatly expands the difficulty of controlling your project.

Remember that there are two ways to handle communication on your project: widen the pipe so more information can get through, or reduce the need for communication through intelligent division of labor. When dealing with long-distance development, you won't have many tools for widening the pipe given differences in time zones, languages, and the lack of face-to-face time. Reducing the dependencies and thus the communication requirements is the best approach. Start by identifying the most obvious lines of separation over the game. Multiplayer vs. single-player. Front-end vs. in game. Pre-rendered cinematics vs. gameplay. Preproduction vs. production. These are all strong division lines that can be exploited for reducing communication requirements. Hand off the multiplayer map generation, the entire front-end interface, or all the cut scenes. There are many pieces of a computer game that can be cleanly dissected for outsourcing.

Perhaps the most important decision about long distance development is whether to even start it. Due to the difficulties I've described, it's obviously not something you want to dive into without careful consideration. When does it make sense? If your head counts won't allow more hires but you need more help often external contractors allow you to get around head count allocations. If you only need help in the short term, external people can save you the pain of having to reduce work force later-, after the push is over. If you require a specialized skill for a limited time on the project, such as costume or set design, then a contractor can make sense. The cost of bringing a person in-house as a full-time employee is extremely expensive in relocation and benefits, so going external will often be the most sensible, cost-effective approach.

As our projects becomes larger, as Sony and Microsoft insist on releasing more complex consoles, as the home user demands more realism, so to must our team and management layers expand. We will require more people adept at the ambiguous art of handling information. People that work in positions that bestow all the responsibility of the project but none of the actual power of the hands-on construction. It's a skill that other industries have had for decades and we're finally catching up to as electronic entertainment becomes a more viable force in the world economy. So put down your code, art, or design work, and spend some time thinking about information as a resource. Consider your own company's communication requirements, how what I've written about can help, and how you can expand on it. Maybe… just maybe… we can use information management to limit these insane crunches that plague our industry.

______________________________________________________