How Funcom Squashed Bugs With Bugzilla

Author: by Pål Frogner Hansen

When Funcom launched in 1994, like many small companies, it lacked sophisticated bug-tracking and project-management tools. With small projects and few people, the company didn't really need such tools in the beginning. As the company grew, however, it began sporadically using tracking tools and instituting production procedures. Today, the company has good tools and a highly efficient production process - and the company spent very little on software to get to this point.

This is the story of how we developed our core production management tools, based a modified version of the Open Source bug-tracking system, Bugzilla, and Microsoft Project 2002. While you're probably familiar with Microsoft Project, let me briefly explain what Bugzilla is. Bugzilla is an Open Source, defect-tracking system that allows individuals (or groups of developers) to keep track of bugs in their product. Originally developed by Terry Weissman, it is now the de-facto standard defect-tracking system against which all others are measured. (See http://www.bugzilla.org for more information.)

At our company, these two products have come together to form an inexpensive yet powerful tracking and planning solution - one that other game development teams may want to examine. If you're currently using Bugzilla, this article will give you some concrete ideas for extending the tool. If you're considering investing in a project management or bug-tracking tool, you should consider Bugzilla. But even if you're not using or interested in Bugzilla, this article might still inspire you to extend your existing project management tools to support some of the functionality we've created.

Bugzilla Versus Commercial Applications

While much of its power is its extensibility, even without any custom modifications, Bugzilla is a good product. It tracks tasks and bugs (really just tasks -- bugs can be considered a special kind of task), and I believe a big reason we reached our current level of efficiency is that we have a single application to store for all tasks. Bugzilla can work like a knowledge database, too, due to its flexible searching capabilities. Once you come to grips with this ability, you can utilize the real power of the application.

Over the years, we have looked at various commercial solutions from companies like Microsoft, Rational, and NXN. There are various reasons we didn't choose any of these tools, but in general each failed meet one or more of our team's requirements:

When Funcom's MMORPG, Anarchy Online (AO), reached the beta stage in 1999, we started using Bugzilla. At first we just used Bugzilla as intended, for bugs. Today the AO team consists of around 40 people, and everything in our development process revolves around a modified version of Bugzilla. If somebody wants a task done by another person, and it takes more than 10 minutes to do, a Bugzilla report is created. (If the task isn't worth spending at least 30 seconds to fill out a report, the task isn't worth adding to the production schedule anyway.)

Our goal for using Bugzilla was to improve our production process. For management, that translated into more control over the project: the ability to measure statistics related to the production, and a better view of the project's status at any given time. For rank-and-file developers, it meant tools to track individual contributions to the project. Each developer is assigned between 10 and 200 tasks at any given moment, and everyone felt that storing a personal daily task list in the same location as the whole project's task list would be a huge benefit.

Problems With Early Bugzilla Versions

When we began using Bugzilla in 1999, we started modifying it. The modification process was fairly organic: somebody got a bright idea to customize the application, and it was implemented. For example, we started to flag bugs or tasks to be "Research", "Implementation", and so on. We also added fields for people to estimate the minimum estimated hours to fix a bug, a maximum-hours estimate, and actual hours spent on each bug. But shortly after introducing these fields on reports, we began to wonder whether we could trust the data, since we weren't sure whether people were entering the correct values into the different fields. That turned out to be a valid concern. For instance, if we discovered that people had different opinions of what "ASSIGNED" really meant, and we found out that many tasks' time estimates were being entered as "0-0-0" (corresponding to the minimum hours estimate, the maximum hours estimate, and the actual hours spent). So we learned we couldn't trust our bug reports. We needed a way to ensure that the information about a bug could be trusted. This meant that the meaning of each field had to be unambiguous, and the definitions had to be quickly communicated to everybody.

To solve the problem, we agreed upon the following Bugzilla changes:

With these changes in place, we saw the following improvements:

The changes we made to Bugzilla involved expanding the tables in the Bugzilla database, which in turn required knowledge of MySQL (the database Bugzilla 2.11 uses) and the SQL language itself. We also needed to know some Perl. The "main" table, bugs, was modified like this:

Since people don't always embrace change, it was a struggle to convince our team to modify their work processes. Initially, the leads were accused of forcing the team to use an overly strict control system. We were met with arguments like "everything is becoming so much more cumbersome and time consuming", "the overhead per task is going to waste the company's time", "making games should be fun", and "this requires too much administration". These comments usually came from people who weren't thinking about the team as a whole. Fortunately, demonstrating the new tools helped everyone on our team understand the big picture.

After we began to use the planning tools (described in a moment), we saw improvements. After a while everybody found out that the current output and control was better than ever before.

We found couldn't abandon our old rules and routines completely, though. One routine that we kept was the one that lets team leads set task milestones. In our system, each developer has so many tasks that they have to be ordered in some fashion, and in our system, the next milestone determines the order of tasks. (Usually we don't work on a task that doesn't have a milestone.)

Funcom's Bugzilla Modifications

One of the key features in Bugzilla we used was the ability to track activity related to tasks or bugs. It lets us see a history of all the events associated with a given bug. You'll see how we try to exploit this data in a moment.

Things that differ from the regular Bugzilla:

The net effect of these modifications is that people can more easily see the data they want. If they want to find a test case, they look in that field. They don't have to read through 40 comments to find it. Also, several operations can be combined which lets you save time and counteract annoyance with the application.

______________________________________________________

Utilities And Custom Reports

The Reopen Tool. Shown in Figure 3, the Reopen tool is typically used by our QA team. It is used to spot weak testing in our organization. It can answer questions like:

This tool is mostly used when we close in on milestones (when QA is in its most active period). There can be valid reasons for a bug to be reopened, so these statistics aren't used as a pillory, just a feedback mechanism. We use the amount of reopened tasks to see if we can make our deadlines. We can also see what a change in the build process (or the whole production process) does to the amount of reopened tasks.

Task Bouncing. Like the Reopen tool, tracking our task bouncing helps us improve the quality of our work. It looks at the history of reassignments, and answers questions like:

The Microsoft Project Exporter. When we have the routines in place for entering meaningful estimates into the bugs, the link between Bugzilla and Microsoft Project is our most valuable planning tool. The export process is semi-automated. From a web page interface, we first choose what bugs we want to export from Bugzilla. The results from the page are then pasted into Project, which has a blank template and correct date set. This template is customized for our company, with every person's vacation and working hours included.

We then analyze the data in Project. Project can order and prioritize the tasks (a process called "leveling"), but to do this correctly requires realistic data about each bug. To have Project prioritize tasks and calculate the optimal timeline for your project, it uses a numbering system in which tasks are given a priority score between 1 and 1000. When exporting data from Bugzilla, we calculate a task's priority score based on its milestone and its priority. For example; all tasks on milestone 14.4 have higher priority than any on milestone 14.5. A P5 task (Bugzilla term for Priority 5) on milestone 14.5 has a lower priority than a P3 task on the same milestone.

In Project, you can have up to 30 custom fields, in addition to the built-in ones the application comes with. We use these fields to capture Bugzilla fields like "milestone", "status whiteboard", "production pipeline", "last updated" and other fields we need sort the bugs by during our analysis. Figure 4 shows how we view things in Microsoft Project.

The Hours Report. The hours report, shown in Figure 5, helps prompt our developers to enter correct estimates. It can tell you the total number of hours developers spent fixing bugs during a given time period, and the number of times a user or department has updated their estimates. Importantly, this tool tells us whether the data in the database is up to date or not. It's important to know whether we can trust the time-related data in the database, and we can easily see whether there's someone on the team that needs to be more vigilant about updating their project data. In this tool the query-interface is very much like the reopen and bounce tools.

The Automatic Bug Hunter. This is a tool we have had for a long time on AO. Our servers create logs all the time, which are parsed by a daemon to create different reports. A crash on a server will generate a Bugzilla bug, accompanied by a log and full stack trace. If an identical bug is already in the db, the bug is just marked as a duplicate. The tool finds asserts, crashes, allocation failures, undefined shutdowns, and it creates charts showing this nasty activity. Once a bug is created in Bugzilla, the normal procedure is followed whereby the owner(s) of those components gets an email about the new bug. The automatic bug hunter is also used to keep statistics on the crash-rate, as shown in Figure 6.

Task Statistics. Our ability to track task statistics has proved to be very handy when it comes to measuring the state of the project. This report counts open tasks on given milestone(s) over time, and generates a graphical representation of it (see Figure 7). Usually the usage of this tool increases when we get close to a milestone. Comparing current milestone with last gives you a good indication of where you are going.

Task Creation Overview page. The task creation overview is similar to the task statistics tool. In the past, the AO team had problems whereby some departments assign lots of tasks to other departments at the wrong time -- e.g., after reaching the feature-complete stage. To easily spot and visualize this problem, we created a simple page that displays a grid showing the number of tasks and links to them (see Figure 8). Clicking on a number for a specific week shows you a task list, using the normal Bugzilla query.

Work Yet To Be Done

Because we aren't always using Bugzilla the way it was intended, we've discovered some undesirable effects, and there are modifications we want to make to the tool. These include:

Calendar Round Trips. I'd like Bugzilla to better integrate with our calendar. A task could have a start and end date, but currently we can't export tasks to Microsoft Project, change them there, and then import these modifications with calculated start/finish dates back into Bugzilla.

Task Clean-Up. Our team leads have to fix many incorrectly submitted bugs, which I'd like to address in the product and/or our processes. Granted, cleaning up the data here and there will probably never go away.

We also want to address the situation in which a task is sent around to different developers, each of whom helps complete the task. Our current system credits all hours devoted to a task to the task owner, but tasks like this should be split up into subtasks. We've repeatedly lectured our staff about subdividing tasks in Bugzilla when this occurs, but the problem continues.

Help Team Leads Manage The Information Bugzilla Provides. Team leads have to track many tasks. If you are the lead and you want to follow everything on your project, you need to be cc'ed on a lot of tasks; this means that you'll get lots of email every day. You have to have a good mail filtering system to survive this.

Latest Updates To Bugzilla. The latest version of Bugzilla (v2.17.4) contains many improvements over the older version we built our system upon. Examples of what's new include:

Threaded Discussions. It would be easier to understand comments about bugs if we could implement a threaded discussion for each bug. I'm sure that this is a requested feature in bugzilla.org's own (publicly available) Bugzilla.

Time Estimate Analysis. We would like to be able to track and report statistics regarding how well a developer (or all the team members) estimates the time needed to complete a given task. This data could be used to help us better plan our schedules going forward. This shouldn't be hard to do, since all the data is in the system already. It's just a matter of making the tools to report the data.

Consider Bugzilla

Whatever you do, whatever solution you use, there's no dodging the fact that project management is hard. Hopefully this description of Funcom's Bugzilla system will help you craft a similar system for your team, or inspire you to add functionality to your existing tools. If you're in the market for an extensible, simple to use tool for this purpose, consider Bugzilla. In the hands of someone who knows Perl and SQL, it's a cheap and simple solution.

______________________________________________________