Your Time Estimates Are Broken

(This article was originally posted at Ananse Production's website. Click here to read the original article)

Your Time Estimates are Broken

My name is Kwasi Mensah, and I've fallen out of love with agile development. While I bought into it when I first started working, over the years I've lost faith in our ability to make reliable time estimates, the cornerstone of this methodology.

My experience, and that of many other colleagues I've informally asked, is that while the majority our estimates may be in the ballpark,[1] when they're off they're really off. There are several reason proposed for this behavior (such as distractions and vacation time). In this post however, I'm going to focus on the aspect I think effects game developers the most: the cost of innovation.

How are we supposed to make an estimate on figuring out the color palette for a game about a psychic summer camp? Or how many iterations are going to be needed to get people to dance to Lady Gaga in their living room? Each game we develop hopefully has work that's never been done before (or is at least new to the team making it).

We have to deal with tasks outside the scope of our experience which agile development doesn't seem to deal well with. And I don't think there's much we can do about it. We don't get better at estimating how long it will take us to do unknown tasks. We gain experience doing similar tasks and then draw on that to give better estimates.

Humans are fundamentally flawed at telling how long tasks are going to take. The planning fallacy tells us that we always overestimate how quickly we can do a task. There's a famous example of this with psychology students severely underestimating how long it would take them to finish their theses. And this is only one of the many psychological quirks that stop us from getting estimation right[2].

Breaking tasks down, documentation and technical design documents help give us a clear road map and disseminate knowledge to the rest of the team and managers. But when you're uncertain about a key dependency in your task any estimate you put down is useless[3]. And there are several cases when doing the work required for an estimation is a lengthy task in and of itself.

So how can we have any hope of shipping a product "on time"[4]? Deadlines are still important for talking to the press, managing PR pushes and getting paid by publishers. I think we can put better emphasis on becoming familiar with the unknown. This is where having a strong pre-production period helps.

Having a period up front that says we're willing to spend x amount of time exploring these y number of specific questions. And then having the discipline to limit production to the questions you know the answers to[5].

If you're going to try to do things your team has no experience in understand that you're going to need a lot of experimentation time or you're going to have to cut down on the number of new things to add. I think the power in being "indie"[6] is that the lack of resources forces you to be conscious of this balance at all times.

So once you've answered your games core questions and are in production what can you do to help the team stay its course? Assuming there are no shenanigans with your bug database, keep your bug count as low as possible before starting new features. Some places advocate charging the time back to the original task that caused them so you can better gauge the true cost of the task.

I just think it's important to keep the total number of bugs down[7]. Even if you've gathered the knowledge needed to have a semi-reliable estimates on getting the first pass of a task done, by definition bugs represent unknown quantities of work. There's also the memory cost of having to remember the specifics of work done months ago instead of days ago.

While I've had a nebulous version of this problem in my head I can't say we followed all of this advice when making Stem Stumper. And this post is largely based on fixing problems I've experienced before and not on how the practical application of these ideas went[8]. Even if you think I'm overreacting (which is quite possible) I think we have to come to terms that we work in an industry where inspiration can't always be forced by milestones.

Notes

Note 1: Small deviations from initial estimates is a problem in and of itself since these add up over the course of the project. As do distractions. Joel on Software has an article on developing a statistical model to take these into account. I like the idea of a confidence range instead of a date. But I think making a game has so many more creative unknowns than making issue tracking software that I think pre-production and not statistical modeling is going to have the most impact.

Note 2: Wikipedia has more examples of why we're psychologically bad at this. Check out this Wikipedia section.

Note 3: Actually the estimate is worse than useless because it gets written down and gives production false hope.

Note 4: The psychology of deadlines is fascinating. There's plenty of research that says intrinsic motivation can have a better effect than deadlines. But there's also the non-scientific but often observed Parkinson's Law, "Work expands so as to fill the time available for its completion". There are parts of the process that we could stay on forever and parts we would rather skip over.

Note 5: But everyone knows this right? So why don't we take the time to answer these questions before going full steam ahead? The problem with this type of planning is that it doesn't fit well with how we currently staff projects. Unless you have ton of independent questions you want answered by your team, you're going to have a fair amount of people with nothing to do. As a fiscally responsible company, you have to try your best to not let that happen but that last thing you want to do is tear apart the relationships and expertise you just finished building up on the last project. "Indie" developers have teams that come together and break apart as projects start and finish. Everyone has enough on their freelancing plate that they're ok doing this. But at larger developers, which try to attract people who don't want to live on ramen, job security is paramount.

Extra Creditz has a really good episode discussing pre-productions which you should all go watch now. By the way, Extra Creditz seems to be having contractual problems with The Escapist. All I know is the videos are a constant source of intelligent conversation about games in accessible language aimed at non-developers. You can follow them on twitter @ExtraCreditz to see what they're up to next.

Note 6: Hopefully the quotations marks around "indie" give away how much of a loaded term I think its become. For the sake of this post, "indie" means a tiny team working out of their bedrooms eating ramen like its going out of style.

Note 7: Yes, I know 100 game crashing bugs is different than 100 bugs about a dumb feature not working properly. You'd probably need some type of stratified system where there are no crash bugs, up to 20 second tier bugs, 40 third tier bugs, etc. And you'd also have to take account bugs that might be fixed by future work. The numbers are completely arbitrary but your team should consciously decide on an acceptable balance between making progress and quantifying the current amount of outstanding work.

Note 8: I guess I'm suggesting time-bounded period for pre-production where you deal with issues in a preassigned priority. This sounds a lot like a sprint, except it'd be over a longer period and you have the disciplne to drop issues that couldn't be figured out in time. I think agile can still be useful during production when you have a better handle on the amount of work in front of you.

Broken Clock image from: http://2.bp.blogspot.com/_0S4RTTd6-v0/Sc2ED9W_IYI/AAAAAAAAA0Q/XA5eZgFTjSE/s320/broken+clock.jpg

Car estimation image from: http://xkcd.com/612/

Balance image from: http://www.dailyhaha.com/_pics/good_balance.jpg