Certified Scrum trainer and veteran developer Clinton Keith takes a look at the state of agile acceptance at gane studios, using survey data to identify common stumbling blocks, and presents here comments from developers on the process at their companies.
[Certified Scrum trainer and veteran developer Clinton Keith takes a look at the state of agile acceptance at game studios, using survey data to identify common stumbling blocks, and presents here comments from developers on the process at their companies.]
Over the past five years agile terms like "Scrum", "sprints" and "test-driven development" have pervaded every aspect of development. Is agile a fad being sold like snake-oil or the savior of the game development industry? Pundits have proclaimed it a "silver bullet", capable of solving all project problems. Others see it as an Orwellian micro-management of their creative freedom.
The truth is, Scrum's simple, iterative, and discipline agnostic practices make it an attractive framework for managers and teams trying to contain schedules and budgets ballooning out of control.
The hype has settled down a bit over these past five years. Many games developed using agile practices, mainly those in the Scrum framework, have shipped. There has been more than enough time to understand how much agile can really help a game team. It's time to look at what is emerging from agile game development teams.
This article starts with a short survey that asked people to spend a few minutes describing their experiences using agile. Read the stories and judge for yourself the challenges, successes and adaptations of agile from the industry.
Research for this article started with a brief informal survey taken in January 2010 of over 50 developers that have used Scrum for game development. The purpose of this survey was to collect a number of rated responses for how Scrum has impacted their studio in the following areas:
Game quality
Planning effectiveness
Quality of life
Project management
Design practices
Art practices
Programming practices
For each area, respondents chose one of the following ratings:
1 = Very negatively
2 = Somewhat negatively
3 = No impact
4 = Some benefit
5 = Major benefit
This was followed by a series of optional questions that would provide some background information to assist in understanding the results.
Overall the results of the survey indicate that there was typically "some benefit" in most of these areas. The bar chart below shows the percentages of responses for each category.
The greatest benefits reported appear in project management, programming practices, and game quality. Art practices, design practices and quality of life areas reportedly receive less benefit from Scrum.
The pool of respondents -- less than 100 -- isn't large enough to properly represent the entire industry. I suspect that many respondents were at either extreme of experience and that the "silent majority" are around the middle of range of responses.
The most valuable part of the survey comes from the written responses. The survey asked about the adaptations that studios have made to their agile process -- and their challenges with it and what they have learned. The responses that were not anonymous are credited.
I've organized some of the responses into three categories: challenges, successes, and adaptations. It's left to the reader to judge the merit and value of each response.
Many of the respondents who rated agile benefits low got right to the point about the challenges they faced:
"My studio changed our entire development process based on a few Gamasutra articles about Scrum. It was a classic case of chasing a trend without understanding it. It was obviously a huge flop, resulting in a dismal game released to market." - Anonymous
"Makes management feel forced to adhere to specific rules and makes interaction with employees less 'real'." - Anonymous
"No one knew exactly what it was, how to do it... and if they did they would run away screaming, because they just want to make games, not appease a cult-like managerial fantasy." - Anonymous
"ScrumBut -- my employer chooses to warp Scrum in ways that guarantee inferior results." - Anonymous
"It's easy to get hung up on the process rather than focusing on the quality of the game." - Anonymous
"People don't want to use something they a) didn't come up with themselves, or b) don't have experience with (or trust)." - Anonymous
"Unfortunately the idea of Scrum being agile can be lost on people." - Anonymous
"[One challenge was the] constant pressure from external forces. It takes a lot of firmness of mind to tell your clients they need to chill on feature creep. Secondly, disabusing management of the mythical belief that more hours means greater productivity. This was overcome by us by having strong data tracking on projects to measure true productivity versus just hours worked." - Chris Oltyan, Director of Product Development, ZeeGee Games.
"One of the biggest challenges for management faced was a resistance to change. Adopting a completely different (and in some cases, unheard of) project management methodology can be a very scary thing. But after a while, we found that people either experienced the benefits of Scrum firsthand, or saw the results speak for themselves on our end-of-sprint demo days.
"Another challenge was establishing lines of communication, pipelines, and procedures between the prototype teams and other teams that weren't working in Scrum (e.g. engine team and external QA team). How do these teams interface with each other when the backlog is constantly changing and being reprioritized?
"I guess the biggest challenge for me, personally, coming from a producer background and had only ever worked in waterfall, was embracing change. Taking on the role of a ScrumMaster required me to complete change my style of management. As Lyssa Adkins said recently in her blog, and I quote, 'You will have to give away your belief that having a checklist makes things run smoothly. You will have to stop chasing the perfect process and, instead, start cultivating your ability to trust the resourcefulness of others. You will cease using line items checked off on a plan as your measure of value. You will face your fears, all of them, about yourself and other people. You will stop making progress and start making products.' I couldn't agree more." - Kim Sellentin, Producer, The Creative Assembly, Australia.
"I attempted an implementation of Scrum on a project that was in need of general management and solidifed creative vision. It did two things rather quickly: 1) created more progress (and more measurable progress) than the project had previously experienced, and 2) caused an allergic reaction with entrenched leadership that resulted in the top-down destruction of anything that wasn't 'The Old Way'." - Keith Fuller, Producer, Raven Software.
Although there were numerous challenges, there were also stories of success:
"Scrum has affected all areas of production at the studio. Everything from meeting structures to report generation is more agile." - Mark James, Lucas, External Technical Director, LucasArts
"Initially, we adopted Scrum with one prototype team that comprised of two programmers, one animator, one artist, one designer, and one quality assurance tester. The increase in productivity was almost immediate. Our cross-disciplinary, co-located pilot team produced great results, faster than we'd ever seen, and with little micro-management. They really took to Scrum, and enjoyed taking ownership of their own work." - Kim Sellentin, Producer, The Creative Assembly, Australia.
"When we founded our studio we trained each and every member of our staff in agile at the same time. This made an enormous difference in getting everyone on the same equal footing and using the same vocabulary." - Jason Robar, VP of Studio, The Amazing Society.
Adaptations
By far, the greatest response was in the descriptions of the adaptations that studios have made to their process and practices for their unique studio culture and process to become more agile:
"Almost immediately, the variety of qualitative demands and concerns has required the exploration of various hybrid forms of Scrum/agile development. This includes using stricter Scrum processes for technology development, while using a much looser form for artistic development and design." - Anonymous
"Scrum was the basis for estimation and planning. A problem we encountered was due to a lack of adequate sprint review. This was compounded with some instances where the ScrumMaster ended up being the product owner. The end result was that quality started to slip as the ScrumMaster realized it would not be possible to meet certain deadlines, so they arbitrarily decided that lower quality would be acceptable.
"Instead, we've now modified this out of the process by firmly separating quality control from production, and by forcing the team to complete their sprints, even with crunching. This quickly resulted in programmers taking more time with planning, and figuring out tasks more accurately to re-establish better (40 hour-ish) work weeks." - Anonymous
"We are a small startup focused on mobile games. We have been using Scrum since day one of the company. Our approach to Scrum is heavily influenced by the size of our team. We are only three people in the studio (the two founders and a graphic artist), and we were not even sure if it was possible to use Scrum with a team that small.
"In the beginning everything was very irregular. We had some fantastic sprints when everything worked as a clock, and others were estimations failed terribly. As we gained experience with Scrum we started to experiment with different sprint times and estimation methods, and in a few months we found the variables that fit our team and keep us more productive and predictable.
"The single most important programming practice we implemented as a complement to Scrum was test-driven development. It has proved itself as a fantastic tool for preserving the game quality as the code base grows." - Manuel Freire, Founder and Lead Developer, Touchcreate.
"We use our own proprietary agile methodology that uses some of the Scrum practices. We use different terminology (closer to XP). We've evolved our own rigorous design/implementation approval process. We added a requirements review process to ensure we aren't fooling ourselves on the backlog." - Anonymous
"We went from waterfall to a hybrid-iterative-waterfall to now full Scrum. We went from project managers being ScrumMasters and sole owners of project plans to now ScrumMaster not necessarily [being] the project manager and everybody is accountable for project schedule and getting it done.
"There is now less micromanagement from the project manager side, who learned to let go and trust the team. The executive team is starting to see and understand the value but have not totally accepted it yet. Resource management went from software engineers being pigeonholed into the same work, to now all software engineers can move around and volunteer for work. Art is one area where it is evolving slower to Scrum, mainly as for our studio, it is a central team." - Anonymous.
"Developers are encouraged to just "trust their gut" when making story point estimates (and not worry about how many days it actually takes them to complete any given user story), since as long as developers are fairly consistent in their intuitive estimation of user stories, the story-point completion-velocity of the team becomes the best metric I know of for long-term scheduling.
"The publisher producer has no official Scrum role, but sometimes feeds into the product owner/creative director's user story prioritization by setting very broad milestone criteria that become more granular as each milestone approaches. We have weekly meetings with the publisher producer, in part to facilitate this." - Nathan Front, Lead Programmer, GiantSparrow.
"When I first started at my studio, they'd just implemented Scrum and things were rocky. We had 11 teams, each with varying sprint lengths, and varying team members every sprint. We pulled our ScrumMasters out of the teams to wrangle stories and tasks, but I gathered the teams' data daily.
"We realized this wasn't working to our advantage, because velocity was impossible to gather, and the developers resented having to work as ScrumMasters. I was given the job of full-time ScrumMaster for half the teams and we promoted someone else to handle the other half. We were sent for ScrumMaster certification and came back and made even further adjustments.
"We locked in sprint lengths and team members, began creating correct stories and having more useful standup meetings. Due to this, we were finally able to get a realistic look at velocity and get a better grasp of the scope of our project. The sprint retrospectives are invaluable and have directly impacted our process as we continue to 'inspect and adapt.'" - L, LaRae Brim, Assistant Producer, NetDevil.
Although the average ratings were positive, surveys of agile outside the game industry have also shown that a large portion Scrum adoptions fail to achieve their full potential. It's difficult to gauge what Scrum's "full potential" really is. In fact, it's definitely designed to help create successful games by great teams.
When developers talk about making a successful game, they often mention the "great team" that made it. Great teams can form in any culture or using any methodology. They are hard to define, but some of the following points apply. Great teams:
Follow a shared vision and purpose: Everyone on the team understands the goal of what they are working on.
Complement other team members' skills: Team members depend on each other to achieve their goals by applying their unique skills to a shared goal.
Exhibit open and safe communication: Team members feel safe to communicate anything to one another.
Share decision-making, responsibility, and accountability: The team succeeds or fails together, not as individuals. Everyone earns their spot on the team daily. There is no room for titles or egos.
Have fun together: They spend time together and enjoy each other's company. They care for one another.
Deliver value: Great teams take pride in their work and deliver high value consistently.
Demonstrate shared commitment: Great teams have a unified cause. When one member has a problem, the entire team will pitch in to help them out. As a result, great teams deliver value because they focus on the whole rather on their own parts. Great teams are committed to their goals. They'll go the "extra mile" to achieve a goal that they believe in.
These attributes sound good, but they aren't the result of any process. The people and the culture of a studio enable them. The goal of Scrum principles are meant to be a framework for organizations to foster great teams, but the hard work belongs to the organization. There is no set of rules and no methodology that can replace that.
So how do implementations of Scrum fail to achieve that? Let's look at some of the warning signs reported:
Is micro-management destroying self-management?
Is management forcing a top-down adoption of Scrum and focusing on the practices alone? Teams need to focus on the principles of Scrum while slowly taking ownership of practices to improve how they work.
Does management allow the team to commitment to what they can accomplish? Teams should slowly take on more ownership of task estimation and achieving goals. This takes time and requires trust to be built, but increased commitment pays off.
Are the agreed upon practices ignored when they are inconvenient? Are sprint goals being regularly changed? Nothing will destroy a sense of team commitment when this happens. Sometimes business reality has to trump sprint goals, but there are practices that allow that to happen without the damage it can cause.
Are the organizational challenges too great?
Are teams being tossed in the deep end of the pool without a swimming lesson? Is the studio deciding to use Scrum without careful consideration and study? Perhaps one team should try it first to see if it will work in the studio culture. Skepticism is good. Proof is better.
Is leadership supporting the change? Teams shouldn't go off on their own to experiment with Scrum. Leadership needs to mentor them and ease them into their new responsibilities (such as estimating tasks). Poor leadership is poor leadership under any methodology. If teams aren't provided with a vision or have no guidance and mentoring from experienced leaders, don't expect them to succeed.
Is the studio "inspecting and adapting" practices through retrospectives? Doing a daily standup alone isn't enough. Continual improvement requires continual inspection and adaptation.
Is the dogma messin' on the Lama?
Is goal to be "agile" and not finding ways to improve the work? The question shouldn't be "Are we doing Scrum?", but "Are we continually improving how we're working?"
Is an agile enthusiast telling teams "this is the way you must work"? Nobody likes to be told exactly what to do. Teams need to explore what works best.
Does agile mean I don't have to eat my vegetables?
Is the team using agile as an excuse not to plan? Planning has a bad rap because the "big documents up front" approach doesn't work so well. Agile teams that abandon planning in favor of pure iteration fail a different way. It's called an "iterative and incremental death march". Agile teams should spend more time in planning. The only difference is that planning is spread though the project.
Is the team measuring key metrics? If teams don't measure their velocity and gauge changes in practices against it, how do they know if they are improving? Some say that it's impossible to measure the size of features, and therefore impossible to measure velocity, but then, how are ironclad schedules being justified? What are the other things that can be measured to help teams explore improvements? Iteration time? Build stability?
Is a round peg being driven into a square hole?
Some publishers refuse to allow much change to the contracted plan. They've been burned by "iterative and incremental death marches" and can often be put off by the mention of Scrum. However, "rolling milestone definitions" and other similar contract/project "tools" have been used for a long time to allow some flexibility to exist. Studios have to slowly build trust with publishers and their own teams over time to allow more flexible scope, but that shouldn't prevent teams from finding better ways to work.
Are practice changes creating "ScrumBut"? ("We're doing Scrum, but...") Is a practice being changed to improve the work or to hide a problem? If you are thinking of adopting Scrum, watch this video first.
If your studio is adopting Scrum, do any of these warning signs apply? If so, they should be addressed. You may not end up with "great teams", but it won't hurt your chances.
In the mid '90s, our new studio was working with Shigeru Miyamoto on a Nintendo 64 launch title. We implemented many of his odd ideas and followed his mantra to "find the fun". One of Miyamoto's talents was to try to quickly prove or disprove something and build on what worked or abandon what didn't. We never felt like we failed him when we couldn't find a way to make one of his ideas fun on the hardware.
We didn't call ourselves "agile" at the time. We felt like we owned our methodology. Perhaps this is the one take-away from this survey: own how you make your games. The challenge is to find ways to continuously improve how you do it.
Some organizations adapt well to agile; others cannot. The main advice is to not take anything on faith. When something doesn't work, address and fix it. Development success depends on talent, leadership, vision and common sense. Once you start following a label, those things tend to take a back seat and that's not good.
[Photos by Drew Stephens, used under Creative Commons license.]
Read more about:
FeaturesYou May Also Like
