I started to use the Earned Value Management (EVM) back in my days in Software Engineering, EVM is a well-known project management technique but mostly unknown in the games industry.  The purpose of this article is to demystify the Earned Value Management and analyse how it can be adapted it for the a game development to become an effective tool and help your team to control the production.  This article is a Project Management article; I hope you will find it useful.

What is Earned Value Management?

EVM originated as a financial analysis from the 60’s during a missile program in the U.S. After a series of programs where the government had no visibility on how much they spent and how much there is left to spend, the Department of Defence decided to implement a directive with 35 criteria (C/SCSC) that were used for the first time on minuteman missile program in 1967 as financial control analysis. After several iterations the technique was simplified and EVM was created. It became so popular that it has been adapted for any projects involving the US government and became a worldwide recognised Project Management Technique in the 90’s under the standard: ANSI/748.

Without going too much in details Earned Value Management, tracks how much work has been earned (EV), versus planned (PV), versus spent (AC) on a periodic basis. With these 3 values, the technique derives multiple Key Performance Indicators (KPIs) that helps to understand the project status.

’Earned’ means the work that has been completed and verified, to determine this, each piece of work are identified in the Work Breakdown Structure (WBS) are budgeted and planned in advance.

The graph above shows a typical Earned Value chart that you will find on the web. The green line is the Planned Value (PV) which is known from the start of project, then in red is the Actual (AC) and in blue is the Earned Value (EV), the values are captured after each period (or milestone). In this example you can see in Period 4, the project was slightly ahead of schedule (EV > PV), the project earned more than planned but quickly become behind schedule for the latest period (EV < PV). In term of cost, the project is under budget (AC <PV) but heavily over budget against what they’ve actually earned (Cost Value = EV - AC < 0).  Then from these values you can determine the following KPIs.

The CPI and SPI are not binary values, usually a SPI/CPI between 0.85-1.10 is considered a good indicator but what is the most important is the trend of these KPIs during a long period. There are many references on the web that will explains much more in details how EVM works.

How can you adapt EVM principles to agile game development?

While with a waterfall project, the EVM technique provides visibility both on the long-term and short-term, this is not possible in game development. The one key element that EVM technique relies in a fully estimated and planned Work Breakdown Structure (WBS)  in order to determine your PV and BAC, without it there is nothing to generate the KPIs, but that’s not possible for games, as most producers know, the game scope is in constant fluctuation during development. EVM does not cope with requirements changes.

The development might start with a Prototype phase but it’s foolish to assume your Backlog contains all estimated feature for release, so applying the basic EVM technique is not going to give what we need. But by separating the short and long term in two separate levels, it is possible to reuse the EVM principles to what is also called Agile EVM.

Sprint level

At sprint level, Agile/SCRUM development are already doing Earned Value, we just don’t know it.

To illustrate the similarities the chart below is a traditional sprint burndown chart but 180° rotated. Instead of having the remaining work burning down, the Earned Value (BAC – Remaining Work) is increasing, that is equivalent to the % completion. The Planned Value is the linear distribution of the estimations over the sprint duration, you can do this shortcut because in Agile the scrum team is usually constant during the sprint and it is encouraged for the work to be shared.

The rules of how the EV is earned during a sprint can vary, in the chart above it’s simply in days when tasks are completed, but depending on the granularity of the work, it may be prefered to only earn the full user story once it has been completed and verified which is more accurate, but for a smaller scrum team with few user stories committed and shorter time, there is a risk to not capture much data during the sprint as the stories may be verified well after.

The SPI is calculated in the same way as the traditional EVM, if the blue line is under green line then the SPI is below 1.0 therefore behind schedule and vice-versa if above. In agile the SPI is known as velocity.

What this chart does not track are the Actuals (red line in the EVM chart - AC) that calculates the CPI. Actuals have a different meaning in Agile EVM and especially in game development, It’s important to note EVM was created for engineering projects where there are lots of fixed costs (materials, outsourcing...) where you can overspend on materials to catch-up, while a video game is all about people, the development team is everything. Within a sprint, the actuals cannot be above the Planned Value (or guideline), the team can’t log more than a day in a day and if you start adding/removing people during a sprint then you are not running agile properly, anyway. Tracking the CPI becomes meaningless as a cost indicator in Agile EVM.

Instead the actuals can analyse the "fat" during a sprint by using the formula: AC / PV. A sprint starts with the theory that all the planned work will be spent during the sprint but as it always happens there are meetings, feature changes, bugs that will always prevent it happening… so this indicator tells you how much the team is efficient against the baseline. The classic reaction is to start planning these events by building more contingency or even adding unplanned tasks, but from my experience this adds an extra layer of control and will annoy the team, it’s much better to keep it outside the tracking environment and monitor to reduce the ‘fat’ KPI by improving the development process.

Another notable difference and a key pillar of SCRUM is having a releasable product at the end of every sprint but in order to achieve this means the sprints have to be delivered @ 100% consistently,  which means a SPI always superiior or equal to 1. If that’s not the case then the velocity needs to be carefully analysed to establish how much the Team is able to earn in a given period. It is understandable sprints at the start of the project or in pre-production can have a SPI to 0.8-0.9 but if it does not improve then less work has to be committed in order to cleanly start/finish a sprint. 

These values gathered on a sprint basis and their analysis help to understand how well the team is executing within a sprint, understanding the short-term productivity helps to become more predictable for the long-term. But that’s only one element of a successful game development, sprints can be very effective but the development can still go nowhere if the scope or creativity is not well managed behind the scene that requires a different level of control.

Project Level

At project level, Agile game development becomes less adapted to EVM. As mentioned previously traditional EVM relies on a pre-planned Work Breakdown Structure (ie. Product Backlog)  while in game development the scope will fluctuate and that’s totally normal, a video-game is to be “fun” where creative leaders wants to make unique experiences, we have to discover, make mistakes and learn to make the best games possible but it has to be achieved within the given constraints.

Thanks to the performance captured at sprint level, it is possible to build an analysis where the predictability can be improved for the long-term. What is needed is a product backlog that links Sprint and Product level, so once a sprint is complete the work earned at sprint level can be deducted from the product backlog. Then the work earned in a given time period is analysed to generate the Earned Value at project level, which finally can help to calculate the date when the backlog will hit "0".

The difficult part is to maintain the backlog and keep the link between the 2 levels. In order to reduce complexity, the backlog should be estimated in a different metric as the sprint. For instance stories in the backlog are estimated in Points or T-shirt size by the leads and then broken down in days at the start of the sprint by the team.

Linking Project and Sprint level

It is important to acknowledge the backlog is never going to be fully estimated, it requires recurring sessions with the leads to estimate and breakdown the backlog.  

The calculated “hit 0” date on its own does not mean a lot, you need to take in account two important variables to become a meaningful metric:

Conclusion

Games are more and more pushing boundaries to offer unique experience in a very competitive market, console games are faced with deep technical challenges to make visuals even more realistic while mobile games are constantly iterating to find the best balance between game experience and KPIs. Agile EVM is a lightweight approach to produce recurring and reliable KPIs. These metrics can be easily communicated upward or downward, either to report project status to Executives or conduct retrospective with your team after a sprint.