Post-mortem from the trenches of a mid-tier console developer

In a past life I was a Producer for a middle/low-end games developer in the UK. We strived for greatness but ultimately never quite got there. I came across this post-mortem that was constructed literally just as the proejct ended. The problems it details seem to be a checklist of how not to run a project. 

It contains a fair amount of project specifics as it was originally intended to be an internal document. I have also edited it slightly to remove a few names and to cut down the length - the original was 24 pages....

However, I hope that the lessons I learned from working within harsh (and complex) constraints can be useful for anyone in, or aspiring to work in the console games space.

miCoach 360/PS3

Post-Mortem

This report outlines the areas that need to be improved should we ever again embark on a large/medium scale asset heavy project.

Key points:

QA

Overview

In general QA was not effectively used during development. For a relatively small studio a large number of QA resources were available to test. On the whole QA needs a greater emphasis within the company. This can be achieved by:

1. Involving QA

Being involved from the design stage gives a greater understanding of the projects and will encourage the creation of test-plans. Currently builds are thrown at QA and they are asked to ‘test’. Equally there needs to be more effort from UK QA to get involved in the process.

2. Formalise the test planning

Structuring the testing over-time would have ensured greater coverage (and visibility to the rest of the company). Laying out exactly when Compliance testing will be run and when each feature will be tested in a project plan (Especially in the run-up to major milestones). Including test-plans as milestone deliverables could force this.

3. Meetings

Set up a weekly meeting between QA Lead and the other leads. Formally go through the serious open issues (The Must Fixes) to discuss real-world severity. Again, this places greater emphasis on QA. This can start in pre-Alpha stages.

4. Structure

Structuring the QA team so that each tester owns a particular area of the product (and rotating that ownership every few weeks).

5. Information

Give QA access to information. Many problems are caused by QA not fully understanding the depth or required functionality of areas of the game. Putting design on wiki would alleviate some of this.

6. Importance

Make QA feel more important. Hold regular meetings between QA and Production. Implement review systems where they can say what they like about the product (not just in terms of bugs). In these meetings QA can move away from comments based purely on functionality and move more into the actual performance of the product (does it meet expectations).

7. Communication

Force two-way communication between QA and rest of dev team. Test coverage was not transparent to anyone during miCoach.

QA

Other issues

1. Towards the end of the project QA was used primarily as a build machine. Spending hours creating and smoke testing isos to deliver for milestones and for external QA. 

To speed this up, and remove human error, we need to automate the build process. Run auto-builds (with automated test tools) overnight. Auto-deploy successful builds to consoles and/or setup a ‘burn machine’ with a rack of burners that can have all discs ready for testing when QA come in. Also implement a ‘one-click’ system, with configurable options, so that when random requests come in for a ‘debug build to test some videos’ the builds can be created by anyone.

2. QA had to wait for an actual iso to properly smoke test a build. Delaying build deliveries/bug reports at critical times. We need to give QA access to a dev kit so an iso can be emulated and checked before burning.

3.Back-and-forth between QA/dev team to narrow down steps to reproduce. We need to install a video capture card on all QA PCs. Bugs can then all be captured as video. This is a common practice now in external QA and other dev studios.

QA Key Points

Code

Overview

Early on in development the code team was limited in size and ability. A junior coder was placed in-charge of the two fundamentally core aspects of the project.

Functional code deliverables in early milestones were hacked together. This built an unstable foundation for the remainder of the project. This foundation did not stabilise until two months before our first submission date.

Key issues:

1. Code Leadership

A lead coder was assigned full-time to the project six months after the project started. Up until this point the code base was Production lead – which willalways result in emphasis being placed on delivering the milestones – at the cost of creating stable and reusable code. In future a lead coder must be involved from the moment the first line of code is written!

The lead coder left the company four months later and was never replaced -Leaving a gaping hole in the code department. Again Production was leading the code team. If this occurs again provision must be made to provide proper leadership in this area.

2. Schedule

A code schedule was started 5 months after the project started. Together with the fact that design was locked down late into project, this contributed to the hacked together nature of early milestones (and the creation of an unstable backbone to work on). A high-level schedule needs to be in-place from the beginning – based on initial design – then developed as design progresses. The schedule can be locked down as and when the design is finalised. This needs to be prioritised by both the code lead and production.

Communication between Code and Production was fragmented and frequently obstructive. Tasks and priorities were shifted without discussion. If Production is to keep projects on track changes like this need to made together or at the very least communicated in a timely manner (rather than by discovery or by probing). Production pushed and pushed for a feature led schedule. However, this was over-ruled and a code led schedule was created.

For the sake of clarity (both internally and for external communication) a schedule must be feature led. Otherwise it is virtually impossible to predict when features are complete and ready to be tested and delivered. A feature-led schedule also makes it easier to slot in work that appears as a result of any change request.

3. Source Control

Experiments with alternative methods of source control were attempted with the installation of Mercurial 3 months into the project. It was abandoned (and reverted back to SVN) 3 months later. The issues we had with Mercurial lost work, time and generally disrupted workflow. In future experiments with fundamental tools should occur before playable code deliverables are due, and certainly never during full production.

4. Design Communication

The design existed only in a massive Word Doc. Putting the design on Wiki would encourage more people to actually read about the feature they are working on and, crucially, allow for information beyond what is usually contained in a design doc to be included and referred to by the coders.

5. Key staff failings

A key member of the team in early milestones – our only UI coder at that stage – was not up to the job. The work completed was poor and endlessly created bugs and fatal flaws. This was made more evident at milestone delivery times. The poor nature of the coders work was communicated multiple times, and subsequently ignored.

Communication of staff failings need to be listened to and considered. Especially when key milestones are being missed as a result of poor quality work.

6. Testing

It was evident that coders were being forced to rush, and therefore committing work without testing. An embedded code-only tester could be utilised to test coders work. But more importantly a culture of “test before committing” should be strictly enforced.

7. Overtime

The project felt like one massive crunch from virtually the beginning to the end. This was definitely felt within Code, QA and Production. Coders would frequently work frankly crazy hours. Mistakes would then be made due to fatigue. Additionally, the late working would result in team members appearing in the office extremely late the next day.

Ridiculous overtime should not be encouraged. There are other factors that can be improved or changed to counter it. Managing the schedule and the expectations of the publisher more effectively is a much better way of meeting milestones rather than hammering the team into the ground.

Code Key Points

Production/Management

Overview

1. Task management/schedule

In general the approach of Production/Management staff throughout the development of miCoach was fractured. Surprisingly for a relatively small studio there were many different methods of assigning/scheduling work between the teams. If we are to ever get everything organised correctly, and bring the various disciplines together, the following needs to happen:

-Decide on a homogeneous and company wide policy on tasks/schedule management. Crucially this must be feature-led.

-Remove the reliance on messenger (banning it entirely)

Multiple task management styles were used:

a. Messenger - Emphasis on talking to the team and asking for updates via messenger. (Whether internal or external).

b. Whiteboards - Using post-it notes to list out tasks in Done/ Not Done lists on whiteboards or simply just written To Do lists on the whiteboard.

c. Excel Doc lists - Lists of tasks in spreadsheet form either emailed round, or in shared folders

d. Email lists - List of tasks in email form mailed round to email groups. With requests for updates requested in reply

e. Campfire - Online collaborative tool with To Do list functionality

f. Verbal/On-paper - Simple verbal requests for work, tracked only on paper

g. MS Project - Gant chart/waterfall style task lists

h. Excel Sheets - One sheet per person or per task to do lists

i. Word Docs - One doc per feature with multiple tasks/multiple people

j. Mantis - Tasks written into Mantis

Defining a global method of managing the project equips the team to deal with the unexpected. E.g. last minute demo requests or slotting in work as required by feature changes/change requests.

2. Approach to the publisher/external stakeholders

Throughout development we were totally beholden to the publisher. For example the GDD and ADD went through multiple iterations until finally being signed off 5 months onto full production.

The lack of a united approach on publisher relations/change request resulted in watering down of several areas of the product. Also, several features pushed by the publisher ended up being dropped as:

At the start of each project we need to define the approach and internal perspective towards the publisher (and other external stakeholders) to maximise our control/input into the project.

3. Features dropped due to alleged lack of time.

A number of features/areas were dropped. If we had spent more time up-front investigating the feasibility of any of these we would have wasted less time trying to implement them:

4. Project scope increasing

The scope of the project increased after signing, without the time-scale increasing.

Areas increased or added:

In future we need to ensure that fundamental changes to the scope are reflected in the project time-line.

5. Attitude to the project

As the project progressed the internal perception of the project flipped to be overly negative. The job of Production is to support and guide the dev team through the project. Adopting a publicly negative standpoint does not help when trying to motivate the team. Aside from the general attitude of the internal production staff this negative attitude is effectively illustrated by the external producer who was brought in partway through to work on the Training Drills. Their approach was negative and destructive. This works in a similar way to pair programming. Where pair programming can increase the quality of a code team’s output – a negative ‘seed’ in management affects the team around them and decreases morale/quality. As a company we need to remain realistically positive and work with the team to solve issues and raise morale where possible. Where negativity exists it requires a large effort to keep it in-check and to bring it out only in Production or closed meetings.

7. Communication

Using Messenger as a primary communication tool is not an effective way to motivate or talk to the team. It encourages laziness and discourages face-to-face talking.

The use of messenger to communicate needs to be discouraged or ideally banned entirely.

8. Meetings

As the project got busier and more frantic meetings became more ad hoc. Need to set meetings in stone and not cancel them even if everyone is busy.

9. Schedule over-run

Milestone 8 was the last milestone the company hit on-time. There was no provision in the milestone schedule for the required one week of testing the milestone deliverables in the run-up to delivery. Meaning that milestones were cobbled together at the last second and submitted with little testing. Resulting in milestones being rejected and further work being required. Coupled with other scheduling problems this helped to enlarge the over-runs as the project progressed. We eventually hit Alpha a full three months late.

For future projects we need to add in time for testing the milestone deliverables. This will of course need to align with realistic unified expectations on what is achievable within the time-scale of the project.

10. Exposure to key information

Throughout the development the budget available to the team was never communicated (or hinted at). Being given information on how much is available arms Production staff with relevant knowledge on how to deal with issues. E.g Can we afford to outsource items? How much is available to pay someone to do a particular job externally?

11. Reviewing the builds

At times Production did not look at the build often enough - mainly because of time constraints. Setting aside set review times (and sticking to them) is the only way to solve this.

Production/Management Key Points

Art

Overview

Considering the size of the committee that had an input in the art/art style what we have ended up with is of a relatively good quality.

The progression from initial concept to the final game art style and UI design is shown at the end of this document (Page 25 and 26).

1. General approach

Given the standard request for a higher budget, more resource and a longer time-frame we could have done some things better.

The initial art style was agreed with the publisher, then presented to the key external stakeholders who were not happy with the proposal. From then an inefficient chain of back-and-forth between all parties continued until we ended up with what we have now. The look and feel was still under discussion 3 months before our initial submission date.

After the ADD was officially signed off the focus of the discussion hinged around one tiny found image.

The core quality of this image is a photographic background with augmented objects on top as a HUD.

If we sat down with THQ and the key stake before we did anything, and established the above image as being the actual approach required, the game (at least visually) could have been a very different one.

If it comes to ever doing anything like this again we should consider dropping rendered backgrounds and using real-world video footage. This would make the job of blending actor video with the background a much more natural process with the end result being “like a TV programme” instead of the obvious look we have now (Real keyed video over relatively cheaply produced rendered backgrounds).

Key point:

2. Reviewing work in-game.

As builds were unstable and unplayable for such a long time it was difficult for the artists to review material in-game. Artists were not given test kits, and the facility to check their work (as they go) until the beginning of August. Prior to this they were relying on QA or a flaky Phyre engine viewer on PC that did not accurately reflect how the elements would be displayed in-game.

Providing early stable builds is of course not always possible so we need to make sure that the Art team have the correct tools so they can check their work (without relying purely on QA or waiting for build roll-outs)

3. Toolchain

The Phyre engine has quite a few rough edges and with little dedicated tech support. Time was wasted on trying to do our own shaders and getting things hooked up in the game. This got slightly better when a new hire was tasked with managing Phyre implementation. Art needed code support earlier. Or, ideally we should use a more familiar engine.

4. UI positivity

Everybody underestimated what an enormous task the UI would be. Looking at the UI one could argue that it’s more an application than a game. However, visually the UI has turned out to be one of the more polished areas of the game. Although some compromises were made to improve the general control and navigation mechanic it still looks polished and is closely aligned to the tone of the game and core branding.

5. Camera alignment

The use of five cameras was a good choice as it gives variety in the presentation of the exercises. However, the camera positions were not fixed. This generated a massive amount of work where we had to manually position every camera angle. A tool was created to do this using an FPS control method to position the athlete footage against the rendered backgrounds. In total this used at least 4 months of the entire UK QA team’s time (and killed them a little inside every time they were asked to work on it!). The manual “by-eye” nature of the work means that we have less than perfect alignment in-game. If ever we use cameras like this in a shoot we need to fix the camera positions. This could be achieved by building a rig that the cameras slot in to. Or by precise measurement using Scatter Cloud scanning technology (which was proposed at the time at a cost of £500 per shoot). This would eliminate the need to align every single shot.

6. Camera tracking

The audio issues we encountered that prevented us from using the MILO meant we fell back on using Techno-cranes. The work this generated was massive. Every presentation clip had to be run through software (Boujou) to generate 3D tracking points, then a spline was created, then the spline was converted to a Phyre engine compatible file. Then it could be tested against the athlete video. The work to generate the splines was outsourced at additional cost. The entire process was overseen by a lead artist – taking up 99% of his time for a full six months.

Additionally the use of a techno-crane generated rotoscope work that had to be outsourced at additional cost. Frankly the quality of this area of the product is very poor. Considering that this was the only way that the athlete’s personality could be injected a better solution should have been sought. The quality of the tracking markers themselves and the quality of the work produced was below standard. So, we have ended up with ‘wobbly footage’ that looks cheap. In future we shouldn’t attempt anything like this and stick with static camera shots (Or use a MILO and solve the audio issues we had)

Art Key Points:

Design

Overview

Over the course of the project we lost several Lead Designers. This left our Creative Director as the designer on the project – as he was rightly involved in other projects/business development he wasn’t able to necessarily give the focus that the project needed. This meant that the design was not led by one individual – leading to, in some cases, a watering down of key concepts and increasing design iterations. There is no real way of avoiding this except maybe extending the interview process for designers (if we attempt ‘game application development’ again rather than just ‘games development’) so they are clear on what is involved.

1. Design Committee

Almost all of the design work required was really focused on problem solving as opposed to actual “Creative Games Design”. In addition a huge committee of 5 separate large organisations all had what was effectively final say on design choice. This is probably one of the largest contributing factors to why we lost so many designers.

All of the stakeholders effectively forced design into the game due to the requirements of the existing brand/products. To a designer this external input removes the need for creativity and reduced the role to one where the requirements of the stakeholders must be met first, then creativity was allowed. Again, there is no solution to this as the nature of the project required this approach. However, in future perhaps we need to formalise the input of stakeholders. Certainly internally we can achieve this by holding regular review meetings and investigating each point/request made before agreeing to the work.

2. Rushing into decisions

A prime example of where we rushed into a decision was the choice to use the mocap animations as exercise tutorials. On first glance it looked simple – all of the animations existed and we can just re-use the mocap data to run our animations. If we had investigated further, before agreeing to include them, we would have realised that there were issues with the approach. The animations only worked from one angle, the mocap data was dirty and incomplete and multiple skeletons/characters we used (to get the animations to work).

Being forced to include them created work and introduced unnecessary cost. We had to run another mocap shoot to fill the gaps in the data and pay an outsourcer to clean the data/rig the skeleton. Then we used an unskilled person to tweak and render the animations.

Additionally the approval process for the animations was a slow and painful process. The final animation was approved five months after our initial submission date.

“Widgets” are another example – it was only well in to the production of the widget feature (when they were actually starting to appear in-game) that we discovered that the idea was fundamentally flawed and not possible to implement with our tools.

So in this case we should have investigated the implications early on to check the feasibility of including the animations as a feature. This same rule can be applied in the future for any other large design suggestion coming from a project stakeholder.

3. Exposing the design to the dev team.

Putting the Design into a wiki format is one way to help communicate design to the dev team. However, having a wiki or a doc still relies on people actually taking the time to read it. Often people will only read the bits they are interested in, if at all. One solution is to force the team to read the documentation (once design is close to being locked down) by having a ‘reading day’. This can take the form of a morning where no-one codes, draws pretty pictures or creates schedules, instead everyone reads the documentation. In the afternoon the Lead designer delivers a ‘Sales Pitch’ to the rest of the dev team – outlining the core components of the design and communicating the ultimate goal of the project. This can be followed by a Q&A session to discuss the implications/implementation of the design.

Design Key Points:

Conclusion

Despite the negative tone of the almost all points raised in this document we should be proud of what we achieved. For an under-resourced and low budget (For what was expected) project with an unrealistic time-scale what we produced is good and looks more polished that anything else we’ve created to-date.

We’ve managed to run video shoots on a global scale and worked effectively with high-profile branding. Also, the first ‘gamey’ features we’ve every created - The Training Drills - are fun to play and function really well. Especially when you consider the tiny resource allocated to them.

Losing the publisher 3 weeks before the initial submission date did flatten the company morale and for certain the work-rate dropped massively. So, it is amazing that we finally submitted the product and it is going to see the light of day.

The issue of not having enough time to produce a product is not going to go away but if we massively tighten project management, define expectations and define what is expected from everybody in a clear way (from the top level down) we should be able to tackle a console project of equal complexity and produce something of higher quality – given what we learned through the development of miCoach.