A Really Big Playtest Failure

"Playtesting is sovereign", I like to say to students.  But it has taken a long time for videogame companies to recognize this, and there're still examples of complete failures to do playtesting right even when they play test. 

An example from last year is Final Fantasy XIV.  Final Fantasy games are normally praised but this one has been trashed by reviewers.  It's score on Metacritic (an aggregator of reviews) is 49, which is really bad for a videogame.  http://www.metacritic.com/game/pc/final-fantasy-xiv-online/critic-reviews

Here are some excerpts:

 "It may be hard to believe, and painful to accept, but the last 'official' chapter of Final Fantasy is a huge disappointment, and it not only becomes the worst chapter of the saga, but also one of the worst MMORPG we have played in a long time. It's obvious that Square-Enix is no longer what it used to be."

"Final Fantasy XIV is in no way, shape, or form ready for commercial release. We suppose there is potential down the line for the experience to improve with patches and tweaks, but that doesn't justify why the game is in such shambles at launch. "

GameSpy (gave a 40).  "Extended stretches of confused incredulity, punctuated now and then by bursts of intense anger. And yeah -- I just equated FF XIV with a filthy bathroom."

PC Gamer (gave a 30).  "A shallow, slow, grind-heavy MMO crippled by a horrible interface and nonsensical player limitations."

So didn't they playtest it?  Yes, they did!  I found evidence of an open beta: (http://www.ffxivvault.com/ffxiv-beta-information.php and http://www.wired.com/gamelife/2009/12/final-fantasy-xiv-beta-test-now-recruiting/).  "Anyone that wishes to be among the first to play the new MMORPG, which will launch on PlayStation 3 and PC in 2010, can sign up at the official beta site. Square Enix says that no prior MMO experience is required, and players selected to be in the beta will test the Windows version of the game."

So what happened?  Did they test only with fanboys and girls?  Did they ignore the results?  It seems unlikely that they ran out of time and put the game out before was ready, because Square Enix is one of the more respected and consistently good videogame publishers.

So what can happen to give a "false positive" from playtesting? 

First, you could be playtesting with people who know the game so well that the problems just don't register.  But this would be likely only if the people within the studio were the playtesters, and in this case it was an open beta.

Second, you can have open playtesting but somehow you only get fanboys and fangirls, people who will say good things about your product no matter what.  The have lost their critical faculties where you are concerned.  Perhaps this is what actually happened.  This is why you want to try to get many playtesters who don't know your products and you aren't prone to this kind of thing.

Third, you could just ignore the results, saying "oh well we know it's good".  I'm convinced that Microsoft has done this quite frequently, adding smugly "we know what's best".  There can be many examples but in the video game category the most famous one is the huge controller that Microsoft included with the original Xbox.  Apparently it was sized for people my height (2 meters) with correspondingly large hands.  According to what I've read Microsoft continued to maintain that it was fine but finally changed to a much more normal sized controller.

Fourth, perhaps they asked the wrong questions and yet didn't pay attention to the actual play.  Frequently if you ask people what they think you'll get a different story than if you simply watch them and listen to them without interfering.  Jakob Nielsen, the guru of web usability, always cautions you to watch people, not ask them what they will do or what they have done, because you get better data.

Fifth, though it seems unlikely in this case, the bean counters can win out in the battle over schedule and the game can be released because it's scheduled to be released even though it's not right.  I supposed Square Enix is big enough to avoid the schedule trap. Maybe not.

When I originally posted this in one of my blogs one commentor who had been a Final Fantasy XIV playtester said that when he saw the title of the post he immediately thought of FF14.  "Everyone was saying it wasn't ready.  All of the flaws that the critics mention were reported to SE over and over again.  In-fact I recall very few truly positive comments on any of the forums. A few people saying "such-and-such is a neat idea, but needs a bit more development.

As far as I can tell, absolutely nothing was changed between the beta and going live, aside from extra areas being opened up and some souped-up hardware."

This brings up another possible reason for playtest failure, that there was no intention to improve the design of the game but only to check for bugs.  They may also be much more interested in building up prerelease hype than it actually improving the design of the game.

I met Jason Schklar at the video game conference in Raleigh (now called East Coast Game Conference), whose company arranges playtesting-the-design (as opposed to playtesting-for-bugs) for video game companies that cannot, or do not have the time to, do it themselves.   Many video game companies only playtest for design internally, which is surely a mistake.  The people who are part of the production team are often too close to what they're doing to recognize when there are design flaws.  You need ordinary players to try the game, people who are your actual target market, to find out whether the design is suitable for the intended audience. 

Furthermore, many video game people seem to make the mistake of thinking that what they like is what everybody likes, even though videogame production people are by definition strange and unusual, and not typical of any commercial market.

Jason said "The real goal of my work is to ensure that players experience the game as the designers intend the game to be experienced. This can result in:

    * Player Experience = Design Vision + "Players enjoy it"
    * Player Experience = Design Vision + "Players do NOT enjoy it"
    * Player Experience <> Design Vision"

In FF XIV some combination of the second and the third happened, but Square Enix didn't recognize it, or wishfully ignored it.