Knights, Pawns, Bombs and Stars: Evaluating your competition to achieve actionable feedback

There has been a shift in how many successful games come to market. We used to predict success based on gamerankings and “on-time & on-budget” development.

The new world of “games as services,” persistent games, “always on” games, F2P and all those games that create a readily accessible community of players have been enjoying the successes of marrying consumer research with production. The accessible data makes knowing what consumers want so much easier. So, why care if a game is developed  on time and on budget if nobody ends up wanting it?

Traditional game development (like disconnected boxed product) suffered from a lack of consumer research, and when consumer research was done – it almost never yielded “actionable feedback.”

You might be in marketing, or you might be in production, but if you make games and give your consumers any importance whatsoever, you probably are on a constant quest for “actionable feedback.”

“Actionable feedback” is – literally - feedback you can act upon. It’s like when you are looking for a spot to park your car and your passenger tells you, “One there!” before you passed the spot, and not after. Essentially, it’s the useful kind feedback, the one that, given in time, makes your life or your product better.

In the digital space, as of 2013, rivers of data stream from consumers.  The industry is enjoying great successes from experimenting on consumers. If there are analytics, we can focus on our own data and see what works. This is truly the great achievement of the videogame industry since its move to more connected experiences. We now know how gamers react to each one of our game designs and marketing decisions, almost in real time.

The emphasis is, however, on one’s own games, one’s own consumers, one’s own data.

Since games are now often a service, we find we invest time in keeping them alive and fighting for an ongoing amount of consumer’s time (and wallets). This is where we can take a lesson from the over-cited Sun Tzu, and figure out that knowing our enemies is good tactics. To help in this, we can use our community to develop a Knights, Pawns, Bombs and Stars matrix.

Getting your Data

Most developers and publishers constantly monitor their own data. This is great! But fewer constantly monitor their competition.

There is however a quite simple way of constantly assessing your game’s performance versus competitors: the “Relative Performance matrix.”

In its simplest form, the steps a team has to go through when creating this kind of tool are:

1)    Determine what are the game’s key features:

2)    Create a first survey (with multiple but limited choices):

3)    Examine the raw data obtained from the first survey and rank the game features by consumer segment.

4)    Create a second survey:

5)    Analyze the data obtained from survey #2 and create a “relative performance matrix” (see below). This exercise should be repeated for every segment of consumers the team is interested in studying and for every competing game that shall be evaluated.

Traditionally, this was done with paper surveys. Remember going to a restaurant and finding a scorecard? To incentivize consumers to fill out the form, small gifts were promised. Like having their name entered in a raffle. I need not say that today, delivering virtual prizes in exchange for valuable feedback is a lot simpler (and can go a long way in creating goodwill).

Developers should use their community. They probably already know what features consumers find important in their game, and which features they prefer. It’s just an extra step to see how they shape up against the competition.

Developers of persistent games (F2P games, MMOs, MUDs) also have a great advantage in creating and implementing this kind of surveys. The first resource are beta testers. They will be more than willing to fill out online surveys on the game they are “testing.” The second resource is, once the game has launched, its online community.

Feature Importance vs. Relative Importance

For simplicity let’s consider we are making an RPG called “MyGame.” The developer of MyGame sends out his questionnaire and has the top features scored for importance. Then he asks the consumers to rate how MyGame does against his arch-enemy game, the dreaded/admired “Competitor.”

Remember:

Once the data is collected, the results for the most important features (>4) should be plotted in a table as shown below.

Relative Performance Matrix Chart

From this data we can plot out a matrix that shows the importance of each feature against the relative performance of MyGame on that feature vs. Competitor.

Relative Performance Map

The score on the Y axis is the rating your segment gives to a game characteristic/feature. The value on the X axis is the relative performance. It is obtained by subtracting the Competitor’s performance score on a given feature from MyGame’s score.

By examining point A, “franchise/brand appeal,” one can understand that:

1)    it qualifies as “low importance,” with a score of 4.7. This yields the consumers point of view on the feature/characteristic.

2)    MyGame, even if it possesses a good brand name due to its hypothetical success, suffers the brand strength that Competitor possesses (maybe its a long lasting franchise). Thus, MyGame’s relative performance is a -0.2 (= 6 – 6.2).

On the other hand, all the rest of the dimensions compared see MyGame as a winner. Most importantly, MyGame scores high ratings on very important RPG features such as gameplay, level of customization and number of classes/races available. Do not be misled, the performance is only “relative.” If we plot MyGame against yet another game, imagine a “Competitor2,” we could find out that Competitor2 largely outperforms MyGame both on customization and number of classes/races available. On the other hand, MyGame’s strong brand name could result in the “franchise appeal” dot in the lower right quadrant.


How to turn this feedback into “Actionable Feedback”

Once the developer knows his game’s strengths and weaknesses and how consumers rate them, he will be able to define all items as one of the following:

1)    Stars: High relative performance, High importance

2)    Pawns: High relative performance, Low importance

3)    Time bombs: Low relative performance, High importance

4)    Knights: Low relative performance, Low importance 

Knights, Pawns, Bombs and Stars Matrix

1)    High Priority – Fix the Bombs: they leak consumers.

2)    High Priority - Maintain your Stars: you don’t want to lose your competitive edge.

3)    Secondary Priority – Evangelize your Pawns: you want players to understand why this feature is so great.

4)    Low Priority – Fix then Evangelize: don’t make the mistake of marketing a feature before fixing it.

Prioritizing actionable items can help establish and maintain a market in a world of constant change. Social gaming, online communities, lowered barriers to entry and analytics have evolved the way games are made and increased potential competition. It just takes an extra step to use all the tools that such connected experiences offer in order to evaluate, not only the game you are making, but also the competitive landscape.