Indiepocalypse? More like INDIESCHMOCALYPSE!

Adam Coster is 1/3 of Butterscotch Shenanigans, and is a card-carrying PhD scientist. That doesn't mean you should believe anything he says, but it does increase the probability that he can do logic and statistics properly. (This article is reposted from the Butterscotch Shenanigans blog.)

It seems every time I open Twitter or look at gamedev-related subreddits someone is linking to yet another article about this scary "Indiepocalypse."   If you haven't heard of it, the idea is (roughly) as follows: The barrier to entry for making and distributing video games has become super low. Consequently, the market is becoming supersaturated with indie games, and it is therefore increasingly unlikely for indie developers to find financial success in the industry even if they make a great game. The idea is that the pie (total games sold) isn't getting bigger, therefore the shares are getting smaller, and soon the market will be untenable for indie studios.  

And that argument sounds legit. I mean, just look at Figure 1. On Steam only 10-20 indie games were released per month at the start of 2013, a mere 2.5 years later there were up to 200 indie games launched per month.

As with any impassioned (or web-based) argument, most discussions about the upcoming demise of indie-game civilization are founded on loosely-defined terms and undefended assumptions. And so mostly I've been ignoring my inevitable indie-doom, because the best move whenever you hear words ending in "-pocalypse" (or "-gate") is to walk away with your fingers in your ears.

"SO WHY ARE YOU WRITING ABOUT IT NOW!?" you might be asking.

Well, we're closing in on the all-platform launch of our pride and joy, Crashlands. While planning launch strategies we came across a different piece of suspect industry wisdom. That is, that indie studios should not launch their games during the pre-holiday months, because that is when the huge developers release their games. Since a small indie game can't compete with a major AAA title, our launch would fail. Or so the logic goes.

That argument is well-reasoned, but its assumptions (that indie games even do compete with AAA titles) have no data defending them. And I couldn't find such data elsewhere on the interwebs. We really want to be able to launch soon, so I decided to test that claim. In the process, I realized that I also had the data to test the Indiepocalypse Hypothesis. So I did.

The result: There is no seasonal conflict when releasing indie games, and there is no Indiepocalypse.

The data

Even the best arguments are hard to defend if they don't have data backing their assumptions. To test the various hypotheses below I used data from Steamspy. For those unfamiliar, Steamspy is a polling system that continuously grabs information from random user profiles to estimate ownership for games on Steam. Along with that comes information about the games themselves, such as their current price, their user rating, when they were launched, etc.

I downloaded the data from Steamspy on 9/29/2015, and included all games tagged "Indie." I also downloaded all games listed under the major publishers EA, Ubisoft, Warner, Activision, and 2K. Since most people would not consider games published by those companies as "indie," any that were in both sets of data were removed.

Because free-to-play and super-cheap games undoubtedly have different markets than premium games, I excluded all games with prices below $3.

Finally, I only looked at games released between January 2013 and July 2015, as these are the years in which Steam Greenlight was active and in which large numbers of indie games began to enter the Steam marketplace. If we are coming upon a rapidly-expanding and soon-to-burst indie bubble, that 2.5 year period should show it.

A word of caution: statistics is a messy business. Saying that "there is no evidence" for something is NOT THE SAME as there being "evidence against" that thing. In addition, just because some piece of data is consistent with your argument that does NOT mean that it is evidence for it! In most cases, there are many plausible explanations for any piece of data you see.

The Indiepocalypse Hypothesis

The loose hypothesis of the Indiepocalypse is this: As more indie developers enter the market, fewer indie developers will succeed. The first part of this hypothesis needs no further definition, but the latter part absolutely does (and the fact that its meaning isn't crystal clear is why I've been ignoring articles on this topic in the first place).

By "fewer indie developers," we could mean a smaller fraction of the overall total, or a smaller absolute number. Defining it as a smaller fraction is pointless because as huge numbers of developers enter the market, duh, of course a smaller fraction will succeed! This is necessarily the consequence of either (1) that player demand for games is lower than the rate of new game releases, or (2) that players do not want most new games (for any number of reasons: lower quality, less novelty, poor marketing, etc). The data I show later shows that demand for indie games is not increasing over time on Steam, so condition 1 is satisfied!

So the interesting case is when "fewer indie developers" means "a smaller absolute number." So our hypothesis becomes clarified as: As more indie developers enter the market, a smaller absolute number of them will succeed.

Great, so we've defined the first part of the hypothesis. But what does it mean to "succeed?" There are all kinds of success: critical acclaim, meeting a niche market, getting emotional fulfillment, etc. Here, though, I only mean it in the pragmatic sense: level of success is proportional to how many games you have sold. Success is relative in this sense, as any particular studio+game combination will require movement of a different number of that game in order to stay afloat.

Yeah, actual revenue and, even better, return-on-investment are even better metrics of success, but that data is not readily available. What is available, via Steamspy, is the number of people who own each game ("owners"), which I use here as a proxy for "success" - games with a higher number of owners are relatively more successful than those with fewer owners. So the specific, testable hypothesis becomes the following:

Indiepocalypse Hypothesis: As more indie games enter the market, each game will have fewer owners.

This makes logical sense at first. If the pie is getting sliced into more and more pieces, doesn't each piece have to get smaller? NOPE! That would only be true if each piece is equal. Spoiler alert: they aren't.

Testing the Indiepocalypse Hypothesis

If the market is saturated with games, the average success per game will be far lower than when the market isn't saturated. But this is just a boring fact of bad statistics. Yeah, sure, if you average all purchasing across a continuously larger number of games, that average goes down. But the average is the wrong metric. The relevant question is: does everyone do worse because there are more games?
 

Player purchases are not distributed equally across games. It's more of an exponential decay (pretty close to log-normal as the above plot shows). So, as more and more games enter the fray, is it the case that fewer total games succeed?

Well, Figure 2 seems to confirm the Indiepocalypse: the average ownership per indie game in 2014 is only 23% what it was in 2013, and it's reduced by 3/4 again in 2015! There are many more games, and it seems that everyone is losing. Game over. Pack up your laptops and find another job.

But hold up. There were way more indie games released in 2014 than 2013, and there were more games in the first half of 2015 than in all of 2014. So we should put all these on the same plot to compare (Figure 3).
 

Wow. That looks pretty frightening. Figure 3 shows the distribution of success moving significantly further to the left every year (and note the log scale, 1 unit left means 1/10th the success). But there is something interesting about the right tail of the 2014 distribution (Figure 3, blue) compared to the full 2013 distribution (red): it looks an awful lot like the 2014 distribution is a mixture of the 2013 distribution plus something else. This is the statistics concept of a "mixture model," where the distribution we see is actually composed of multiple distributions "mixed together."

That means this data is consistent with (at least) two hypotheses: (1) The Indiepocalypse is now, and everyone is going to lose even harder as time goes on; or (2) there are two kinds of indie games, successful and unsuccessful, and these games do not compete with each other. To find out if the second option makes sense, I re-plotted the Figure 3 data with only those games that received Steam user scores of at least 70% favorable (Figure 4).

Well, now, that looks a bit less terrifying. Figure 4 shows that "good" games in 2014 performed just as well as in 2013. It does seem to show that good games in 2015 are struggling; there appear to be fewer successful games (lower peak) with overall less success (peak shifted to the left). But this assumes that the distributions will not change over time. So, I wondered, is it likely that, given time, the 2015 distribution will shift right and get taller?

As for the question of getting taller, what needs to happen is that roughly 2x more good games come out. But guess what! Figure 4 only has 2015 data for the first half of the year, so we need to see if it is a reasonable guess that the second half will provide (roughly) as many good games. Plotting the release rates of only "good" games over time results in the right-hand plot of Figure 5. Note the y axis: roughly ten good indie games come out a month. The variation is high, but this is not surprising since the numbers are so small. From a statistical standpoint, then, the rate of "good" indie game releases has been constant over the timespan tested! And so, we can reasonably guess that the 2015 peak in Figure 4 will end up (roughly) twice as high as it is now.
 

But will the same curve shift to the right over time? About a year from now I can just test that idea directly. Unfortunately, for now we have to make a reasonable guess. A super-important point here is that success (measured by number of owners) can only increase over time as new players buy it. Meaning that, if the curve is going to shift, it is going to shift right, towards the 2013 and 2014 curves of Figure 4. Recently-released games have undoubtedly not yet maximized their potential success, and half of this year's "good" games haven't even been released yet.

If I plot the data over time (Figure 6), instead of throwing it all together as in Figure 4, it becomes even more obvious that "good' games (right plot) do not show the plummeting trend that indie games do overall (left plot). (I discuss this figure a bit more, below.) Still though, there is a downward trend for good games; only time will tell if this is because they haven't saturated their potential sales or if there is a downward sales trend for indie games. In any case, the effect is not nearly as dramatic as is the increased rate of new games, or the plummeting rate of success of those games.

(Note that, if you define the Indiepocalypse to mean that the fraction of successful games will plummet then, not only are you totally right, but the above plots show that this has already happened!)

In short, I find no evidence for the Indiepocalypse Hypothesis on Steam.
 

Does this "good games" thing make you nervous?

It totally should, so let's discuss it!

First, the caveat: games that aren't in the "good" category are not necessarily bad. Most of them are unrated, meaning they have so few ratings that the average is statistically meaningless.

When you saw Figure 5, with the dramatic rise in indie game releases (left plot) compared to the static rate of good game releases (right plot), you probably wondered what the hell was going on there. Surely, as more games enter the market there must be more good ones, right? What are the possible explanations for the stagnant release rate of well-rated indie games?

There are many possible reasons. Below are two, along with their consequences:

This could be due to marketing (e.g. no matter how many good games there are, there is a Steam bottleneck that allows only a handful to be successful), meaning that players only are aware of and/or rate a small number of newly released games. In the mobile markets it is well established that being on the front page of an app store, or having a huge advertising budget, are the only two ways to success. Indies typically only have access to the former, and the front page shows a very limited number of games. Built-in Steam discoverability may similarly limit indie success to this handful of games per month. In that case, the Indiepocalypse already happened years ago.

Another possibility is that the barrier to entry to the Steam marketplace wasn't a barrier to games that could be successful. As the barriers continued to fall, potentially-successful games came in just as they used to, but then so did more and more games that did not have potential for success (whether due to issues of quality, originality, marketing, or small-niche interest). In that case the Indiepocalypse, as an idea, simply doesn't make sense: of course you can't be successful if you aren't making a product people want! There is no evidence that success of "good" games is being impaced by the influx of all the other games.

The Indiepocalypse Punchline

If by "Indiepocalypse" you mean "a larger and larger fraction of new indie devs will fail," then of course you're right. But that has always been true, so it doesn't seem to warrant a freakout. In that case, things are exactly the same as they were: the market wants the same amount of indie games as it always did, and you can be successful by making games that players want.

Probability of success is not defined by how many other devs there are. It's defined by how good your game is and how well you market it. Most games fail. But most indie games are some combination of unpolished, unoriginal, poorly priced, and poorly marketed. So that isn't surprising in the least. As the barriers to making games go down, the barriers to successfully selling games stay right where they are.
 

As for the original question: how does release month affect indie success?

So the Indiepocalypse is (probably) nonsense, but what about the original claim I came here for, that indie games should not be released in the few months before December? I decided to instead address the broader Release Month Hypothesis:

Release Month Hypothesis: The success of an indie game is correlated with its release month.
 

Looking at the first plot in Figure 6, there is no clear seasonal pattern. There is however, a visible trend: the per-game success of indie games (as measured by median ownership) is decreasing dramatically over time while the overall success (Figure 6, blue squares) decreases much more slowly. Looks like the Indiepocalypse all over again!

There are two obvious interpretations for this decreasing trend. As discussed already, one is that it may take several more years for a game to saturate its level of potential ownership, so that games released earlier are closer to that saturation point than those released later. Indeed, there is a slight downward slope in overall ownership (Figure 6, blue squares).

However, considering that the rate of new good indie game releases has remained fairly constant (Figure 4, middle), as has overall ownership of indie games (Figure 6, blue squares), the downward trend may, additionally or instead, be due to something else: the increasing number of games that no one wants!

Indeed, if we only look at the well-rated games, that downward trend becomes way smaller (Figure 6, right). Well-rated indie games were seeing similar success rates over the entire studied period. While a statistically significant correlation between release month and ownership remains, the size of that effect is small and carries no obvious seasonal trend.

In short, there is no evidence for a seasonal relationship between release timing and eventual success for indie games.

Overall Conclusion

[Edit: Shortly before I published this, Sergey Galyonkin of Steamspy published a related article that came to a similar conclusion, though he did so through a different line of argument. Give it a read!]

Everything above suggests that good games aren't incompetition with the deluge of NEW ones, so indie devs need not fear competition from the deluge of new games coming in. In addition, if it really is the case that AAA titles release in a seasonal pattern, then good indie games aren't even in competition with those titles (they effectively exist in separate markets!). Therefore, as has always been the case, the key to indie success is to make a fantastic, polished, original game that a lot of people would want, and then market the hell out of it.

The Indiepocalypse (probably) isn't real.

And you should launch your indie game whenever you damn well please!

So, you disagree?

I'd be shocked if you didn't! The fact is that most scientific results are wrong, and there is no reason that I would be the exception. Perhaps my logic has a major flaw, perhaps there is an error in my code, or perhaps the data doesn't mean what I think it does. But, hey, you're about as likely to be wrong as I am, and there are many more of you than there are of me, and I have no time to argue with the horde (I've got games to make!). I have no stake in being right and I'll continue to make games regardless!

Aruging is fun though, so if you're going to do it (with others...) I encourage you to carefully define the words you use and to back up your assertions with data whenever you can. You can download the data and the code I used to generate the graphs in this article. Use that as a starting point for your own analyses (or to test mine).