What went wrong with XNA/XBLIG?

[Thomas Steinke, one of the most successful developers on XBLIG (CastleMiner, Avatar Laser Wars), gives his perspective on the system's biggest problems.]

If you haven’t heard, the internet has been buzzing with the release of information indicating the death of the XNA platform and by proxy Xbox Live Indie Games.

It's official: XNA is dead

I know there has been tons of articles written like this in the past, but as the most successful  XBLIG developer and one that has been on the system since nearly the beginning, I would like to offer my inside perspective on XBLIG and more specifically the shortcomings it had, for the sake of posterity.

If you would like to know more about me it is probably best to read some of my other blogs about XBLIG:

Lessons from CastleMiner

Myths about Xbox Live

XBLIG was a great platform that got a bad rap but is also had a lot of serious problems. People have always been quick to chide XBLIG for being a cesspool of low quality fart apps, avatar games and Minecraft clones, but the problems ran much deeper than that. I am not going to sit here and say there weren’t low quality games on the system, but marketplaces like the ios app store have them as well and have been far more successful. So what was special about XBLIG?

Microsoft’s treated XBLIG as a second class citizen on the Xbox

I can sit here and tell you anecdotal information about insiders that I have talked to telling me about the friction that Microsoft had with an uncontrolled Indie system on the Xbox, but you don’t need to hear them to “get” that Microsoft never really embraced the idea of an open channel on their box. This relegated XBLIG to a corner of the marketplace which had a broad impact on the system as a whole. As time went on it seemed like XBLIG found itself in progressively more obscure places on the Xbox dashboard.

Around the same time, Apple was building one for the most successful digital marketplaces that ever existed on just that premise. To make an open market place on the Xbox really work like the AppStore, Microsoft would need to make EVERYTHING go through it. For example the next Call of Duty game would be released the exact same way your indie title would.

Deep inside, I sort of hoped that Microsoft was using XBLIG as an experiment with plans to eventually embrace this idea. You could see the obvious apprehension that Microsoft would have with this, and if I was talking about this in 2006 I would seem like a crazy person. However since then the Apple App store has illustrated that a free market actually works very well.

So why didn’t XBLIG turn into the next Apple Marketplace? The problem lies in that Microsoft needed to create a barrier between the “real” Xbox Live Marketplace and the indie one.

For people that may not be familiar with the game industry, getting a “real” game on the Xbox is a horrendous and monumentally expensive undertaking; one that isn’t really feasibly for an Indie developer. There are publishers involved, a lot of money, middle men and approval processes. So right away you see a problem. If you could just put game up on the Xbox for the $100 a year subscription fee, why would anyone ever NOT use that method to getting there game up there?

Microsoft solved this problem with pricing. The only available prices for games on XBLIG are $1, $3, $5. Right away by doing this they were prescribing a certain type of content on the system. The next state of the art $50 shooter just couldn’t be there. The next $20 XBLA game couldn’t even be there. There was only a certain type of game that could exist there. Immediately you then have a partition between expensive high quality games and cheap indie experiences.

It is worth saying at this point that although this was bad for the marketplace as a whole it was an amazing opportunity for indie developers. The problem with marketplaces like the AppStore is that you have to compete directly with the top tier developers/brands. Microsoft had successfully created a safe little “pond” where small indies could make tons of money on a real platform with 10’s of millions of users, without having to compete with the “big fish”. It was a gold mine. However, growing beyond these cheap experiences from a business standpoint becomes problematic.

To make the situation worse, Microsoft gave the most exposure to games that had the most unit sales, not gross sales, greatly disadvantaging higher priced titles. This drove the price of all the games on the system down to the lowest tier of $1 which became the de-facto price for an Indie game. It was a great disservice to the developers on the system and could have easily been avoided.

Getting a game on the system was too hard/frustrating

The peer review system just plain sucked. Most people may not know how games actually made it to the system. Microsoft, to avoid obvious legal problems, devised the following system: You would submit a game or update to a game on the system and it would go in a review list. At this point other developers on the system (yes, your direct competition) would review your game (yes, before it is released to the public or anyone has seen it). At this time the reviewer would either submit a pass or fail vote on the game.

If the game received 7 passes it would appear on the system, if it received 2 fails for any reason, it would be booted out of the system. Whether your game passed or failed, a 7 day wait period was imposed on the title preventing you from resubmitting or updating the game in this period. If you resubmitted the game without fixing the issue you risked being permanently banned from the entire system. 

After 30 days if your title did not clear the system (for example people weren’t interested in it enough to vote on it at all), you failed automatically and needed to restart the process. If you needed to release a Multilanguage title, in addition you need 2 reviews for each language you want to release in (this made releasing or even updating localized titles nearly impossible).

Aside from the obvious problems that arise from letting your competition see early releases of your titles and ultimately deciding if your game should make it to the system, Microsoft intentionally left the criteria for review vague with rules like “Your game cannot be frustrating or confusing”. With rules like that you could have fun imagining the creative ways that one could judge a title unfit to release. Each fail costing the developer another week plus the time it takes to accumulate the votes again, potentially holding titles in limbo for months.

It would seem that this would quickly degrade into the chaos of developers maliciously failing each other to prevent them from competing with them, however in practice, as if in some sort of odd social experiment, the opposite actually occurred. Everyone had a mutual goal of getting their title though this tedious process so it became common practice to do “kickback reviewing”, where one developer would “review”(pass) your game with the expectation that you would do the same.

It became so prevalent that it actually degraded the quality of the titles on the system as a whole. You see it wasn’t really in your interest ever to fail a game; it just meant that that developer would come back and invent some reason to fail your title. For that reason, fails even legitimate ones became fairly rare. As a reviewer myself (an honest one I felt) there were many times when I elected just not to vote on a game instead of failing it.

Surely Microsoft would step in to enforce order. Due to legal reasons, no one at Microsoft was allowed to view any content on the system until it was officially released to the public. Not even the discussion that occurred in the game’s review forum.

Their solution to this was the moderators or MVPs. These MVPs ran essentially unchecked by Microsoft and had ultimate power over the system. They single handily had the ability to make it so your game would or wouldn’t go out. To be fair, these people had a VERY hard job and were not compensated at all for it.

They were reigning in complete chaos and it always amazed me how devoted they were to the system. With that said, their “war-torn” demeanor at times became combative and rude. Some of them had very strong opinions that they were quick to bluntly express and they were the ultimate law on the system.

If there was a serious issue or debate about your game it could be deferred to Microsoft. However since Microsoft could not see your title directly or the discussion that transpired, it would become the moderator’s job to present your case to them and deliver the verdict. You could imagine if the moderator was not on your side this would never go well.

Regardless, Microsoft’s de-facto response was always “You guys decide as a community” which in reality meant, “The moderator makes the decision”. If the moderator gave the community the license to pass or fail a game that is what happened. As a reviewer, if you disagreed and didn’t follow suit it usually resulted in some sort of disciplinary action. Any meaningful discussion about how the system could be improved or be changed was almost always immediately squelched.

The review system limited the quality of games released

The peer review system was a complete failure. It didn’t really work and was a frustrating experience for developers. However what a lot of people don’t realize was that it ultimately had another huge detrimental effect on the marketplace. This was the “upper bar” for quality on the system.

As the largest developer on XBLIG I had quite a bit of resources available to me to make games. I had some of the absolute best talent in the Triple A industry working for me and millions of dollars at my disposal but I had one big problem. If I made a large investment in an XBLIG title, I always had to be ready for some 18 year old kid to stop me from releasing my game.

For that reason we had to limit the risk that we would take on a particular title effectively limiting the bounds of what we could do in an XBLIG title. In an ironic twist, the review system setup to enforce quality and ultimately limited the quality of games on the system by limiting the risk you could take on them.

The system had tons of problems

The system, meaning the marketplace itself, was always experiencing serious problems. There were all sorts of problems, from games not going up correctly, the review system breaking and purchases/updates not working. The most common and probably the worst problem were “list freezes”, where the critical lists that drove all sales in the system would stop updating some times for weeks at a time. Any game released in this time was completely doomed.

I understand that any system is going to have hiccups from time to time, but what made this bad was the way it was handled. The official response to any of these problems was basically “too bad…” Now immediately you would say, “no problem, just release it again.” There was a strict rule against this which would result in a ban, and the system made no exception or concession for this when it was obvious what was going on.

I personally had at least 3 games that were destroyed by glitches. To me, this was XBLIG’s greatest travesty. Imagine if you had worked two years on a game and spent $100k of your own money just to have your game swallowed up by a glitch. Ultimately though, what I don’t think that people realized is this again affected the overall quality of submissions on the system. For me when I considered projects, I had to always be prepared for my game getting swallowed up by a system glitch which greatly limited the overall scope and budget of the projects I took on.

The submission process was too cheap

It only cost $100 a year to submit 10 games, which was later increased to 25 games. Right now you are thinking “What?! Too cheap? Isn’t that good for indies?”  It would seem so but actually it isn’t. The biggest complaint about the system, especially from the moderators, was the appearance of cheap apps, vs. actual robust games.  The reason that these existed was that they had setup a system that made this work on an economic level.

By putting your game up in front of millions of people, especially with something quirky and a catchy box, you could easily get a few thousand dollars. My first game was a screen saver that made over $35,000. If you were savvy you realized that your were far better off making a “Craplication” every 2 weeks and getting $2000-$5000, then spending two years on a project and getting incrementally more.

If Microsoft really wanted this to stop they could have just made a submission cost something like $5000. There would have been far less titles on the system, but higher quality titles that would get a greater share of the revenue. 

The platform itself was very limiting

Although XNA Game studio is great for making games, it was very limiting. Everything needed to be coded in C# (using only the compact framework) so the use of any sort of existing middleware or engine was out of the question. You saw some people trying to capitalize on this by trying to sell XNA specific solutions, however it was obvious to me right away that the small developer community would make this kind of business impossible. As a result, these solutions tended to be subpar and poorly maintained. Many games ended up being produced on technologies that were long abandoned by their authors.

On top of this, the framework itself was severely limiting for “security” reasons. Games couldn’t even access the internet in any way at all, making things like a simple leader board in your game an epic feat of engineering. It became standard practice to implement leader boards with an elaborate peer to peer system. I sort of bought into the idea that all this sandboxing was necessary until I took a look at the iPhone that doesn’t have any of these restrictions and works just fine.

The ecommerce options were extremely limited

XBLIG had a neat try/buy system (although the forced implementation put on it by the moderators/community was sort of ridiculous), it was cool to be able to complete transactions so easily. However, it lacked a micro transaction system. Today many of the highest grossing mobile apps are based on a free to play, micro-transaction experiences. Customers would commonly ask us for DLC etc, which was impossible.

We were the first developer to actually implement a system where we included personalized content unlock codes in our games for other titles that we had. This allowed us to use another game as a DLC pack. Due solely to the popularity of our games this ended up being a very successful program for us. However a real micro-transaction experience would have worked better for everyone.

With all of this said, XBLIG was an amazing system while it lasted and XNA was an absolute pleasure to work with. It was amazing to me that Microsoft actually took the time to solve all the problems with getting XBLIG up on the Xbox at all and I couldn’t appreciate it more. It completely changed my life and the lives of the people closest to me. The days I worked making XBLIG games were by far the most rewarding days of my career as a game developer.

As for DigitalDNA Games

XBLIG was a great stepping stone for the company to get recognition and allow us to setup some really excellent deals. This will allow us to continue making games for our fans in the future. You haven’t heard the last from us.