Five things we learned from the Windjammers Beta

Hi, we’re Christian Cortez and Hugo Lefevre, producers at DotEmu!

From our Paris office we’ve spent the last 10 years bringing beloved retro games to modern platforms. Our portfolio is stocked with the classics you love – games like Double Dragon Trilogy, Another World, and Titan Quest. Last year we released Pang Adventures, our first console game, followed by Ys Origin on PS4 and PS Vita and Wonder Boy: The Dragon’s Trap on PS4, Xbox One, Nintendo Switch, and PC. Our console experience is still growing, but these recent successes have inspired us to do more. In fact, we hope to reach the same level of success with our next game: Windjammers.

For those of you who’ve never played it, Windjammers (aka Flying Power Disc in Japan) is an incredible, underrated arcade title originally developed by Data East and published by SNK in 1994. The game was released on NEO GEO MVS/AES/CD, and though it developed a devoted following, we still don’t think it achieved the acclaim it deserved. That’s why we decided to revamp it for the world of modern gaming!

Windjammers is Pong meets Street Fighter. In Windjammers you’re a Flying Disc Master and your goal is to throw your disc past your opponent. Simple. The fun factor is increased by the fact that there are different athletes with different skills, different custom super throws, etc. But though the game is easy to grasp (just a stick and two buttons), its speed, complexity, and strategic layer give it a deeper, more competitive vibe, something we hope everyone will (re)discover with our relaunch of this masterpiece.

Before the release, we decided to host a closed Beta from June 8 - June 12. We added this Beta to our production calendar a few months earlier in order to test our network environment and validate the choices we made for the game: menus, online modes, “how to play”, etc. This was the first time we Beta-tested one of our games, and it went pretty well - a few thousand players participated. Of course, the Beta was also a huge learning experience. This is precisely why we’d like to share our Beta adventures, so that fellow devs can gain insight into organizing their own closed Beta on console.

So here we go: five things we learned during the Windjammers Beta!

Betas aren’t that complicated

For small studios, a console-based Beta might seem daunting. We are a small team ourselves, and a Beta was beyond our scope when the Windjammers project began. Yes, it takes time to prepare the build. Sure, working with the first-party and preparing for submission and release is time-consuming. And no doubt you’ll need to allocate funds for QA. Staging a closed Beta costs time and money, but not as much as you might think. And here’s why:

In sum, with a little anticipation and a reasonable investment of time and money, running a Beta is a worthwhile enterprise. Not only will it vastly improve your game’s quality, it will get people talking about your game. And we all know how valuable word of mouth is.

Track tons of metrics

When considering your Beta’s scope, make sure to allot enough time to clearly define and list all of the metrics and statistics you want to track/gather from your application and servers (pure data), and your beta testers (user feedback). This step might seem obvious, but to be honest we ran through it too quickly and ended up a little light on “pure data” collection. If we could do it over again, we would include more tracking and analytics.

In the end, the juncture to glean valuable, countable data is one of the most important and interesting upsides that a Beta offers, so consider it thoroughly. It shouldn’t take too much time to implement if the metrics are clearly defined early in the development.

Below are some metrics that we consider mandatory for an online 1-vs.-1 Beta for a game like Windjammers:

This is just a partial list. There are plenty of other categories worth investigating, depending on your game, your community, your goals, etc. Spend time early on figuring out what you want to know, what you’re looking for, and what could help you in the future. Don’t hold back: the more, the better.

Even though we didn’t collect a ton of data during the Windjammers Beta, we were still able to gather vital feedback via a 20-question Google form sent to Beta subscribers. We were blown away by the response: we received tons of completed forms, most with detailed feedback. We definitely recommend doing this.

Make sure your “Beta story” has a clear beginning and end

Handling the lifecycle of your Beta is very important. Generally speaking, a Beta is free and limited in time, so you want your players to have access to it only on precise days and hours. For this reason, it is very important to define how and when your Beta will begin and end.

We sent out beta keys 24 hours before the actual Beta start, without realizing it was actually possible to redeem some of them and play the game before the defined launch time. Our unpreparedness resulted in some confusion on social media, since some testers were able to play before others. The lesson learned is to pre-test some beta keys on your own a few days before the actual Beta launch. Also, don’t forget to set up a “Beta ending” message when you close your servers, so players know why they can’t access the game and don’t see it as a bug.

A Beta is the best way to test your ladder and matchmaking system

Very early in production, we decided to implement an online ranking mode. Nothing fancy: you play a match against another player, the winner earns points and the loser drops some. The challenge was to ensure that the matches pitted players with approximately the same ranking. Even though the system we chose is pretty common - one hidden MMR to match players in the Quick Match mode and one visible MMR to match players in the Ranked Match mode - we still needed to test it live to confirm whether the ladder would be effective. It’s important to note that this was our first experience designing a Matchmaking system.  

Thankfully, the Beta was reassuring. People were happy with the matchmaking system and had little trouble finding similarly-ranked opponents: 70% of the players of the Beta have declared that their opponents had the same level they had, and more than 70% of our players have found the Matchmaking system to be Good or Very Good.

Matchmaking is essential for “versus” games like Windjammers. It comprises a huge part of the online experience, and doing a Beta is a great way to test its quality. Thanks to the Beta, we confirmed that our system was solid, and after a few adjustments, ready for global release.

Don’t fear the feedback

Although our closed Beta’s main objective was to test the strength of our netcode and matchmaking, we also prioritized player feedback regarding our core decisions for the game.

Even though Windjammers is an adaptation of a 90s title, we added new features and options that were not in the original version (new menus, online play, new game modes, a “how to play” section, the “retro” option, etc.). The Beta helped us determine whether we were on-point with these changes. NEO GEO games have a huge, hardcore fan-base, and our biggest challenge was to meet their expectations as well as those of newcomers. Thanks to the Beta community, we received numerous suggestions for adding new features or altering the ones we’d already implemented. And while it’s impossible to address every suggestion, the Beta lets you implement the best ideas right away and save the others for updates.

A well-managed Beta is essentially a giant playtest - one that taught us not to be afraid of sharing our game with the public and to welcome their feedback with kindness. Any game, regardless of genre, can benefit from other players’ intuition and ideas.

Conclusion

It might seem strange that we organized a Beta for a 23-year-old game whose features, gameplay, and design were already “approved” by the public and integral to the original game’s success. Our job (and the Beta’s purpose), aside from trying to create the most faithful emulation of the original game, was to add the features that weren’t available in 1994: online multiplayer modes.

Windjammers represents our first online console game, and our first game with advanced matchmaking and a ranking system using external servers and a specific infrastructure. Developing these online features was much more challenging than we expected, especially since our capacity for internal testing was limited. But taking the time to facilitate a closed Beta helped us immensely in terms of understanding the best practices for online multiplayer development, not to mention providing us with plenty of measurable results, feature-related suggestions, and key information about how our adaptation could be improved. Furthermore this Beta was an excellent way for us to not only get people talking about our game, but to create a dialogue with our future community and that’s priceless.

After weighing the relatively short setup time versus the valuable results obtained, we highly recommend a Beta to anyone who is or will be working on a console game with an online multiplayer component.