The Making of Sonic Beat: Solving rhythm gaming for mobile devices

I’ve played rhythm games all my life. I’m a collector and enthusiast. I have metal dance pads, bongos, ALL of the guitars, an electronic drum kit with all the trimmings. I stream to Twitch. I host Rock Band parties at local businesses. I have dumped thousands of dollars over the years into equipment and DLC.

Knowing this about me, imagine my excitement these last 12 months. A new way to play Guitar Hero? Sign me up! My Rock Band library on new consoles? Absolutely! A growing music game scene on Youtube and Twitch? How many webcams can I plug into this computer!

But GH Live has a free-to-play model with no buy-out. Half of my Rock Band songs won’t import. I can’t play my beloved Pro Drums (until October or so). The legacy instrument adapter is laggy. I’ve been as heartbroken as I have been giddy.

Then around Christmas my boss Jason asked me something wonderful: “How would you like to make a music game?” Stars shined in my eyes and I immediately started listing the ideas that had been accumulating during our three years making apps together.

Some of them we’d talked about before, but this time it was serious. We had always stopped short for one reason or another. Certainly, it didn’t seem as if anyone else had really figured it out on mobile yet. Rhythm gaming on phones hasn’t been as fun as it should be. Fingers covering up notes. Poor performance and issue of timing accuracy. A million StepMania clones. Limited to one platform or another. Total lack of content. A bias toward one handed-ness or the other. Unskippable, insanely wordy Japanese-style “tutorials.” Forcing you to hold the phone in a way that covers up the speaker. The list goes on and on.

We could do better.

What finally convinced us was a little 3DS game called Theatrhythm. I brought it in and had Jason play it. Before long, he was as excited as I was. Not being a music gamer himself, he had been rightfully shooting down many of my ideas for fear of function or scope, as well as the fact I had not really offered any solutions to the failures of the apps that came before. But Theatrythm held the keys to not only a great rhythm game (which it is andwhy hasn’t anyone played it, what’s wrong with you), but a great mobile phonerhythm game. The “tap, hold, swipe” gameplay was uniquely mobile, and it solved so many problems. And it could be even better on a capacitive multi-touch phone.

“So just make Theatrhythm, then” I can already hear you thinking. You see, Square Enix already did that. If you’ve played the iOS version of the game, you understand it suffers from nearly all of the problems listed above. It even chugs at about 15fps on our office iPhone 6+. And if you’ve played the Android version... oh, there isn’t one. There’s only this:

“FantasyBeat” by I&C SOFT is a well-produced game and probably the closest thing to Theatrhythm available on Android, but it’s mired by problems common to mobile rhythm games and bogged down with heavy-handed microtransactions and what must be the longest tutorial in gaming history.

But we had other ideas for the formula. Several excited whiteboard drawings later and we had a design that appeared to work! We quickly boiled down the aspects of what would eventually become Sonic Beat at a level of scope our team of two could finish in six months. We knew our game:

So let’s look at these requirements and explain the design process along the way.

The very first playable prototype of Sonic Beat [LEFT] compared to the final product [RIGHT].

GAMEPLAY and ACCESSIBILITY

The transition from the resistive stylus controls of Theatrhythm to the capacitive thumb controls of a phone game brought new challenges to consider - challenges I feel Square Enix did not handle well for their own mobile port. For instance, upon building our first playable prototype we became certain the game was meant to be played in portrait orientation. Not only does this accommodate the differences in hardware, it makes for an ambidextrous interface.

And with that first big change, more design decisions seemed to fall into place. Unlike the 3DS, we had access to multi-touch. With two thumbs tapping simultaneously, the mechanical and skill ceilings of the original format could be raised significantly. One tester gleefully likened it to learning how to alt-strum in Rock Band all over again.

Another thing Theatrhythm did differently from other DDR-style games was to not require the player to tap specific places of the screen. Unfortunately this can be a little confusing in practice, so Jason simplified the concept down to a single note lane. This felt so much easier to read and play, and the new design stuck. This decision moved the gameplay away from one that required precise aiming of inputs to one that could focus just on the fun part - rhythm. This is much more enjoyable than trying to force what works for a four-panel dance game into the confines of a five inch touchscreen.

Yet early testing showed this wasn’t enough. Players still desired to tap the middle of the note target, covering it with their fingers. We struggled with how to fix this, as we believe strongly that nobody (read: NOBODY) reads tutorial prompts. Our eventual solution was to add two pseudo tap zones that animated when the screen was touched, like pinball flippers. Behind the scenes, the player could still tap anywhere, but the added panels ensured that even if the player ignored our short tutorial and starts pressing the tap zone, they are drawn to the sides thanks to the constant visual cue.

“Beat MP3" by CREAPPTIVE was a huge inspiration for Sonic Beat, both visually and as a prime example of how to make a mobile music game fun - and of what design pitfalls to avoid.

ACCURACY

One of the biggest problems with every music game I played on mobile, even the really big ones, was accuracy. Accuracy is really four separate problems from a UX perspective. The obvious one is “lol are the notes on time.” The answer to that is more difficult to answer than you’d think.

Going into this, we had no idea how much of a challenge it would be. There are tons of factors, any one of which can go wrong and collapse the whole effort. Many mobile rhythm games rely on programmatic beat detection, but this can be finicky. Perhaps the best example I can find is Beat MP3, which is genuinely fun to play, but you can tell concessions were made regarding timing windows to accommodate for the imprecise nature of beat detection algorithms.

We chose to go with authored song charting so as to eliminate poor beat detection as a factor, and this decision ended up being a cornerstone of the project. The excellent “StepMania” dance game simulator is well known for its accuracy, and its charting format is easy to understand and tested against years of users attempting to break it in hilarious ways. While Sonic Beat does not itself use the StepMania simfile format, our choice to author songs using StepMania’s editor and convert the output for use in our game solves the problems of charting accuracy and music timing offsets as inputs for the game engine.

Then there’s gamefeel (further reading: everything by Vlambeer). A surprising number of music games do not offer calibration tools, and this comes from the mistaken belief that your target gaming hardware will always produce perfect results. It also assumes that the ‘technically perfect beat timing’ visible in a waveform would be the correct time to trigger each note. Theatrhythm makes this mistake, even on the 3DS. Without calibration tools, wearing the wrong headphones can destroy the experience (40ms delay on some noise cancelling headsets, and as much as 200ms over bluetooth). Even then, your player may subjectively prefer the beat ahead or behind. Perhaps s/he’s tired, or wired on energy drinks, or a percussionist forever cursed to be more correct than your best efforts. Offering players choice is almost always preferable to a one-size-fits-all solution.

The third factor is legibility, or how well can you read the rhythms presented on screen. We had several examples to choose from when designing our note lane. One way to help players stay in time is through animation or color shift. The osu! series employs this strategy, as does Dance Dance Revolution using its default arrow set, but it’s not something I personally endorse. DDR also has the option to color notes based on their quantization, which is closer to reading sheet music and would have worked well for Sonic Beat. We instead went with Rock Band’s approach, adding measure lines to the note track. It’s very subtle, and it allows us to keep a standard set of note gems.

Finally, repetition. I strongly feel it’s important to be able to play the same track over and over again, improving as you go, learning new rhythms by rote practice. Lack of practice-friendly gameplay has been the sole reason I’ve uninstalled several other music games over the years. And truthfully, I understand why many of these games have randomized notes - it’s easier to program and allows for virtually infinite content and levels of play. However, it also replaces the skill ceiling with a mechanical ceiling. Anyone who’s played (the excellent) Piano Tiles games has probably run into this problem.

That’s not to say random is all bad. By ditching random charting and audio beat detection, we are surely limiting ourselves to the content we can create for the game by hand. We do have ideas on how to implement procedurally generated song charts that we might explore in a future version of Sonic Beat, likely as a separate mode. It’ll be like making a whole new app,though.

All of this - authored charting, timing accuracy, legibility and repetition - I feel this is the magic formula for a music game worthy of beginners and experts alike. A large portion of development time was devoted to making all these things come together, and I think it’s well worth the effort. But it does come at another kind of expense.

CONTENT and VIABILITY

We had seemingly backed ourselves into a corner. Now we needed authored content on a shoestring budget. We already had an excellent (and fast!) authoring solution using the StepMania editor, but we couldn’t afford to license more than a couple of tracks for the game. And music licensing itself seemed to be reserved for companies like Harmonix, or those friends in the music industry and weeks-long content pipelines.

At first, we played around with the content model employed by BeatX, a particularly good StepMania clone. BeatX boasts “more than 100,000 songs,” which is technically true. BeatX directly supports StepMania simfiles, and they can easily be played via a phone’s external storage. We already had the ability to import simfiles into Sonic Beat, so we expanded the option and used it for testing. We liked how the 4-panel dance charts played so well in the Sonic Beat format that we turned it into a feature that still exists in the final product.

But we knew this wasn’t enough. People would enjoy reliving their DDR Supernova tracks in a new way, but ultimately people would really want to play marathons of popular music like, um, Justin Bieber. It was Jason’s idea, and his dedicated programmatic wizardry, to do something much more bold.

He engineered a way to take music already on a player’s phone, optimize it for use in the game, and apply my authored song charts to the result. The first time I saw this work in real time I was floored! The flood gates had opened. We could make gameplay content for any songs we wished, so long as the player already had the songs!

Bingo! No more legal grey areas surrounding packaged StepMania simfiles. No more worry over licensing budget, which we had already depleted after two included tracks. Even better: using songs available across iTunes and Amazon MP3 normalized the tracks we could support. By making the game content free and encouraging players to legally download supported tracks via their favorite music stores, every song downloaded for the game directly supports the artists. And unlike discrete DLC models for games like Guitar Hero and Rock Band, when someone buys a track for Sonic Beat, they own that track.

This is all done by one person, the above selection completed in about one work week. Just imagine how much this can expand once other people start joining in.

Combining this “must own the song” requirement with our StepMania importer offered yet another benefit: user generated content. Legal user generated content. Over the lifespan of the game, we plan to support players who go the extra mile and chart their own favorite tunes into the game. The StepMania content pipeline allows anyone to easily test their work in the game without expensive dev kits or complicated mods. With only a few lines of extra metadata added to a simfile, users can share their works with each other minus any actual music files, all without breaking copyright law. And after all that, we have the option to feature community charts in the game without jumping through licensing red tape. Literally everybody involved wins.

But the best part of our new content pipeline was that it narrowed our monetization options. Other games had tried a multitude of gross strategies, none of which we liked. The otherwise excellent Cytus forced you to wait 30 seconds for every song until you paid them money. Theatrythm’s iOS port came with two songs and charged $3 each for song packs that had to be purchased blindly. Beat MP3 threw advertisements at you every chance it got.

Instead, we chose to employ a sort of “unlimited demo” approach that would allow players to enjoy every part of the game without paying while still ensuring we got something for our efforts. Instead of a hard play timer, we implemented a free-to-play style “energy and rest” solution where a player has a limited number of recharging play tokens. We added minimal advertising at the end of each game. But MOST IMPORTANTLY, we added an inexpensive buy-out option.

This was a very heavy point of discussion, as it goes against the existing monetization strategies employed by the likes of Candy Crush or Clash of Clans. Those games, while fun, are designed to infinitely siphon cash out of players in small doses over time. That’s effective, but it sucks. At least in our opinion, every time we’ve ponied up cash to briefly extend a hard paywall it’s been done begrudgingly. We ultimately decided a few bucks would always be more than we’d make on any free-to-play player on average, and reasonable expectations of player counts would mean the project might at least break even. We can also adjust the buyout up or down as we get a feel for how well it can succeed. Of course, we have no idea if players will take to this strategy. Perhaps I can answer that in a future post-mortem.

ALL COMING TOGETHER

I distinctly remember the first time it all just... worked. We were huddled in front of the same computer, reading each line of code carefully looking for an error in our math. Jason jumped at the keyboard and entered the likely missing piece, built the project, and played the output. And smiled. He handed it to me. I powered through an Expert chart as if it were as familiar to me as all my years of Guitar Hero. I couldn’t believe it. I tried another. Everything was in time. Every touch was handled correctly. It was perfect. And I loved it.

We exchanged laughter and a high five. The weight of failure was gone. We had solved the problems we had set out to solve. And now there was so much more to do.

The game was missing something to keep people playing. We looked at all of our user feedback and compared it to our feature wishlist. We spent more time playing other music games serving as our inspiration. Within the time we had left, we decided to implement a more robust scoring system. We added star progress and score multipliers to the game. Facebook was integrated as a way to handle leaderboards cross-platform, as well as give an ID for players to store their progress to the cloud.

And with that, we finally had a complete game. Ship it! Right?

YOUR iPHONE SUCKS

Jason will write a more technical breakdown of some of the platform challenges we faced, but I want to take a moment and point out the things that impacted me the most.

This was our first project using Cocos 2d-x - indeed our first project using any game engine - and I have to say there was a lot to love about ditching native app development. On the iOS side, it solved a lot of common problems regarding how to support multiple screen sizes and how to handle art assets and animation. Traditionally, we have been a studio that builds Android first and ports the project to iOS ourselves, but Cocos is pretty much designed for iOS with Android more an afterthought. This meant for the first time, we were developing primarily on iPhones and, also, that making it shine on Android was going to take a little extra dedication.

The Apple ecosystem is very much a walled garden, and if you have to stray from the game engine in any way you have to use the toys Apple deems fit for you. It adds an extra layer to everything from what audio format to use to wondering whether you can support sideloading songs into the game.

It would have been nice to have support for OGG files, for instance. A majority of StepMania simfiles employ Vorbis Audio because it’s free. Apple won’t stand for that, though. It’s also worth noting how poorly MP3 is supported on Apple devices, with system level bugs in audio muxing causing a slew of problems for people who use MP3 for their music library. As it stands, our site has to officially recommend converting your music into the AAC format (m4a) - a non-solution to a problem that should not be.

Sonic Beat was in danger of being cancelled over concerns of how to access files on iOS. One such issue was how to allow players to add StepMania songs to the game. We ended up using apple’s lesser-known file sharing support via iTunes, which is a phenomenal and easy solution when all is said and done. This of course threw the Apple app testing department for a loop and initially got us rejected from the App Store. Once we explained, in pictures, that this really wasn’t such a sinful use of the feature we were allowed this privilege.

That overbearance extends to how Apple allows you to beta test your games as well. Let’s count the steps. Upload a special version of the game for testing. Wait a day for them to glance at it and empirically wave you onward. Gather e-mails and names from likely testers (who want your game NOW). Submit the information line by line to Apple. Approved testers have to download a separate app that basically informs Apple that a sanctioned test is going to happen on a very lucky consumer iPhone, just this once. Finally, the tester may download the special version of your app, through the other app, which you submitted days ago and have since improved upon. Oh, and Apple makes sure to delete your game from the device after 30 days.

Got all that? Good.

We had only one other squabble with the Powers That Be. We originally declared a permission to let Sonic Beat control music in the background. This had a single and very practical function: if the player turned off the screen in the middle of gameplay, we needed to pause the music. Without this permission, iOS does a lot of things it considers in your best interest that you cannot detect, intercept, nor avoid.

The first problem is the music will fade out, pausing at the incorrect time and desyncing it from the note track when you turn the screen on. If that wasn’t bad enough, iOS then unpauses the game music when you turn the screen on and before you unlock the device. Controlling this ourselves provided a clean solution on par with the Android build.

But this only confused the Apple app testing department. See, the permission we asked for was originally designed for apps that played a music library, such as iTunes itself. Apple rejected our app because, as they correctly put it, Sonic Beat isn’t a music player. When we explained why this permission was crucial to the project, some other app tester ignoredour explanation entirely and rejected us again, informing us they couldn’t determine how our app used the permission. When we appealed this, the appeal was immediately closed without discussion.

So we did they only thing we could do. We removed the permission and called for the music to stop when we can first detect our app has lost focus. This still desyncs the music from the game, but at least it won’t unpause when the screen comes back on. Because Apple’s obstinance, this aspect of the game is decidedly inferior to its Android counterpart. And, we can only imagine was due to us poking the bear, our app review time went from 24 hours to 7 days causing us to miss our launch date on the platform.

YOUR ANDROID SUCKS

Let’s not pull punches here; there are reasons so many developers don’t bother developing for both iOS and Android. That doesn’t excuse laziness, but there are reasons. Audio in a music game would have been one of those reasons.

The very thing that makes Android great is also a problem for many game engines. Even if you’re running “on the metal,” you still have to interface with various systems through Java. When those systems weren’t designed for real-time interaction, you’re at the mercy of Android itself to get you what you need in a timely manner.

Audio on Android is a crap shoot. Depending on which hardware your phone has, and depending on which version of the operating system is installed, the audio latency on Android is anywhere between - and I’m not kidding - 35ms and 1,000ms.

Now before you run away in terror, know that this actually doesn’t impact gameplay. Because there is no meaningful video or tap input delay, any audio delay can be completely countered and the game will run as if the audio delay doesn’t exist (much unlike your expensive, terrible-for-gaming HDTV).

The problem comes when trying to explain that to your players. I do not doubt we will have to put more UX work into audio compensation after launch, and until then I am sure less technically-minded players will uninstall the game in frustration. The best we can do from our end is choose a default value that hits the most widely used class of audio hardware.

The geniuses at superpowered are tackling this issue, however. They make a free custom audio engine for Android, and it really does do as they claim - near perfect and cross-platform. We almost employed it too, but as yet the software is a little unfinished and lacks key codec support for our needs. Still, I recommend them highly to any other developers reading this article.

Moving on from audio, Cocos is a problem in and of itself. As seen on message boards around the internet, Cocos is poorly optimized for Android. At least at the time I write this, of course. We had to throw out particle systems, physics, and 3D features of the game engine after seeing it chug on Android. Luckily, we were able to find and disable features of Cocos used for backward compatibility with incredibly old phones and now the game runs swimmingly even on lower-powered hardware. Still, we had to raise the minimum OS version requirement on Android significantly to avoid devices that, for inexplicable reasons, ran poorly even when tasked with builds of a blank default project.

Hopefully, the developers at Cocos are aware of this setback and are working to improve performance out of the box.

BUT, UH, HOW DO WE LAUNCH THIS THING

...we ask to ourselves this final week before launch. Being completely honest here, we’re really bad at the whole marketing thing. Even knowingwe’re bad at the whole marketing thing and every week discussing how tonot be bad at it, we find ourselves at our launch date with virtually no idea how to get this thing in the hands of players.

We certainly tried some new things this time. For the first time ever, we put serious effort into a beta test. We recruited players from all over the internet, focusing on existing user bases of other games. While I don’t believe our product directly competes with the likes of Rock Band or StepMania, I want to mention how grateful I am that their forum moderators let me use their off-topic boards to gather interest and crucial feedback. What we gleaned from those testers has been invaluable, and it directly influenced the direction of the game.

But while that did get a few players and much needed critique, it could not be called an effective marketing strategy. Neither has reaching out to media outlets yielded any results. We had read, and hoped, that offering early looks at the game and possibly exclusive stories on the project several weeks before launch would have gotten us an article somewhere, or an app review on YouTube. I can say now none of those is ever a sure thing.

The only “for sure” advice we can gather is “throw a bunch of money at it.” Which we don’t have. Or failing that, we could always go the Evony route.

Without the interest of local or internet publications, we instead spent some resources on in-person testing. We were so proud. We had a giant printed sign and everything. We teamed up with a local deli to host a testing session and give away free cinnamon rolls to anyone who showed up. You’d be surprised how spectacularly that failed. Instead of gaining a bunch of input on the game, we heard opinions from patrons that what we were doing had to be too good to be true. A free delicious treat seemed more like a trap. Ironically, we would have done better to give out $1 cinnamon roll vouchers. We... learned a lot, in retrospection.

But what we do have? Universally positive feedback from gamers and non-gamers alike. A game that anyone can pick up after a quick app store search. Something streamable to Twitch and Youtube, and the first product where we really dedicated ourselves to the social features in the game. That’s gotta count for something. Reports of success along similar lines such as that of Rocket League are encouraging, even if they’re the first to admit that, yeah, a lot of it is probably sheer luck.

But perhaps not all of it. We’ve also been especially fortunate to have caught the attention of a guy who’s really, like really into marketing and has been course correcting us with daily e-mails the size of short stories. It merits its own article, and I’m certainly not qualified yet to discuss the intricacies of Facebook advertising or how to determine your target demographic when you didn’t know who would be interested in your game from the start. The point is, it’s helping.

SO WISH US LUCK!

Regardless of how this ends up, I know one thing absolutely: This has been a dream come true. I’ve wanted to make a music game ever since Mario Paint’s music composer captivated me back on the SNES. To scratch off one of my bucket list line items at the age of 29 is an immense privilege.

In these first few days after launch, it’s wonderful watching people enjoy the game and converse with us on social media. And watching the song requests pile onto our server, and the several precious dots blip in and out of existence on Google Analytics. And to be among the right people at the right time to make a game I always wanted to play for myself. EVEN NOW as I finish charting that Bieber song into the game... It makes all the effort, all the late nights and short lunches, totally, totally worth it.

Sonic Beat released worldwide on June 1st 2016 on iTunes, Google Play, Amazon, and SlideMe.
http://www.sonic-beat.com