Middleware Postmortem: Quazal Technologies

Online gaming has been the 'next big thing' for quite a while now. First it was LAN and modem gaming in Doom and Duke Nukem 3D. Then, systems like Dwango let gamers find each other more easily. Next came the big breakthrough of massively multiplayer gaming into the 'mainstream', and all the while online gaming was still 'the next big thing'.

Today, millions of gamers hook their home consoles and PCs up to the Internet, game and chat with each other (with full voice communication), play in massively multiplayer games with users in other countries and on other platforms, and enter into worldwide online competitions for real-world prizes. Therefore, more developers are finding ways to allow their users to interact with each other, in both a direct way (competition and cooperation) and in an indirect way (sharing results online and chatting about their game experiences). This is the area of middleware that Quazal specializes in.

In fact, Quazal started originally as Proksim Software, way back in 1998. Originally, we were three guys with an idea (I say 'we' here loosely, since I was still in university at the time, and hadn't really thought about entering the gaming industry). Back then, Sylvain Beaudry, Martin Lavoie and Carl Dionne had seen the outlook in the massively multiplayer market, and decided that creating a system to manage the massive amounts of network data required for one of these games would be a worthwhile business. Thus, work began on Eterna.

Fast forward to 2004. Now, Quazal produces three major products. The aforementioned Eterna is a core technology for networking in massively multiplayer games. Net-Z is a subset of the Eterna featureset, designed for smaller scale multiplayer games of 2-32 players. Both of these products are built on the idea of 'duplicated objects', where each networked game object is treated as a set of attributes with specified update policies in-game, and duplicated across to all the stations that need to know about them. The system is designed to abstract networking into a high-level function, though, as it's very different than what developers are used to, they can find it complex at first glance. We have worked hard to make the learning curve as smooth as possible, though.

However, networking isn't sexy. You can't send out a screenshot of a new networking technology, like you might with a new rendering effect. Initially, this lack of a 'sexy' demo hurt, but ultimately is became a non-issue as word of mouth spread, and we now have a decent list of well-known clients, including Eidos (25 To Life, Snowblind and Commandos: Strike Force), Ubisoft (Splinter Cell: Chaos Theory) and Sammy Studios (Darkwatch).

From 1998 to 2004, though, a lot of things happened, and a lot of decisions were made, both good and bad. Let's take a look at some of the successes and some of the mistakes that were made along that road, and how other middleware vendors, especially smaller ones, can learn from these decisions and hopefully make the right ones when faced with similar challenges.

______________________________________________________

What Went Right

Good Programming Techniques. Our R&D team here at Quazal is a heavy proponent of planning before coding. Generally, we work in teams of two for planning, and then separate out to accomplish their tasks. Sometimes we'll devote entire days to pure planning sessions for major features. With the development of Rendez-Vous, several weeks were spent planning how the overall product would be constructed, and ultimately how each service would work and interact with the other services. Once actual coding started, things progressed quite rapidly towards an alpha version, pretty much meeting all of the planned functionality.

We have an automated build system that we use, and employ CVS for build tracking and code check-in/check-out. Luckily, we don't have to worry much about any other assets (as mentioned before, networking isn't the most visual subject). Nightly builds are run on multiple platforms, with comprehensive internal automated tests being done to ensure that there are no memory leaks or other preventable errors in the products. Major feature upgrades are locked in the core build tree (Win32 version), and branches are then created for service packs, custom builds and console versions. This lets us implement custom features quickly for clients who ask for them, without disrupting other clients who don't need these features. When requested features are deemed worthwhile from a more general perspective, we can quickly incorporate them back into the 'trunk' of the build tree for the next major revision. Also, we keep extensive off-site backups of our codebase, which came in very handy when our offices were once broken into, resulting in the theft of our CVS server.

Finally, as middleware providers, we have to make sure our code is clean and well documented. Our R&D staff adds most documentation within the code, and then our tech writer can automatically strip the programmers' comments in order to more easily and accurately build both our paper User Guide as well as our searchable reference guide. This has proven a great benefit to teams evaluating our tech.

Shift The Focus. Initially, the focus was purely on creating a core technology suitable to help run a massively multiplayer game, with up to one hundred thousand players active in-game. MMO games were right on the cusp of huge mainstream success, following up from the success of Everquest, Ultima Online and Asheron's Call. After looking at the average development time for an MMO title, Quazal quickly realized that relying on teams who might not have anything to show, or have proper funding (and therefore not be able to afford your product) for well over a year, was most likely a mistake. In 1999, a talk with Strategy First showed that they were having troubles with networking in a 4-player multiplayer game. This finally convinced us that focusing on smaller scale, 2-32 player games, was a better choice, and would open a much broader market to us.

Thankfully, the Eterna technology was at its core not limited to the massively multiplayer genre. It was really just a hybrid topology, object-based networking engine. By removing certain features that were focused on the MMO development community, Quazal was quickly able to put together the companion product, Net-Z. At GDC 2000, Net-Z was unveiled and received a very solid response.

The multiplayer gaming community on PC was pretty much well established by then. However, the console online development scene was a fledgling market at best. Within a short while, Quazal had picked up its first few PC licenses. However, this was about the time that we hired a new CEO (see 'What Went Wrong'). Today, most of Quazal's business is standard multiplayer gaming, not MMO development.

Support, support, support and truth. Any middleware company has to provide good support, or they might as well not bother trying to sell their product. Middleware is supposed to save time and money, but if you provide good technology and the client has to spend a huge amount of 'man-hours' figuring out how it works, they probably would have been better off creating their own tech. Not only that, but if you aren't responsive to questions and issue reports, you can pretty much kiss any follow-up business goodbye. This isn't something that is solely mandated by management by any stretch, and relies on the passion of the engineers to work outside of regular hours to provide strong support to developers around the world.

Quazal has definitely focused a lot of our efforts on support. All of our clients, including evaluators who ultimately did not choose to use Quazal tech, have praised our support responses and our quick turnaround. Does this cost us in terms of time, and therefore money? You bet it does, but we decided that it was absolutely necessary to differentiate ourselves from our competition not only on a technology front, but also on the support front. Ultimately, this has resulted in us winning a multi-title contract with Eidos, and hopefully several similar deals with other publishers in the relatively near future.

Also, never forget that bending the truth will likely come back to bite you. If you can't do something, say you can't do it. When planning out features, be honest about the scheduling, especially to your clients. Over-promising will kill your staff. Under-promise constantly, and your clients will likely go elsewhere.

The bottom line, for any other middleware providers out there caught in the looming shadow of a big-time competitor, is that if you put a ton of effort into running a top-notch support program, you'll find that shadow receding very quickly.

Releasing the Personal Edition and the DevZone. Around GDC 2003, we decided to create a developer's area for our website (we call it the DevZone). This started out (in theory) as a simple place where developers could access our technology without having to be provided with links manually. It quickly became an area where anyone could download a version of our various technologies (at the time, just Net-Z and Eterna) for a relaxed evaluation. For non-commercial games, student projects and the like, developers could download the relevant distribution, grab a Personal Edition key, and start working. We provided forums for users to ask questions and put up articles, documentation, FAQs and more to help them learn more about our technology.

As of the time of this writing, we have over 1500 registered users. The vast majority of these users will never become paying clients (I can't even imagine managing hundreds of licensees) but they gain access to a solid networking core, full documentation and the limited support of our staff and any other users who have themselves used our tech. We've had a number of students post responses to other users' questions, and we've had several of our professional developers chime in with responses of their own.

It didn't cost us much to develop, it's a bit of work to manage sometimes, but ultimately it is a huge boon and provides a lot of 'street cred' to allow anyone and everyone (yes, even our competition) to download and evaluate our tech, even allowing professional developers a way to openly take a look at our tech without having to contact us first. Worth. Every. Penny.

Give'em What They Want (to a degree). Originally, we had our Net-Z product, providing a duplicated-object core for in-game networking. It was solid tech, it worked, and we got a lot of interest in it. VERY quickly, however, developers and publishers voiced their concerns that we didn't provide a 'total solution', meaning we couldn't handle the 'out of game' element to online gaming (this being the lobby and matchmaking). This caused us to lose a number of promising licensees, since we didn't have what they wanted. In their mind, and this is totally understandable, it was far easier to develop their own in-game networking tech and license our competition's lobby engine, rather than licensing two disparate techs or licensing our in-game tech and developing their own lobby system (which is more difficult than many might think, and very few developers or publishers want to deal with operating it 24/7!).

So, we started work on Rendez-Vous, our lobby and matchmaking technology. This took about a year of solid work to go from design to a version 1.0, and it has become the product that developers and publishers now show the most interest in. The lesson? When your clients say they need something, be it a major feature or in this case a wholly new technology, LISTEN. It'll be a lot of work, but in the end will be totally worthwhile. With Rendez-Vous, we now had a product that didn't compete with developers' internal technologies, which made things much easier.

______________________________________________________

What Went Wrong

Postponing Rendez-Vous. The counterpoint to the last 'What Went Right' point is that we took a long time to react and start on Rendez-Vous. We had received a lot of hints from developers even before GDC 2003 (actually, as early as 2000) that our potential clients, especially on the publishing side, really wanted a lobby solution to go alongside Net-Z. However, we decided to focus on Net-Z development instead and only started Rendez-Vous around July of 2003. We viewed Quazal as a product-development team, and not a service-development team. That's about a 5-6 month delay between really understanding the need for Rendez-Vous, and committing to actually developing it.

Originally, we planned to focus on in-game networking, and work with the existing lobby providers on PC and PS2 to let them handle the out-of-game solution. After several months (through GDC 2003), we realized that this relationship simply wasn't going to work from a technical and an ideological point of view. Also, several potential clients decided that they didn't want to license two technologies for online play, so they chose to develop their own in-game solution and license the lobby system elsewhere. This made for a pretty compelling argument to start Rendez-Vous.

We've always strived to be open to new feature requests, but since then, we've tried to be a lot more responsive to things like this, and the message to other middleware developers is to always have an open mind to new features, and to always be able to criticize your current plans. Now, no company has unlimited financial or manpower resources, so you can't just immediately start work on any newly proposed feature or product, but we really should have reacted to the need for Rendez-Vous much sooner than we did. In the end, this likely cost us a good half-dozen licenses, which is a lot of revenue for a small company like ourselves.

Curse of the Grey Hair. Back in 2000, the Quazal management (then known as Proksim) saw that they had the potential to really explode onto the scene if they played their cards right. After getting a significant round of financing from the investors, they scouted around and hired a CEO to ensure the success of the company. This was someone who had experience handling millions of dollars, and someone who had success in the past. Turns out, it was someone who pretty much broke the company.

With a healthy bank account and a small company, the new CEO immediately decided that gaming wasn't where the money was, and started looking at other revenue streams. Business plans were made for servicing the oil and gas industry, the banking industry, the airline industry and more. Even after a very promising GDC 2000, focus was shifted away from gaming and toward business applications. Every few months, the target market would be changed as new potential opportunities, or even hints of opportunities, arose, development would shift to accommodate the new market, and very little R&D work got done. No real sales happened in this time period either. Add to that a new Texas office and a reasonably massive increase in spending, and that healthy bank account rapidly dwindled to a stack of loose change.

The CEO was ousted, Proksim attempted to shift their focus back to gaming, but the damage was done. In late 2001, Proksim changed their name to Quazal, as the view in the development community was that we had abandoned the gaming market (and after the wild goose chase for business licenses, I can't blame them). Eterna was launched a month later, after receiving a small amount of investor funding. However, the debts from the prior CEO's era were too great, and Quazal went into stasis for most of 2002.

In late 2002, Quazal was resurrected with a new round of financing. The lesson here? Just because someone has had success in the past doesn't mean they have any clue how to manage your company, and the results can be disastrous, and when you're looking for a CEO, don't turn over too much power to them too soon - bad things can happen! Since then, however, we've been running quite smoothly and have had much more success with far more limited resources. An interesting side note: We launched around the same time as Havok. If we hadn't made this detour, could be have been as large as Havok is today?

Finding That First Customer. Demos are great. Solid technology is important. Good documentation can't be overlooked. However, all of that is pretty much worthless unless you find a team willing to take a risk with your product. Getting that first reference customer for our Net-Z technology took far longer than it really had to.

At GDC 2003 (three years after our first appearance, but the first tradeshow after our resurgence), we started up relations with dozens of teams, including some large publishers and developers. However, as promising as many of these were, none wanted to be the first to use our tech. At E3, we started talking with Eidos, and the relationship ebbed and flowed on through the summer. Finally, in October 2003, we closed our first deal with Eidos (and several followed quickly after that). The most asked question at all the trade shows and conferences was 'Who is using your technology?' We didn't have a good answer for that, and ended up spending a lot of time showing off demos and explaining how the technology actually worked.

At GDC 2004, we ran a much smaller booth, but we were able to talk about several teams that we were working with, including Eidos, Ubisoft and Sammy Studios. Seems that once you mention your client list, people don't care HOW your technology works, since they know that it DOES work. Now, the obvious next question is 'what games have SHIPPED with your technologies?' We'll have a good response to that in early 2005.

Did it have to be this way? Probably not. Looking back, we probably could have made some relatively major concessions and discounts with one of our other evaluations to lower their risk, but we didn't, and we suffered for it. Again, live and learn.

Blurry Focus and Funding Strings. With a reasonably small staff, and even with products that share a lot of their core technologies, focus is key to successfully delivering what your clients need, on time. After Quazal started back up in 2002, we still spread our focus between Eterna and Net-Z, and then started up Rendez-Vous development. This has been a huge stress on our staff, and impacts what we can deliver to our clients.

Looking at our client base now, pretty much everything has been based on Net-Z and Rendez-Vous. Eterna still hasn't really started paying off. Looking back, we probably should have put more focus purely on Net-Z, while letting Eterna slide a little. This would have let us get Net-Z into a 'completed' form sooner (nothing's every really complete in middleware), and would have let us devote more resources to Rendez-Vous, possibly letting us have it ready sooner too.

However, even if you realize that a change of path is beneficial, sometimes you can't act immediately due to your funding model. We love our VCs. Without them, we wouldn't exist. However, when you gain funding with a certain business plan, shifting away from that business plan is something that has to be cleared through your board of directors. Any other middleware company should realize this too, and be prepared. Also, part of this is simply a hesitation to 'let go' of the technology you started with, even though the market is screaming at you that your focus shouldn't be there. It took us a little too long to really realize this, though.

Don't Flinch. After the failure outlined in the second point, and the 'wasted' time outlined in the fourth point, Quazal became a little gun-shy and hesitant to invest. Basically, while we can now much more accurately figure out where our resources and money should go, we still hesitate to actually commit these resources, to the detriment of our marketing and our development. We now have decided to hire a few more staff members (though finding the right people is really quite difficult) to help speed up development on Net-Z and Rendez-Vous, and to help with support requests, which we know will start becoming an issue as we gain more clients.

Earlier on in our history, we ran an advertising campaign in trade magazines, with little tangible response. That cost a good amount of money, and didn't give any return, leading us to be more skeptical of the need for traditional marketing. Also, we licensed a certain console for development that it turns out nobody was interested in for online gaming. This made us more hesitant to jump on new platforms.

In 2005 we'll be starting up a more robust commitment to marketing and promotion. While our 'free' marketing has been worthwhile, and word of mouth has helped immensely, we understand that more traditional marketing is needed to really take us to the next level of awareness. We hope that by spending a little more on these things, it'll turn around and actually give us some better returns. It's a hard attitude to change, though.

A Look to the Future

What's next for Quazal? Well, everyone in the industry knows that there is a major transition looming on the horizon. With the next consoles from Microsoft, Sony and Nintendo in the wings, and with the PSP and DS introducing online play to the mainstream handheld scene, there are some great opportunities for middleware, as well as some great challenges. Quazal does indeed plan on supporting several of these new platforms. Also, we'll be devoting a lot of our time to Rendez-Vous development. While the technology in Rendez-Vous works really well, there are a large number of things we could enhance to make it easier for developers to adopt it as their own, and we're working on these. Last, but not least, it's time to really take on the Goliath in the online middleware market. With Net-Z and Rendez-Vous, we like to think that we have a pretty powerful slingshot in our pocket (with a laser sight and everything!).

______________________________________________________