Developing Online Console Games

All the major consoles - Playstation 2, Xbox and Gamecube - now support online gaming. PC developers have been writing online titles for years, but this is relatively uncharted territory for console developers. Until recently, only a handful of multiplayer online games had been created for consoles, and even fewer had been successful. The target audience for consoles has different expectations from the PC market. Console game designers are faced with limited input devices and console manufacturer requirements that demand a different set of design decisions. Console game developers are faced with adding network programming expertise to their list of skills. And now console producers have to schedule new online features and carefully consider network skills in hiring decisions.

This article examines the new world of multiplayer possibilities with current-generation consoles, along with the limitations online console game development presents. It contrasts the different approaches the console makers have taken, and details the design and development considerations involved in creating online console games.

Online Strategies: The Big Picture

Each console manufacturer is taking a different approach to enabling online play. Microsoft is building an integrated service that emphasizes a uniform experience across titles, while Sony and Nintendo are investing less on back-end services and allowing more flexibility across games. Each scheme has its benefits and drawbacks. Microsoft requires that customers have a broadband connection, whereas Sony and Nintendo are also supporting players with dial-up connections. Xbox game designers have fatter pipes for game traffic, at the cost of a smaller potential market. PS2 and Gamecube developers have a larger potential market, but a wider range of bandwidths to support. The next section examines each company's approach in detail.

PS2 Strategy

Sony is the undisputed leader in terms of console market share. Sony launched its first online game in Japan in May 2002 and in North America in August 2002. Sony encourages PS2 game developers to support both dial-up and broadband users, although some games have chosen to support broadband only. For instance, SOCOM: U.S. Navy SEALs is a broadband-only game. PS2 online players must purchase a separate network adapter ($40), but there is no sign-up fee. Players can use their pre-existing dial-up or broadband connection. Some games support additional peripherals, such as a keyboard for online chat, a voice headset, or hard drive. Sony encourages its developers to provide multiplayer online gaming for free as an added feature, although developers can charge a subscription fee for games such as persistent-world games. For instance, the PS2 version of EverQuest will charge a monthly fee.

From a development standpoint, Sony provides an online API via the SCE-RT library. This library is free to licensed PS2 developers. Sony is building out facilities for hosting game servers, but they also encourage developers to build and host their own services or use third-party middleware and services. The advantage of this approach is that the game developer has maximum flexibility. They can build or buy, and they can manage their own customer base. A downside of this flexibility is the inconsistency of online capabilities and UI from title to title.

Sony is allowing considerable latitude for online developers. For example, developers can pick and choose how to implement multiplayer services like matchmaking. The possibility exists of having cross-platform compatibility with Gamecube or PC games. Buyers of online PS2 games could conceivably find a large existing community of PC players the very first time they log on. One potential disadvantage of this open system is the lack of a global security infrastructure. As online PC game developers have discovered, security is critical to preventing cheaters from ruining the game experience. With many different network library options available, there is a greater possibility that any particular online title may have unsatisfactory security built in.

Sony has placed few requirements on online games. For instance, voice is not a mandatory feature but rather left up to the developers to decide if it's appropriate for a game. In general, the Sony strategy is to maximize the potential audience for online games and give game developers broad flexibility in choosing what features their games should support and how those features should be implemented. However, Sony is reportedly developing a more integrated service for the European market called the Network Gaming Service (NGS), with indications that it will support single player IDs and other global features for titles that use the SCE-RT library. Whether this indicates Sony is moving toward a more consolidated service philosophy, at least in Europe, remains to be seen.

Gamecube Strategy

Nintendo is taking a cautious course, with the view that online gaming is not yet a viable market. Nintendo launched its first online title with Phantasy Star Online Episode I & II for Gamecube in October 2002. Players must purchase a separate dial-up or broadband adapter ($35) to play online. Like Sony, Nintendo is not charging a sign-up or subscription fee. Online services must be built by the game developer or accessed via third-party middleware.

Nintendo has been characteristically tight-lipped about future online plans for Gamecube. At the time of this writing, Nintendo does not appear to be enforcing any types of policy decisions about voice communication, keyboard chat, or global identities. There are also no indications that Nintendo is committing large resources to back-end services or other online infrastructure.

Xbox Strategy

Microsoft's online approach for Xbox is bold and hence, risky. Xbox itself was designed with online play in mind; all consoles include a built-in Ethernet port and hard disk. Microsoft launched its online service, Xbox Live, in North America in November 2002, and it has announced plans to launch in additional territories over the course of 2003.

To use Xbox Live, players must have broadband, typically via a cable or DSL modem using any ISP. Microsoft chose not to support dial-up connections, limiting their market significantly. In addition, unlike Sony and apparently Nintendo, Microsoft is charging players a fee for the service. For $50 players get a one-year subscription, a voice headset, and two games (MotoGP and Whacked in North America). Games may charge additional fees if they wish, although there are no current games doing so.

Microsoft has built a suite of online services and online game APIs, some of which are optional and some of which are required. From a development standpoint, this is both a benefit and a limitation. The benefit is that standard services like matchmaking are automatically available, tested, and free to use. Developers don't need to roll their own, pay for middleware, or pay to host their own matchmaking servers. The disadvantage is that game developers may find themselves required to use services that they normally wouldn't use or that aren't flexible enough for their needs. Developers also need to budget additional time for coding and testing required services. Currently, Microsoft provides services for peer-to-peer matchmaking, buddy lists (including online presence), game content delivery, voice chat, billing, and persistent storage of player statistics. Developers may write their own game servers if they wish, a necessity for massively multiplayer games.

Unlike Sony's (and presumably Nintendo's) relatively "open" approach, Xbox Live is considered a "closed" service. Xbox Live games cannot communicate with other consoles or with PC games, nor can they access web sites or the Internet at large. Custom game servers must be hosted in secure data centers approved by Microsoft. A closed system is a disadvantage from the standpoint of the community-building issue. It's hard to build community within such a limited environment. The benefit of these restrictions is that games are heavily resistant to cheating.

Microsoft's online service also allows players to have a global identity across all Xbox Live games. When players sign up for the service, they choose a "Gamertag" that becomes their name in every Live game. Gamertags are in turn used to build buddy lists (Friends lists, in Xbox Live parlance), which are also consistent across games. In addition, all Xbox Live games must support voice communication. Microsoft is banking on voice becoming a key differentiator for Xbox Live.

Microsoft's strategy is to provide a consistent experience for players across all Xbox Live games. Microsoft is investing heavily in its online service, and it appears committed to the long-term success of online play. Microsoft is providing a wide range of built-in technologies and services to online game developers, but in turn requires developers to make an engineering commitment to many global Live features. Obviously, Microsoft is also making a calculated bet that the broadband market will grow substantially in the coming years.

Online Console Game Design Issues

It's no secret that the typical console player is different from a typical PC game player. How does this apply to game design issues for multiplayer console games? The first difference is the couch versus back-room mentality. When you ask PC game players how they most enjoy playing games, their answer usually involves something about sitting at the computer - by themselves - and beating the snot out of players they don't know. When you ask console game players the same question, their answer will typically include something about sitting on the couch - with their friends alongside - and beating the snot out of each other. Console players tend to play with their friends, and they tend to play in a more social atmosphere.

The second difference is that the average console player is less technically knowledgeable and less forgiving than the average PC online gamer. Console game players don't care about round-trip ping time, they don't run traceroutes, they don't know an IP address from a P.O. address, and they certainly don't want to configure routers. If multiplayer gaming doesn't work for them, they will blame the game - not their modem, not the ISP, and not the Internet. This means that multiplayer gaming must appear to be an extension of single player gaming. Techniques for doing this effectively include disguising latency, avoiding unfamiliar terms like ping time, and not putting players into sessions that don't have good bandwidth characteristics.

Consoles are designed for the living room. They output to television screens, not high-resolution monitors. They typically take input from controllers, not keyboards or mice. Multiplayer console games must be designed for the living room as well. Suggestions for online console game designers include:

Writing a game with voice communication presents additional design challenges. First of all, it probably doesn't make sense, both from a bandwidth and discernability standpoint, to allow every player to talk with every other player during the game. That means that the game needs to somehow limit players' use of voice. The most effective games limit voice in ways that make sense to players in the context of the game. Perhaps players can communicate only with their teammates, or only with people near them in the game world. Or maybe the game uses phones, radio channels, or other well-understood means of filtering voice communication.

As with online PC games, online console games must carefully budget bandwidth and understand how to effectively tolerate latency. Even with broadband-only games, broadband is not a panacea. All the lessons PC network game developers have learned about disguising latency, limiting network traffic, and handling dropped packets still apply to broadband.

Topologies

Multiplayer game topology is a key design decision that can have a huge impact not only on gameplay but also on engineering time and long-term costs. There are three general models to consider: peer-to-peer, client-server where one console acts as the server, and client-server when the server is external (Figure 1). These models are applicable to PS2, Gamecube, and Xbox.

In a peer-to-peer game, each console communicates with every other console in the game. External servers are only peripherally involved, perhaps during the matchmaking phase or for downloading new content. The advantage of a peer-to-peer game is that no game servers need to be designed, coded, or maintained. Peers communicate directly without server involvement, so latency is minimized. The disadvantage is that the number of peers is limited due to bandwidth constraints, particularly with dial-up connections.

In a client-server game where one console is the server, the console could be either a dedicated server or an active participant in the game. Like the peer-to-peer model, no external servers have to be maintained. Unlike the peer-to-peer model, bandwidth becomes less of an issue, since each console communicates only with the server console. However, choosing the server console to maximize bandwidth and minimize latency becomes important, as does handling host migration when the server console shuts down or leaves the game.

A client-server game with a custom external server typically provides the ideal bandwidth characteristics, since the custom server can reside in a high-bandwidth data center. However, this topology introduces the most expense, both in terms of server development and testing, as well as long-term maintenance, server management, and data center bandwidth fees. Experienced server developers who also understand gaming issues are rare.

Another possibility is to mix and match topologies. In a massively multiplayer game, for example, it may make sense to use the client-server model for gameplay, but use a peer-to-peer model for voice communication. The important thing is for developers not to confine their thinking to any one model. Evaluate all the possibilities carefully.

Development Issues

All the console makers provide socket-level interfaces for network programming. On PS2, developers can choose from a variety of network stacks, including Sony Inet, Sony libeenet, SN Systems' NDK, and Access AVE-TCP (Japan). Each library has its own unique performance characteristics and features. For instance, the NDK stack includes tools that simulate various network rates and connection types, while Inet and libeenet provide access to ISP setup data that may be used in all titles across the platform.

On Xbox, the network stack is a combination of Winsock 1.1 and additional security-related functions. The Xbox network stack automatically handles packet encryption and authentication. On PS2, developers can use SCE-RT for traffic encryption, use a middleware solution, or provide their own security mechanisms.

On PS2, developers have many choices when it comes to online service libraries. The SCE-RT library, developed by Sony and RTime (a networking company Sony acquired) provides game server and lobby functionality, including account management, matchmaking, voice chat, buddy lists, clans, and ladders. This library is free to PS2 developers. GameSpy, one of several popular network middleware vendors, provides libraries for peer-to-peer matchmaking, statistics, security, and voice chat. GameSpy provides similar libraries and services for Gamecube. GameSpy's history with PC network solutions makes it a good choice for cross-platform development on PCs, PS2, and Gamecube. Pricing is title-dependent and is negotiated with GameSpy. On Xbox, the Xbox Software Development Kit (XDK) provides all the APIs and libraries required for accessing services like matchmaking, friends, statistics, content delivery, and voice chat. The XDK also includes tools used for Internet simulation.

Xbox provides technology developed by Microsoft Research for measuring connection bandwidth and latency. Game developers can use these functions for detecting the quality of service between various consoles and servers. On PS2, each network stack varies in its ability to provide ICMP-level functionality. For instance, SN Systems' NDK does not provide a ping interface, although the stack will respond to ICMP pings.

Each console vendor is taking a different approach to dealing with network address translators (NATs). A NAT is a device (typically a router or gateway) that allows multiple network devices to share an IP address. NATs are prevalent in home environments where an Internet connection is shared among PCs and game consoles. Some NATs also act as firewalls, blocking suspicious packets. Such NATs can be difficult for games to negotiate, particularly peer-to-peer games. The only viable option requires a mediator informing both peers about their port mappings. On PS2, games can use Sony's SCE-RT library or middleware libraries and services like GameSpy. On Xbox, NATs are handled automatically using a combination of the network stack and the Live service.

On Xbox, the Xbox Dashboard handles all network configuration issues. If an Xbox game can't establish a connection to Xbox Live, the game simply allows the player to run the Dashboard. The Dashboard saves network settings on the Xbox hard disk. Xbox games don't need to provide custom network management. PS2 players configure the Sony PS2 network adapter for connection to their ISP using the Network Startup Disc that comes with the adapter, which then saves these settings to the memory card for use by all games. Developers of PS2 games may also choose to embed a standard network-configuration UI library provided by Sony, in case players haven't yet set up the network configuration with the Network Startup Disc.

Because of their architecture and memory constraints, consoles have other limitations in comparison to PCs. Even though console network stacks require a comparatively small amount of memory, every byte is precious on a console. For instance, on PS2 the IOP (I/O Processor) memory can be a limiting factor, since it's used to load drivers for USB, keyboards, the network adapter, game pads, and memory cards. On Xbox, memory is not generally an issue, since the unified 64MB can be utilized by all devices. However, the automatic Xbox packet encryption and NAT traversal algorithms require developers to be aware of underlying performance issues they're not used to if they've done networking on other platforms.

On PS2, the first-generation network stacks required multithreaded implementations to achieve asynchronous network I/O. These first-generation stacks also contained many operations that would block due to data transfers between the Emotion Engine CPU (EE) and the IOP. The latest Sony network library provides a complete BSD socket API on the EE. This library eliminates previous blocking issues, and also abstracts away the underlying multithreaded I/O implementation.

On Xbox, the networking libraries are designed to work primarily in a single-threaded environment, although multithreading is also supported. The Xbox Live libraries use a task-pumping architecture. Each online task (searching for game sessions, for example) is "pumped" once per game loop. During this period, the task performs a minimal amount of work and then returns control to the game. Typically, the minimal amount of work simply involves checking to see if data has arrived on the network.

Testing

Perhaps the most difficult part of developing multiplayer games is testing and tuning. In the PC world, the answer to this problem seems to be to let the players test the game, then release patches, ad infinitum. The console world does not follow this pattern. Once a console game has been certified and released by the console manufacturer, it is virtually never updated. That means console multiplayer games must undergo extensive real-world testing long before they ship.

There are multiple approaches to the problem of testing multiplayer console games, and the console manufacturers still have a long way to go to improve life for developers in this area. For instance, early PS2 online game developers had to test their games on every possible USB modem, which Sony now discourages supporting in favor of the official network adapters. Early Xbox Live game developers had to build custom network-testing tools because tools provided by Microsoft were limited.

The primary problem in testing multiplayer games is emulating real-world Internet conditions. Even sophisticated tools fall short, because the Internet is such an amorphous and changing environment. For instance, the Internet doesn't randomly drop or delay packets - drops and delays tend to clump together as router queues become clogged. Imitating this behavior in a realistic fashion is extremely difficult.

All the console manufacturers are supporting online beta programs to various degrees. Microsoft runs an Xbox online beta program with selected players on custom Xbox hardware. PS2 developers have two options: Sony can administer the beta, drawing from a pool of network adapter owners, or the publisher itself can administer the beta with beta discs mastered by Sony. From a development perspective, the important part is planning for a beta period, and including time in the schedule to respond to issues raised by the beta.

Console manufacturers also include online testing as part of their normal game certification process. Microsoft has many additional certification requirements for Xbox Live games, mainly around specific features like friends and voice. Microsoft also has minimum bandwidth and latency requirements which games must work under without noticeable lags or stutters. Sony has no such official requirements. Games are simply expected to perform well for the number of players they officially support.

Microsoft is alone in requiring that all games support a patching mechanism called AutoUpdate. During Live sign-in, if the game detects that a newer version is available, it automatically downloads the updated version to the hard disk. The primary purpose of this feature is for fixing security flaws or scaling issues in the game. Sony also has a limited ability for games to patch after release. Currently, only one PS2 title supports patches, which are stored on the memory card.

The Future

The future for online console games looks bright. The fact that every console manufacturer is supporting online play is a good indication that it will become an important feature of new games. The ability to use the Internet affects single-player console games as well. Console games now have the opportunity to persist game scores, download new levels or rosters, inform players of tournaments and game-related news, advertise fresh content within the game, and even access real-time data like weather and sports scores.

One major frontier for online console games is figuring out ways of building community. Some new games are already starting to do this in simple ways. For instance, MotoGP, a motorcycle racing game that comes with the Xbox Live Starter Kit, shows how you rank, not just overall, but against your friends. Only a handful of players have the incentive to be the best racers in the world, but almost everybody has an incentive to beat the people they play with every day. This is just one example of using the community aspect of multiplayer gaming to enhance the game.

In the future, expect console games and services that integrate player feedback mechanisms (like Ebay.com customer ratings), clans, tournaments, ladders, and detailed game stats. Expect web interfaces with features like sign-up, statistics, session reservations, game ladders, and so forth. Microsoft already supports a minimal upload service for game scores and leaderboards. Future versions of Xbox Live may support content upload for things like replays, game levels, and perhaps even game mods. With the advent of voice as a viable communication mechanism (as in SOCOM, for example), developers have the opportunity to innovate in new ways, including command and control mechanisms, character-specific voice masks, and other clever voice tricks.

Online console games have the potential for changing gameplay in the living room. The ability to invite your friends to your virtual couch and enjoy a high-quality gaming experience is irresistible to most players. The market itself is still small - not many folks have an Internet connection next to their TV - but it's growing quickly, and the technology for enabling online play within console games continues to improve.

For More Information: Online Console Middleware Providers

Gamespy - www.gamespy.com
SN Systems - www.snsystems.com
Access - www.access.co.jp/english