Conference Wrap-Up - Mastering the Craft of Online Gaming Infrastructure

  On April 19, 2007, a conference of game developers, IT entrepreneurs, internet security experts, telecom executives, and technology investors gathered in San Francisco to take part in Mastering the Craft of Online Gaming, a day long event in which the complex issues surrounding the online gaming market were discussed.

The Future of Digital Content Distribution

The day began with a conversation on future trends in digital content distribution moderated by Jerry Godwin, Director of Online Services for the Themis Group. Panelists included Brian Taptich, VP of Business Development for BitTorrent, Mike Gordon, Co-Founder and Chief Strategist at Limelight Networks, Romain Nouzareth, Founder and CEO of Boonty, and Gary Croke, Director of Marketing at CacheLogic.

Taptich, began the conversation with a warning; “The current internet is broken and will not scale to meet demand,” he said. However, he remained optimistic about the future, believing that game designers will adapt to network constraints by making consumers an integral part the online gaming business where they can contribute resources such as CPU cycles and creative assets. He was also hopeful that new distribution and business models would help alleviate the current hits driven model of game publishing.

Croke said file sizes were too large, which can create a margin issue for publishers who are trying to distribute online. He also noted that frequent, large patches to online games have a negative effect on game play and the number one impact on customer satisfaction is slow game performance. “Broadband speeds are lagging behind the rest of the world, part of the problem is geography, but the problem will only get worse,” he said.

“The PC will really take the dominant position,” Nouzareth predicted. In the future he expected to see more casual players drawn to PCs over consoles, with women comprising much of the new audience. He envisioned new kinds of games, simpler in focus, with business models based on micropayments.

Gordon felt that games such Guitar Hero II pointed the way forward by creating an ongoing revenue stream for publishers through its use of online Xbox Live micropayments. He then described how implementing online features in games can help to unite the needs of the publisher with the consumer, extending the life of a game.

When Nouzareth suggested that retailers might have a role to play in online distribution, the audience agreed, pointing to Nexon America’s Maple Story and the publisher’s partnering with Target to sell pre-paid cash cards in its stores.

Nexon's free, 2D, side-scrolling MMORPG Maple Story

Taptich marveled at the growth of MMOs in the wake of Blizzard’s tremendously popular World of Warcraft but worried that the online business may possibly be over invested. He also questioned whether retailers were truly interested in synergy with online distribution, noting that big box retailers such as Wal-Mart had so far enjoyed only limited success in the online space. “Retailers are not going away,” he said.

  Security Issues for Online Games

Security is always a prime concern for online business where the safety and reliability of network operations is essential for commerce. In addition to issues of IT, financial, and identity security, online game business face a unique set of challenges in which an aimbot can be as detrimental to profits as a stolen credit card.

Moderated by Steven Davis, CEO of IT Global Secure, the Security Issues for Online Games panel included David Lee, VP of Engineering and Infrastructure for K2 Network, Scott Parcel, VP of Engineering and CTO of Cenzic, Varun Nagaraj, CEO of Net Continuum, and Micah Quinn, Senior Engineer at Even Balance. They discussed the various techniques used by hackers to disrupt online games, from web hacks to game cheats.

“Web site hacking is a nightmare for publishers and it happens constantly,” Lee said. Some of the major vulnerabilities that web sites face are from Cross Site Scripting (XSS) and Cross Site Request Forgery (XSRF) in which attackers take advantage of form inputs in web applications to execute malicious code. “Most hacks are web hacks, not network hacks,” Parcell emphasized, and integrating security testing with early development is a prudent defense. Developers must test early and test often, he said.

Nagaraj addressed hardware based Web Application Firewalls that can provide another layer of defense when dealing with web hacks. By filtering the data flowing into a host’s server, application firewalls are able to prevent the execution of harmful commands without the expense and time required to fix source code. They can also improve server speed by load balancing and accelerating encryption algorithms.

Blizzard's phenomenally successful World of Warcraft

Rampant cheating can be a pernicious threat to online games, eroding the user experience until honest players are driven away. Quinn spoke about the history of game cheating and the development of cheat countermeasures. The problem had its origins with the release of Quake when clever hackers designed aim-bots that would automatically move a player’s aiming cursor with computer precision. The practice has since spread to most popular first person shooters, severely hampering genuine players’ enjoyment.

Online role-playing games are subject to a bewildering variety of cheats, from macros that automate repetitive activities to elaborate confidence scams. The practice of gold farming can also severely unbalance the in-game economics of MMOs. To deal with these threats, several third party anti-cheat software packages have been developed to aid game developers such as Valve’s Anti Cheat and Even Balance’s PunkBuster.

Maintaining and Updating MMORPGs

Ideally, MMOs are long-term propositions that require years of ongoing development work. The extended life of these games present new challenges for game developers and in a presentation by Shannon Posniewski, Director of Technology for Cryptic Studios, he examined some of the strategies his company used to insure City of Heroes/City of Villains’ continued viability.

“Development continues indefinitely and the classic game industry “crunch” doesn’t work,” he said. To remain creative, Cryptic’s staff follows regular work hours over a steady cycle of production and beta testing, providing regular updates every three to four months. “That seems to be the sweet spot. It prevents players from getting bored and the game feels more alive,” he said.

Because network operations are costly, Posniewski described how Cryptic worked to simplify and automate as many tasks as possible, such as log file rollover, load balancing, and fault detecting. Thorough data logging was very important to the process and over a terabyte of information on City of Heroes has been captured and archived. He added that a database keeps a record of every event in the game, from player chats to combat actions, all of which can be easily accessed by developers and customer support.

Cryptic Studios' super hero focused City of Heroes

Posniewski also discussed the importance of good customer support to the longevity of an online game. Although support is costly, it can mean the difference between an unhappy player walking away from the game and never coming back, versus one who continues their monthly subscription. The key to success is to minimize the number of customer support “tickets” or instances in which a customer support person has to get involved. By anticipating problems during the design phase, game developers can enable ways for players to drop difficult missions or avoid abusive situations, thereby helping themselves without the aid of customer support. “A lot of the design of our game is to reduce griefing,” he said.

  The Changing Landscape of Online Payments

When writing a business plan for an online game, the acceptance of online payments brings a host of considerations that must be addressed including revenue deferral, the managing of fees and rates, tax liabilities and value added taxes, and the ever present issue of fraud management.

In a panel moderated by K2 Networks’ David Lee, Rob Uhrich, Senior Director of Digital Markets at PaymentOne, Shane Happach, Business Development Manager for Global Collect, Peter Huang, Senior Director of Web Technology at Trion World, and Edward Sullivan, CEO and Founder of Aria Systems discussed some of the billing concerns that publishers of online games face in a global market.

In the United States the monthly subscription model is the most successful, but in Asia the micropayments model is more prevalent, Huang noted. However, as Happach explained, micropayments are problematic because customers are making multiple small purchases rather than one large transaction. Credit card merchant fees make this impractical and online publishers have had to work around this by selling fixed amounts of in-game currency or company scrip such as Wii points or Microsoft points.

Another issue for publishers to consider is the varying payment methods that exist across the globe such as credit cards, PayPal, regional debit cards, and bank transfers. “Giving more payment options can be a competitive advantage,” Uhrich noted. Accompanying the acceptance of credit card payment is the matter of Payment Card Industry Data Security Standard (PCI DSS) compliance. PCI is an industry standard created to prevent credit card fraud and businesses that accept credit card payments must adhere to it or risk losing their merchant account.

Complicating things for online publishers is the enormous rate of chargebacks that they must contend with. A chargeback occurs when a customer disputes their credit card bill resulting in the merchant being responsible for payment along with a penalty fee. Because refunds are cheaper than chargebacks, Sullivan urged publishers to have a system in place for crediting customers forward. He also emphasized keeping detailed records to use as evidence when disputing chargebacks. “There is an unlimited downside to fraud and chargebacks,” he warned.

Latency can kill online games. How do we make sure this does not happen?

In Shannon Posniewski’s earlier presentation, he noted that one of the top customer support tickets that online games face was over frame rate problems. Focusing on network latency, Vlad Ihora, Head of Gaming Community at TeliaSonera International Carrier described what publishers should look for when choosing an IP carrier for their online games.

Ihora emphasized the necessity for a carrier to have full ownership their networks, ensuring their optimal configuration along with wide coverage, extensive bandwidth, and private peering with other Tier 1 networks. He urged online game publishers to look at the various broadband carriers within their target territory and then examine whom these carriers were getting their IP service from. “You have the power of dictating how your service is configured,” he said.

Audience members wondered if there was an advantage to using lower cost “best effort” networks in which traffic load dictates the rate of data packet exchange. While acknowledging that it is impossible to eliminate latency, Ihora responded that networks with guaranteed service and minimized packet loss justified their higher costs by providing players with the most satisfying game experience.

  Your MMO Infrastructure: Build or Buy?

As the industry matures, online game developers and publishers are enjoying the luxury of being able to choose from a variety of products and services that can potentially reduce costs and increase quality. In a panel moderated by James Hursthouse, CEO of OGSi, George Dolbier, CTO Games & Interactive Entertainment at IBM, Michael Steele, VP and Evangelist at Emergent Game Technologies, Derek Wise, CEO of Global Netoptex, Inc., and Mark Rizzo, VP of Operations & Platform Engineering for Perpetual Entertainment examined the pros and cons of outsourcing online game infrastructure.

A proponent of outsourcing, Steele declared that all of the big problems facing the MMO business had been solved over the past three years by market forces, leaving efficiency and scale to be addressed in a commoditized market. Further, he dismissed the notion that relying on third party solutions resulted in homogenized products. Drawing an analogy with the film industry, he pointed out, “Directors don’t operate the cameras and they shouldn’t have to.”

On the question of building online infrastructure from the ground up, Steele felt the risks were extremely high. Rizzo thought it was possible, but that finding qualified people to make it happen was difficult and costs were prohibitive. Wise was cautious in his advice saying, “If you can take that side of the company and spin it off into a successful IPO, then yes.”

Ultimately, when faced with creating an online infrastructure from scratch versus outsourcing, most of the panel preferred to rely on outside service providers. However, as Rizzo noted, “If you look at the companies that derive most of their revenue from online, not necessarily even game companies, but companies like Amazon and Google, the most successful ones build their own infrastructure.”

NetDevil's vehicle-based MMO Auto Assault

The Business of Online Game Infrastructure

Rounding out the day was a look at the business of online game infrastructure through the eyes of a diverse group of developers, including Daniel James, CEO of Three Rings, Jon Hörddal, COO of CCP Games, Scott Brown, President and Co-Founder of NetDevil, and Steve Goldstein, Director of Business Development and General Counsel for Flagship Studios.

Moderator Matthew Le Merle, CEO of Gameplay Holdings, LLC began by asking the panel to describe their companies’ online goals. Growth was the common answer with James adding, “I want to create something that grows virally. I want to create a forest fire.” However, as Le Merle pointed out, expansion brings with it new challenges. Hörddal responded, “Even if I have to over invest to stay ahead of growth, I’ll do it.”

The conversation turned to outsourcing and the panel described the benefits of distributing parts of the online business to third parties. “Billing systems are not fun. If you make a mistake it can cost a lot of money and it would be nice if that was somebody else’s problem,” James said. Goldstein added, “We have to be light on our feet and we’ve done a lot of outsourcing.” Describing the interrelationship between outsourcers and clients, he said, “I take comfort in knowing that somebody else’s business is at risk as well.”

Hörddal agreed, saying, “We have an aggressive plan to outsource everything that can be done better than ourselves,” adding that there is nothing wrong with outsourcing as long as it is carefully managed. However, Three Rings’ Daniel James sounded a note of caution saying, “I think it’s insane to outsource customer support.”