Guns of Icarus Online Post-Mortem - Part V: What Dumplings Taught us about Service

After our Kickstarter campaign concluded and pre-orders went up on our site and Steam, people started to interact with the game.  We started to coordinate testing with Kickstarter backers and people who pre-ordered, and in the beginning, everything was really rough.  Mechanics were being refined, Ux/UI were still rudimentary, and we were just setting up our servers and peeking into that terrible rabbit hole that was networking.  With people having their first experiences with the game, we had to think through just how we would shape and interact with our community.

From the outset, we had to decide what type of service we would provide to our players and how we would define service.  We felt that the players who believed in us and braved all the rough implementations and experimentations (ex. the servers that were only on one day every two weeks) deserved the best that we could possibly offer.  With the Kickstarter campaign, far more important than the money and the PR ripples were the people.  For everyone who took a chance on us, we wanted to repay them with the best service and to make sure that they did not misplace their faith in us.  We would make mistakes aplenty, but from the time of the Kickstarter campaign onward, we were no longer just making a game or a product.  We were now offering a service, and our game no longer belonged only to us.  It belonged to our players as well.  We knew we always had creative control, but we also had a responsibility to serve our community.  As Yancey Strickler (Kickstarter’s CEO) said in a recent interview, “This thing (Kickstarter) is more important than me or any of us.  It is like a public trust.  It should exist like a place of opportunity for anybody.”  We took this belief to heart, and interestingly, a Kickstarter campaign was where our service creed was born.

Typically, we associated better service (faster, more personable, friendlier, more flexible) with higher price paid.  But for us, we realized Guns of Icarus Online was a game and experience where people paid a (hopefully) reasonable price one time.  Even with the in-game purchases that supplemented the cost of offering customization, which counted for between 10-15% of the game’s sales, people generally didn’t spend a lot on average, which was just fine by us, but then the question was, could we offer a premium service without passing the bulk of the cost to our players?  In other words, how could we charge people the price of a couple of McDonald meals yet aspire to the service of American Express for example?  We might have found answers in delicious, melt-in-your-mouth dumplings.

Every time I went back to Taiwan recently, whether to stress myself out in publisher meetings or visit family, I always made my way to this dumpling place called Din Tai Fung (鼎泰豐).  Started in 1974, they expanded into a chain that spanned Asia Pacific.  Relative to other restaurants ranging from mom and pop shops to any average-joe eatery in a neighborhood, Din Tai Fung would be reasonably priced and affordable, especially considering that it served up some of the best “small eats” Chinese dishes in the world.  The melt-in-your-mouth dumplings, chicken soups, and fried rices were so good that Din Tai Fung was rated the only Michelin Star restaurant in Taiwan last year.  People typically lined up for 30-60 minutes to get into any location in Taipei, and regulars and tourists alike consistently found the service superb.

Here was a place that succeeded by offering premium quality and service at mom-and-pop prices, thereby creating amazing value.  It was exactly the model we sought, so we started to think about and observe what they did well, and here are the points we took to heart.

Service by Principle, Not Rule

At Din Tai Fung, I would argue that service started the moment one got in line at the door.  Someone would hand out a menu with a smile, and people would order while they waited.  If a family had small children, the staff would note and set up high chairs and drawing crayons before the family seated.  People would make requests off menu, and the answer would pretty much be yes, and people could make a request to any staff they saw, and the request would be answered and acted on promptly.  Within a couple of minutes of being seated, tea and drinks would be poured and food would start to arrive.  If measured by speed and pricing, Din Tai Fung would be a fast casual restaurant, but if measured by food and service quality, it would be a fine dining restaurant.  Incredibly it would be both.

I think one way it is able to achieve this level of attentive and flexible service is that it is principle driven and not rules driven.  I’ve never heard of a claim at the restaurant that something couldn’t be done because of a rule.  The staff at Din Tai Fung were trained to respect customers and to treat service itself as a noble profession.  The speed and niceness had to be systematic because everyone in a Din Tai Fung restaurant embodied these qualities.  The empowerment is evident because the staff was not just flexible in what type of tasks they performed, but in who performed them.  A server could take the order and a busboy could take the check.  As long as someone was available, that person could step up and perform any task that a customer demanded.  Unencumbered by rules, the staff often got to say yes with a smile, and the combination of yes and a smile left a power impression on the customers.

About System and Tools, not Just Individual Personality

Not to pick on USPS, and I’m sure most people who work at USPS are really nice, but whatever great individual personalities or attributes generally don’t translate into great service there.  On the other hand at Din Tai Fung, there is a standard and an expectation to smile, to be polite, to be fast, and to do whatever it takes to serve.  At a place where everyone flies to meet demands but can still take the time to engage in quick conversations, where food is always delivered with a smile, the level of service can only be achieved by setting a standard and by training.  Just a person being nice alone can’t attain this level of systemic excellence.  The takeaway is that, deciding a standard and then training everyone to meet that standard is incredibly important.

Training explains the niceness, but it only partly explains the speed, and this dumpling place is fast.  In fact, it has one the highest table turnover rates in the world.  In other words, they serve more people at a faster rate than most any restaurant in the world.  To achieve this type of speed, an efficient underlying system looks to be in place.  Before anyone ever sits down at a table, the order is already taken and processed.  Since dumplings and other steamed dishes take time to prepare, timing is important.  By the time someone sits down, the timing is synced so that within a couple of minutes, dishes would start to arrive one by one, and it takes no more than a few minutes for the longest prep time items to be served.  The staff all wear blue tooth headsets and are in constant communication.  The takeaway for me here is that, building an efficient system as well as constant communication and situation management are as important as the attitude in performing great service.

A Service of our Own

With the two lessons in mind, we started to shape our own service credo and tools while working within constraints (such as the not so small problem of having very little money).  The first thing we did was to decide on a few principles to aim for:

1.  Speed -  For all player ideas, feedback, and issues, we wanted to respond to them all and respond fast.  We knew we could not (or should not) implement all the ideas we came across, but we wanted to make sure we always listened.

2.  Respect - To our players, we would always be nice and courteous, even in the face of hate sometimes.

3.  Empowerment - Everyone chipped in with support and was empowered to act and reward players.  Support and service are hard, and we’re small, so we know it would take all of us.

How We Got Faster

With “responding fast,” we strived for 24 hours for a first response to a player report.  We then tried to update whenever we made progress on an issue.  Very soon after setting this goal, we would be thoroughly tested.  Early on, in the days following launch or a featured sale on Steam, we really struggled to keep up.  During launch week in the eye of Hurricane Sandy with the game at the top of Steam’s Indie topseller list, we were swamped with player reports because whatever could have gone wrong went wrong.  We had time to exhale for a brief moment when the game went live and then sailed into a maelstrom, working pretty much all waking hours to support players and keep the game up.  Issues came at us from all directions.  We definitely had a lot of issues with bugs, optimizations, and unstable servers (because of the storm) that we didn’t uncover during alpha and beta.  With more players in game, these issues manifested like wildfire.  On top of everything we caused, we had to deal with Steam servers teetering.  And if we hadn’t learned that everything from a player’s client, the local network, to his or her ISP was going to be our fault, we learned that lesson then.  Not matter what the causes were, as long as the game didn’t work, players would look to us for help and to fix the issue.  We realized that the reason was simple - where else were the players going to turn to?  So the responsibility was on us to help people.

Two big problems emerged for us early on:

1.  Despite everyone’s efforts, if we wanted to grow, we lacked bandwidth to sustain the response speed.

2.  As much as we tried, we did not have a good enough system to track player feedback while allowing everyone to interact with it fluidly.

Support Library

To improve bandwidth, we needed more help, but we also needed to do things more efficiently.  Not only did we collect more and more solutions to various player issues, such as what model of router disrupted Photon’s data transmission, what antivirus blocked which ports, what VPN software affected routing tables, what weird and obscure issues resulted from XP use, what GPU over-clockers overclocked, what ethernet card had configuration issues, etc etc., we also detailed them in an FAQ on our site as well as built a training manual for ourselves, with what info to ask for and the fixes for all the scenarios we knew.  With a support library, we got faster at figuring things out, and our successes in solving player issues increased significantly.

There were always failures along the way.  My worst support case was when a player had a start up crash, and I had been sure that it was due to Steam client’s install.  I asked the player to reinstall Steam client, not knowing that it would affect his game and save files (reinstall usually didn’t cause problems before).  His reinstall caused him to lose files for 10s of games in his library.  He was beyond furious and fairly chastised me for my carelessness.  As it turned out, he had older adventure games, and client reinstall affected those games.  I had lost a player forever and caused him a lot of pain, all because I was careless.  I printed out and nailed his email to the wall next to me for a month as a reminder to not let that happen again.  And the safe way to reinstall Steam client was  promptly added to our FAQ and support library.

Community Ambassadors

Building up a support library ultimately improved efficiency.  Another amazing development increased bandwidth itself, and we called it the Community Ambassador program.  It was born from players asking us how they could help more.  As we were still learning how to troubleshoot ourselves, letting players be responsible for that aspect of service would be setting them up for failure.  We thought long and hard about how to get players involved, and we decided to focus on two areas initially.  One was to teach new players, and the other was to help manage our forum.  Teaching was, and still is, supremely important.  In game, players were experiencing something really new.  New players were learning not only the different mechanics and tactics, they were also learning to truly communicate and teamwork.  Our players often said to us, “teamwork is overpowered in this game.”  We didn’t have the ability to be in game all the time to teach players. (Also, in the beginning we didn’t have much of a tutorial, and it would take us many months to build and improve on our tutorial.)  In this respect, the Community Ambassadors (“CAs”) were immensely helpful.

Logos of Player Run Tournament

Another area where CAs were integral was organizing and helping with community events.  Most of our events were player run, and the CAs not only organize events, but help other organizers run them, serving as refs, casters, and just general helpers.  Over time, CA’s number would grow, and they would volunteer to take on more as we became more confident with support  We now notify CAs of all our events ahead of time and have them participate in early testing of new features.  Our player volunteers are truly amazing, and it’s really humbling to have this strong of a bond with our players.

With CAs, we looked for players who had experience with the game, great sportsmanship and citizenship, and experience managing game communities.  We formalized the program with an application and a play session so new and existing CAs could get to know each other.  The CAs were listed on our site and in game, so players can reach out to them easier.  We would also honor CAs with badges, status in game, as well as gifts that we made over time.  From an idea to help, players helped us create a path for them to volunteer, then intern, then work with us in community management as part of our team.

Support Pipeline

To create an effective system for support, we first had to identify all the points of interactions with players - all the points where we could fail.  Firstly, there were places where players would reach out no matter what - Steam forum, our own forum, in-game, facebook/twitter/whatever social media, and youtube.  Once we had our Kickstarter campaign running, our Kickstarter page would need our attention as well.  Then, we thought it would be a good idea to setup a web support/bug track portal.  That along with everything else turned out to be way too overwhelming, to the point where we were overrun.  Pushed by desperation to the precipice insanity, we realized we had to simplify and streamline.

Out went the web portal.  We instead tried to direct all player feedback and reports to one support email that gets distributed to the team.  Two or three of us would serve as first responders, answering all incoming emails, getting the right player information, solving issues directly, and escalating issues to others if needed.  While we responded, we also tracked all incoming issues and requests in a support center (first built in google docs, then transferred to Trello).  In the meantime, we developed a pipeline for feedback to flow to production.

Every couple of weeks, we went over the ideas/feedback from players or team together.  If we decided that there were enough feedback or that the ideas were cool, we fed them to a repository and prioritized them.  Before the start of every sprint, we would sprint plan based off of the repository and balance the ideas against anything else we decided to be important.  Bugs were directly added to our sprint board, and we would always reserve time each sprint cycle to fix bugs.  The advantage of using something like Trello was that everyone could see the progress of a task and comment on it at anytime, and this allowed us to update our players on where we were a lot easier as well.

Be Personable, but not Take Things Personally

In order to realize our second tenet of niceness, we had to be ourselves but also train ourselves.  Our support knowledge base provided the contents of training, and we held sessions to anticipate and practice responding to different player issues (We held training sessions for our CAs as well).  But there was really no better way to learn than actually engaging with the players.  We learned quickly that service was hard, very hard.  Here was a post on Steam forum during the first week of release:

Thinking of getting GoI?...  I’d recommend holding off until it’s complete...  Endless excitement. From being kicked midgame, to 30 second long lag freezes, to not being able to hear the voice chat system, there will always be something to keep you on your toes.  Yep that's right, after spending $20 to get a game that would still be considered a beta by any number of other companies, you have the pleasure of paying $1 for a single new hairstyle.  Oh and one more thing, they are advertising features that don’t actually exist in the game. Why don’t you try the “Resource Race” game mode. If you can’t find that in game, you can always earn “cosmetic upgrades through your achievements” Oh what’s that? You can’t find either of those? That would be because they don’t exist.

Reading this post for the first time, I for one definitely felt crushed.  The sarcasm obviously masked the player’s anger and frustration, and Steam forum was as public as it got.  This post alone generated 67 comments, and at launch, we had plenty more like this one.  Before responding to this, We had to first take a step back and focus on the actual contents and not the way it was written.  We also had to remind ourselves that from the moment the game went live, it was our job to serve, and we couldn’t take anything personally.

The cause of the connection and performance issues were in part really apparent (because of the hurricane knocking out our servers and our servers being overloaded) and in part detective work (we had to figure out what lag freeze meant and what caused it).  We opened up more servers on the US West Coast, troubleshot server performance, got his info, and tried to figure out what was happening.  For paying for cosmetics, he was not wrong either.  We offered in-game purchase cosmetics in game.  The main reason was to cover the cost of offering cosmetics, which we spent significant efforts to craft.  Without charging more, we simply could not afford offering them.  The other reason was to raise more funds to keep us going.  We also had to make clear that, no one had to pay for this.  The game was by design not pay to win, and everyone could reach the same heights through skill and skill alone (the purchasable items were cosmetic only).  For advertising features that didn’t exist, that was partly us being unclear and partly because we didn’t deliver.  We had hoped to commit the cosmetic item progression unlocks as well as Steam achievements before the storm, but the storm derailed everything, and we weren’t able to edit the store page fast enough.  All this we just had to be upfront and honest.  Regardless of attitude, he was right, and we had no problems owning up and apologizing (a lot!).  Once we started to stabilize and fix his issues, he was nice enough to amend his post:

**EDIT: Howard has since informed me that they have fixed the crash to desktop bug. I haven't seen it again over the last couple days(Yaaay).  I also just read(see post 36) that even though nothing is defined as “Resource Race” in game(that I have found anyway), it is in fact included.  **The Steam store page was updated to more accurately reflect the game. So if you haven't purchased yet, anything about misrepresenting the game will be irrelevant to you. There is a lot explained in the developer replies as to why things are the way they are, and changes for the future.

With a post that garnered 67 comments in launch week, this player was one of our biggest haters.  But taking a step back, we recognized that he spent his hard earned money on the game, the game didn’t work, and we didn’t do good enough of a job regardless of circumstances.  Owning up to mistakes and ultimately fixing his issues went a long way to build a relationship.  I thought our approach was right, because after almost a year and a half, the player is still in the game, logging well over 1000 hours and playing over 4000 matches.  We became friends on Steam (most of my Steam friends now are players), and we still chat.

Once the game’s community took shape, there was another service issue - moderation.  We drafted a community standard and incorporated it in the player registration process.  The community standard let the players know what type of community we were cultivating, and it served as a foundation to build our reporting and moderation tools.  To draft it, we referenced other games and Steam, but we also wanted to make sure we balanced the sword with the shield.  A community standard needed to be enforced.  If we did not enforce it, then the community at large would not respect or trust us.  But it could also encourage.  When we defined the rules of enforcement, we had to make sure we take a human approach.  We were here to serve, and we were not running a police state.  That balance between policing for what we deemed to be the greater good of the community and courtesy towards people who spent their hard earned money on something we loved had always been delicate, but we reminded ourselves to err on the side of accommodation.

Would this be tested?  You bet!  Just recently, we got a support email and it read as follows:

Ban me? I'll let everyone in my community know about how your hired goon...likes to troll players that don't immediately follow their whims and dictates, and encourages everyone to "report" them when a person argues back. You're struggling little ship game doesn't need me? For fuck sake, then who cares? Plenty other great games built on proper game engines that aren't completely bogged down with bugs. Oh and by the way, I'm a grown adult and I'll use what ever fucking language I want you fucking freedom hating Nazi. To think I used to support this bullshit game and even bought the 4 pack to help you assholes. Fucking eat shit if you don't like players like me... Bitches.

We learned the hard lesson of investigating both sides of a player report of significance.  The reporter was not always completely valid, and the reported was not always at fault.  Sometimes the truth was somewhere in the middle.  In event of a report, we would pull up chat logs of the parties involved to see what was said and also the context.  The process was tedious, but critical.  In this case, what we received was harsh, but the context was pretty clear.  From the chat logs, we knew the player crossed the line, and we had sent an email to him to warn, not ban.  Yet, the player was offended.  We figured we had a few options:

1. Do nothing and lose the player forever.

2. Be a stickler to the rules of moderation and lose the player forever.

3. Just reach out and appeal to the player at a personal level and see what happens.

We chose option 3 and wrote him back with a few points:

1.  We really appreciated the effort he took to write us, meant that he cared.

2.  We honored and respected that his hard earned money spent on the game, but we had an obligation to serve the community as a whole and this job is tough.

3.  Clarified his status - a gentle warning and not a ban - and explained in detail why we acted

4.  We did our due diligence (We attached the chat logs in question for his reference)

5.  Just encouraged him to have fun and be a bit more mindful

6.  Thank him again for listening to us.

We debated this response a bit because the things he said to warrant a report were blatantly offensive.  We weren’t sure if the player would write back and what the player would say.  But then the player did write back:

I will move forward, and when I play GoIO,  a game I hope rises above the rest of the crowd, I will do my very best to be respectful to fellow players. You're not "freedom hating Nazi," and I sincerely apologize for the disrespectful language I've used to lash out at your company and community.

We were really moved, so we bought him a little gift for Christmas and in this way made another friend through the game.  Interactions didn’t always end up this way, but when they did, it made all the work worthwhile.  The important lessons we learned were to not be afraid to engage, not hesitate to apologize, take responsibility, and see player issues through to the end.

Empowerment - Be ready to make mistakes, but be ready to do something about them

When we empowered ourselves, we had to accept that we would make mistakes.  And mistakes could be corrected if we had the right mindset and were persistent.  Empowerment was hard, because it placed the burden on us to make a call and to live with the consequences.  It was hard also because the better action could sometimes be counterintuitive to us at least initially.  A lot of situations were judgement calls, and there wasn’t necessarily an answer that was exactly correct.  All we had to guide us were the principles we set.  When we moderated, a first reaction might be to enforce punishment of some sort, but the better choice could actually be to reward and compliment as in the case mentioned earlier.  To empower ourselves to reward was at times harder than to enforce a rule.  But on the other hand, if by giving extra copies of the game or items in game, we made our players happy, the gifts were well worth it.  To determine how or what to reward players, we did it by trial and discussion.  Sometimes, it was almost an experiment for us to see how far niceness could go.

To facilitate empowerment, not only did we communicate with each other and with our volunteers, going over situations throughout the week, we also pooled game and item keys in a master google spreadsheet for example.  Everyone had access to it, even our CAs and interns, and could gift things to anyone at anytime.  Another thing that we found important was to support each other and make sure we didn’t leave anyone on an island.

Don’t Forget Why We’re Here

Ultimately, we served so players could have fun.  We have a small percentage of players who spent over 1000hrs in game, playing over 3000 matches.  Overall, over 600K matches are played, with 2.5 billion shots fired to date.  3.5 million ships would explode in the air.  We have a dedicated player base, and it’s a lot of fun for us to get to know our players and participate in everything that they do.  What they create in and around the game only fuels us to create more fun and have fun doing it.

Fan Art Selection

Summary

Since beta, we probably responded to 10-20K player emails, comments, and posts.  We didn’t always have success solving everyone’s issues, but the our rate of success increased as we learned and improved our tools and flow.  Gauging how we’re doing or whether what we do is worth it isn’t easy, but we can still look at a couple of things.  One is Steam reviews.  If we look at rate of positive reviews ((Total Reviews - Negative Reviews)/Total Reviews), then on Steam we have over 96% positives.  The other is the Metacritic user score.  Our critic score isn’t very good, which is topic for another time, but our current user score of 8.8 out of over 200 reviews is not bad.  While a lot of the positives are for the game itself, but I think the way we approach serving our players plays a part.  We’re a long ways off from where we need to be, but we think that at least we’re on the right path.