Localizing MMOGs

Last Fall, Turbine Entertainment released Asheron's Call 2, a massively-multiplayer online role-playing game (MMORPG) that shipped simultaneously in English, French, and German, with a Korean version following last Spring. Instead of using language-restricted servers, all AC2 servers allow all languages, so players speaking different languages can play together on the same world. I'd like to share some of the things we learned as we made the game "localizable". I'd also like to present a technique that improved grammar quality in all the supported languages (including English).

In many ways, localization is just a catchall phrase for a long list of smaller tasks. For instance, every language has its own keyboard, and the game needs to have reasonable key mappings on any of these keyboards. To allow Korean or Japanese users to type in their native language, you need to integrate support for the Microsoft Input Method Editor (or provide similar Eastern character features). German words tend to be longer than English ones, so German needs 30% additional space for each field in the UI. There are also cultural and legal differences that can cause the content of the game to need variation - green blood is the classic example, although a "very low gore" option is often just as useful these days. Fortunately these issues are relatively well understood - they are essentially the same for any PC game, and a good book on localization will help you overcome these hurdles. The biggest difference between localizing MMORPGs and other games is the number and format of their strings.

Most of the MMORPGs released so far have a lot of strings - far more than a typical game. This is because the number of strings keeps growing for as long as they run - text is a cost-efficient way of adding content after a game ships, and most MMORPGs add new content on a regular basis.

Storing the Strings

Early MMORPGs such as Asheron's Call 1 tend to send strings to the client in the most straightforward way possible: they send the string to the client each time the string is needed. When the player talks to a vendor, the server sends the client a string to be displayed: "Bob says, 'I'll custom fit any purchases for free!'" When the player leaves and returns later, that same string is sent to the client again. The biggest problem with this system is that it uses more bandwidth than is necessary. How many times do you really need to send Bob's greeting string to the player? After the first time, it shouldn't need to be sent again unless it changes.

The approach for using strings in localized single-player games is completely different. In this method, all the strings are placed in a resource file. Translated versions of these strings are made, and the data for every supported language is shipped on the CDs with the game. When the program needs to display a particular string, it looks up the appropriate ID from the resource file, fills in any variables (such as player name), and displays it. This is a great solution for games that don't change much after ship, but if strings are changed post-ship (such as due to a bug-fix patch), the entire resource file must be re-downloaded. Also, because all the strings are contained on the client in a well-organized resource file, it becomes rather easy for hackers to read every string in the game. This isn't an issue with traditional PC games, as they take the stance of "if you cheat and look ahead, you just ruin your own fun," but one of our selling points is monthly updates that add new quests and secrets to the game. Some of these secrets are meant to appear a few weeks after the player has downloaded the monthly update. If we sent all the new strings to the client during the initial monthly download, they would appear on spoiler sites instantly and we'd have no way to surprise players between updates.

We took the model used by other types of games and modified it to better fit the requirements of an MMORPG. The most important enhancement is to allow the resource file to be updated on the fly. Each monthly patch can send only the strings that have changed or been added. And not all the new strings have to be sent to the client during the monthly patch; a handful of secret strings can be kept out of the big download. These strings are downloaded only when needed.

All the strings are stored in one file on the client, but that file contains thousands of individual tables. When the server sends strings to the client, it doesn't send them one string at a time; it sends all the strings of a particular table. A typical table contains a half-dozen strings. There is a table for every monster, item, and quest in the game. The idea is that if the client is asking the server for the name of a particular monster, it will soon be asking for the description and other text relating to that monster, so we might as well send them all at once. If any of the strings in that table change, the whole table must be re-sent, but the odds are that if the name of a monster changes, for instance, other strings relating to that monster will also change.

Representing the Strings in Code

Most strings have fill-in-the-blank spots, such as "You take the $ITEM$!", where $ITEM$ needs to be replaced at run-time with the name of the item being taken. (In our system, these variables are surrounded by $ characters.) In a non-networked game, filling in the variables is easy. But our server often needs to manipulate strings without knowing what the variables will be yet. Should the server fill in "$ITEM$" as "Sword" or "Épée"? It can't tell without knowing what language the client is running in, and that could be different for each player. It's easier to just send the IDs of the variables to the client and let the client pull that string from its file. This way most of the server code doesn't care what language each person is running in.

We use a simple structure called a StringInfo to encapsulate our strings and all their variables. They have two integers, the table ID and the string ID, to indicate which string they're referring to. They also contain an embedded StringInfo for each variable. (Alternatively, a StringInfo can simply store a literal string that a user typed in, such as a player name.) Each of the embedded StringInfos contains the IDs for one particular variable in the original StringInfo. This is shown in Example 1.

In the most extreme cases, the embedded variables can contain variables themselves - for instance, the "Fiery Sword of Doom" might be the $ITEM$ in our earlier example, but this string is actually a randomly generated name created by filling in a string with the variables "$PRE$ $MID$ $POST$". As is shown in Example 2, $PRE$ becomes "Fiery", $MID$ becomes "Sword" and $POST$ becomes "of Doom." Then the whole StringInfo gets embedded in the first string as the $ITEM$ variable for that string.

Since a StringInfo can represent a string from a table and all of that string's variables, or even just a literal string, it is a very convenient storage format. Objects can store their names in a StringInfo without concerning themselves whether the name is from a table or not, or where the variables come from. When the client wants to display a StringInfo, it calls the RenderString() function on the StringInfo. If the string is a literal value, that value is immediately returned; otherwise, the table is fetched, the variables are recursively rendered and inserted, and a literal value is finally returned.

A Meta-Language To Improve Grammar Quality

In a text-heavy game where players are going to be reading your strings for hours upon hours, it pays to have good grammar. This is a polish item - people don't even notice when the grammar is good, but they certainly notice when the grammar is bad. We wanted to avoid the grammatical errors that are typical of table-based solutions. However, since keeping the number of strings low saves on bandwidth cost, we wanted to avoid using multiple versions of a string just to change parts of speech. For instance, if one of your strings is

"You give $PLAYERNAME$ an $ITEM$. He inspects it and hands it back."

You don't want to have a separate version of this string just to change the "He" to a "She" based on player gender. The common solution to this problem has been to reword the sentence to avoid using a pronoun. But in other languages, even items have genders; if this one string must fit all possible languages, it becomes extremely awkward.

And what about items with plural names? If you substitute "pants" for the $ITEM$ in the string above, you need to change the modifiers and pronouns, even in English. Similarly, words that start with vowels need to be prefaced with "an" whereas words that start with consonants use "a", but items that have proper names shouldn't use either "a" or "an." And there are several other common cases that require the sentence to change based on the variables. Since the game adds content every month, it isn't reasonable to just rework the strings to avoid the problem. We can't really tell our content designers, "Never add a new item whose name starts with a vowel," nor can we tell them to never have an NPC refer to an item by pronoun.

After a bit of thinking about the problem, we decided upon a simple scripting language to let content designers embed the different cases into the string itself. For instance, the first part of the above sentence would be something like this in our data:

"You give #1:{the [!n]} #1:$PLAYER$ #2:{a[!n] | an[v] | some[p] } #2:$ITEM$."

This "meta-language" lets the content creator embed a lot of meaning into one sentence, at the expense of legibility in the tables themselves. Some of our content designers found the language intimidating while others learned it very quickly. Fortunately, most people only needed to use the meta-language to name items, which is pretty easy.

Each item, monster, and place name must be annotated with the proper tags to indicate what rules to use for that thing. We do this right in the string itself - for instance, if an item name starts with a vowel, the string might be

ID_Name = "Apple[v]"

Here the [v] indicates that it starts with a vowel, so it should be prefaced with "an" instead of "a". Multiple letters could also be used; for instance,

ID_Name = "Eric[mnv]"

Indicates that "Eric" is a proper noun ([n]), a male name ([m]), and begins with a vowel([v]). The complete list of tags we used for AC2 is listed below.

[m] = male ("he" or "him")
[f] = female ("she" or "her")
[i] = inanimate or gender-neutral (an "it")
[p] = plural name (as in "those pants")
[v] = starts with a vowel (so use "an" instead of "a")
[n] = name (proper noun - don't use "an", "a", or "the")
[s] = ends in the letter 's' (so use "'" instead of "'s" to make it possessive)

If the name of an item didn't need any tags at all, it should still be marked in some way (we used [E], which stands for "empty") so that the translators can tell at a glance which words might need meta-tags or not. Even when a name doesn't need any tags in English, it might still need tags in French. Each language can have different tags for each string according to the rules of that language.

The game engine could have programmatically figured out which letters to use - for instance, if a name starts with a vowel, it could automatically have appended [v] to the word without requiring us to manually assign it. However, there are too many special cases for that to be safe - some words that start with "h" should be prefaced with "a" while others should be prefaced with "an": compare "an hour", with "a house." Only a human being can tell at a glance what language rules should be followed for a particular word, so we assign that task to the human being naming the item. In the case of "an hour", the [v] tag should be applied, even though the word doesn't actually begin with a vowel, because the word should use the same semantics as a word that really does start with a vowel.

Thus in general, the game engine doesn't actually know what each letter means; the letters are just flags assigned to the word as far as the engine is concerned. The exception to this is player names. Since we obviously can't know the meta-letters to assign to a player name until the user types that name in, we have to programmatically generate the tags for that name. So the character-creation system automatically adds the appropriate letters to the players' names. There is a chance the game will get these letters wrong because it is using such simple rules of thumb to assign the letters, but fortunately the English rules for names are simpler than the rules for objects. For instance, it doesn't really matter whether a name starts with a vowel or not - you don't add "a" or "an" before it. I am not "an Eric," I'm just "Eric." The fact that my name is a proper noun trumps the vowel rule.

(We were lucky in that we didn't need to add any special tag-assignment rules for French, German, or Korean. Had we localized to other languages, or wanted to allow other grammar forms, we might not have been so lucky.)

Once all the items and creatures and whatnot are tagged, they can be used as variables in sentences. These sentences can choose different words based on what meta-letters are in a variable. The basic structure of an optional block of text is to enclose it in {} characters, with different options separated by "|" characters. Each option lists the meta-tags that it should be used with, or none if it is the default:

"I want {an[v] | a} $FOOD$."

This sentence can become "I want a banana," or "I want an apple," depending on whether the value of $FOOD$ has the [v] tag or not. You can also indicate that a case is supposed to be used specifically when a tag is NOT present, by prefacing the letter with an exclamation:

"You kill {the[!n]} $NAME$."

This would only insert "the" into the sentence if the $NAME$ variable does not have the [n] tag. So this sentence becomes "You kill Timmy," or "You kill the Gurog Shaman," depending on whether $NAME$ has the [n] tag.

Cases can also indicate that they should only be used when two or more tags are present; this is most useful in French and other languages that have genders for inanimate objects:

"You {murder him[mn] | murder her[fn] | destroy the feminine-gender object[f] | destroy the masculine-gender object[m] | destroy the gender-neutral object}!"

This isn't needed in English, but it is a useful construct in French, where even inanimate things can have genders. You don't want to use "murder" as your verb for a barrel, and you need to use the proper French form of "the" based on the gender of the word "barrel."

In cases where there are lots of choices like this (which is pretty rare, especially in English), the game chooses the option that has the most letters in common with the variable. (Where negated letters count as a match if they aren't present.) In the case of ties, the last tying match found is used. Here is a contrived example:

As you kill {the[!n]} $NAME$, {it[i] | they[p] | he[mn] | she [fn] | that named machine[n]} {screams | scream[p]} and {explodes | explode[p]}!

If $NAME$ is "Eric[mnv]", then the string comes out "As you kill Eric, he screams and explodes!" This happens because "he[mn]" has two letters in common with "Eric[mnv]", and that's more than any other choice. If $NAME$ was "Sir Roboto[ni]", then it would come out "As you kill Sir Roboto, that named machine screams and explodes!" This is because two different choices match here: "it[i]" matches with one letter, and so does "that uniquely-named machine[n]". Since there is a tie, the last valid match is chosen.

Then we come to the case of sentences with multiple variables in them. When there are multiple variables, each {} must specify what variable it references. The variables are numbered (such as "#1:$NAME$"), and the same number is used for blocks that depend on that variable (such as "#1:{he[m] | she[f] | it}"). Here is a torturous example of using lots of pronouns in a complex sentence with three variables:

"#1:$PLAYER1$ gives #2:{the[!n]} #2:$ITEM$ to #3:$PLAYER2$. #3:{He[m] | She[f] | It} thanks #1:{him[m] | her[f] | it} and goes about #3:{his[m] | her[f] | its} business."

This works out to "Bob gives the sandwich to Sue. She thanks him and goes about her business." Or, if Sue gives the sandwich to Bob, the gender of all the pronouns is reversed.

We can also do fancier things. Sometimes we want to refer to an object without even displaying the name of that object at all. If a variable is given a negative number, it does not appear in the string, but it can still be used by meta-cases:

"#-1:$FIRSTPLAYER$ The withered old man shakes his head sadly and says, 'I already gave it to #-1:{him[m] | her[f]| it}.'"

We also added a meta-tag that is automatically applied by the engine, instead of being applied by the string's authors. When a variable is a number, it is examined by the game, and if it is singular, it is given the [1] tag automatically. This lets us put both the singular and plural cases into the same sentence:

"You find $NUM$ {coin[1] | coins}."

This is a little like the [p] meta-tag, but [p] means the object is inherently plural, whereas [1] means that the variable is a singular number.

The [1] tag is an example of feature creep. In this case it's very useful feature creep, but nonetheless it's an example of improving a system to do things beyond what it was originally intended to do. The [1] variable has proven its worth many times over, but in general I tend to be wary of adding arbitrary new features into the engine on a whim. When it comes to human languages, it's never as simple as it looks. When you add a feature, you have to evaluate the underlying grammar assumptions you're making, and then have those assumptions validated by your translators, and then make any alterations necessary for foreign languages. Even the [1] tag has a special case: in English, only the number 1 is treated as a singular number, so only variables that are "1" get the [1] tag applied to them. But in French, the number 0 is also singular; when the client is running in French it has to tag both "0" and "1" with the [1] case. There are other languages with even more complex rules about singular digits; if we had been translating to other languages this feature may have proven impossible to add.

Some underlying wisdom can be gleaned here: first, don't assume you know how other languages work unless you speak them fluently. Consult your translation team a lot. They care about translation quality and will be happy to help you get it right. Have your translation team available as early as possible so that you have them available for consultation early in the project, when you still have time to add or change features as necessary. It's also important to know what languages you should be targeting as soon as possible. Even a somewhat over-ambitious list is better than a list that omits languages you end up needing to target. Better to be safe than sorry.

Because meta-languages are dramatic simplifications of complex human languages, thereare many times when they cannot cope. If you can foresee problems with your game's features, talk to your translators and come up with a plan before you start coding. For AC2, the biggest problem was our randomly generated treasure. After consulting our translation team, we decided to name our treasure using the form "$ADJECTIVE$ $NOUN$ $PHRASE$" ("Mighty Helmet of Extreme Comfort"). If we had chosen a different form, like "$ADJECTIVE1$ $ADJECTIVE2$ $NOUN$" ("Mighty Comfortable Helmet"), translation into German would have been much more difficult, because German adjectives modify each other's conjugation form.

But even though we chose the simplest form for translating, it was still not as straightforward as we'd hoped. The meta-language had a hard time coping with random treasure items that were used in other sentences. If we wanted to correctly use an "Accurate Sword of Malaise" in a string, we needed to somehow tag this item with the [v] tag. "Bob's Sword of Maiming" needs to have the [n] tag automatically applied to it. That way when we get to a string like

"You pick up {an[v] | the[!n] | a } $ITEM$."

the random item works correctly. So we had to add a feature to the meta-language so that it automatically pulls certain tags from certain words. For instance, if the adjective contained [v] or [n], the final object would have those tags. If the name component had [m], [n], or [f], the final object would get those tags. If the phrase on the end had the [s] tag, the final object got that tag. These rules were different for each language, but were easily specified in a per-language configuration file. This wasn't a hard feature to add, but because we realized the problem very late in the development cycle, it was a pain to schedule even the single day of work needed to add the feature. Then we had to spend time explaining how the feature should be configured by the translators, documenting it, etc. This should have been done earlier.

A couple of face-to-face meetings early on would have saved us some headaches, but in the end the meta-language proved worthwhile. The entire system took two weeks of programming time to implement. It took additional time to teach the designers and translators how to use it, of course, but the result is very high quality text in all of our target languages.

As more and more MMORPGs hit the market, they will begin to differentiate themselves by their level of polish as much as by which features they offer. MMORPGs are unique among games in that they are constantly improving and can add features over the lifetime of the game - typically at least three years. But after the game ships it is impractical to rework all the strings to incorporate a meta-language. If it is going to be done, it has to be done during development. As such, I'm very glad that we were able to pull it off. Our translators are very pleased with the results, too. As a simple solution to a complex problem, a meta-language is worth investigating for any text-heavy game, not just MMORPGs.

Implementing the Meta-Language

The meta-language runs on the client in real-time - that is, the grammar parses every string right before it is displayed to the player. It's implemented as a simple black box interface that takes a StringInfo as input and returns a string as output. I implemented the meta-language with a yacc-like tool. I was unable to use a lex program as my tokenizer because the input was Unicode text and none of the lex tools I could find supported Unicode input very elegantly. So I just wrote my own tokenizer.

The yacc grammar started off as astoundingly simple, almost not worth the effort of using yacc at all. A top-down parser would have taken only slightly longer to write. However, near the end of the project when I needed to add a few last-minute features to the meta-language very quickly, the yacc grammar proved its worth.

I spent a bit of time working on the tokenizer to make it as user-friendly as possible. For instance, when someone wants to display a string with a '#', '{', '}', etc. in it, they are supposed to put backslashes in front of those characters - entering '\#' to display '#', for instance - so that the meta-language can discern which '#' are part of a meta-language expression and which are characters that are supposed to be displayed. However, since '#' can only be used in one place as part of the meta-language, the parser automatically decides that if the '#' is out of place, the user probably just forgot to precede it with a backslash. The meta-language was configured in such a way that you very rarely need to actually remember to insert the "\" before a special symbol.

Also, the meta-language automatically combines whitespace. If given

"You die!"

it will return

"You die!"

This proves useful for allowing arbitrary spacing in between blocks of meta-language to improve legibility. Thus the designers can type

"When you pick up {the[!n]} $ITEM$, {he [m] | she [f] | it } explodes in your hand!"

without having to worry that there will be two spaces after "he". They can put extra spaces wherever they want and the string will come out okay.

One feature we thought we would use a lot turned out to be very rarely needed. If you precede a variable with a ^ symbol, the first letter of the variable is automatically capitalized (using the towupper() C function). That way you could say

"^$NAME$ bows gracefully."

And the first letter of $NAME$ would always be capitalized correctly. However, it turned out that we almost never began a sentence with a variable, and when we did, it was usually in situations where we knew the first letter was going to be capitalized already anyway. In all, we needed to use the ^ feature exactly once out of all of our thousands of strings. On the other hand, we didn't expect to need the negatively numbered variable feature (which lets variables be invisible in the string) much at all. It was really just added for completeness, because it was trivial to add and it might prove handy. We ended up using that feature quite a lot, and in many creative ways. In general, my guesses as to which parts of the meta-language would be most useful were completely wrong.

The documentation, however, proved very useful all the time. We gave it to the translators, modified it based on their comments, and kept it up to date as we added features. Good documentation is completely mandatory for something that you're going to hand off to another team - in this case the translators. However, one mistake I made was to use very technical terminology. Some of the translation team quickly understood how it worked, but others took a while to understand it. It pays to remember that translators tend to be brilliant people but NOT programmers. Hence the documentation should be as approachable as possible and use lots of relevant examples. Meta-languages are relatively rarely used in games because they are only appropriate for games with lots and lots of text; thus it is quite possible that your translation team will never have worked with one.

That points out another possibility - that your translation team will balk at the idea of using a meta-language. This is an understandable position because a lot of the effort will be put on the translator's shoulders. Your designers need only write the strings once; the translators will need to rewrite those meta-sentences for every other language. This will definitely increase the translation time somewhat, and your project simply might not be able to afford it. In this case you can at least use the meta-language for English. The translated versions would not have correct pronouns and modifiers, but you can still get them right in the English sentences. Simply write a script that goes over all your strings and condenses any meta-language into the last case from each section:

"You take {the[!n]} $ITEM$ and store {him[m] | her[f] | it} away."

Becomes

"You take the $ITEM$ and store it away."

And then give these resulting strings to the translators. Of course, if you do this you can only use the meta-language for simple grammar bits, such as "a", "an", "the", "he", or "she." You cannot insert alternate verbs or adjectives, like one of the earlier examples that allowed both "You murder him" and "You destroy it" in the same string. Anywhere that you would have used the meta-language to insert verbs or adjectives like that, you must instead create separate sentences for each verb.

Talk The Talk

The importance of high quality sentences is a subjective thing and varies between individuals. To a large extent, then, the need for good grammar depends on your target audience. It certainly isn't mandatory, especially if your game doesn't rely heavily on text. On the other hand, it isn't as difficult as it may seem.

The biggest concern people had when we were developing the meta-language was that it sounded like it would greatly increase the time needed to author strings. Actually, though, only the combat messages and item usage strings need to have meta-language in them. Big blocks of NPC dialog, exposition, or descriptions tend to need absolutely no meta-language because they aren't interactive - those strings don't contain any variables.

The meta-language was perhaps the most interesting part of the localization effort, but not the hardest. A fair amount of time was taken in integrating the IME to support Korean. Even though our publisher (Microsoft) provided us with plenty of assistance in this area, we still had to integrate it into our engine, which proved tricky. But the most time-consuming task of all was the documentation. The translators needed a dictionary of game-specific terms, of course, but more than that, they needed to know the context and possible values for every variable in every string. They needed to know when various ambiguous strings actually appeared in the game so that they could understand how to translate them correctly. They needed, in short, documentation on the strings themselves. Our string editor allows comments to be added, but most people didn't bother - we didn't realize their importance until the translators started sending us long emails daily, each with 20 or 30 ambiguous strings that needed comments. Then we'd have to dive into the code and figure out where that string was actually used. It would have been a lot less stressful and time-consuming if we had just added some quick comments when the strings were originally added. The hardest part of localization on our end was just preparation and organization - a task that will come easier next time.

Resources

The Unicode Standard Version 3.0, from the Unicode Consortium. Good to have for the reference CD, if nothing else. Partially available online at http://www.unicode.org/unicode/standard/standard.html

The Inform Designer's Manual, 4ed, by Graham Nelson - Inform is a system for writing text adventures, and has multilingual support. The designer's manual has some interesting discussions on meta-grammar concepts. Available on the web at:
http://www.inform-fiction.org/manual/html/index.htm

______________________________________________________