Usability evaluation for video games: Formal evaluation methods

[In this reprinted #altdevblogaday in-depth piece, European University Cyprus's Georgios Christou continues his discussion on video game usability evaluation by looking at formal evaluation methods.] In part one of this series, I talked about Norman's seven stages of action model (Norman, 2002), and how that can be used to start an informal usability evaluation on the part of the game designer/developer, after the User Interface (UI) of a game is pretty much completed. In this post, I will talk about more formal evaluation methods. Usability evaluation can occur during various times during the design and development lifecycle of a game – early, in the middle, late, and at the end. However, not all types of usability evaluation methods can be applied during all phases of design and development. Also, different methods of evaluation can lead to different results about the UI of a game, so it is crucial to decide when and how to evaluate our games, or we run the risk of not getting our metaphorical money's worth out of the usability evaluation. Types of usability evaluation There are many different types of usability evaluation, all aiming to test different parts of our UI design. Let me discuss two, that are common, and then describe the process for one of them, that I think is more feasible for indie developers and small houses. The first type of usability evaluation is called Expert Review, and consists of experts in the selected type of usability evaluation going over the game design. This evaluation's goal is to bring about errors in the way that the UI has been designed. It may, for example, answer questions such as: are all allowable actions reachable whenever they should be reachable, are the UI elements laid out in the way that they should be according to some UI design specification that has been decided, and others. The drawback to this type of evaluation is that to carry it out we need the help of an expert in the usability evaluation method that has been decided. Thus, for many indie developers and hobbyists this is a type of evaluation that is not usually feasible. On the other hand, this evaluation can proceed as soon as the first draft of the UI is laid out, not necessarily as a prototype. And it saves huge amounts of time after the game development has been completed, because once the UI errors have been caught as early as the design stage, there is little to no coding required to fix them. A second type of evaluation, and the one that I will focus on, in this post is one that follows the protocols of psychology experiments. This type of usability evaluation requires players that will play the game, under a controlled environment. By a controlled environment, I mean that they should all play it on a machine with the same specs, under (as much as possible) the same conditions, and for as much time as is decided to be meaningful for the game in question. You may, for example, decide to allow players a fixed amount of game time, or you may give them infinite game time, as long as they manage to complete specific tasks that are given to them before they start playing. Or it could be that the players need to complete one level in the game. Bear in mind that the amount of time played, and the amount of controls used to play the game will ultimately decide how good the results are for the evaluation that is taking place. But even before the players are called in to play the game, the person responsible for the evaluation will need to answer a few questions about the usability evaluation. Usability evaluation design This section's advice will refer to the second type of evaluation that I briefly touched upon earlier. I will try to explain how this type of evaluation works, and why usability should be in every game designer/developer's list of things to do. The first question that the evaluator needs to answer is "What do I want to test?" This will be the goal of the evaluation, and it could range from overall ease-of-use, to ease-of-learning, and could even go into other features of the game, such as correct level design, visibility of the obstacles and objectives, understanding of the gameplay, and whatever else you might think! It is also possible to perform a usability evaluation that targets more than one objective. Once the first question is answered, then the second question is "What is the meaningful amount of game that will give me results that make sense?" This question prompts the evaluator to think about the logic of the game and how the player interacts with the game at various intervals. For example, in an FPS, allowing players to use only one weapon during the play time, does not reveal anything about how the player will interact with the "change weapon" control, nor about how each weapon in the game affects the player's perception of the gameplay. Finally, there's always the "I need more participants" question. The more people attending the study, the better (and usually more representative) the results are. However, remember that the goal of the evaluation is to identify specific aspects that have been decided when the first question was answered. So my answer to this question, especially for someone with limited resources is: find as many people as you can, and stop testing once your data give you a sense of what you are doing right and what you are doing wrong. In this post I have not described the way a formal usability evaluation occurs. The reason is that my goal is to get people started performing usability evaluations, and hopefully convincing a few game development teams that usability evaluation is not something that should eat up a lot of time, or that it should be something difficult to perform. On the contrary, usability evaluation should be part of standard testing, because the benefits will outweigh the costs. In fact, it has been proposed that for every dollar spent implementing usability techniques, the organization will realize a benefit between $10 and $100 (Gilb, 1988). And while your mileage may vary, your players will definitely thank you for it, and maybe even support your next effort! References Gilb, T. (1988) Principles of Software Engineering Management, Addison Wesley. Norman, D. (2002). The Design of Everyday Things. Basic Books. [This piece was reprinted from #AltDevBlogADay, a shared blog initiative started by @mike_acton devoted to giving game developers of all disciplines a place to motivate each other to write regularly about their personal game development passions.]