ON GOING: Methodology to Evaluate Gameplay Design in Action-RPGs

Well, Hi. My name is Gustavo Martins de Souza, I'm from Brazil, a not-so-well game development culture country and I'm trying to do some effort in this area (and trying to conclude my university course as well). So, I'm doing a open framework of game design evaluation with some objectives:

1 - Improve the quality analysis of game design (on my country if so).

2 - Approach game design of user and critic opinion.

3 - Keep together and entwined the concepts of methodology analysis and heuristic evaluation, keeping on the pace the pre and post design phase.

So I will tell you what I did until now, what is next, what is my purpose and so on:

What do I have now:

I started by making my theorical embassis on three topics: Methodologies that already exists for data extracting about design and another characteristics, to make a data organization and definition of gameplay; critic opinion of action-RPGs on three critic portal websites (IGN, GameSpot and Destructoid) by comparing three revisions of three games of the genre for three revisors of each portal (yes, I like the number three <3), so making some pre-heuristics that I will use later to refine the heuristics that I already extracted; and the studies of heuristics from Federoff (2002) until now, extracting the heuristics that, by the gameplay definition, have relation with that.

Why ONLY gameplay and ONLY action-rpgs:

It's a first start approach and with only one year to do, so I had (by orientation) to keep up only one single focus and, if allowed, increase it in post work.

What is next:

As I said, one of the objectives is the approach to final users (not only test users), what they expect and their relation with the heuristics. To do that, I will pick up ten gamers with multiple preferences (as I said, only one year =( ) and do two qualitative questionnaires: one BEFORE the gameplay experience to define the game perfil of the subject AND one AFTER the gameplay experience based on the heuristics but trying to extract their opinion about the importance of each one. With that, I will have the last part of the embassis and will start to work on the method by refining the heuristics that are generic with the opinion of the critic and users.

Gameplay and Action-RPG definition:

By Gameplay definition, since many writters have different opinions by exactly what it means, we defined it as a three-part aspect: Rules of the Game; Controls and GUI. We do not consider mechanichal issues as we evaluate gameplay, even if they maybe have relation with the controls or with the GUI.

As for action-rpg, we defined it as the game genre with enphasis on character progression and a gameplay based on reflex and fast paced action NOT THE INVERSE, so a game with emphasis in the fast paced action but with some RPG characteristics will not count as an action-rpg, but as an action or action-adventure game with RPG characteristics (even now, as game progression is a must have [deadpool words XD])

The Games Used on the User Test:

The games are still on definition. Since we want to englobe most game types as possible, we do not want to use AAA games, as they are most for hardcore gamers (In here they REALLY are for hardcore gamers, casual types usually use Indies and with really low hardware needs, as for consoles, people who have an actual-gen console are considered riches, believe me).

The Final Purpose:

With all that embassis, the objective is to define a method to trace the gaming perfil of the target public (not considering aesthetical and mechanical aspects YET), refine the heuristics with the critical opinion and user opinion, implement importance points for them based on the definition of gaming perfil (for example, as Dark Souls was made for a hardcore public, the punishment is constant, as with must not be on a more popular game, for example Skyrim or Torchlight 2). The data extracted with the methodology can be used to make a topic approach to data analysis or a direct text, even if you want to describe all the idea of the design and even use that to extract user opinion by publishing these direct texts and asking for it. The game genre can be exchanged by shifting the gaming heuristics.

So...what is your opinion about it?