Foreword

In a recent article, I put forward the notion of “Compression and Funnelling” – a technique which offers a rational approach to the design of 2D levels. In the concluding section of that piece, I discussed the potential for the theory to be applied in the contexts of 3D level design to help augment the already potent notion of portals and occluders. This article extends the concept of compression and funnelling by examining the “kinaesthetic” language associated with the concept. This article also explores some of the underlying theoretical concepts that where intentionally left out of the previous article.

An Introduction to Movement

From a purely post-structuralist view, the world is a creation of our own minds. All objects are given meaning by us and our own experience. From an aesthetic perspective, visual stimulus is often powerful simply because it triggers an existing memory or experience in the gamer, not because the visual stimulus has any inbuilt meaning. From an aesthetic design perspective, this is somewhat problematic as to design effective aesthetic forms, we need to carefully understand the existing experiences of the particular target demographic. For the most part, we can base these inferences on common sense. However, by only focusing on static aesthetic elements within a game environment, are we missing an essential step?

In games, moving an avatar consists of two distinctly different processes. In the case of a 3D mesh of a bi-ped, when we go to move the object, the game engine moves the asset within the 3D world and a related animation is played at the same time. The animation is often carefully considered from a semiotic perspective and the code that moves the object is perfectly functional – however what is rarely discussed or thought about is the emotion that gamers take away from the movement of various on screen objects in relation to each other.

Often games are designed as systems, with various elements coded or created in isolation – an animated mesh can be observed walking on the spot and the movement of the mesh in the game world is left up to another independent system, developed by another person. The human brain is interesting in the way it processes movement and movement information. For movement to be meaningful, we use a model of comparative visual cognition which compares a moving object to other objects in an environment in order to understand its motives and motivation.

This particular article proposes a model of analysis which helps designers and animators alike understand the often, taken for granted elements of the semiotics of movement. For some, this will be an exercise in identifying the obvious, however it is an often overlooked element of game design and animation, especially from a novice perspective.

The Theory

Semiotics is a field of study concerned with understanding how meaning and emotion is conveyed using a number of different stimulus materials. To add context to this particular piece, my original academic background is in the study musicology which is largely concerned with the semiotics of sound (not just the semiotics of composition). Once my own interest turned to the analysis of games, I quickly began to adopt the systems that I had learnt in the semiotic analysis of music. This interest eventually led to the creation of the term “viseme” which is derived from musicologist, Phillip Tagg’s (2004) usage of the term “museme” to identify a short, section of music which infers a particular affect; the term “Kinetic Viseme” is a term used to define the relational difference between objects on screen without semantic interpretation of the object’s individual form or appearance impacting on the underlying relational meaning.

Barthes (1964, p. 10) acknowledges the ability of “objects, images and patterns” to signify on a “large scale,” albeit with conscious need for “linguistic admixture.” Birdwhistell (1971, p. 101) as cited in (Elam, 1980, p. 63) observed over 20,000 facial expressions and found that within the kinetic systems of body language lay kinemes, distinctive units of movement which are comparable with phonemes. When combined, kisemes form “kinemorphic classes” (ibid), similar to how letters are the building blocks of words. Barthes (ibid) notions that these individual component initially cause a process of signification that it is then linked to a semantic definition when combined. In this regard, we can see that kisemes initially cause a moment of sensation, followed by affect once the conscious mind is able to combine these smaller elements and link them to semantic definitions. However according to Elam (ibid) “Birdwhistell insists however, that any comparison is formal rather than substantial: there is no direct correspondence between kinesic and linguistic units, but only a more general analogousness between language and movement as overall syntactic systems.”

Daniel Ster(1973, p. 117) provides rationale for why Kinetic visemes should be treated as unique from other types of symbols stating that “while language is a “symbolic event” kinetic units are not symbols.” Birdwhistell (1971, p. 117) as cited in (Elam, 1980, p. 64) also states that kinetic visemes make “statements about the context of the message situation,” and not necessarily providing any well defined semantic definition. Elam (ibid) goes on to state that “such signals [kinetic visemes] serve in the first instance to draw attention to and so designate the protagonist in an interaction sequence, placing him in relation to other present and to the communicative situation.” Finally, Elam (1980) provides his ultimate justification for the differentiation of movement to visemes by stating the importance of the kinesic message:

At the same time, kinesic messages have their own material and formal autonomy even in the most memetic of performances: the establishment of patterns and recurrent movements, rhythmical variations, changes in tempo and constellations of positions has been an essential part of the aesthetic information of the spectacle throughout its history and has, indeed, been the very “raison de-tre” of highly visual displays (p. 69)

 In the case of this piece, kinetic visemes are a form of symbol that identifies affect by analysing the spatial relationship of one or more object(s) over time. The metrics of this particular symbol are based around the work of Elam (1980, p. 66) and can be defined as the following:

Kinetic Visemes are based on kinaesthetic semiotics, which originally began as a style of understanding the semiotics of movement in dance and theatre. Semiological studies of theatre started with the works of Jean Alter, Keir Elam, Erika Fischer-Litcher, Patris Pavis and Jiri-Veltrusky (Aboo Backer, 2007, p. 21) and culminate in recent writings by Arts scholars such as Jane Desmond in her book “Meaning and Motion: New Cultural Studies” (1997). The main reason for the using the different term of Kinetic, rather than Kinaesthetic, is that theatre and dance studies have an additional focus on form and aesthetics whilst the model being proposed here is solely based on movement of nondescript objects, and are hence devoid of any bias introduced by aesthetic representation.

Theory by Example 

Kinetic Visemes are particularly useful in video game analysis as an observer can identify relationships and subsequent affect based on this relationship. To observe the significance of Kinetic Visemes, one must first strip back the viseme layer of representation to only show the kinetic aspect of representation. To do this, let us analyse the Kinetic Semiotics of a small section of a boss fight from the Shmup Ikaruga.

Figure 1

Figure 1

Figure 1 represents the first object in the Kinetic Viseme. “Object” A is emitting a consistent string of fast moving sub-objects all travelling along the same vector, as represented by the red arrows. As these objects are moving away from the larger object, we can infer ownership, or a relationship between the two sets of objects. The fact that these sub-objects move away from Object A at a high velocity infers the power of object A over the sub-objects. Power is also conveyed by observing that the vector at which these sub-objects move away from object A is consistent and regimented as if they were soldiers following strict instructions. Now let’s look at the next layer of objects in the exact same scene.

Figure 2

Figure 2

Figure 2 is the very same scenario, with the addition of Object B. If we look at the movement of Object B over time in relation to Object A and its sub-objects, we can see that Object B has a tendency to move away from Object A. As the movement vector of Object B changes a few times we can also infer a lack of certainty in the movements of Object B, as if it were at the whim of Object A. From this series of Kinetic Visemes we can infer that Object B is inferior to object A, as its spatial movements seem to be dictated by other objects rather than it dictating the portioning of virtual space. The affect would therefore be power, Object A being the representation of power of oppression as is demonstrated by its dominance over Object B.

Figure 3

Figure 3

Alternatively, Figure 3 identifies a set of Kinetic Visemes that demonstrate how Object A could become subservient to Object B. In Frame 1 of Figure 3, Object B encroaches on Object A with a consistent approach vector demonstrating equal aggression and confidence. In this example, Object B has identified with the aggressor and has begun to emit a consistent stream of sub-objects aimed directly at Object A, inferring a sense of purposefulness. Frame 2 of Figure 3 is the next step of the Kinetic Viseme, whereby Object B continues to defy Object A’s monopolization of virtual space by further encroaching upon it in the same consistent, powerful manner. Finally, Frame 3 of Figure 3 demonstrates Object B’s dominance of Object A as it forces Object A to quickly move away from Object B without emitting any type of sub-objects towards it. This particular kinetic visemes first demonstrates repression of Object B, then defiance and then finally dominance of Object B.

Figure 4

Figure 4

Figure 4 is the very same scenario as depicted in Figure 1, Figure 2 and Figure 3 with the iconic, indexical and symbolic visemes in place. What this aims to demonstrate is the importance of Kinetic Semiotics in order to understand the relationship qualities of certain objects within video games. Interestingly, the above scenario could be represented using a number of different visemes, however the affective quality of the relationships being demonstrated on screen would remain the same. Importantly, Kinetic Semiotics demonstrates relationships. Once one can observe the relationship, they can ascertain the relative affective quality of that particular relationship. In the case of video game and especially Shmups, Kinetic Semiotics is a way of observing power and dominance.

The Missing Link

At the beginning of this piece I discussed how we take semiotic meaning away from aesthetic experience, yet movement and its importance as a communicator of emotion goes largely under the radar. As there are no existing models or exemplars, I have prepared a selection of very basic movement relationships and listed them by the emotions that they convey. In all instances, the examples below are taken from an above, third person perspective. It is essential to point out that emotion will only be conveyed in a first person perspective, so long as the player can compare the relative positions of objects on screen.

Purposefulness: Figure 5

Figure 5

Figure 5

Fast, purposeful movement of an object infers confidence and to some extent a sense of empowerment.

Indirection: Figure6

Figure 6 

Movement that is erratic and inconsistent suggests a relationship with the environment in which an object is confused. This can also be interpreted as indecisiveness and lacking in power or control.

Forcefulness: Figure 7

Figure 7

Fast, purposeful movement of an object away from a stationary object infers a hierarchy of power. Object B belongs to Object A because it moves away from Object A in a controlled, linear and purposeful manner. As its vector does not change, it infers that it has no free will or sense of empowerment over itself: the object at its point of origin therefore determines its centre of control.

Forcefulness and Control: Figure 8

Figure 8

When an object (B) shares the same movement vector as its point of origin (A), Object B can be determined to be a sub-object of A. Affectively, as the sub-object shares the same movement vector, Object A can be deemed to be forceful and direct.

Lack of Control: Figure 9

Figure 9

If a sub-object (B) moves away from its point of origin (A) but does not travel on the same movement vector, the relationship can be interpreted as accidental, similar to the way objects react during an explosion. In this case, although Object B can be identified as a sub-object of Object A, the movement can be seen to be unintentional and hence almost accidental.

Playfulness, Disempowerment: Figure 10

Figure 10

Alternatively, indirect, erratic and inconsistent movement away from an object can infer that the object at the point of origin has little or no control of the sub-object and is not a source of power or control. This type of relationship can also depict child-like playfulness as Object B does not try to control the space which Object A controls; rather Object B questions the authority of Object A, indirectly.

Victim Relationship: Figure 11

Figure 11

Playfulness can also turn to evasion depending on whether the space has certain occluders in it that prevent line of sight or inhibit movement (Objects C, D, E and F). In this example, Object B can be seen as being “scared” of Object A as it attempts to evade Object A by putting occluding objects between them. This relationship infers that Object A is dominant, however Object B is not subservient to Object A, but rather terrified by it. This terror can also be observed in the erratic movements of Object B. A lack of constant speed and movement vectors infers a state of panic and lack of purpose.

Hunter /Prey Relationship: Figure 12

Figure 12

When one object (A) uses occluders (C & D) to evade another object (B) this can be a sign that Object A is the more powerful in the relationship as it is aware of B, but B is not aware of A. This type of relationship is that of a hunter and its prey; there is no indecisive movement and everything on the hunter’s side (Object A) is planned and methodical.

Power Relationship: Figure 13

Figure 13

Repulsive forces around an object which cause other objects (not necessarily sub-objects) to move away can infer power and dominance of the main object (A) and submissiveness / fear of the other objects (B, C, D, E). In this case, Object A is a force to be reckoned with and holds the power in this scenario.

Inferior Relationship: Figure 14

Figure 14

Inversely, when an object (A) is pursued by faster moving objects along the same movement vector, Object A can be deemed to be the inferior in this relationship, especially when other objects (B, C, D, E) move as close as possible to Object A as fast as possible.

Mutual Relationships: Figure 15

Figure 15

If Object A and Objects B, C, D & E move towards each other with consistent speed and a purposeful vector, the relationship can be seen as mutual on both sides. However, superiority of one object can be observed by the total distance travelled by the objects in relation to each other. In this example, Object A is the dominant because it chooses to travel less distance in the same time when compared with Objects B, C, D & E. However, the key element that differentiates this Kinetic Symbol from that of aggression is that all parties will decelerate logarithmically before touching.

Fearful Relationships: Figure 16

Figure 16

When objects share common approach vectors, they can be assumed to be part of the same family of objects. In this case, Object A is the focal point of all approach vectors for all other objects. This infers that everything other than Object A is part of the one group of objects. This can make Object A seem like a “god-like” entity as all other objects feel the need to challenge Object A’s space. As such, Object A can be viewed as not the most powerful, but that with the most potential.

Fight or Flight Relationships / Underdog Relationships: Figure 17

Figure 17

In this example, as there is no central focal point for approach vectors, it is difficult to ascertain which objects are more significant than others. The only way to deduce the superiority of objects in this scenario is to observe over a longer period of time and see which objects become deleted first. The last object standing can be considered the superior; however this scenario demonstrates that no one object is perceived to have any higher power or potential power than another.

Practical Applications  

The most obvious practical application for this theoretical framework is in games which have limited resolutions or frame budgets. These types of games do not have the ability to convey semiotic meaning as effectively as their high resolution peers due to a lack of detail in the individual world entities. This lack of detail is often manifested in reduced polygon counts, resolution and texture depth meaning that the visual semiotic channels reserved for facial expression are compromised and need to conveyed using other means.

Based on personal experience, many student game makers looking to create third person, 2.5 games often spend far too much time creating detail and animations for entities which only occupy five to ten percent of the screen. A recent example of this was when a 2.5D, Contra styled game was pitched using extremely high resolution mud-box meshes for in game characters. Although a Mudbox mesh is often an ideal starting point for games with high resolutions, large frame budgets and equally large financial budgets, any detail in the mesh which is meant to convey semiotic meaning will be compromised by how small it will be in the game world.

Kinetic semiotics can be invaluable in any game where it is difficult for the player to perceive any small scale changes on the in game entities by themselves. As such, any third person game (or first person) game can benefit by factoring kinetic semiotics into the design and subsequent implementation. Higher resolution titles have also benefited from actively thinking about how movement relationships impact on the player.

The controlling powerful, controlling movements of Wesker during the second boss battle in Resident Evil 5 are effective on an emotional level and convey additional meaning beyond the normal representational channels. Serious Sam is also another interesting example as the player is constantly disempowered by the fact that they are constantly forced to backpedal in the game world.  Sure Serious Sam is fun because of it ridiculous amount of enemies, but does this disempowerment as conveyed through kinetic semiotics come at the detriment of creating a satisfying play experience? By using the metrics provided within this article, the evidence would suggest that the experience of playing Serious Sam is not about empowerment or disempowerment, but rather anxiety and relief.

Concluding Thoughts

If the assumption about the potential for kinetic semiotics to create sensation is true then third person games stand to gain the most from precise application of well thought out kinetic semiotics and this has been explored somewhat in the previous section, but only in regards to practical applications. One of the advantages of the first person experience is that it literally involves the player in the play field – they are them, looking directly into a virtual world which is full of conventions and practices similar to their day to day lives. Third person games on the other hand rely on a player being able to identify with their chosen avatar and subsequently perceive the virtual world through this abstracted third person perspective. Put into personal terms; I am me in a first person game, and this is me in a third person game. Therefore anything that can decrease the amount of conscious cognition required in a third person experience would be beneficial to the player’s sense of immersion. Even if the connection between kinetic semiotics and the sub-conscious proves to remain problematic, there is room to explore the potential of sensation in regards to spatial relationships of onscreen objects within games. Research conducted by Calvert and Tan (1994) on the impact of observation versus interaction in virtual environments is a good starting point for anyone considering pursuing this further and goes some way as to proving the legitimacy of the theory.

 No matter where the discourse on the related theory goes, a holistic perspective of animation will definitely assist in providing powerful emotive states and as such, the more informed the production team is of these relationships, the more potential there is for establishing mood and most importantly – sensation.

Works Cited

Aboo Backer, M. B. (2007). Semiotics: Signs and Meaning in Contemporary Dance in Malaysia. WACANA SENI Journal of Arts Discourse , 6, 71-76.

Barthes, R. (1964). Elements of Semiology. London: Jonathan Cape.

Birdwhistell, R. L. (1971). Kinesics and Context: essays on body-motion. London: Penguin.

Calvert, S. L. & Tan, S. (1994). Impact of Virtual Reality on Young Adults Physiological Arousal and Aggressive Thoughts: Interaction versus Observation. Journal of Applied Development Psychology, 125-139.

Desmond, J. (1997). Meaning in Motion: New Cultural Studies of Dance (Post-Contemporary Interventions). (F. Jameson, Ed.) London: Duke University Press.

Elam, K. (1980). The Semiotics of Theatre and Drama. Methuen & Co. Ltd.: London.

Stern, D. N. (1973). Theatre and the Social Sciences. The Drama Review , 17 (3), 114-126.

Tagg, P. (2004). The Case of Anguish. Retrieved 17-Decemeber 2006 from Media Music Studies: www.mediamusicstudies.net/tagg/articles/musemeuse.html