Semiotics and Intertextuality of Videogames

CONTENTS

INTRODUCTION

A SHORT PRIMER

PLAYING WITH SEMIOTICS

CODES IN VIDEOGAMES

INTERTEXTUALITY

CONTINUE/CONCLUSION

BIBLIOGRAPHY

.................................................................3

.................................................................6

................................................................11

................................................................19

................................................................27

................................................................23

................................................................29

2

3Semiotics and Intertextuality of Videogames

There is terrible irony present in the scholarly study of videogames: though the

medium is an effective communicator, it requires a tremendous amount of communica-

tion to convince those that hold it in derision, and think of it only as a child’s play-thing.

The videogame is a medium that speaks to its generation, its culture. It possesses a

language all its own, and yet due to its progenitors, it can speak to the same discourses

of traditional media. Videogames are a model of expression that utilizes several modes

of communication within itself. The visual: text, words and narratives borrowed and

derived from a great wealth of literature and invention; computer-generated imagery,

evocative of animation and film techniques and styles. The aural: sounds, both in the

realm of diegesis and beyond, used to manipulate ambience, depth, and function; more

recently, human voice acting, further blurring the line between the videogame medium

and one of its antecessors, the film. The haptic: the tactile sensations between a player

— the medium’s actor — the physical topology of the controller, and the medium itself.

These modes coalesce to provide the user feedback at prescribed moments when

meaning can be inferred.

The videogame is a profoundly sophisticated means of communication, even

more so than aforementioned. “Games are models of experiences rather than textual de-

scriptions or visual depictions of them. When we play games, we operate those models,

our actions constrained by their rules...Videogames are a medium that lets us play a role

within the constraints of a model world,” (Bogost, How to do things 4). While I would be

cautious to downplay the significance of the textual and visual elements of the videog-

ame, it is impossible to ignore the intellectual significance of the medium’s gestalt, the

encapsulation of distilled experience. And yet we do. Or at least, we have. The academic

inquiry of new media, and consequently, videogames, is dwarfed by the stalwart canons

of liberal studies. Due in part to its relative adolescence, with the earliest non-commer-

cial games dating back to the 1960’s1, and perhaps as well, its general perception among

the masses. “(Videogames) are a part of the ‘entertainment software’ industry, and

1There are examples of computer games (we cannot really call them digital because of the analog computers used) being created from the early 1950’s, like the tennis simulator, Tennis for Two. This inspection of the medium examines more commercialized titles, beginning with Pong, released for the Magnavox Odyssey in 1972.

4 Semiotics and Intertextuality of Videogames

they are generally considered a leisure practice by players and the general public alike.

Videogame play is considered an unproductive expenditure of time, time that fills the

breaks between works,” (Bogost, “Rhetoric of Videogames” 120). Regardless of the

validity of the medium, videogames are almost exclusively seen as a form of entertain-

ment, predominantly skewed towards children — not something to be studied by

academics.

However, I, along with others, do not hold that to be true. If McLuhan was cor-

rect in his magnanimous affirmation that the “medium is the message” (McLuhan 7),

and that the delicious, meaty morsels of content are communicatively superfluous, then

it does not matter that the medium is routinely used to slay dragons and play war; if

anything, the distractive tendencies of its content plays perfectly with McLuhan’s model,

as really any “gamer” would tell you, it is easy to become lost in those “model worlds.”

As I have expressed, I feel the videogame’s penultimate function is as a mode of

expression, of communication. But what is it to communicate with one another? How

are ideas formed, framed, and rendered from one mind to another? Because of the

videogame’s close relationship to the differing media and modes that compose it, I

propose that we can often use the same points of analysis to better understand its parts.

Umberto Eco wrote that “every act of communication to or between human beings — or

any other intelligent biological or mechanical apparatus — presupposes a signification

system as its necessary condition,” (Eco 9). This, I believe, is the essence of communi-

cation, a system of significations, called semiotics, which decode abstractions that we

have previously given meaning to, and encode further interpretations. “At the heart of

semiotics is the realization that the whole of human experience, without exception, is an

interpretive structure mediated and sustained by signs,” (Deely 6). The study of semiot-

ics, a contentious field of interdisciplinary research that spans across the social sciences

— moreover linguistics, psychology, and philosophy — attempts to explain the “action of

signs” (Deely 22). Early 20th century linguist, Ferdinand de Saussure wrote of and


created the term, semiology2, a synonym for the field to describe what he foresaw as a

field of academic interest. “A science that studies the life of signs within society is

conceivable; it would be a part of social psychology and consequently of general

psychology; I shall call it ‘semiology.’ Semiology would show what constitutes signs,

what laws govern them” (Leeds-Hurwitz 4). Saussure went on to write of critical fixtures

of semiotic studies that will be later described within a videogame context. Additionally,

Saussure’s contemporaries were also breaching the field. Charles Sanders Peirce, an

American philosopher, wrote of nearly identical relationships3, albeit more explicitly,

and declared his study “semiotic.” From these two scholars alone comes a tree of

academics interpretively defining communication.

There are several different families of semiotic research. The one investigated in

this paper is anthropsemiosis, a system used to describe “all of the sign processes that

human beings are directly involved in, and, looked at another way, names those sign

processes which are species, specifically human,” and as Deely puts it, “the one closest to

us,” (28). Can a relatively young academic field be used to explain an even

younger medium like the videogame? Absolutely, almost shockingly so. The videogame,

as I said, is a valuable method of communication, and as such, is a veritable trove of

signs — some obvious and familiar, and others foreign, but all work together through

their different modes to signify meaning. Much like the gestalic models that Bogost

earlier mentioned, “meanings in multimedia are not fixed and additive...but

multiplicative...making a whole far greater than the simple sum of its parts,” (Lemke

72). And the agency of communication in videogames is not derived from a disorderly

mess of signs, but rather a hierarchy. When Michel Foucault spoke the relationship of

the verbal (referring to text he observed) and the visual, he said, “what is essential is that

verbal signs and visual representation are never given at once. An order always

2 From the Greek semeion:, a sign, mark.3 Without knowledge or collaboration with Saussure’s work.


hierarchizes them, running from the figure to discourse or from discourse to the figure,”

(33). As I will describe, signs work in the videogame in much the same way Foucault

described paintings: in order to establish discursive, rhetorical structures, meaningful

signs must combine to form systems within videogames — these systems combining

to make larger systems, which in turn beget even larger systems. I will demonstrate,

starting with the medium’s basic semiotics units, that this organization creates a grid

of intertextuality, encompassing the videogame medium, the sub-culture surrounding

the medium, and the culture enclosing that — all of which is a part of the paramount

semiotic structure, human experience.

In order to analyze the role of semiotics in videogames, we first must have a solid

understanding of the subject’s fundamentals. As mentioned earlier the most succinct

definition of semiotics is a field in which examines the actions of signs. Interesting, you

may say, but what are signs? What sort of actions can they exhibit, and why should we

care?

The two aforesaid scholars, Saussure and Peirce, developed two distinct but

similar methods for analyzing semiotic structures. A sign is something that represents

something else. I say “something” not out of a lack of words or an inability to find

an appropriate one, but rather because what I am presenting is more an equation;

a proposition that states that any variable material or immaterial concept or object

is representative, significant, of another. Saussure in his book Course in General

Linguistics offered not only what a sign could be, but more specifically what constitutes

a sign. Saussure interpreted signs as a dichotomy, comprised of two parts, the signified

and the signifier. The signified is the “visible part”, or rather, the most obvious part, as

it is the “the explicit aspect of a sign, present during the interaction, a material presence

A Short Primer


of some sort,” (Leeds-Hurwitz 9). The signified is what you literally see when you gaze

upon the sign; literally, meaning not at all figuratively, as it is the figurative which is the

signified (even if there is a great similarity between the two at the time). The signified is

the “invisible part”, or the “tacit element of a sign, what might be termed an ‘immaterial’

presence, something literally absent yet functionally present because it has been

invoked,” (Leeds-Hurwitz 9).

I feel that it is difficult to understand the concept without the use of a visual aid,

so let us briefly consult a sign that we are all familiar with. The object in Figure 1A is a

cartoonish depiction of a tree, made with simple lines, curves, and angles, but when it is

looked upon and registered, the individual components that comprise it go unnoticed,

and instead you think, “tree” — oak, evergreen, sumac leaves green with rough, dried-

out bark, earthy smells. As the viewer of this sign, you play the role of the “social actor”,

one whom uses their knowledge of contextual evidence to extrapolate the action of

the sign. When you see the tree sign in Figure 1A, what you are viewing is the active

signifier: representing what you can gleam out of your cultural knowledge of what the

object should mean, the signified.

Figure 1A

Leeds-Hurwitz suggests perhaps a

better example in her description of Saussure’s

relationship: instead of a tree, let us imagine a

white wedding dress — an actual dress, and not

just a cartoon depiction as I presented before,

because a sign is a function, and not married

to the mode of the image. The wedding dress

signifies a wedding (that is, the dress meaning

the signifier, while the concept and practice of

the wedding is the signified). But there’s more to


this particular sign that could be said. Depending on the cultural situation of the social

actor that is interpreting the sign, a wedding dress could be representative of many

things — in some cultures the white color of the dress is indicative of purity which

extends into an idea of a virgin bride, while in other circumstances the wedding dress is

symbolic of the marriage itself and not just the wedding ceremony, and furthermore by

other’s interpretations, the dress could be representative of an industry geared to take

monetary advantage of lovelorn brides who want a perfect wedding (22-27). There are

many semiotic principles at work within this simple, conjured image.

First, the sign is dependent on context, whether it is contextually from the

location of its introduction, or the awareness of the interpreter. “The sign first of all

depends on something other than itself. It is representative but only in a derivative way,

in a subordinate capacity. The moment a sign slips out from under this subordination,

as frequently happens, at just that moment does it cease for a while to be a sign...

Thus on its own, it is a mere object or thing become object, waiting to become a sign,”

(Deely 35). Deely continues to say that without this content, or in my own preferred

terms, the context, “the sign ceases to be a sign.” He explains that it is still something,

often an “object”, as it cannot become inexistent (or if it can, that is a stasis more

suited towards philosophers like Deely to contend with), but instead it must wait to

be re-contextualized, to be given further meaning. What is important to realize here

is that signs cannot develop meaning on their own, “that objects, images and patterns

of behavior can signify, and do so on a large scale, but never autonomously.” (Barthes,

Elements of Semiology 10). Signs require human intervention and invention in order to

function and to be understood.

Second, it is necessary to note how many different interpretations of the wedding

dress that I alone was able to produce. As I have shown, signs are not created from a

void. They are reliant upon the context of their environments and intentions, and allow


for further definition by semioticians. Because the wedding dress serves as a signifier

to several different signifieds4, we can declare the wedding dress a polysemic sign.

Alternatively, if a signified is tied to multiple signifiers, it is polysemic as well, and signs

can often be both.

Figure 1B

There are other terms that dictate how a

sign is interpreted. Referring to Figure 1B, notice

how the same sign trying to evoke the tree is now

displayed in color, with a green top and a brown trunk.

“Motivation...refers to the degree in which the signified

determines the signifier,” (Lee-Hurwitz 26). Because

the sign more closely illustrates what an actual tree

looks like, the signifier is more closely constrained

to the signified, and is described as “being highly

motivated or highly constrained.” We can continue to use the tree example to describe a

sign’s attribute of “convention.” Convention is used to refer to a sign’s sense of tradition,

or how it is often used in a particular system. If one views Figure 1, they are likely to see

“tree” because that is what they are conditioned to do. Lee-Hurwitz explains that highly

conventionalized signs are often ignored due to their ubiquity, and include other popular

signs such as the male and female symbols. Also, when analyzing signs, it is imperative to

be able to distinguish “detonation” from “connotation”. “Denotation refers to the explicit,

obvious, straightforward, first meaning of a sign; the related term connotation refers to

the implicit, conventional, second meaning of a sign, imposed by a specific culture,” (Lee-

Hurwitz 27). If we reuse the example of the wedding dress from earlier, it is simple to see

that the dress denotes clothing, but it connotes wedding. Or at least, it connotes wedding

for the culture of the intended audience of this paper. “Denotation often crosses cultural

boundaries; connotation almost never does,” (Lee-Hurwitz 27).

4Which will differ depending on the type of reading occurring: dominant, sub-ordinate, oppositional, etc.


Of these semiotic principles that have been laid out, the most pertinent one to grasp

and isolate is the sign’s dependence upon human direction — an interpreter must be

present in order for a sign to be interpreted, but even more so is the sign’s dependence

upon other signs (which were earlier signified by people). Umberto Eco describes this

relationship as unlimited semiosis.

“In order to establish what the interpretant of a sign is, it is necessary to name

it by means of another sign which in turn has another interpretant to be named

by another sign and so on. At this point there begins as process of unlimited

semiosis, which, paradoxical as it may be, is the only guarantee for the foundation

of a semiotic system capable of checking itself entirely by its own means...the very

definition of ‘sign’ implies a process of unlimited semiosis...semiosis explains

itself by itself,” (Eco 69-71).

In other words, every sign relies upon another in order to be interpreted by the social

actors within its realm.

Eco, you may have noticed, uses unfamiliar terminology within his description

of the semiotic process, which he actually borrows from the aforementioned forerunner

of semiotics, Peirce. I chose to introduce Saussure first because understanding his

methodology allows us in turn understand Peirce’s viewpoints5. Peirce believed

the sign to instead be a trichotomy: “the sign or the representatum, the object, and

the interpretant,” (Leeds-Hurwitz 23). The representatum corresponds directly to

Saussure’s signifier and acts in the same way. However, instead of presenting the

signified as a complete unit of expression, Peirce splits it into two — the object, the

idea or figure that the representatum reflects, and the interpretant, “over and above

the unique essential structure that makes signification possible in the first place,”

(Deely 25), the invisible component that represents the meaning generated by the

representatum in reference to the object. The process that a social actor uses to

5A phenomenon not unlike that which was discussed with the interconnectivity of signs.


interpret, to go from the representatum to gather the interpretant about the object,

Peirce calls semiosis — a term now commonly accepted as a synonym for signification.

As Leeds-Hurwitz explains, Saussure and Peirce’s differing ideas are not grounds for

contradiction, “Peirce was simply slightly more explicit than Saussure,” (23). It is

because of the validity of both of these methods — or maybe as Deely would argue,

points of view — that I choose to examine the semiotic structure of videogames in

accordance with Saussure’s more direct system, but with the accoutrements of Peirce’s

further inspection of sign variations.

Playing with Semiotics With this primer of semiotic theory under our belts, we finally have a working

knowledge of fundamentals that can be used to analyze the semiology in videogames.

But again we should address: can we, and if so, should we? I feel that the disconnect

between new and old media is not a drastic one, and that all media (media in the

McLuhan sense, that is, almost everything) have a common lineage. “Rhetorics

inevitably vary by their substance (here articulated sound, there image, gesture, or

whatever) but not necessarily by their form; it is even probable that there exists a

single rhetorical form, common for instance to dream, literature, and image,” (Barthes,

“Rhetoric of the Image” 161-162). That rhetorical singularity Barthes references is what

I am referring to myself (because what is a rhetoric if not a chain of signifiers of an

“ideology”, as he would put it). These commonalities allow human communication to

be analyzed with generally the same methods — just as we analyzed a tree or a wedding

dress, we could analyze the latest Call of Duty.

So we are able to apply elder methods to new analyzes, but is there a point to it?

Do we stand to learn anything from close readings that we could not have gathered from

extraneous study of the medium? I believe there is still much to learn from individual

fields like semiotics through the inspection of unfamiliar environments.


As Bogost would retort, “The content and context of a media artifact is not as inessential

as McLuhan would have it. The medium is the message, but the message is the message,

too,” (How to do things 5).

For the sake of simplicity along with establishing a temporal continuity of sign

play in videogames, it is interesting to begin discussing semiotics with a game from early

in the creation of the videogame industry. Older videogames and those created by non-

commercial developers have a long history of using representational visuals to exhibit

meaning. Adventure (1979), a game created for the Atari 2600, allows the player — a

term specific to videogame culture and conveniently coincides with our semiotic idea of

the social actor — to control a small, colored square through a digital environment. This

square whom you control starts out in front of a similarly colored structure featuring

a forward-facing black grid and it all is surrounded by a semi-permeable perimeter of

rectangles (Figure 2). That’s how the game begins. There is no foreword in-game text

that provides context to your objectives, surroundings, or even who you are, and yet

without direct effort you are able to collect this context through semiosis. As you direct

your avatar throughout this game environment, you notice things; you realize that when

you go through openings between the perimeter’s boundaries the environment changes

its appearance either through its layout or the color (Figure 3). This signifies that your

Figure 2: Starting point of Adventure. Figure 3: Adventure - Environmental differences


character, this square, is able to traverse this area and you are able to differentiate

between various locations. But then upon your wandering you discover objects familiar

to you within the negative space that you walk in: door keys, arrows (resembling

directional arrows, like on a keyboard) and goblets.

In this realm of abstraction where all you can infer are shapes, you finally come

upon something of substance, a key. As a social actor from a system that understands

what that shape confers, you know what a key is and what it can do. Adding to the

significance of this object is a relationship unique to the videogame, the ability to

interact, manipulate the object. Your character picks up the key and carries it along

with them (Figure 4) while exploring the environment, but you decide instead to equip

yourself with the arrow to further investigate its uses, wherein you note that it points

Figure 4: Adventure - Interacting with found artifacts.

Figure 5: Adventure - Found objects interacting with others through semiosis.

towards you when you hold it. Along the

way you encounter an object resembling

a serpentine dragon, moving aggressively

unlike anything that you have seen before.

Just as you knew what the key was, you

are additionally able to decipher this sign

to be a dragon — a beast not known for

its benevolency — and this suspicion is

confirmed when it opens its jaws to attack,

followed with a harsh, deep rumble. You

accidentally run forward directly into the

path of the dragon forcing it to produce a

defeated noise signifying its demise. It takes

a new position symbolizing its condition

(Figure 5).


Using your innate semiotic skills you are able to gather many things about this

videogame from the encounter: first, you are in a world where dragons exist and

the tool you possess seemed to enable you to slay them. This jumpstarts a series of

significations much like the unlimited semiosis effect Eco described earlier. Using your

knowledge as the intended social actor for this game, you realize the arrow was a sword

the whole time, because what other than a sword to slay a dragon in popular culture?

This secondly, in turn, compels you to examine the sword more closely. The arrow that

we earlier saw is the signifier to the sword’s signified — the reason we did not realize it

sooner was because the motivation of the sign was not closely constrained. Regardless,

due to the convention of the sword being a symbol of dragon slaying and knighthood,

we are able to identify the sign accurately within the game without being given further

subtext. Lastly, by knowing this we are better able to signify the rest of the signs that

we have come into contact with thus far, we allow chains of semiosis to take effect. The

structure we started out in front of in the game’s beginning now more obviously is a

signifier for a castle — thereby connotating a medieval aesthetic — the keys we discover

open the corresponding color’s castle gates, and the character in which you play — the

square-shaped congregation of yellow pixels — now resembles a knight. All of this can

be inferred within a minuscule period of time due to semiotics at play.

I mentioned earlier that there was no “in-game text” provided for context for

Adventure, but what I did not mention was that was that there was accompanying

literature with the game that did so. Often instruction manuals and booklets would

provide a deeper backstory and explain game mechanics in games that did not have

the time or graphical (or even narrative) resources to do so — for instance, there was

little indication that Super Mario Brothers 2’s (1988) narrative was set within a dream,

without having read the manual where it was much more obvious. But that system,

of reading supplementary literature, does not break the semiotic structures that we


have supposed in Adventure, nor any other game that we could analyze, but rather it

makes for a more descriptive experience — one would understand signs more readily

having have read all the available context, just as a social actor would better understand

particular signs having understood the cultural system that the signs came from. Luckily

Adventure’s designers understood the mechanics of semiotics enough to imbibe the

game with enough meaning to allow players over 30 years in the future to comprehend

its gameplay.

Modern games appear to have moved away from the model where an instruction

manual is necessary, instead relying on other forms of media to provide backstory, like

books, websites, and animations that may or may not be canonical, and rely furthermore

on semiosis within the game itself. In contemporary videogames these moments can be

juxtaposed between playable moments like in loading screens (Figure 6), or outside of

those times (Figure 7). Notice in Figure 7 that I am intentionally describing the words

under the images as signs. More specifically, words are a part of a semiotic and linguistic

system in which language is composed of groups of signs we call words, and words are

composed of units of signs called letters. Semiosis still occurs when meaning is derived

— the idea of a sign simply being a “picture” is not accurate as we can see with this

display as well as with the sound of the dragon’s roar from before. The representational

Figure 6: Braid - Instructional loading screens. Figure 7: Braid - Diegetic, active in-game instructions.


style shown in Figure 7 is especially interesting because the player is able to actively test

out what the signifier, the instruction suggesting to jump with the spacebar, is supposed

to signify — even without knowledge of the higher systems at work here, a player could

interpret the sign’s meaning along with consecutive tutorial signs.

When we discuss the role of semiotics in more current games like Braid in Figure

6 and 7, we are invoking more complicated relationships than what was observed

in Adventure. Videogames, just like signs, grow exponentially in complexity and

sophistication, and necessitate the use of more specific types of signs. In his scholarly

research of semiotics, Peirce “identified 66 potential varieties of signs, 3 of which have

gained wide acceptance: the concepts of icon, index, and symbol,” (Lee-Hurwitz 23).

Because we are able to treat the videogame as a serious medium worthy of semiotic

study, it is possible to locate these types of signs within games, and often due to the

graphical fidelity, it’s easier to do so in slightly more modern games than Adventure

where we first identified Saussure’s principles. To further prove the validity of the

assertion that classical semiotic thought is active in videogames, as well as these specific

varieties identified by Peirce, let us look briefly at a slightly older title, The Legend

of Zelda for the Nintendo Entertainment System (1986) — a spiritual successor to

Adventure.

Figure 8: The Legend of Zelda

The Legend of Zelda plays out much

like how Adventure started: you begin

the game with little narrative context of

what you are supposed to do or who you

are, and you are tasked with completing

your objectives by manipulating objects

that interact with one another. But when

compared to Adventure this game is

relatively more realistic (figure 8).


The Nintendo Entertainment System was capable of producing more robust 8-bit

graphics as opposed to the Atari 2600 because of expanded memory, which allowed it to

produce detailed and vibrant virtual environments. This also meant that signs intended

for use on games on this console could also be more detailed.

Figure 9: The Legend of Zelda - Comparison of in-game representation v. extratextual representation.

Figure 10: Assassin’s Creed 2 - Comparison of in-game representation v. extratextual representation (of a modern game).

Peirce’s icon “has a relationship of similarity or

resemblance... (and displays) a similarity be-

tween the present and the absent components

(of the sign),” (Lee-Hurwitz 23). An icon is a

sign that closely resembles what it is intended

to represent; it is a highly-motivated sign. Refer

to Figure 9 for example: the green and brown

sprite that you control serves as an icon to what

to what you are supposed to see as according

to the game’s literature (situated to the

right), because of the close proximity of

its representation. With improvements of

graphical integrity it has become increasingly

easy to denote icons in videogames, and extra-

game content often has a 1:1 signification with

in-game content (Figure 10: Assassin’s Creed

2 comparison). That is not to say that games of

higher graphical quality are intrinsically better

at at communicating through visual semiosis,

however. Describing cartooning, Scott

McCloud sees simplification as more a way of

amplification. “When we abstract an


image through cartooning, we’re not so much eliminating in details as we are focusing

on specific details, (McCloud 201). Additionally, realism, through its emphasis on

denotation as opposed to connotation, makes it more difficult for “the artist to make his

point symbolically,” (Arnheim 141). Iconography viewed through the lens of semiotics

can make these distinctions.

Figure 11: The Legend of Zelda

Indices use a part of something to stand for the

whole (Lee-Hurwitz 23). A common trope in

The Legend of Zelda series is the character’s

representation of vitality. As depicted in Figure

11, the character’s health is represented by

an orderly group of hearts. Upon injury, the

player is alerted by a disastrous noise and the

visual of the heart being halved or quartered, signifying that they have taken damage.

The index present here is subtle and more complex than anything we would have seen

in Adventure; the heart works as an index by representing a vital part of the body — not

the whole body, as it is an index, but enough to make it obvious to what it represents —

the body, as an idea, then acts as signifier to the signifed of the concept of health, a small

intricate subset of semiosis.

Perhaps the most important sign that Peirce identified was the symbol, a

structure so prevalent it has nearly become synonymous with the entirety of semiotics,

and is often the one of most interest in the communication field. “A symbol has the

relationship of arbitrariness...any sign using an arbitrary connection between the

present and absent components is a symbol... (and can often include) objects, behaviors,

texts, ideas, and people,” (Lee-Hurwitz 23 and 30). It is this arbitrary aspect, despite its

pervasiveness, that makes the symbol difficult to isolate and signify. The arbitrariness of

a symbol speaks to its situation in the culture in which it was created, or was intended to


be used in — without being privy to that system it is initially difficult to interpret these

particular signs. I feel the best example of symbols in the videogame is the “health bar”

and its multitudinous variations. The health bar, as I will touch on when speaking of

medium intertextuality, is present in several games of varying scopes and time periods.

The health bar is in a way represented by the heart containers of The Legend of Zelda

series earlier discussed, but it appears in more generic terms such as in game series like

Mega Man (1987), Mortal Kombat (1992), and the even the first Halo (2001) game6.

There is no health bar for real life, nor in any other variety of media — only

videogames. “Symbols are a form of shorthand; encapsulating cultural knowledge

in particular ways, they serve a valuable role in deliberate passing on of traditions...

People use symbols as a way of conveying considerable amounts of information in a

small space or short time,” (Lee-Hurwitz 31-34). The health bar is shorthand unique

to the videogame, and can only be “picked up through osmosis” as Lee-Hurwitz would

say, through contact in that medium. I will soon suggest, however, that because of the

transcendal properties of both semiotics and videogames, other social actors could

become of aware of videogame symbols and other signs without ever playing them.

The word “code” in videogames connotes several different signifieds: primarily

cheat codes, like the strings of characters and commands used to in order to unlock

special privileges allowed by the game’s designer, and the videogame’s coding,

the machine language that controls and shapes the medium’s visuals, sounds, and

mechanics. But in semiotics, codes are something else altogether. Like with the

language analogy presented earlier, signs congregate into increasingly larger systems.

A group or system of signs is called a code by semioticians. “Placement of signs into

appropriate grouping stresses that meaning arises not solely, not even primarily, from

Codes in Videogames

6Subsequent titles in that series did employ a similar mechanic, but the first most closely identifies with the aesthetic I am speaking towards).


the relationship of signifier to signified but from relations between signs,” (Lee-Hurwitz

51). Semiotic codes work the same way in videogames just as they would in other forms

of media.

While we could easily reanalyze the videogames examined before, it would be

more prudent to introduce an even more recent game to illustrate coding systems. Mass

Effect 3 (2012) is a third-person action/role-playing game developed to function in a

much more fast-paced manner than its predecessors, and substantially retooled the

series’ game interfaces to reflect this change. These alterations required that the player

be able to receive a wealth of information in a short period of time — just what signs

were made to do. To meet these ends, groups of signs are often used in conjunction

with one another in what is called a “heads-up display”. Not unlike the dashboard

of your car, a heads-up display provides the user a quick synopsis of necessary

information to them; in Mass Effect 3, this often occurs during combat (figure 12).

Figure 12: Mass Effect 3 - Gameplay Heads-up Display


The image in Figure 12 reflects a singular moment7 of this code system, where the focal

point in the middle of the image features a segmented circle with several different signs

on it. Some of these signs can innately go through semiosis; they are icons and obvious,

some color coded to signify futility or blank to reflect unavailability, but still others are

symbols and require an intimate knowledge of this particular videogame to understand.

This circle within Mass Effect 3’s mechanics is called the Power Wheel, that is, a code

developed by the game’s designers for users to be able to quickly identify what actions

they can take. This type of constructed system is called a digital code because the signs

that make the code up are clearly identifiable and distinguished by the social actor, the

player.

In the lower right hand corner, however, we see a compass. This, we may not

notice, is a code as well. The individual signs consist of red dots to signify hostile non-

playable characters, blue ones representing friendly ones (including the central most

one that represents your own avatar), and the arrow pointing due to the next objective

and its distance in meters consolidating and signifying that this is indeed a radar

within the realm of this videogame. But the average player, or even the above average

player, would not see these parts — they would only see the radar. Analogic codes, of

which this is, contain “signs that run together, being separated only by the analyst (or

determined semiotician) for the purpose of interpretation,” (Lee-Hurwitz 52-53). While

the radar is an analogic code, it’s not the only apt determiner we can assign to it. It is

also a paradigm. A paradigm, as defined by Saussure, is a code in which social actors

only choose one central sign to display, or focus upon — Lee-Hurwitz compares it to

us wearing different pieces of clothing, but calling the ensemble one outfit. The power

wheel is a paradigm, the health and shield bars below it is a paradigm, as well as the

enemy health bar above it8.

7 This type of analysis is called synchronic analysis; if we observed the code’s evolution through time it would be called a diachronic analysis (Lee-Hurwitz 64).8 As I elaborated before, the health bar motif is as prevalent as the use of arbitrary, numerical scoring in this medium.


There are more codes here, though, and larger ones at that. The largest one

shown in Figure 12 is the aforementioned heads-up display, another common element

in games. We saw the beginnings of this code in games like The Legend of Zelda, with all

the pertinent signs relaying information at the top of the screen, and this is the natural

evolution of that idea; signs, and codes as a result, are always growing. The HUD in

Mass Effect 3 is a special type of code, also identified by Saussure, called a syntagm. “A

syntagm is that new set relating from the combination of elements drawn from different

paradigms...meaning is primarily located at the level of the syntagm,” (Lee-Hurwitz 55).

Meaning is located here, but it is still necessary to break it down to the paradigmatic

unit in order to be situationally comprehended, as Barthes presents the syntagm as a

fundamental “chain...which must be carved up,” (Elements of Semiology 65). Mass

Effect 3’s HUD is a syntagm composed of several active paradigms, but as Barthes

suggests as well as practice dictates, the paradigms must be interpreted individually so

the syntagm’s meaning can be deciphered.

It is at the syntagmatic level where the breadth of the videogame’s meaning is

derived, often displayed through the same long, overarching methods: systems of rules

delineated by Bogost in his analysis of the videogame’s operative mechanics; perceptions

of space, whether “hyper-realized” in sports or racing games, or “highly abstract” in

Super Mario Bros (Wade 78); genre, used not unlike its application in traditional media,

as well as narrative; and perspective, whether it be top-down or isometric, first-person

or third, the perspective is perhaps the most complex system of signifieds of them all.

Perspective is a syntagm that influences how an individual game will function: its play

mechanics and interface, visual style, and even influence how the social actor — the

player — perceives themselves, their avatar, or both as one.

All videogames communicate via a cocktail of syntagmatic clusters, a language,

which varies in paradigmatic composition, like dialects, from game-to-game, but they all


must singularly communicate from these previously introduced high-level syntagms. All

videogames are kin to one another.

Intertextuality As we know, high functioning syntagms and lesser paradigms form larger

systems, which beget even larger structures until they encompass a language completely

their own. But these languages are bound to context and proximity. Even at the highest

levels of semiotic processes within a domain, be it videogames or anything else, a foreign

social actor will not be able to interpret every sign or relationship, while at the same

time they could in another domain, or culture, from which they are a part of. A culture

is essentially a grouping of symbolic codes (Leeds-Hurwitz 17), and is the highest term

given to semiotic structures. The whole of videogames — the medium, its games, its

social actors and all that surrounds them — is a culture. Videogame culture viewed from

the outside stereotypes its users as “those who forgo all other cultural, social or even

hygienic activities in favor of videogames,” (Bogost, Unit Operations 52). But as the

consumer base expands and record financials are set each sales cycle, more and more

uninitiated actors become acquainted with the culture, disregard or even reshape those

stereotypes, and thus they become more familiar with the medium and its peculiarities

and its specificities — they learn to interpret the signs and structures that were once

unintelligible to them. “Every time we make meaning by reading a text or interpreting

a graph or picture we do so by connecting the symbols at hand to other texts and other

images read, heard, seen, or imagined on other occasions...which connections we make

(are) characteristic of our society and our place in it...”(Lemke 73). Participants in this

culture are able to understand individual videogames because they have played other

videogames, representatives of the genre.


Figure 13: Mortal Kombat

It is Hodge and Kress’ metasign,

“markers of social allegiance which permeate

the majority of texts” (Leeds-Hurwitz 27), that

allows social actors to interpret other signs

with signs. In our discussion of other sign

types we have already touched upon examples

of metasigns in videogames, most notably and

ubiquitous, I believe, the health bar (reiterated in Figure 13), a sort of volumetric

percentage representation of an abstract quantity. As a metasign, which are mainly

comprised of important codes and some individual structures, the health bar is

dispersed evenly throughout the medium, and even though it can be represented

different visually, its meaning is universally understood.

The most important semiotic aspect in the videogame medium involves the use

of metasigns and their ability to speak across discrete units. The breadth of videogames’

ability to communicate comes from its intertextual qualities. Intertextuality is a concept

derived from Mikhail Bakhtin, describing the ability of a text to make reference to

another or to several others (Leeds-Hurwitz 41). Intertextuality allows games to speak

amongst themselves through semiotic means, transcending time, bureaucracy, and

physical media.

Several intertextual elements have already been introduced: of course, any

metasign like the recently mentioned health bar, but any other shared trope in

videogames as well — numerical scoring systems, narrative clichés9, HUD elements, and

environment depiction standards. Additionally, there are videogames that include less

than subtle homages to previous titles. Software development itself has a rich history of

its developers inserting messages into their coding, and videogames are no exception

to this. Many videogames contain Easter eggs, included materials hidden by developers

and often meant to be humorous to or lightly editorialize the

9 Many already shared with classic literature, such as Joseph Campbell’s monomyth.


game (Bogost, How to do things 37). They are often obscurely located within the game

environment and difficult to view, but often make meta-jokes, about the game itself,

other games, or even people or events (Figure 14).

Figure 14: Grand Theft Auto : San Andreas - In-game Easter egg

Intertextual elements are not only

within and around videogames themselves,

but also manifest themselves as interlopers

exploring other semiotic domains, who then

retrieve new elements. As the videogame

industry continues to grow, it begins to

adopt business practices increasingly similar

to that of traditional media and that of the

the entertainment complex. To increase awareness of the product, videogame icons

and symbols now saturate various markets — clothing, food, ringtones, toys, bumper

stickers, nearly any sort of merchandise you can think of has had a Call of Duty or Halo

emblem on it. You see trailers for videogames on television just as you would films

before, and print advertisements everywhere else. This is intertextual and extratextual

because it illustrates the videogame’s ability to make reference to itself even while being

outside of its original text.

At the same time, videogames show the capacity of recursiveness, mirroring its

ability of impacting surrounding cultures by displaying the action of cultures in itself.

In-game real world advertisements exist in videogames where it makes diegetic sense,

ones that emulate a sort of factual reality like in sports or racing games. These games

can display advertisements from actual companies, updated dynamically through

constant internet connection, and adjusted to suit the advertiser’s target demographic.

For example, in 2008, the Barack Obama presidential campaign paid for advertisements


Figure 15: NBA Live 08 - In-game advertisments

to be displayed in games like NBA Live

0810 (Figure 15). Advertisements especially

legitimatize the videogame as an intertextual

medium, because as Barthes would point out,

the semiotic properties of the advertisement

are purely intentional, “formed with a view

to the optimum reading,” (“Rhetoric of the

Image” 152). This (along with the millions of dollars spent to produce such exchanges)

signifies that the videogame medium is mature enough to clearly accommodate its own

semiotic processes, an individual game simultaneously referring to itself and games

before it, in addition to incorporating outside significations.

However, not all intertextual elements are as obvious nor as integral as others.

The most subtle element in videogames, gameplay, is its most influential and also

unique to the medium. The process of gameplay — that is, the gestalic culmination

of elements that compose the play mechanics of a videogame — is made up of

interactions of codes; codes in the semiotic-sense, and the coding of machine languages.

Videogames, just as other completely digital media, are created from a myriad of

computer languages, complicated structures complete with their own systems of

signifiers and signifieds. To create a videogame from basal codes is complicated, time

consuming, and fiscally unadvisable. In order to circumvent a portion of the menial

labor in the creation of videogames, many developers use pre-configured software

frameworks called game engines. Game engines are provided to developers by other

videogame developers — thus establishing semiotic sub-culture systems amongst

themselves — to aid in the production of their medium. One engine can create several

different videogames. The engines are often presented as proprietary or open source

software suites containing development tools to render graphics, alter in-game physics

or artificial intelligence, to script events, sounds, or any number of manipulable 10 Source

http://www.reuters.com/article/2008/10/16/us-usa-politics-videogamesidUSTRE49EAGL20081016%3Fsp%3Dtrue


elements. “Game engines move far beyond

literary devices and genres. Unlike cultural

categories like the modern novel or film noir,

game engines regulate individual videogame’s

artistic, cultural and narrative expression,”

(Bogost, Unit Operations 57). Bogost seems

to suggest that while game engines take the

“drudgery out of game development”, they are

also “partly responsible for the massive growth

of the game industry”, allowing developers to

focus more on discursive opportunities rather

than the mechanical agency of a game. Because

of the intrinsically similar language, and there-

by semiotic structures, that game developers

use for particular titles, we can observe clear,

intertextual commonalities in gameplay.

Figures 16 depicts different videogames

that have been produced from the same game

engine, the Unreal Engine 2.While these games

do not belong to the same genres, or share a

great deal of high level syntagmatic codes, we

can observe basal similarities in these images,

namely perspective.

Figure 16a: Star Wars Republic Commando

Figure 16c: Deus Ex: InvisibleWar

Figure 16b: Lineage 2


Click Here to Continue

While game engines, to a degree, reign in artistic expression within the

constraints of the software (in the same way iMovie does film for novices), they are still

an efficient method of videogame production, of progenerating videogame symbols, to

sustain and disseminate those signs, and display the most lucid influence of semiotic

structures in videogames, along with the intertextual relationships they evoke.

Videogame developers have their own rhetorical purposes, outside of their

individual games, when creating a game engine. Videogames have great communicative

properties, but to the average social actor, signification is only for their own reception.

Some game engines address this by using graphical user interfaces to allow one to

manipulate game mechanics without working knowledge of a programming language,

and some engines simplify this even more by being geared toward beginners.

http://jarrodt.me/ThesisGame


BibliographyMcLuhan, Marshall. Understanding Media The Extensions of Man. Cambridge: MIT Press, 1964. Print.

Bogost, Ian. How to do things with videogames. Minneapolis: University of Minnesota Press, 2011. Print.

Bogost, Ian. Unit Operations. Cambridge: MIT Press, 2008. Print.

Eco, Umberto. Theory of Semiotics. Bloomington: Indiana University Press, 1976. Print.

Lemke, J.L. “Metamedia Literacy: Transforming Meanings and Media.” Visual Rhetoric in a Digital World. Ed. Carolyn Handa. Boston: Bedford/St. Martin’s, 2004. 71-93. Print.

Kress, Gunther. “Multimodality, Multimedia, and Genre.” Visual Rhetoric in a Digital World. Ed. Carolyn Handa. Boston: Bedford/St. Martin’s, 2004. 38-54. Print.

Arnheim, Rudolph. “Pictures, Symbols and Signs”. Visual Rhetoric in a Digital World. Ed. Carolyn Handa. Boston: Bedford/St. Martin’s, 2004. 137-151. Print.

Barthes, Roland. “Rhetoric of the Image”. Visual Rhetoric in a Digital World. Ed. Carolyn Handa. Boston: Bedford/St. Martin’s, 2004. 152-163. Print.

Barthes, Roland. Elements of Semiology. New York: Hill and Wang, 1964. Print

McCloud, Scott. “Understanding Comics: The Invisible Arts”. Visual Rhetoric in a Digital World. Ed. Carolyn Handa. Boston: Bedford/St. Martin’s, 2004. 195-208. Print.

Wade, Alex. “Spatial Typologies of Games”. Associazione Italiana di Studi Semiotici,ed.5 2009, Print.

Foucault, Michel. This is not a pipe. Berkeley: University of California Press, 1983. Print.

Leeds-Hurwitz, Wendy. Semiotics and Communication: Signs, Codes, Cultures. Hillsdale: Hove and London, 1993. Print.

Deely, John. Basics of Semiotics. Bloomington: Indiana University Press, 1990. Print.

The digital work produced in this paper was created with the Construct 2 engine by Scirra Inc. (2012), operating under the free license. http://jarrodt.me/thesisgame

Games

Robinett, Warren. Adventure. Atari Inc., 1979. Atari 2600.

Ubisoft Montreal. Assassin’s Creed 2. Ubisoft, 2009. Multiplatform

Number None, Inc. Braid. Number None, Inc.. 2009. PC.

Rockstar North. Grand Theft Auto: San Andreas. Rockstar Games, 2004. Multiplatform.

Bungie. Halo: Combat Evolved. Microsoft Game Studios, 2001. Xbox.

Nintendo. The Legend of Zelda. Nintendo, 1986. Nintendo Entertainment System.

NCsoft. Lineage 2. NCsoft, 2004. Windows/PC.

Bioware. Mass Effect 3. Electronic Arts, 2012. Multiplatform.

Capcom. Mega Man. Capcom, 1987. Nintendo Entertainment System.

Midway. Mortal Kombat. Midway, 1992. Arcade.

EA Canada. NBA Live 2008. EA Sports, 2007. Multiplatform.

http://jarrodt.me/thesisgame

Documents

Semiotics and Intertextuality of Videogames