Embed Size (px)
Citation preview
Videogames and Multimodal LiteracyNelson Zagalo, University of Minhohttp://nelsonzagalo.googlepages.comhttp://virtual-illusion.blogspot.com
Institute of Education, University of Minho 7 June 2016
DigiLitEY Training School 2016
…been doing
Authored and edited books on:Storytelling, Aesthetic Emotions, Virtual Worlds, Videogames and Creativity
Research in the domain of Human–Computer Interaction, within a Communication Cognitive approach.
In specific, Interaction Design, and in concrete, Engagement.
…domains
a) SyntaxCode, symbolic objects of expression.
b) SemanticMeaning, interpretation of the symbols and its interrelations.
c) PragmaticContext, understanding the relation between participants.
Multimodal literacy: communication
For the multimodal literacy purpose, problems arise when different objects of expression (modes) are used to support the syntax, affecting semantics but mainly pragmatics.
Levels of the communication process (Watzlawick, 1967).
Communication can occur between humans, between humans and machines, between humans mediated by technologies, and even between humans mediated by artifacts (like videogames).
Defining pragmatics
The human communication occurs through body and can be externalized to objects of expression (media), which add new modes to human expression, each making use of different codes.
Modesverbal (voice expression), non verbal (body language), sound (object produced),
picture (drawing, photo…),text (visual symbols),animation (moving pictures), audiovisual (moving pictures with sound),interaction (action simulation).
SyntaxEach mode develops its own conventions, languages, creating what we call Media Arts (painting, sculpture, literature…), which we use to express ourselves and communicate.
Multimodal literacy: modes
AnalogueSculpturePaintingLiteratureMusicTheatre
DanceFashion
Industrial Cinema Photography
Animation Television Video Comics Graphic
design
Digital Videogame Virtual Reality 3D Animation Generative
Music Physic
Computation Programming Interaction
Design
Multimodal literacy: the media arts
Integration of modes
Multimodal arts: theater, film, and videogames
Videogames are an integration of previous media arts, like film, and even theater.
Videogame: Cinema + Computer Processingnew expression: Interactivity (user’s performance)
Cinema: Theater + Photographynew expression: Montage (sequence of photos)
Theater: Sculpture + Painting + Literature + Music + Dance + Fashionnew expression: Drama (actor’s performance)
Multimodal path: from text to videogame
Through literature the main goal of the writer is to tell the narrative as detailed as possible in order to develop a strong “fabula”, or mental story, in the receptor’s mind.
Film doesn’t need to spend time telling details because they are shown. The story world is given already built to the receptor, purposing direct perception of the visual world that enhances sensorial emotion and learning.
Coppola, The Godfather, 1972Puzo, The Godfather, 1969
Story is no more an act of “telling”, neither “showing” but an integrated set of active participation, of “doing”. Interactive media open a new space (virtual) for the mediation of knowledge. The story containing learning messages will evolve only through the actions and decisions of the participator.
Videogames go beyond representation to become simulation.
EA, The Godfather: The Game, 2006
Do Something Do Something
Multimodal path: from text to videogame
Representation is made of conventions of symbols, different from reality, involve strong need of interpretation. More demanding in abstract reasoning, cultural background, and imagination activation.
Simulations are illusions of the real, purposing a sort of direct access to the real without its insecurities. Permits learning through infinite repetition, with little change, modelling for a fixed mental model.
Representation vs. Simulation
Acting upon the real, upon objects, simulations, makes learning become a task memorized through somatic actions, using body perception instead of mere mental activity, configuring an optimized situated cognition environment.
One of the greatest examples of the gains of simulations have been flight simulators, which were able to lower drastically accidents.
The simulation gain
In the 1960’s, Seymour Pappert (MIT) following his studies with children, purposed a sort of visual programming language, LOGO, which he defined as "body-syntonic reasoning”, because students could write code putting themselves in the place of the turtle, mentally simulating, doing the reasoning and predicting based in the turtle’s position and movement.
The computing evolution: becoming visual
Then in the 1970’s, Alan Kay (Xerox) purposed moving from text-based to image-based interfaces. The idea arose from Pappert studies, the evaluation done for the interaction with objects and written words, understanding that people would be more able to act upon objects than on written words.
The computing evolution: becoming visual
In the 2000’s, Mitchel Resnick (MIT) developed a real visual language for kids, Scratch, which in 2016 was brought into the physical world by Osmo.
The written symbols became visual elements, and finally physical objects. Programming was born in abstract modelling but is now made of concrete objects, sort of “programming simulator”.
The computing evolution: becoming physical
The computing evolution: in the future
“Holodeck”, machine presented in 1998 in the Star Trek series, previously dreamed by André Bazin in the paper “The Myth of Total Cinema” (1948).
Multimodal expression evolution
Symbolic, abstraction (text)
Audiovisual, realism (image/sound)
Simulation, surrogate (interactive)
Why multimodal literacy evolved for simulation?
Transforming the abstract topics into realistic objects, and adding to them a fully interactive layer, to guarantee a virtual learning by doing.
What can you “do” in videogames: 3 levels
1 - ManipulationAct upon the representation, change and transform, express yourself trough virtual objects.
3 - ParticipationTake decisions, choose, decide, experiment with moral values, and check your real beliefs.
2 - NavigationFollow conventions or define your own approach to the virtual world. You’re what you do in the world.
Doing to Learn
Brenda Romero, Gaming for understanding, at TEDxPhoenix (2011)
Learnign about Middle Passage at schools
Families in Africa Boat to take people to the US
New World , board game improvisedby Brenda Romero
Doing to Learn
Rules
. Takes 10 turns to cross the ocean.
. Can not take complete color sets of pawns.. We have 30 units of food.. Each turn roll the dice.. Dices determine the food units to be consumed.
Motivation
CompetenceAutonomy Relatedness
Motivation model from the “Self-determination theory” (Deci e Ryan, 1985).
What you do and why?
Choice / Power
Autonomy means, to empower the receptor to the limit of experimenting with moral values, testing beliefs, trying in “practice” your own theories.
To Do: autonomy
When your friend Chloe asks you to put an end to her life, you need to decide, and choose if you let her die or not.
“Life is Strange” (2015)
Skills / Progress
Competence means, to develop the skills requested by the simulation to keep progressing. Progress is the main fuel of interest.
No text manuals to read, you’re taken by the hand, each action done, you go up another step in comprehending what’s expected from you. A proper scaffolding process.
To Do: competence
Others / Recognition
Relatedness means, to be part of a community, be as capable as the others, which can be easily translated by Leaderboards, or school friends recognition.
To Do: relatedness
Videogames are interactive audiovisual experiences, everything we take into account in movies must be taken into account here, but the center of the cognitive effort, the real challenge for the players, goes beyond all the interpretation you need to understand the audiovisual medium, you need to become part of the experience, and in order to fully understand it you need to perform, to do.
The videogame’s core of the multimodal literacy
Videogames and Multimodal Literacy
Nelson Zagalo, Universidade do Minho
http://nelsonzagalo.googlepages.comhttp://virtual-illusion.blogspot.com
