Electronic reading - My English Portfolio · type of item, eg shampoo, deodorant, drinks, frozen meals, junk food. One advertisement should be on paper, eg magazine, newspaper or

LIBRARY www.library.unisa.edu.au

University of South Australia

Extract title The characteristics of multimodal texts: implications for the teaching of reading and writing

Extract author

Book title Evolving pedagogies : reading and writing in a multimodal world

ISBN 9781742003436

Citation details Ch. 2, pp. 21-46

Publication details Carlton South, Vic. : Education Services Australia, c2010.

Book author Bull, Geoff and Anstey, Michele.

Total pgs in book vi, 170 p. :

Edition

Electronic reading

COMMONWEALTH OF AUSTRALIA

Copyright Act 1968

Notice for paragraph 135ZXA (a) of the Copyright Act 1968

WARNING

This material has been reproduced and communicated to you by or on behalf of University of South Australia under Part VB of the Copyright Act 1968 (the Act).

The material in this communication may be subject to copyright under the Act. Any further reproduction or communication of this material by you may be the subject of copyright protection under the Act.

Do not remove this notice

Book extract

In this chapteral We define and examine the characteristics of multimodal texts and consider how

these characteristics influence the ways in which we consume, use and produce text.

Et We discuss the implications of these characteristics for teaching readingand writing and more particularly, how the Four Resource Model might bereconceptualised and used to facilitate the literacy teaching and learning process.

We define the elements of multimodal texts and how these elements cohere toproduce meaning

We examine how the reading and writing processes take place with multimodal texts.

E We consider how the Four Resource Model can be reconfigured to assist in planningfor teaching reading and writing with multimodal texts.

SECTION ONE

Defining multimodal texts

-The charOcteristics Qf .

,u.ltim

In this section, we introduce and explore six concepts about multimodal texts. As partof the exploration of each concept we have provided a Reflection Strategy to assist youin understanding and applying the concept. Each strategy will require engagement withthe two texts that you choose, therefore, before you read this section we suggest youacquire the two texts that you will use. Obtain copies of two advertisements for the sametype of item, eg shampoo, deodorant, drinks, frozen meals, junk food. One advertisementshould be on paper, eg magazine, newspaper or advertising billboard, and one should beretrieved from television or the internet. Both should be in a format you can view andexamine a number of times, eg the digital-electronic advertisement should be able to beviewed with and without sound, paused and played in slow motion; or if you are usingan advertising billboard, photograph it and reproduce it on an A4 page in full colour.

Website Reference 21Access Section 2.1 of the website for more detail.

If you are having difficulty finding items to use a useful website is www.bestadsontv.com wlshowcases TV, print, radio and interactive advertisements that can be viewed free of chargeand downloaded for a small fee.

Concept One: Multimodal texts convey meaning through a combination ofelements duddraw upon several semiotic systems.

When a text combines elements that draw upon two or more semiotic systems (t hatis, the linguistic, visual, gestural, audio and spatial) in order to convey meaning, i tis defined as multimodal. Examples of multimodal texts are a live ballet performancein which gesture, music, and space are the main elements; a picture book in wh ichthe elements of images and words are arranged in space on individual pages whichcontribute to an overall set of bound pages, or the home page screen of a website inwhich elements, such as sound effects, oral language, music and still or moving images,are combined. All these texts are multimodal, and each is delivered to the reader/viewer/consumer via a different medium or technology, that is, live (the ballet), paper (thepicture book) or digital-electronic (the website).

REFLECTION STRATEGY 2.1

The purpose of this strategy is to reflect upon the six concepts about multimodal textthat we introduce by comparing two multimodal texts on the same topic but deliveredvia a different medium or technology.

1 Analyse the two advertisements you have chosen in terms of the elements theyuse and the semiotic systems present. We have completed an example using caradvertisements in the sample analysis. You may wish to use a similar table butchoose a different product.

TABLE 2.1 Sample analysis

22 Evolving Pedagogies

Type of Text Elements Semiotic systems

Billboard

advertisement for

small hatchback

sedan

Image of car

Young people surrounding it and gazing at it admiringly

Caption identifying brand and model

Phrase describing car

Logo of brand

Visual

Visual, gestural, spatial

Linguistic

Linguistic

Visual

TV advertisement

for small

hatchback sedan

Sound effects

Pop music

Inaudible conversation and gasps from people

Voice-over

Sequence of images of car being driven byyoungattractive female through the restaurant and

entertainment area of a city

Groups of people looking at parked car and commenting

Caption identifying brand and model

Logo of brand

Audio

Audio

Audio and linguistic thovve)

linguistic is inaudible)

Linguistic

Audio and Linguistic

Visual, gestural, snotiril

Linguistic;

Visual

2 Identify the common and different elements and semiotic systems used. Why arethere differences?

3 Which elements and semiotic systems are dominant in each text and why do youthink this is so?

4 Consider how the use of a particular medium or mode of delivery (paper or digital-electronic) has afforded the advertiser different options in terms of delivering amessage about this product. Think about the implications of this when reading(consuming or using) or writing (producing) multimodal texts.

Concept Two: Multimodal texts draw upon and cross the boundaries of the arts,performance and design disciplines, their knowledge, understandings and processes.

When explaining Concept One, we drew upon three examples of multimodaltext: a ballet, a picture book and the home page screen of a website. What is alsointeresting about these three examples, and multimodal texts generally, is that theycross the boundaries between the disciplines of art, performance and design (Kressand van Leeuwen, 2001:1). In the examples given, each text draws upon the arts(consider the dance of ballet, the art of the picture book or still or moving images on thewebsite), performance (consider body position, facial expression and movement of theperformers in the ballet and people depicted in the images of the picture book or thestill and moving images on the website) and all three texts had to be carefully designedin terms of the combination of the elements within a space (stage, page and picturebook or screen). Proficient users or producers of multimodal texts need the knowledgeand ability to move across and between the boundaries of performance arts and design,accessing and using the knowledge and processes they afford to make and communicatemeaning.

Website Reference 2.2Access Section 2.2 of the website for more detail.

Links to two ballet performances (one modern, one traditional) are provided to enable you to consider

how the disciplines of arts, performance and design are displayed.

www.youtube.com/watch?v=910802X31

www.youtube.com/watch?v=n8CUDVpkGk4


Re-examine the two texts you have chosen to analyse for this series of ReflectionStrategies.

1 Identify how each draws upon the disciplines of the arts, performance and design.2 Identify the differences between the two texts in terms of the disciplines they

draw upon and how they use them.3 Why have these differences occurred and how do (or don't) they contribute to the

success of each text?

Chapter 2 23

Concept Three: The role of language in multimodal text varies, is not aIwuv, (wi hintand is only one part of the whole.

The fact that multimodal texts combine elements and semiotic systems, .1 tit I oss

disciplines means that language (the linguistic semiotic system), vvhich has lur so longdominated meaning making, is now only one part of it (Jewitt and Kress,'2.008. it).In the three examples previously provided, one (the ballet) does not use language althe picture book may foreground the image over the words (language) and the !minepage website design may foreground images and symbols to capture the reader/v iew er'sattention initially before engaging with the less obvious linguistic elements. Consequ en I lyto engage productively with multimodal texts, proficient users or producers mus I notonly be competent in the linguistic mode but with all modes and their semiotic systems.They need to use this competence to make decisions about the appropriate selection,combination and use of modes and semiotic systems in order to communicate, use anddeliver information most effectively.

Concept Four: In a multimodal text, meaning is distributed across all elements (parts)and each element has a role in contributing to the overall (whole) meaning of the text.

The meaning of multimodal texts is distributed across all elements present in amultimodal text and each element or mode has a specific role or function in contributingto the overall meaning of the text; this is the partwhole concept. Kress and van Leeuvven(2006:20) pointed out that these roles might be complementary (eg in a picture bookwhere the picture shows a child's bedroom but the words lead you to look at the bedroomin a particular way), reinforcing (eg in a ballet when music and gesture both signify thesame emotion), or hierarchical (eg at another point in the ballet when the music fadesand [he action on the stage becomes the dominant signifier of meaning and the music issecondary). Proficient readers and viewers of multimodal text need to be able to identifyhow each mode is contributing to the overall meaning and attend to those that areproviding the most pertinent information at the time. They need to differentiate [heirattention while engaging with the multimodal text, attending to the modes that bestaddress the purpose of their reading or viewing at the time. Producers of multimodaltext need to be able to make informed decisions about how the role or function of eachelement will contribute to achieving their communicative purpose.


Use the table you commenced for Reflection Strategy 2.1 as a basis for further reviewand analysis of the texts.

1. Identify the role of each element and semiotic system (part) in fulfilling the overallpurpose (whole) of this advertisement. In Table 2.2, we have completed theexample we started in Reflection Strategy 2.1 to demonstrate how this might bedone. In the example we started, the goal of the advertisement was clearly toshowcase the car and encourage people to buy it. However, once we analysed therole of each element and the specific contributions of the semiotic systems, webegan to see that the advertisement was directed very specifically at young, imagr.rtconscious females. In other words, our analysis of the roles of the elernent.-; dudsemiotic systems (parts) showed us how some meanings were being augmentedor reinforced by each mode and element contributing to one (whole) message, Itis important to note that we needed to be able to understand and use the specific:

grammars (codes and conventions) of each of the semiotic sysiern't iti rdei

analyse the texts and identify this overall message.

24 Evolving Pedagogics

TA

BLE

2.2

Ext

ende

d sa

mpl

e an

alys

is

Typ

e of

Tex

tE

lem

ents

Rol

e of

Ele

men

tS

emio

tic s

yste

ms

Bill

boar

d

adve

rtis

emen

t

for

smal

l

hatc

hbac

k

seda

n

Imag

e of

car

Dis

play

pro

duct

Vis

ual

Col

our

of c

ar is

pop

ular

am

ong

youn

g fe

mal

e dr

iver

s

You

ng p

eopl

e su

rrou

ndin

g it

and

gazi

ng a

t it a

dmiri

ngly

Indi

cate

that

pro

duct

is p

opul

ar w

ith

the

youn

g

Vis

ual,

gest

ural

, spa

tial

Gaz

e an

d bo

dy p

osi t

ion

of y

oung

is to

war

d ca

r, fa

cial

expr

essi

ons

indi

cate

adm

iratio

n an

d en

vy

Cap

tion

iden

tifyi

ng b

rand

and

mod

elIm

prin

t bra

nd m

emor

yLi

ngui

stic

Font

,co

lour

and

siz

e ha

ve y

oung

, fem

ale

appe

al, p

lace

men

t

acro

ss to

p fr

ames

imag

e

Phr

ase

desc

ribin

g ca

rR

einf

orce

bra

nd m

emor

yLi

ngui

stic

Pla

ced

at b

otto

m w

ith li

near

art

com

plet

ing

fram

e of

imag

e an

dle

adin

gto

logo

Logo

of b

rand

Rei

nfor

ce b

rand

mem

ory

Vis

ual

Pro

min

ent b

otto

m r

ight

for

mem

ory

TV

adv

ertis

emen

t

for

smal

l

hatc

hbac

k

seda

n

Sou

nd e

ffect

sE

nhan

ce d

ispl

ay o

f vis

ual f

eatu

res

of c

ar

Aud

ioQ

ualit

y of

sou

nd (

purr

ing

thro

aty

exha

ust)

attr

acts

atte

ntio

n,

indi

cate

s po

wer

but

is n

ot m

ascu

line

Pop

mus

icE

ngag

e po

tent

ial p

urch

aser

s of

car

Aud

ioS

tyle

of m

usic

mos

t pop

ular

with

you

ng fe

mal

es a

t pre

sent

Inau

dibl

e co

nver

satio

n an

d ga

sps

from

peo

ple

Indi

cate

pop

ular

app

eal t

hrou

gh s

ound

Aud

io a

nd li

ngui

stic

(how

ever

, lin

guis

tic

is in

audi

ble)

Alth

ough

the

wor

ds a

re in

audi

ble

the

pitc

h, p

ause

ano

'

into

natio

n in

dica

te a

dmira

tion

Voi

ce-o

ver

Aud

io id

entif

ies

feat

ures

of c

ar u

nabl

eto

be

show

n vi

sual

ly a

nd r

einf

orce

s

bran

d an

d m

odel

nam

e

Aud

io a

nd li

ngui

stic

You

ng fe

mal

e vo

ice

pitc

h, p

ace,

pau

se a

ndin

tona

tion

indi

catin

g

enth

usia

sm

Seq

uenc

e of

imag

es o

f car

bei

ng

driv

en b

y yo

ung

attr

activ

e fe

mal

e

thro

ugh

the

rest

aura

nt a

nden

tert

ainm

ent a

rea

of a

city

Indi

cate

loca

tion,

con

text

s an

d

segm

ent o

f pop

ulat

ion

to w

hich

car

is s

uite

d

Vis

ual,

gest

ural

, spa

tial

Cam

era

angl

es s

how

off

car's

vis

ual a

ppea

l, w

ide

angl

es a

nd

over

head

sho

ts s

tres

s at

trac

tive

loca

tions

and

eas

e of

ste

erin

g

thro

ugh

city

Gro

ups

of p

eopl

e lo

okin

g at

par

ked

car

and

com

men

ting

Vis

ually

rei

nfor

ce p

opul

ar a

ppea

lV

isua

l, ge

stur

al a

nd

spat

ial

The

cam

era

zoom

s in

on

youn

g fe

mal

e th

en fo

llow

s th

eir

gaze

to fe

atur

es o

f car

Cap

tion

iden

tifyi

ng b

rand

and

mod

elV

isua

lly r

einf

orce

bra

nd a

nd m

odel

mem

ory

Ling

uist

icFo

nt,

colo

uran

dsi

ze h

ave

youn

g, fe

mal

e ap

peal

, pla

cem

ent a

t

begi

nnin

g an

d en

d of

adv

ertis

emen

t seq

uenc

e fr

ames

imag

es

and

rein

forc

es b

rand

and

mod

el m

emor

y

Logo

of b

rand

Vis

ually

rei

nfor

ce b

rand

and

mod

el

mem

ory

Vis

ual

P/a

cem

ent a

t beg

inni

ng a

nd e

nd o

f adv

ertis

emen

t seq

uenc

e

fram

es im

ages

and

rei

nfor

ces

mem

ory

of b

rand

and

mod

el

(-2. Having completed your analysis of the role of the elements and semiatic -;ystemsin contributing to the overall meanina of your selected texts, analysa ihn role of

the semiotic systems in relation to one another. For example, identify tha semiotic

systems that are providing complementary, reinforcing, or hierarchical meanings.When we re-examined our sample analysis in this way we found that while allsemiotic systems fulfilled a reinforcing role, different combinations reinforceddifferent meanings or messages. For example, the visual, audio, spatial and gesturalsemiotic systems combined in elements of the advertisement that focused onappeal to young females; the audio, visual, linguistic, spatial and gestural combinedin elements that reinforced car features and only the visual and linguistic combined

in elements that reinforced branding.3. VVhen you have completed your analyses think about why particular elements and

semiotic systems have been chosen by the producers of the texts to reinforceparticular messages.

Concept Five: Different modes have different functional specialisations and are usedselectively in multimodal texts.

Time and space are also critical issues in multimodal text; they affect the roles of the

modes in a text and how we use or process them. Some multimodal texts, such as videoclips, movies, dance, or theatre; are dominated by time-based modes, that is elements ofmeaning presented over time. In these texts, speech, gesture, body movement and actionprovide meaning over time, rather than instantly, as the video clip or play unfolds. Othermultimodal texts, such as sculpture, a still image, an architectural arrangement, a page

or individual website screen are dominated by space-based modes that simultaneouslypresent all the elements for meaning making through the layout and organisation of theelements. Kress (2003:45) suggested that understanding more about how time-based andspace-based modes operate and convey meaning is important to the proficient readingand production of multimodal texts. Jewitt (2006:25) referred to this as the functionalspecialisation of modes, that is, we select, attend to and use modes because they are best

suited to the task we are trying to achieve. Therefore if you are limited to a screen or

page, you will choose space-based modes, but if you are using a technology that enablesyou to build up meaning over a series of episodes or scenes (such as video) then youwill draw on both space- and time-based modes and particularly the special featuresafforded by time-based modes (for example, building atmosphere through music).

Kress (2003:44) pointed out that the functions of some modes (eg gesture and writing)

are mixed (that is they can be effective in space and time) and a good example of this

is in the picture book. While each individual page is a space in which the layout of theelements is critical to meaning, these pages are bound into a book that has an overallmeaning and is examined over a period of time. On each individual page meaningis afforded through the gestures and language (writing) and over time we use thisinformation to build up the narrative. Thus gesture and writing are used effectivelyon each page to build up meaning over time, that is, they fulfil both space and timefunctions.


REFLECTION STRATEGY 204

The paper conveyed text you have been examining is dominated by space-based

modes because all the information mus,t be processed simultaneously to make

meaning, however, the digital-electronically conveyed text uses both space- and time-

based modes.View the digital-electronically conveyed text and identify how different modes are

used over time to build up meaning. For example, in the sample that we analysed,

sound, which included music, sound effects, voice-over and inaudible language, vvas

carefully orchestrated to enhance particular meanings at particular times.

Concept Six: The meaning of a multimodal text is afforded through the cohesion and

coherence within and between its elements and the context in which it is used.

One of the complexities of multimodal texts is that in order to make meaning of

them we have to work out how the text operates or comes together to make meaning

at several levels. How the text operates or comes together is referred to as cohesion or

coherence. We are all familiar with the concept of cohesion in the grammar of the English

language, where we understand that cohesion may occur at the sentence level (eg using

conjunctions and pronouns), paragraph level (eg using repetition of key phrases) and

whole text level (eg through the use of familiar generic structures). However, we now

need to expand our understanding of this concept to include the cohesive devices of

other semiotic systems. In the visual semiotic system, the repetition of colour or the use

of vectors in an image are cohesive devices that lead the eye to particular parts of the

image and potentially shape the viewer's meaning-making.The cohesion that occurs within the elements and semiotic systems in a multimodal

text is one level of understanding about how the text works. The other level of

understanding that is necessary is the cohesion between the elements that make up the

multimodal text. The cohesion of these elements may be chronological as in a narrative,

or spatial when visual or diagrammatic material is presented (Bearne 2009).

In many multimodal texts, both spatial and chronological cohesion of elements

will occur. For example, in the video car advertisement we have been analysing in the

Reflection Strategies, the overall cohesion is chronological. The elements develop a

narrative around a girl driving a car and the reactions of people to the car. However, in

individual screen shots of the advertisement the cohesion of the elements is spatial as

the moving images are overlayed with the caption (name and model of the car) and the

company logo. The caption and logo elements of this overlay are organised in the most

effective spatial arrangement to ensure we attend to them and remember them. They

frame the moving image top and bottom. The logo is placed at the bottom right corner

because in Western society we read left to right and top to bottom, so that placement

ensures the logo is the last thing our eyes will alight on and remember. One of the

ways in which the overall cohesion between the elements in a video advertisement

can be ensured is through the development of a storyboard. This storyboard will show

each screen shot of the overall narrative sequence of the advertisement ensuring the

chronological cohesion of the narrative of the advertisement.

However, cohesion also occurs within the elements of a multimodal text, and this

is the other level of coherence we have to process. One or more semiotic systems may

contribute to the meaning of an element within the text. For example, in the driving

sequence of the car advertisement, the visual, audio, spatial and gestural semiotic

systems are employed and the linguistic is added with the voice-over. To fully understand

the effectiveness of the driving sequence, we need to understand how each semiotic

Chapter 2 ' 27

system contributes. This necessitates an understanding of then. eud ( I

and their cohesive devices. For example, when we watch the I ithe

advertisement, we need to understand how the codes and coliveld1. ; IIu ah

semiotic system are realised in moving images through devices, so, I ,/ .it'uu uIìul,wide angles, and point of view. We need to think about how II tese I wen

combined to showcase the car.The producer of the advertisement has achieved this by onsoriiir h,, u lieyin izird

identifies each screen shot for the driving sequence, has detailed ilo.c!: i lu 1;InieraLohesionangles to be used, points of view, etc. This results in an advertisement o w111,.Ii

is developed through camera shots that move smoothly from one 10 iIi (,t her, loading

the viewer's eye to the areas of the car to be showcased. In this exri)1,!, 1,,,,wHige and

use of the contributing semiotic systems together with the use ()II:;.(ii (insured

cohesion within and between the elements in this advertisement:.

It must be remembered, however, that this advertisement was d :led ;It young,

image-conscious urban females. Therefore, all the decision making ond design was

influenced by the advertisement's purpose, audience and the con t,eX. w vIlic11 it would

be used. The meaning making, cohesion and coherence is ultimately depend enl. upon

audience and context.


Analyse how the cohesion between and within the elements of the tc,ubeen analysing in Reflection Strategies 2.1 to 2.4 works.

1 Is there chronological or spatial cohesion between the elements and how (loo.. I he

cohesion between the elements contribute to meaning making?

2 What semiotic systems are operating within the elements and whot v. the

relationship between and within them?3 How well does the cohesion of the elements and semiotic systems oddiel~. Hie

purpose, audience and context of the text?

TABLE 2.3 Key concepts about multimodal texts

As we discussed in Chapter 1, design is a key concept dnd

nowhere is it more evident than in the production and consul-111)i. il)ii loxls.

The summary of key concepts about multimodal texts prw-ierile(1 u .111)1i' A ill(iii1;110S


''( iii IW.IVe

nts tInt (11.1W upon :!,c-!vori.ilConcept One Multimodal texts convey meaning through a combination of den

semiotic systems.

Concept Two Multimodal texts draw upon and cross the boundaries of the arts,

disciplines, their knowledge, understandings and processes.

Concept Three The role of language in multimodal text varies, is not always domi

the whole.

Concept Four In a multimodal text, meaning is distributed across all elements (1

role in contributing to the overall (whole) meaning of the text.

Concept Five Different modes have different functional specialisations and are 1

texts.

Concept Six The meaning of a multimodal text is afforded through the cohesi(,

between its elements and the context in which it is used.

that multimodal texts are the deliberate and strategic combination of elements to achievea particular communicative function in a particular context at a particular time: they areall about design. Consequently our effective use of these te-xts is contingent upon ourunderstanding and use of design.

As indicated in Concept Six, design when applied to multimodal texts does not simplyrefer to the arrangement or coherence of content and eleme-nts but to the influence ofthe social and cultural contexts in which the text is produced and consumed. Kress(2003:49) stated:

Design does not ask 'what has been done before, how, for whom, with what?' Designasks 'what is needed now, in this situation, with this configuration of purposes,aims, audience and with these resources and given my interest in this situation?'(his emphasis)

These questions apply equally whether producing (writing) or consuming (reading)a text because engagement in design will occur. Regardless of whether producing orconsuming text, consideration needs to be given to:

the purpose of the reading or writingwhat the audience or reader knows and brings to the textthe context or conditions in which the text is, or will be, read, used or producedthe resources available to use when reading or writing the text.

Having considered these issues, we can then design a text or design our approach toreading the text we have to be more strategic and reflective about how we engage withmultimodal texts.

Walsh suggested that the whole approach to learning was being changed bymultimodal texts, because of their complexity and the role of design in how readersand writers engage with them. She concluded that reading, writing and learning withmultimodal texts required greater metatextual awareness and the use of metacognitiveskills (Walsh 2007: 48). Our investigations in this section confirm this.

THEORY INTO PRACTICE 2.1It is possible to adapt any of the Reflection Strategies in Section One to classroominvestigations, by selecting texts appropriate to the age and interest of your studentsand limiting the investigations to their conceptual capabilities, that is, simplifying themvvhere necessary.

How multimodal texts have reshaped reading

SECTION TWO

After a detailed analysis of the process of reading a monomodal paper text (linguistic) amultimodal picture book text (linguistic and visual) and a multimodal digital-electronicwebsite text (visual, audio, linguistic, spatial), Walsh (2006:32) concluded that allrequired meaning making or interpretation of signs. The differences she identifiedin reading these texts concerned the processing of the modes and how these modesactivated the meaning making of the reader.

Chapter 2 29

Bearne (2009:163-164) suggested that in order to process multimodai lexls we need

to consider:

the elements presentthe role or contribution of each element to the overall meaning of the textthe medium by which they are conveyed through time and spacethe weight or influence of the modes which are conveyed in the mediumthe modes' patterns of cohesion and how emphasis is created within them.

Walsh and Bearne's work indicated the need for a reader of multimoda I texts lo beable to choose a reading path and identify and impose what Kress (2003160) referredto as criteria of relevance. These criteria of relevance relate to the reader's purpose inaccessing the text and the characteristics of the text itself. In other words, reading amultimodal text is about continually having a conversation with oneself about:

what to attend tohow to make sense of itwhat to ignoreidentifying the relationships and links that are relevant, not only among the coo lent

but among the elements and modes presenthow the whole text is making sense.

Reading multimodal texts requires sophisticated metacognitive work. However, allthis thinking, sense-making and metacognition relies upon very specific knowledge aboutthe affordances of the medium by which the text is conveyed (paper, live and d igital-

electronic), the modes used, the semiotic systems drawn upon and how tbee systemsuse time and space. Therefore it is essential students have a detailed 111)(101!-;1 and ing of

what semiotic resources can achieve (their affordances)

1. Affordances of the medium: what is made possible oravailable given the roles of space and timeTexts delivered by paper, live and digital-electronic technologies offer differentopportunities and experiences to the reader and are therefore processed differently.These differences are not only in terms of the modes and semiotic systems available butthe physical and sensual experience of engaging with them, which in Urn influences

reading.

Affordances of texts delivered by paperThe affordances of texts delivered by paper appear more limited thaii t hose deliveredlive or by digital-electronic means, and certainly one's reading pal 11 is limited toconsideration of words and images. Therefore the complexily ut r eInig i1 conveyedby paper would appear to be more dependent on familidrily wait) die Lon telit andits terminology and ability to process the visual and lingtilsus Si.:111ii)i 1(1 sysl ems. As

we discussed in Chapter 1, while paper appears to offer only two setiults systems

(linguistic and visual), it may be read aloud and we may sill........nvoealnw i hei/r i II our heads

what we are reading if it is not read aloud. Therefore the and in itsiid lc - y-;l(,:!1» is

available, if not physically present, through the tecIiiudt)i.,.,y ()I 1.1 1.;esturai

and spatial semiotic systems also contribute to Lexts ni vi \s t ),Iitiu. I hrot.0.1 the

images and the layout and organisation of page ;Ind I)(.)()1. t l I t i i i j., and

complexity of paper texts are not as simple as they may tie..1 ppa b-,1,. i sud bypaper tech-nology also offer sensual experiere:esds lii lid paper andink (for example, thick, heavy, textured, thin or stuiv iii i is I h.,1 I I. These

experiences together with the layout of the conteld tm I a wslhetic


present. For example, we may unconsciously accord more authority to a full colourdocument on good quality shiny paper than a poorly photocopied black and whitedocurnent on thin paper. Our values as well as aesthetics may also influence response,for example we might feel more positive towards the authors of a document printed onboth sides of recycled paper.

The physical affordances of paper relate to how it is assembled and the context inwhich it is used at the time. Pages assembled conventionally can be turned back andforth to aid meaning making, for example to review information. Advances in printingtechnology and reductions in printing costs now mean that in many texts conveyedby paper, flaps, folding, half pages, cut outs and overlays afford other opportunitiesand reading paths. Advertising, pamphlets and postmodern picture books are goodexamples of how these advances in technology have made the reading paths of textsconveyed by paper more complex. Nevertheless paper documents generally offer morelimited reading paths than texts delivered via live or digital-electronic technologiesas the conventions organising paper documents are more rigid and the signals clearand usually conventional. For example, the linguistic and visual text on the page willbe organised in linear fashion, top to bottom and left to right, columns may be used,images or diagrams will usually have captions and reading of the linguistic text will beguided by the conventions of punctuation and formatting (font, size, colour and styleunderline, bold or italics). The visual text will be limited to still images and diagrams.Text delivered on paper is dominated by the affordances of space rather than time. Weprocess the elements of each page of text simultaneously as they appear on the page,although longer texts do need to be processed over time.

It is important to note that not all text delivered via paper will follow the moretraditional conventions of layout and organisation. There is more and more crossoverbetween the layout of text and screen (digital-electronic texts) as texts are produced forsimultaneous or shared transmission via different technologies. Recently this has becomea feature of postmodern picture books (as discussed in Chapter 3) making postmodernpicture books an excellent vehicle for investigating some of the characteristics ofmultimodal texts and how they influence reading paths and meaning making. Thisintertextuality, or borrowing of layouts and organisation from one medium to another,means that readers have to be prepared to read unconventionally at times. It would befoolish to think that texts delivered via paper have not evolved and require less activeprocessing than other multimodal texts. Clearly our ability to use the linguistic andvisual semiotic systems, particularly still images, will influence our reading processand understanding, but we must be competent with the other three semiotic systems inorder to use all the affordances of texts conveyed by paper.

Affordances of texts delivered liveTexts delivered live afford similar and different opportunities and experiences to thereader/viewer to paper texts. Clearly they afford more sense rich and physical experiencesbecause they involve live face-to-face interaction and have the potential to draw uponall five semiotic systems. Sight, touch, hearing and smell may all be part of makingmeaning with a text delivered live or face-to-face. Physical proximity, movement andattitude will also influence meaning making in these situations.

Texts conveyed live may afford the opportunity or expectation of participation orinteraction by the reader/viewer. A performance, such as a play, oral presentation, balletor show, may not require or offer the opportunity of participation, except in terms ofthe social conventions of showing appreciation at particular points, for example,clapping, calling for encore or making appreciative sounds, such as whistles or cheers.

Chapter 2 31

However, sometimes an oral presentation may offer the opportunity to participate byasking questions or participating in activities as part of the presentation.

Everyday use of texts delivered live and conducted as part of our social, commercialand municipal lives necessitate participation by the reader/viewer in order to getthings done (for example, conversation, discussion, interactions at shops, banks andworkplaces). Once again, reading, viewing and participation will be shaped by purpose,but the affordances of text conveyed live will offer the following to aid meaning making.There vvill be oral linguistic and non-linguistic sound, words and grunts or other non-verbal but meaningful utterances.

Other audio components may include sound effects and music. Gesture, includingfacial expression and body position, will also influence meaning making and spacebetween participants in the text will influence meaning. Consequently there are manymore reading paths offered in text delivered live than in a text conveyed via paper andmany more elements to process. Decisions about what to attend to and the developmentof criteria for making these decisions when engaging with the text will be essential andcomplex. Furthermore a text delivered live operates over time: meaning making doesnot depend on the immediate and simultaneous processing of all elements. One of themore interesting aspects of texts conveyed live is that they are often conducted with oraround texts delivered by paper or digital-electronic means. This adds an even morecomplex dimension to meaning making, as decisions about the role of the other texts aspart of the interaction around the text conveyed live, are ascertained and played out. Thepresence of other texts immediately offers many other opportunities for differentiatingattention and meaning making that need to be considered.


The purpose of this Reflection Strategy is to consider the decision making that takesplace when one is engaged with a text delivered live together with either paper ordigital-electronically conveyed texts and to reflect on the complexity of processingmeaning in such a situation.

Identify a recent situation in which you were required to engage with a textdelivered live (eg a conversation or transaction) together with text conveyed viapaper or digital-electronic means in order to achieve your outcome. For example, atransaction at a bank or service centre that required you to fill in a form as well asconduct a conversation with the service representative, or a situation where you gaveor received directions or instructions live but also used texts conveyed by paper ordigital-electronic means.

1 Briefly construct a timeline of the situation and what happened at each stage.2 At each stage of the timeline identify the technology by which thet texts were

transmitted (live, paper, digital-electronic).3 Identify the semiotic systems used at each stage.4 Look at the graphic representation of the situation and circle or artnotdt, to

indicate what you attended to when and the reasons for your attention rt thattime. For example when was the image on the text delivered via papei locti:-; of

your attention, when were aspects of the live interaction the focus Ow Irdcningand speaking, the gestures or facial expression?

5 Did you understand everything at all times? Use a highlighter In inch( ,rjeswhere things vvere not clear, that is, when your processing of 011 Ow .1.,,111,;ble

reading paths was not leading to effective meaning making 01110 orprevented understanding?


In a traditional English or literacy curriculum, the role of texts conveyed live hasoften been referred to as the listening and speaking part of the curriculum, an areaoften neglected or focused on performance (eg plays, poetry reading and oral reportsand presentations) rather than everyday situations. However, texts delivered live thatinclude listening or speaking are part of almost every aspect of our lives and occur withand around other texts. Therefore it is crucial that the affordances of text deliveredlive (including listening and speaking) be actively analysed and taught as part of amultimodal approach to teaching reading and writing.

Affordances of texts delivered via digital-electronic meansTexts transmitted digital-electronically share the affordances of texts delivered liveand via paper. They may offer opportunities for interaction with others, for examplethrough gaming and social networking, or the opportunity to interact with and createnew texts. Texts conveyed digital-electronically use both space- and time-based modes.While the screen affords similar space-based characteristics to a page of paper, it mayalso provide the opportunity to access moving images and sound. The links availablemean that reader/viewers are required to construct relationships over time, not simplybetween content, but between screens and modes as they move from still images tovideo; processing visual, gestural, linguistic, audio and spatial semiotic systems. Thereare many reading paths available and decisions about which to take will be based, notsimply on the reading purpose, but by developing criteria that enable identification ofrelevance among the many possible links and elements. The opportunity for distractionand loss of focus is much greater than in texts delivered by paper.

The ability to use texts conveyed by digital-electronic means is dependent not onlyon facility with all the semiotic systems and modes but familiarity, comfort with andability to use the hardware and software of the digital-electronic technology beingused. As we discussed in Chapter 1, research such as that of Dragona and Handa (2000)indicated that gender, age and experience influence attitudes and facility with digital-electronic technology and therefore the ability to process and make meaning of the textsthey convey.

Similar to texts delivered live and via paper, the senses are used when processingtexts conveyed by digital-electronic technologies. Tactile senses are used to operateconsoles, mouse and keyboard but probably do not invoke the same reactions oraesthetic and emotional associations as the quality of paper. Hearing is used to processsound effects, music and language. Sight is used in very sophisticated ways as eyemovement, scanning and focus is used differently on screens, and with still and movingimages. Therefore processing texts conveyed digital-electronically requires decisionmaking about how to use one's senses as well as making links and sifting the relevantinformation from the irrelevant. While the stereotypical image of people who work withdigital-electronic technology such as computers is that they do not engage in physicalaction, in fact, skills of visual analysis and rapid and focused eye movement, manualdexterity and even physical movement are a part of many leisure pursuits with textsconveyed by digital-electronic technology. Consider the physicality of products suchas Playstation and Wii, in which hand-held game consoles may require both manualdexterity and efficient whole body movement. Reading texts conveyed by digital-electronic technology is therefore a semiotic and physiological act that engages allthe senses.

Jewitt (2006), Gee (2004 and 2008) and Kress (2003) have pointed out that computergames require sophisticated and quick decision making based on rapid processing ofinformation from multiple modes, semiotic systems and senses. This indicates a need

Chapter 2 33

to review the ways in which we construct and teach reading and learning to studentswho engage in these activities as part of their everyday leisure. Kress and Gee are notsuggesting that because our students engage in these activities successfu Ely they donot need to be taught about them and how they might be applied in other settings(eg in learning, school and workplaces). They are suggesting that we need to know moreabout the skills and processes students have learned to apply in settings other thanthe classroom, so that we can be more effective and relevant in our teaching of literacyand learning.

2. Affordances of the semiotic systems: their cclas andconventions, their use of space and timeIn order to read and make meaning of a multimodal text, the reader must understand thecodes and conventions of the semiotic systems within the text and how they contributeto meaning making. Kress and van Leeuwen (2006:15), drawing on the work of Halliday,suggested that it is useful to look at the function of the semiotic system in contributingto meaning making, that is, what type of information it is contributing. They reportedthat Halliday identified three functions:

The Ideational function is about representation how the grammar of the semioticsystem represents or portrays the state of affairs or events being communicated inthe text.The Interpersonal function is about relationships how the grammar of the semioticsystem represents or portrays relations between the participants within the text. It isalso about how the grammar of the semiotic system creates or shapes a relationshipbetween the viewer or reader of the text an.d what is being represented in the text.It is therefore, also about the relationship between the maker of the text and his orher audience.The Textual function is about composition how the elements of meaning portrayedthrough the semiotic systems in the text have been framed and placed in space andin relation to one another in order to shape meaning. It is about coherence among,and within, the elements of the text.

Understanding these functions is useful when considering the way in which thecodes and conventions of the semiotic systems assist meaning making, as it ensures thereader/viewer attends to all the possibilities of the text, that is, what is being represented,what relationships are being represented or shaped and how is it all coming together tomake meaning. We examine the application of the functions in more detail in Chapters4, 5 and 6.

The codes and conventions of the semiotic systems are the tools tha I enable thereader/viewer to work out the meanings of all the elements of the text and 111(111 considerhow they come together to make meaning. The codes of each semiotic system providea grammar and terminology that enable the reader/viewer to identify and dei.;cribe howattention is captured, how emphasis of particular elements is creaied iind thereforehow meaning is shaped. The auditory codes employed by a radio coinromitater includepitch, volume, pace, speed, intonation. When we listen to a radio broad ast er ha sporting event such as a football match, these codes are coin hi nctl iinninding toaccepted conventions in our social and cultural experience and Iliereforn \ a(;(:ordcertain meanings. For example, as the pitch climbs, the paceI I Io: v(.1 morerapid, the speed of the speech increases and intonation beconie-: n u I infer,

34 Evolving Pedagogies -

even i,vithout understanding the words or perhaps the game, that something excitingor significant is about to happen (for example a try is about to be scored, or a goalkicked). The radio broadcaster has effectively combined the codes and conventions ofthe audio semiotic system into an audio text that fulfils all three of Halliday's functions:the ideational function (what is happening), the interpersonal fuinction (how theparticipants in the game are relating to one another and how the commentator is shapingour relationship to the participants in the game), and the textual function (the audio textis a coherent sequence of descriptive narrative about the game).

In the television broadcast of a sporting event, the visual semiotic system ofmoving images is accompanied by the audio semiotic system of commentary. Visualsemiotic codes, such as camera angles, pace, close-up shots, long shots, top down,bottom up and eye level views, are edited by the producer to provide a visualrepresentation of the match. For example, fast-paced cutting between close-up and longshots increases excitement and the choice of camera angles shapes focus on specificaction or the setting in which it takes place. In addition, linguistic and symbolic textmay appear on the screen to augment the visual and auditory information, for example,a clock might show how many minutes have elapsed and numbers and symbols mayindicate a score.

As we read/view a television broadcast, we choose which semiotic system to attend todepending upon the information we want and which is providing the best informationat the time. For example, if watching a football match, as we view the close-up of agoal being scored we might check the time and the score provided on the screen inorder to calculate the likelihood of any further goals before the end of the game. Ifthe commentary has become unintelligible, we might concentrate more on the actionprovided through the moving images.

An important point to remember about the use of semiotic systems is that the easewith which we process the information afforded to us by the semiotic systems is relatedto our background knowledge that is our knowledge of the content and genre or type ofinformation being presented. In the situation we have just described, viewing a footballmatch for leisure, our familiarity with the genre of 'sports broadcast' and the content,(the game of football) means we can automatically process the codes and conventionsof the semiotic system.

However, when we are reading or viewing a topic and/or genre that is less familiar orwhere we are trying to learn or do something new, then our knowledge of the semioticsystems and our ability to use what they afford us becomes critical. In these situations,we will need to consciously employ knowledge about semiotic systems and how theyindicate cohesion and emphasis to help us process (read and understand) the text. Forexample, we may look for cohesive ties such as 'therefore', 'first', 'after' or 'before' in thelinguistic element of the text to aid processing the chronology of the text. When listeningto the audio element of a text, we might listen for changes in volume or pitch or pausesbecause we know that this is the way emphasis is indicated in speech and therefore thesechanges may help us identify important information in the text. Similarly, we might useknowledge about layout in the visual element of the text (for example foreground andbackground) to identify what is important in an image.

In summary, knowledge of the semiotic systems helps shape metacognitive activitywhen reading by providing information that aids choice of reading paths and informsdecisions about the parts of the text to focus upon.

Chapter 2 35


The purpose of this Reflection Strategy is to develop more explicit understandingsabout:

the function (ideational, interpersonal and textual) of each element in a textthe role of each semiotic system and the codes and conventions of that semioticsystem in the texthow the elements and semiotic systems work together to shape meaning.

Overall the strategy is designed to reinforce the complexity of reading andprocessing a multimodal text and how understanding the affordances of each semioticsystem can assist in meaning making.

1 Find five minutes of a sport with which you are familiar in a format that you canview several times.

2 VVatch it once, as you would normally watch such an event, and list the majorevents or action.

3 View the excerpt again, each time attending to only one of the semiotic systems.Describe how each of the semiotic systems are used in the text, note any codesand conventions that you can identify that aid or influence your meaning making.Identify the function of the elements and semiotic systems in your processing ormeaning making.

4 Review your summary. Are there points at which one element is more useful ordominant or where they complement one another to clarify what is happening? Anexample is provided in Table 2.4. Note that italics highlight how space and time areutilised in the semiotic systems and modes to shape meaning.

THEORY INTO PRACTICE 2This activity could also be conducted in a more simplified fashion as a carefullysequenced whole-class discussion where students contribute to the joint constructionof the analytic table. It would be important to conduct the analyses yourself priorto engaging with the class to ensure you use the cortect terminology and cancarefully structure the discussion (guided discovery). However, beware that your prioranalysis does not result in the discussion becoming 'guess what's in the Leacher'shead'. Be open to students' additional observations, asking them tb oxplain their.reasoning.

Our discussions of the reading process in this section indicate that molt i od al textsreshape our reading in a number of ways. It is clearly a more demaoding process asthere are multiple elements, modes and semiotic systems present to 1)roces;. There aremore choices about what to attend to and therefore more decisions to be [wide aboutappropriate reading paths. These decisions have to be monitored as many ()I tliose textscomprise elements that are time based and therefore one's stratogìs und wndiuç. pathsmay need to change as one engages further with the text. The reading tn.sk itself willrarely take place in isolation in terms of people or place. In t eractioi I \A' i h lilnis (Jr withthe media itself may be demanded as part of the engageMelli ri 1;1 V Illt111 that

several texts are operating simultaneously (eg texts delivered I-.)v li :it'd 1 i,"11)! A' or live


TA

BLE

2.4

Sam

ple

anal

ysis

of f

unct

ion

and

role

s of

sem

iotic

sys

tem

s in

a te

xt

Exc

erpt

from

a te

levi

sed

tenn

is m

atch

bet

wee

n F

eder

er a

nd N

adal

.

Eve

nt/A

ctio

nV

isua

l sem

iotic

sys

tem

(mov

ing

imag

e)F

unct

ion

Ling

uist

ic s

emio

ticsy

stem

(au

dio)

Fun

ctio

nLi

ngui

stic

sem

iotic

sys

tem

(prin

t on

scre

en)

Fun

ctio

n

Fed

erer

ser

ves

Nad

al r

etur

ns

Ret

urn

ball

goes

out

Nad

al w

alks

to

othe

r se

rvic

e

cour

t

Clo

se-u

p of

Fed

erer

's fi

rst s

erve

Long

sho

t of w

hole

cou

rtP

anni

ng to

sho

w w

here

bal

l

land

s ou

t

Clo

se-u

p of

Fed

erer

's s

econ

d

serv

e

Long

sho

t of w

hole

cou

rt

Pan

ning

to s

how

whe

re b

all l

ands

Cut

to c

lose

-up

of N

adal

's g

roun

d

stro

ke r

etur

nP

an fo

llow

ing

ball

Cut

to lo

ng s

hot s

how

ing

that

bal

l

goes

out

Clo

se-u

p of

Nad

al's

face

show

ing

his

gaze

mov

e to

war

d

Fed

erer

and

mou

th g

rimac

ing

in

frus

trat

ion

Slig

htly

wid

er a

ngle

sho

t sho

ws

who

le b

ody

post

ure,

sho

ulde

rs

slum

ped,

sla

sh o

f rac

quet

in a

ir

Idea

tiona

l: se

quen

ce o

fim

ages

ove

r tim

e sh

ow

wha

t hap

pens

Tex

tual

: The

sel

ectio

n

and

editi

ng o

f thi

s

sequ

ence

of s

hots

prod

uces

a ti

ght,

sequ

entia

l and

coh

esiv

e

pict

ure

of th

e ac

tion

that

took

pla

ce

Inte

rper

sona

l: U

se o

ftw

o se

mio

tic s

yste

ms

(vis

ual a

nd g

estu

ral)

draw

s at

tent

ion

to-

rela

tions

bet

wee

n

Nad

al a

nd F

eder

er a

nd

posi

tions

the

read

er/

view

er to

see

the

actio

n

from

Nad

al's

poi

nt o

f

view

Sof

t voi

ce q

uiet

ly

desc

ribes

ser

ve

Exc

lam

atio

n ab

out s

peed

and

accu

racy

of s

erve

(into

natio

n ris

es a

nd

volu

me

incr

ease

s)

Nor

mal

vol

ume

and

pace

resu

mes

and

des

crib

es

Nad

al's

app

aren

t

frus

trat

ion

with

Fed

erer

's

abili

ty to

ser

ve in

way

s

Nad

al c

anno

t ret

urn

Idea

tiona

l: te

lling

wha

tha

ppen

s an

d

Inte

rper

sona

l: po

sitio

ns th

ere

ader

/vie

wer

to fo

cus

on th

e

skill

of F

eder

er

Tex

tual

: A c

oher

ent a

nd c

lear

com

men

tary

. The

lang

uage

uses

app

ropr

iate

tech

nica

l

term

s, a

djec

tives

and

adv

erbs

in

desc

riptio

ns o

f act

ion,

iden

tifie

s

the

play

ers,

and

pro

vide

s a

com

plem

enta

ry a

dditi

on to

and

inte

rpre

tatio

n of

vis

ual t

ext

(mov

ing

imag

es)

at a

ppro

pria

te

times

Idea

tiona

l: te

lling

wha

tha

ppen

s an

d al

so In

terp

erso

nal:

indi

catin

g re

latio

ns b

etw

een

Fed

erer

and

Nad

al. P

ositi

onin

g

the

read

er/v

iew

er to

hav

e so

me

sym

path

y fo

r N

adal

's fr

ustr

atio

n

and

to fu

rthe

r ad

mire

Fed

erer

's

skill

Top

left

of s

cree

n:

Tab

le s

how

ing

scor

e by

set

and

gam

e

Spe

ed o

f ser

ve

flash

es o

n sc

reen

Rev

ised

gam

e sc

ore

show

n

Idea

tiona

l: P

rovi

des

spec

ific

info

rmat

ion

abou

t act

ion

taki

ng

plac

e

Tex

tual

: App

ears

at

the

appr

opria

te ti

me

toco

here

wel

l with

vis

ual

and

audi

o el

emen

ts.

Tab

le a

nd la

yout

adhe

res

to c

onve

ntio

ns

of te

nnis

sco

re n

otat

ion

and

is th

eref

ore

easi

ly r

ead

(coh

eren

t)

Spa

tial a

rran

gem

ent

(pla

cem

ent t

op le

ft

on s

cree

n) d

oes

not

inte

rfer

e w

ith m

ovin

g

imag

e

Inte

rper

sona

l:C

ontin

uous

upd

ate

of s

core

ove

r tim

e

indi

cate

s ch

angi

ng

pow

er r

elat

ions

betw

een

play

ers

(rei

nfor

ces

mes

sage

s

of v

isua

l and

aud

io te

xt)

and digital-electronic technologies). The role of live or face-to-face encounters as part ofpaper or digital-electronic reading or viewing in everyday life is increasing. This in turnreminds us that our background knowledge, ie what we bring to the text socially andculturally, together with the context in vvhich the reading takes place are also factorsthat readers need to be aware of and take into account as they read.

To be competent, readers of multimodal texts must know and understand howall semiotic systems work, both in isolation and when combined with each other. Theyneed metacognitive skills and metatextual skills they need to be aware of and ableto monitor their cognitive processes and be aware of and understand how texts work.Readers need to understand the role of background knowledge and the context inmaking meaning.

Kress (2003:50) suggested multimodal texts mean that reading is moving from readingthe world as told (reading as interpretation) to reading the world as shown, that is readingrequires the imposition of relevance and order on what is being shown. He suggestedthat when reading the world as shown, reading becomes a process of designing. This isbecause the reader has to take control of the process and impose hypotheses and rulesfor processing all that is afforded by the text. The reader has a role in designing thereading path and therefore controls the outcome of the reading.

SECTION THREE

How multimodal texts have reshaped writingOur discussion about how reading has changed provides insight into how multimodaltexts have reshaped writing. Just as the reader has more reading paths and decisionmaking, so writers or producers of text have more writing paths and decision making todo when composing or producing a text to achieve the desired communicative purpose.Writers/producers need to consider many factors in order to ensure the text fulfilsits ideational, relational and textual functions effectively. The opportunity to conveyinformation live, using paper or digital-electronic technologies (or some combinationof these), means the writer has more modes available and more semiotic systems fromwhich to select and combine. While consideration of purpose, audience, context andavailable resources will still be critical, the questions the writer or prod ucer mightconsider will be more complex. We will now discuss these areas of decision ma king.

How purpose, audience and context influence selectionof resourcesPurpose will not only have to be considered in terms of desired communicat lye outcomebut also the best way to convey it. Previously, when communication was dominatedby the word (linguistic semiotic system), selection of genre and la nguage W;IS mostimportant in terms of meeting purpose. However, now consideration has lo be givento whether some aspects of the communication are more suited to particular modes orsemiotic systems. For example, would images an.d diagrams or an oral explatialion bebetter when providing instructions, or is there opportunity to choose a technology ormedium of communication that can combine all these in an effective wavY


When thinking about shaping the text to appeal to, and be understood by, theidentified audience, consideration must be given to the age, gender, socioculturalbackground and general literacy skills of the audience, together with their experiencewith digital-electronic hardware and software and their ability to process differentsemiotic systems. For example, as we have discussed previously, particular age groups,genders and cultures will be more experienced with and attuned to process and engagewith particular technologies (paper, live or digital-electronic) and may have a preferencefor particular semiotic systems. For some, the actual technology or medium of deliverymay impede their ability to process the information.

Context will need to be considered in terms of what is available for conveying the text.It is useless planning a PowerPoint® presentation if there is no suitable equipment inthe context where it will be used, the software is incompatible, or the electricity supplyis unreliable and subject to surges. Similarly, other forms of electronic communication,such as email and SMS, rely on the audience and context having reliable equipment andcoverage. These are real considerations in a country as large and diverse as Australia.In addition to this, consideration needs to be given to how the text will be accessedand used in particular contexts. Will it be mediated, not only by the technology, butalso by someone on site such as a teacher, presenter or employer or employee? Will oneperson use the text individually, or will it be used in a group? How many people will beengaging with the text at once? What sort of interaction might occur around the text?The answers to these questions, all of which are about context, will also inform theshaping of the text.

Selecting and weighting available resourcesOther decision making relates to selection and weighting of modes and semiotic systems,which are in turn influenced by selection of the medium or technology of delivery(paper, live or digital-electronic).

Kress (2003:51) pointed out that the selection of modes and semiotic systems caninfluence the meaning of a communication. The focus of attention may be changedand different aspects of knowledge may be emphasised because of the combinationof semiotic systems and which one is dominant. Bearne (2009:189) suggested that thewriter or producer of the text should consider what content or messages they want thereader/viewer to attend to and what would be useful to the reader/viewer in makingmeaning of the text. The writer/producer of the text should then select modes, semioticsystems and media that facilitate processing of that content and weight those modes andmedia that best facilitate the message, that is, make them dominant.

If the information or content is complex, then it may be appropriate to provide thereader/viewer with opportunity to review the information either by returning to it or byhaving it presented to them in another form. For example, if using the medium of paperor screen the same information could be provided through the linguistic and visualsemiotic systems (words plus image or diagram). If using a digital-electronic mediumto deliver film then there is also the opportunity to add voice-over, sound effects ormusic to clarify or draw attention to particular content. However, when putting thesesemiotic systems and modes together, the writer/producer has to consider where theemphasis or weight should be, and that will depend on the purpose, content, mediumand audience.

In order to demonstrate the complexity of writing a multimodal text and theconcepts we have been discussing, we composed two texts about the life cycle of a frog.

Chapter 2 39

We monitored our cognitive processes as we composed and have provided a descriptionof the decision making we engaged in and the reasons behind our choices. We havedescribed choices about semiotic systems, their relative weight or dominance in thetext, the selection and composition of images and language, the organisation and layoutof text. The projected audience for our texts is an eight-year-old school student. Thecontext or place of use we envisage is a year 3/4 primary classroom where studentsmight use the text independently, in groups or as a text mediated by the teacher whenreading it to the whole class.

Our knowledge of audience and context influenced our choice of language,organisation and layout of the texts. We chose to use conventional layout and organisationas the emphasis was on providing information and therefore this information shoul dbe easily accessible through familiar conventions and organisation.

When composing Text One, we identified two purposes: to describe the life cycleof the frog and to emphasise their importance in the environment. We chose to weightthe linguistic semiotic system more heavily as it was more suited to conveying thisinformation. Therefore in Text One, the linguistic is dominant, both in quantity (morewords than image) and in the organisation and layout of the page. The organisation andlayout is dominated by linguistic conventions rather than visual, there are headings andparagraphs, and the linguistic text comes first and takes up more space on the page. Thevisual semiotic system (in this case an image) is smaller (takes up less space overall),is placed after the linguistic text and does not provide any additional information. Itis a labelled illustration of the frog environment and includes pictures of frogs at thevarious stages of development referred to in the linguistic text. It represents visuallyonly parts of the linguistic text and does not provide information about relationshipsor sequence.

To fulfil our two purposes, the linguistic text not only provides facts about the life cyclebut also presents information about the relationship of frogs and the environment anda conclusion about their usefulness in monitoring the environment. This information ispresented in the first paragraph and re-presented in the final paragraph, reinforcing theimportance of this information. A convention of linguistic text is to repeat informationthat is important at key points in a text (in this case as an introduction and conclusion).Text One is presented in Figure 2.1.

In Text Two, our purpose was to present only information about the life cycle of thefrog, the emphasis being on the fact that it is a continuous cycle. Therefore, we decidedto make the visual semiotic system dominant in terms of space (more images than words)and the conventions of layout and organisation. The concept of a cycle was foregroundedin the text by using a circular diagrammatic layout to organise all informti on in the text.To further reinforce the cyclic nature of a frog's stages of development, we used arrowsto indicate the relationship of one stage to the next. Numbers were used lo indicate thepoint of entry into the cycle (1) and reinforce the progress of the cycle. In tl t is way, weachieved clear cohesive ties between all elements on the page. images were used toillustrate and foreground the main chauges in the frog at each stage of ils lile cycle. Weused the linguistic semiotic system for labels on the diagram, naming t lie stages andproviding additional information about time, diet and development,i1 t.haracteristicsthat could not be represented in a simple diagram. Because we clime to nlake t he visualsemiotic system dominant the language is concise and presenten in nnIn forn TI is textis presented in Figure 2.2.


FIGURE 2.1 Text One: The life cyc/e of the frog

THE LIFE CYCLE OF THE FROG

Frogs are an important part of our habitat. Because they like to live in areas that have clean

water and do not have large amounts of pollution, their presence or absence can alert us about

the health of the environment. Their life cycle has five main sta9es.

Stage One: The Ea9

Frogs l9 their es in water in a slippery jellylike clump. There may be several thousand

es in the clump and this helps protect them as female frogs do not stay and look after their

es or young. Not all the es will survive because some will be eaten by fish, birds and other

water animals. Gradually the embryo in the egg forms organs and 9iliS and after 21 cla9s if

leaves the jelly and attaches itself to weed in the water.

Stage Two: The Tadpole

The embryo will become a tadpole within 3 cla9s to 3 weeks, depending on what tye of fro9 itwill become. Tadpoles eat ver9 small plants or algae and must hide from larger pond animals

that might eat them. They have very long tails that help them move around in the water.

Stage Three: The Tadpole Be9ins to Cliange

After about five weeks, the tadpole rows hind le9s and soon after it arows forelegs. Next,

their tails become smaller and lungs develop so they can live on land when they become adult

frogs.

Stage Four: Final clian9es

Gradually the tadpole begins to look more like a frog as its mouth widens, the tail becomes

mucb smaller, and its leas arow. It also learns how to use its lunas more effectively.

Stage Five: The Adult Frog

After eleven weeks, the frog is fully developed. Now it will live mostly on land and only swim

sometimes. It will begin to eat insects and worms. And eventually it will find a mate and the

whole cycle will loe9in a9ain. As 9ou can see, durin9 its whole life the frog depends on the pond

and land to provide shelter and food, so it is important for the frog that the area is clean and

not polluted. Otherwise they will not survive. This is why frogs are an important part of our

life; their presence tells us the environment is healthy.

Chapter 2 41

FIGURE 2.2 Text Two: Life cycle of a Frog

cmIcpeSWQdun, calle fro3 staampn&AO,

ce&F'

Mcd-i_kre

11 v\iee

levelce-s LArvs,+IAA kscleeear-s

"Embro leave_,sClUrv

4-ckclede.

\It_tr ko 3 weeks

Tqct pole. hotsloy-o +cal.Ect-t-s cal9qe,

lecci pole gro.asshori-e_r, e.ccl19(33z3(

Ectis play-tfs ctret qnwr-tql


Strategy is to further reinforce concepts about decisiond weighting of elements, modes and semiotic systems

cing multimodal texts.

1 Review Texts One and Two and plan a text for the same audience and context onthe same topic but using a digital-electronic technology to convey the text. Forexample, you could plan a PowerPoint presentation including the actual slides andthe oral text that will accompany it, or you could plan a video clip. Before you beginidentify the communicative purpose and focus of the text.

2 Use a storyboard to help identify the sequence of content in the text, the semioticsystems used and their roles, the balance between them and at which points theywill enter and leave the text. For example, for the video you would plan what is ineach shot, the camera angles, auditory commentary, sound effects or music.

3 As you plan, try to be metacognitively aware of the decisions you make and thereasoning behind them. Make notes about the decision-making process, focusingon the reasoning behind the decision and how it is contributing to the text fulfilling

your purpose.

e processes we demonstrated in the description about how we wrote the twotexts and that you employ as you write this text are those that you should be

delling to students when teaching about writing multimodal texts. By engagingand reflecting upon the task you will be teaching to students, you become more

aware of the processes that need to be taught and practised. Thus your teaching and

modelling becomes more explicit.

The purpose of this Reflectionmaking and the balance anwhen composing or produ

Th

frogMO

in


The analysis and discussions in this section indicate that composing or prodtici lig

text is not simply about juxtaposing modes (eg image, text and sound). It is also aboutthe design creating multiple interconnections between the modes and elements of thetext in order to create a sign or text that shares meaning with others. Some of theselinks and interconnections will be explicit, achieved by using the shared codes andconventions of the semiotic system that will be recognised by others. Others will beimplicit, potential links that may be identified because of what the reader/viewer bringsto the text, their purpose, knowledge or socioeconomic background. Ultimately, theproducer of the text cannot control what the reader/viewer will take from it, but theknowledge and conscious decision making the writer/producer brings to the task willresult in a text that influences the way the reader/viewer enters the text, the readingpaths taken and the particular parts of the text they to which they attend. This in turnwill have some influence on their meaning rnaking.

The Four Resource Model and multimodal textsPrior to considering the role of the Four Resource Model in assisting us with the readingand writing of multimodal texts, it is appropriate to reflect on what we now understandabout the design and composition of multimodal texts and how this influences ourproduction and consumption of them. This will help us understand what students needto know and be able to do in order to produce and consume multimodal texts andthus provide links to how the Four Resource Model can assist us with this. Figure 2.3provides a summary.

Our exploration and discussion in Sections Two and Three indicated that particularknowledge and understandings are necessary in order to read or write (consume orproduce) multimodal texts. Students need to know how texts are designed and producedand how the elements within them work individually and as a whole, that is, theyrequire metatextual knowledge. Reading or writing multimodal texts requires strategicthought, hypothesising, decision making and metacognitive processing in order to usethis metatextual knowledge effectively (see Figure 2.3).

The complexity of multimodal texts, their technologies, and constant change meansthe concept of reading and writing has moved from the passive interpretation and writingof signs or texts to active choice-making and ordering; taking control of, designing andtransforming meaning In passive reading and writing, the power is attributed to thetext. In active reading and writing, in imposing order and relevance, the power is in thereader or writer.

In Chapter 1, we reported the origins of the Four Resource Model; it arose from aconcern that any approach to the teaching of literacy should be flexible and able to copewith change. The technologies by which multimodal texts will be conveyed, and the waysin which multimodal texts will be designed, will continue to change; so the flexibilityof the Four Resource Model is very useful. The Four Resource Model also focuses onachieving a balance between resources (knowledge and skills) and practices (processes).This makes the processes of reading and writing more transparent and ensures studentssee reading and writing as strategic problem-solving processes, responsive to change.The focus on what you need to know and be able to do (knowledge, skills and processes)

SECTION FOUR

Chapter 2 43

FIGURE 2.3 The design and composition of multimodal texts

facilitates the development of metatextual knowledge, strategic thought, hypothesising,decision making and metacognitive processing, all necessary to reading and writing

multimodal texts.The concept of active reading and writing, of reading and writing as design, indicates

that knowledge, skills and processes beyond code breaking and meaning making are

required beyond simple interpreting and sign making or sign reading Decisions about

using or developing texts appropriately for particular audiences and purposes and, inparticular for contexts also require knowledge skills and processes, such as hypothesising,

monitoring and making judgements, that is, text user and text analyst resources.In Figure 2.4, we have illustrated how an active reader of multimodal texts might

move between the resources available for use as code breaker, meaning maker, text userand text analyst when reading a text over time. There is constant movement betweenthe four sets of resources, drawing upon those appropriate to fulfil the reading or writing

purpose at the time. The reader moves from one set of resources to another as needed,

that is, as he imposes order and relevance among the elements of the text, and fulfils hisreading purpose. It is important to note that resources are often used simultaneously.For example, when scanning the text to identify the elements present, the reader willalso be identifying known language and images therefore employing meaning-makingand code-breaking skills together. Similarly while conducting the scan, the reader isconsidering how the text might best be used (text user activity). Ultimately the reader

uses all this information to make a hypothesis or judgement about the best source of

information in the text, that is, where to focus attention (text analyst). The reader also

engages text analyst resources when monitoring his use of the text and decides that the

video is not relevant to his needs.


Texts operate in time and space

Texts are designed to achieve particular communicative purposes.

The design of a text is influenced by each of the following on its own and in combination with one another:

purpose

audience or user

the context anci conditions in which the text is accessed or used

the technologies by which the text is conveyed

the availability, selection and use of resources available for the design of the text.

Resources can include:

elements or modes, such as speech, sound, writing, or image

semiotic systems, their codes and conventions (linguistic, audio, visual, gestural and spatial).

Coherence within and between the resources in a text is essential.

Coherence between resources (ie between the elements or modes and semiotic systems in the text) can be

achieved through:

the weighting or dominance of each one of them

the relationships among them (complementary, reinforcing, hierarchical).

their function (ideational, interpersonal or textual)

how they operate individually and together in time and space

the sensual and aesthetic effects they invoke.

FIGURE 2.4 How the Four Resource Model might be employed by a reader during the readingof a multimodal text

Context: The reader, a year 3 student, is accessing a website that provides information about frogs. His

reading purpose is to identify the life cycle of a frog for a report to be collated and presented as an oral

presentation, aided by PowerPoint. The website has linguistic information presented in conventional

paragraphs with headings, dot points and diagrams. There are also photographs, and a link to a video

about frogs in Australia.

Chapter 2 45

74a,....,

.E

1E

rcl

Ts

i =

o

lc

en

a,

a,c4

c

=ö

0-

g

Skills, processes and knowledge employed Main focus of skills, processesand knowledge being employed

1 Prior to accessing the text, reader considers purpose for.

reading and decides what criteria will be used to judge

relevant information on the site.

1. Text analyst

2. Reflects upon what he already knows about the life cycle

of frogs.

2. Meaning maker

3. Decides to look for key words, such as 'life cycle',

'breeding', 'tadpoles', 'eggs', 'adult'.

3. Text analyst

4. Reflects upon what he already knows about websites,

how to use them, their format and what might be available

to him and decides to look for both visual and print

information as they would be most useful and appropriate

for a PowerPoint presentation.

4. Text user and text analyst

5. Accesses website and decides to scan text to identify the

elements present that he will need to access (linguistic

information, paragraphs, headings, dot points and

diagrams, photographs, links to video).

5. Code breaking and meaning

making

6. While scanning, looks for keywords and images and

diagrams that illustrate them.

6. Meaning maker

7. Having identified keywords and diagrams, considers the

most relevant by evaluating against previously determined

criteria. Identifies paragraph headed 'Life cycle' and a

diagram with similar title and video on Australian frogs as

best sources.

7. Text analyst

8. Uses knowledge about diagrams and note-taking from

other experiences to read and note-take from diagram

and paragraph.

8. Meaning making and text user

9. Starts watching video and has trouble finding any

information on life cycles; it simply describes different

types of frogs. Decides this is not relevant to reading

purpose and stops video.

9 Text user and text analyst

10. Reviews notes and original purpose for reading. Scans text

once more for any other information that meets criteria.

Decides he has all that is relevant and closes down site.

10. Text user and text analyst

It is important that students learn to employ the four resources strategically andunderstand the need to monitor their effectiveness when employing them. If studentslearn to consciously employ the Four Resource Model, they will be more aware of theirresponsibilities as a reader and become more strategic, self-conscious and active readers;all characteristics necessary to becoming an effective reader and writer of multimodaltexts.

ConclusionThis chapter has focused specifically on exploring the characteristics of multimodaltexts and how these characteristics influence the ways in which we consume, use andproduce text. In Section One, we defined multimodal texts, the elements of which theyare comprised and how these elements cohere to produce meaning. Sections Two andThree examined how the reading and writing processes take place with multimodaltexts. In Section Four, we considered how the Four Resource Model can be used toassist students in becoming more strategic and active consumers and producers of text,characteristics necessary to using multimodal texts effectively.

We have very deliberatively constructed Reflection Strategies and Theory intoPractice sections that encourage you and your students to carefully examine how youread, and the decision making involved. Similarly, we have engaged you in tasks thathave promoted your understanding about the very complex and structured design ofmultimodal texts. The purpose of such detailed investigation was to reactivate yourconsciousness about hovv you read and write, because as competent adult readers muchof what we do is automatic. We are not aware of the thinking processes and decisionmaking that we now do automatically, therefore it is difficult for us to really understandthe complexity of the tasks our students engage in or to adequately model them.Modelling and conversations around the processes of reading and writing have alwaysbeen important aspects of literacy pedagogy, but are probably even more important astexts and literacy become more complex and subject to rapid change.

We hope this chapter has provided a firm foundation of knowledge about multimodaltexts and a beginning understanding of the knowledge, processes and skills necessary toengage in consuming and producing them. We hope the activities have also demonstratedthe way in which pedagogy, particularly talk about and around text, is an importantaspect of teaching and learning about multimodal texts.

In Chapter 3, we will commence more detailed practical classroom investigationsof multimodal text with postmodern picture books. We have chosen to do this for anumber of reasons. Firstly, almost all schools have access to a collection of picturebooks, therefore it facilitates immediate engagement with multimodal texts for you andyour class. Secondly, there is a danger that in addressing multimodal texts we considerthat all multimodal texts involve technology. While many do, many do not, and manyof the multimodal texts that we engage with in everyday life are conveyed by print andpaper, for example, books, newspapers, maga2dnes, information pamphlets, instructions.Therefore, we thought it useful to start with what we see as familiar (picture books) butexamined from a new perspective, as a multimodal text that enhances our understandingof multiliteracies. Finally, investigation of the modes of picture books (language andimages) and the semiotic systems employed (visual and linguistic) together withtheir design and cohesion, provides a knowledge base that can be used with all othermultimodal texts and technologies and is therefore a very good place to start.


Documents

Electronic reading - My English Portfolio · type of item, eg shampoo, deodorant, drinks, frozen meals, junk food. One advertisement should be on paper, eg magazine, newspaper or