Embedding Digital Literacies in English Language Teaching: Students' Digital Video Projects as Multimodal Ensembles

Embedding Digital Literacies in EnglishLanguage Teaching: Students’ DigitalVideo Projects as Multimodal Ensembles

CHRISTOPH A. HAFNERCity University of Hong KongHong Kong

As a result of recent developments in digital technologies, newgenres as well as new contexts for communication are emerging. Inview of these developments, this article argues that the scope of Eng-lish language teaching be expanded beyond the traditional focus onspeech and writing to the production of multimodal ensembles,drawing on a range of other semiotic modes. The article describes anundergraduate course in English for science at a university in HongKong, which incorporated elements of digital literacies. Studentswere engaged in a project to conduct a simple scientific experiment,reporting their findings (1) as a multimodal scientific documentary,shared through YouTube with a general audience of nonspecialists,and (2) as a written lab report aimed at a specialist audience. Thisarticle focuses on the multimodal scientific documentaries created bystudents and evaluates their potential in terms of language learningby drawing on data from student interviews, student comments on acourse blog, and the students’ documentaries themselves. The analy-sis shows that students met the challenge of writing for an authenticaudience by combining a range of modes (with language playing animportant role) to develop an effective rhetorical “hook” and appro-priate discoursal identity in their efforts to appeal to their audience.

doi: 10.1002/tesq.138

50 years after the computer was invented, we do not have old languagelearning plus the computer, but we have a different language learning.(Warschauer, 1998, p. 760)

In the 40th anniversary special issue of TESOL Quarterly, the editoridentified digital technologies as one social movement which has

transformed language and communication practices, altering thescope of English language teaching and learning (Canagarajah, 2006).Digital media facilitate the creation of multimodal texts that combinewriting, image, sound, and other modes, and which can easily be

TESOL QUARTERLY Vol. 48, No. 4, December, 2014

© 2013 TESOL International Association

655

shared in online affinity spaces (Gee, 2004) where readers and writersinteract. In addition, digitally mediated communication is becomingmore and more commonplace for English language learners. AsThorne and Black (2007, p. 149) observe, “for many individuals, per-forming competent identities in second and additional language(s)now involves internet mediation as or more often than face-to-faceand non-digital forms of communication.” In view of these develop-ments, a number of scholars in language and literacy education havecalled for the development of new pedagogies which “account for theburgeoning variety of text forms associated with information andmultimedia technologies” (New London Group, 1996, p. 61).

Considerable work has gone into the description of new literacypractices in digital media, and the ability to engage in these contextsis referred to in a variety of ways. These include, among others, digitalliteracies (Jones & Hafner, 2012; Lankshear & Knobel, 2008),electronic literacies (Warschauer, 1999), new literacies (Lankshear &Knobel, 2003), and multiliteracies (Cope & Kalantzis, 2000; New Lon-don Group, 1996). However, it remains unclear what role (if any) lan-guage and literacy educators should play in promoting the newmultimodal forms of representation found in digital media. As Street(2004, p. 12) points out, writing remains a culturally privileged semi-otic mode in important contexts like the academy, with the genres ofthe research article and dissertation dominating. Therefore, the“traditional” concerns of English language education, especially as theyrelate to literacy practices in the written mode, remain.

In response to such concerns, this article suggests a pedagogicalapproach which embeds digital literacy practices alongside more tradi-tional literacy practices in a university course in English for science.On this course, second language (L2) students took part in an Englishfor science project and produced (1) a digital video scientific docu-mentary, shared through YouTube with a nonspecialist audience, and(2) a written lab report, designed for a specialist audience. This articlefocuses on the first task, that of creating and sharing a multimodalensemble using digital video, and investigates its potential languagelearning benefits, with reference to the following questions: (1) Whatrhetorical challenges do students perceive in this task? (2) What multi-modal rhetorical strategies do they draw on in response? (3) What isthe role played by language and other modes in this task?

THEORETICAL FRAMEWORK

This article is informed by a sociocultural approach to L2 languageand literacy development. In particular, the article draws its theoretical

TESOL QUARTERLY656

underpinnings from a perspective on literacy advanced by scholars inthe tradition of the new literacy studies (Barton, 1994; Gee, 2008; Street,1984). According to these scholars, literacy (in both first and secondlanguages) is more than a “mere” set of cognitive skills located in indi-vidual minds, because it is socially situated in the context of literacyevents (Barton & Hamilton, 1998). In other words, reading and writingare accomplished in the service of social goals in particular socialcontexts. Thus, from the perspective of the new literacy studies, liter-acy is multiple: Different literacy events call for different kinds of read-ing and writing. For example, people draw on very different discursiveforms (genres, registers, and styles) when they write a lab report foran academic audience, compared to, for instance, when they write aFacebook update for their friends. Audience, purpose, and mediumare all factors which have the potential to shape the way that a writtenmessage is created and interpreted.

If L2 literacy is seen as socially situated, it follows that the develop-ment of literacy works best as a process of situated learning (Gee, 2004;Lave & Wenger, 1991), embedded in real or simulated social contexts.According to this perspective, developing literacy is a matter of learninghow to use language and other discursive resources, including multi-modal semiotic resources, in a variety of communicative contexts. Inaddition, it is a matter of appropriating a discourse, which goes beyondthe use of language and includes “ways of acting, interacting, feeling,believing, valuing, and using various sorts of objects, symbols, tools, andtechnologies—to recognize yourself and others as meaning and mean-ingful in certain ways” (Gee, 2005, p. 7). Thus, learning to communi-cate through a specialized discourse like the academic discourse ofscience means both (1) learning to use the appropriate technical regis-ter, genre, and style, and (2) adopting the appropriate subject positionsand social identities made available by the discourse, in order todemonstrate affiliation to an academic scientific discourse community.

According to the new literacy studies, the learner’s social identity isstrongly implicated in the process of developing literacy. Reading andwriting involve the construction of social identities, with every textualchoice also functioning to construct a particular presentation of self.In this view, identity is seen as fluid, with people using discursive prac-tices in order to construct a range of identities as they perform differ-ent roles in different social contexts. Recent studies show how thedevelopment of academic literacy is linked to writer identity, especiallyas an important part of academic literacy is constructing an authorialor discoursal identity (Hyland, 2002; Ivani�c, 1998). Such a discoursalidentity is a textual presentation of self which aligns the author withthe values of the relevant discourse community, thereby identifyingthe author as community member.

EMBEDDING DIGITAL LITERACIES IN ENGLISH LANGUAGE TEACHING 657

As Hyland (2002) notes, constructing a discoursal identity poses achallenge for those who are new to the academy, because “they arelikely to find that the discourses and practices of their disciplines sup-port identities very different from those that they bring with them”(pp. 1091–1092). Consequently, for many learners, academic dis-course, with its impersonal, nominal style, may represent a kind ofloss: a constraint on expression that they are unable or unwilling toaccommodate. For such learners, developing academic literacy meansunderstanding the ways of thinking and valuing of the specialist dis-course, and learning to assume an unfamiliar identity. In doing so,they learn to play the role or talk the talk of the disciplinary specialist(Gee, 2004; Ivani�c, 1998) and gradually gain access to the specialistcommunity and its discourse practices.

MULTIMODALITY AND LANGUAGE LEARNING

As suggested above, multimodality, the combination of various semi-otic modes (e.g., speech, writing, image, gesture) in texts, can be seenas one important consequence of developments in digital media.Research suggests that these developments are leading to a paradigmshift in forms of representation, moving from the logic of the page tothe logic of the screen (Kress, 2003). This shift goes beyond digitalmedia, crossing over to more traditional media like books and newspa-pers. For example, Lemke (1998) shows how visual modes of commu-nication have gradually gained prominence, with more imagesappearing in scientific publications over time. In these texts, meaningis carried only partly by writing, with an increasing load now carried byimages as well. Kress (2000) maintains that it is now impossible tounderstand the meaning of a text without taking into account all ofthe modes that contribute to it. Science textbooks are a good exampleof this phenomenon, with much of the meaning now carried by theanalytical diagrams interspersed with the text.

An important concept in a theory of multimodality is the idea thatdifferent modes have different affordances for meaning making. Theterm affordance was introduced by Gibson (1979) in his work on visualperception; for the purposes of this article, it can be defined as anopportunity for meaningful activity or interaction which may or maynot be taken up. Kress considers that modes offer different affor-dances, and he develops this theory by considering the logic of the pageand the logic of the screen. Meaning on the pages of a book is typicallymade through the mode of writing, which operates according to alinear, sequential logic; readers are expected to begin on page oneand proceed through the book in a sequential manner. In contrast,

TESOL QUARTERLY658

meaning on the screen draws heavily on the mode of image, which fol-lows a spatial, simultaneous logic: Viewers can take in the whole screenat a glance and are typically not expected to process it in a linear way.The different affordances of writing and image mean that they canconvey different kinds of meanings, leading to what is termedfunctional specialization, with writing tending to convey particular kindsof meanings, whereas image conveys others. For example, writing issuited to the discussion of abstract concepts because these can be cate-gorized by words, but may be difficult to show visually. Conversely,images are suited to the depiction of finely graduated properties, suchas the many different shades of color one sees when looking at a tree(using language, we would likely describe the color in categoricalterms, e.g., green).

The affordances of modes should be seen as mere tendencies,rather than as hard and fast rules that dictate how meaning can bemade. Although the work of Kress has served to greatly raise awarenessof the semiotic contribution of different modes in text, it has alsobeen criticized for creating unnecessary “binaries” (Prior, 2005).Indeed, it is clear that the logic of the page and screen are both pres-ent and combined in texts like the digital videos described in this arti-cle. Video is a highly visual medium and so the affordances of image,to create a spatial representation processed immediately and simulta-neously, can be taken up. At the same time, however, video unfolds intime and so includes a sequential element, similar to a written narra-tive.

Furthermore, a theory of multimodality should not limit itself tocombinations of image and writing alone, and consequently the theoryelaborated here aims to take into account all modes of communica-tion. Although it recognizes that, in a given text, certain modes maydominate at certain points, it avoids privileging any of those modes.Thus, in their manifesto, the New London Group (1996) identify atotal of five different modes of meaning that they consider meritsystematic attention in literacy education: (1) linguistic, (2) visual,(3) audio, (4) gestural, and (5) spatial. Each of these modes can bedescribed as a systematic semiotic resource, with its own internal gram-mar (with reference to the visual mode, see Kress & van Leeuwen,2006). Multimodality describes how these various resources are“orchestrated” in order to create a multimodal “ensemble,” a materialproduct which combines “a plurality of signs in different modes into aparticular configuration to form a coherent arrangement” (Kress,2010, p. 162). Different kinds of meaning-making combinations arepossible. For example, work on image–text interaction shows that themeanings in different modes can be in a relationship of (1) concur-rence, where the meanings of each mode reinforce each other;


(2) complementarity, where the meanings are supportive but different,coloring in elements sketched out in the other mode; and (3) diver-gence, where the meanings contradict one another (Unsworth, 2008).Because of such relationships, the meaning carried by one mode inter-acts with the meaning of others to create new meanings; consider, forexample, the tension created in a newspaper article whose writtenwords denounce a person’s criminal act, but which simultaneously pre-sents a sympathetic photograph of that person.

One final (for present purposes) point to note about multimodalityis the way that it is bound up with the concept of design. In part, theterm design is needed because the terms writing and reading are nolonger adequate to describe the construction and interpretation ofmultimodal texts. According to Kress and Jewitt (2003, p. 17), “designrefers to how people make use of the resources that are available at agiven moment in a specific communicational environment to realizetheir interests as makers of a message/text.” Here, the characterizationof meaning makers as interested or purposeful also highlights the waythat multimodal ensembles are considered as strategic rhetorical com-binations. Kress (2010, p. 23) makes the point that design is prospec-tive: it is “a means of projecting an individual’s interest into theirworld with the intent of effect in the future.” Design therefore involvesthe concept of rhetorical strategy, sometimes used in research on writingto refer to an author’s use of linguistic and structural forms (Bhatia,2004; Hyland, 1998) to achieve a particular communicative purpose.In this article, I focus on students’ use of rhetorical strategies in thedesign of multimodal ensembles, where such strategies are defined asgoal-directed acts of meaning making in multiple modes that aremeant to realize the designer’s communicative intent.

All of the foregoing has implications for language teaching andlearning. If it is accepted that modes other than writing are becomingmore prominent, then one question which arises is: How can a focuson multimodal forms of representation be incorporated in the lan-guage teaching curriculum? The concepts described above provide auseful starting point for reflection on the ways that meaning is madethrough multimodal ensembles. Broadening the curriculum to focuson multimodality would cater to the needs of second or foreign lan-guage learners, who are increasingly involved in the consumption andproduction of multimodal texts such as web pages. More particularly,engaging with multimodal representations of science in the academiccontext has the potential to encourage reflection on the use of imageand other modes as an essential part of scientific communication, ahighly multimodal activity (Molle & Prior, 2008; Tardy, 2005).

The question of how to incorporate multimodality in the curricu-lum has already received some attention from researchers in fields

TESOL QUARTERLY660

such as rhetoric and composition (Selfe, 2007) and literacy studies(Hull & Katz, 2006; Vasudevan, Schultz, & Bateman, 2010). In the con-text of L2 language education, however, relatively few studies investi-gate the pedagogical applications of multimodal composition, with afew notable exceptions. For example, Nelson (2006) studied L2 univer-sity students in the United States taking an experimental course inmultimedia writing, and Yang (2012) studied Taiwanese undergradu-ate English majors taking a course in technology and language teach-ing. The present research, which describes the mainstream adoptionof multimodal composition as part of what has become a large-scaleuniversity course in discipline-specific English at a Hong Kong univer-sity, provides a valuable addition to this literature. The following sec-tion describes the pedagogical approach adopted, designed to provideL2 students with opportunities to engage with and reflect on thedesign and orchestration of multimodal ensembles.

PEDAGOGICAL CONTEXT AND COURSE DESIGN

The course described here is a one-semester course (thirteen 3-hoursessions) in English for science at a university in Hong Kong. It istaken by undergraduate students majoring in biology, chemistry, envi-ronmental science and management, and mathematics. The courseadopts a project-based learning approach, with the course syllabusstructured not around the introduction of communicative functions orlinguistic forms, but around stages in the completion of a self-directed,experiential project (in this case, a simple scientific study). In the con-texts of English as a second or foreign language, such an approachcan provide “opportunities for language learners to receive compre-hensible input and produce comprehensible output” (Beckett, 2006,p. 4) and is also considered to be supportive of “principles of learner-centred teaching, learner autonomy, the negotiated syllabus, collabora-tive learning, and learning through tasks” (Hedge, 1993, p. 276). Inaddition, projects lend themselves to computer-assisted approaches tolanguage learning, because they involve students in the creation ofmeaningful texts, which draw on the affordances of digital media toengage with an authentic audience in the L2 (Warschauer, Schetzer, &Meloni, 2000).

The focal project on this course is an English for science project inwhich students conduct a simple scientific study on an assigned topic.The study involves background reading, data collection and interpreta-tion, and presentation of findings, and is reported in two ways: (1)through a digital video scientific documentary, aimed at a nonspecial-ist audience, shared publicly through YouTube and viewed in class


(group work); (2) through a written lab report designed for a special-ist audience (individual work). The project integrates knowledge (ofcontent and language) and skills, such as finding, evaluating, and syn-thesizing information, and collecting, interpreting, and presentingdata (orally, through multimedia, and in writing). In presenting theirfindings, students engage with both a more popular genre (scientificdocumentary) and a more academic genre (lab report). The topicsused for the period reported on in this article are Blind as a Bat, aboutthe blind spot in the human eye, and Taste Me if You Can, about theinteraction of smell and taste in humans.1

The project-based learning approach engages the teacher in a rangeof roles. First and foremost, the teacher acts as planner and organizerof learning experiences, designing the project task and process, to bejointly navigated with students. On the course in focus, this meansdesigning a quasiexperimental study for students to carry out and set-ting interim goals for students (e.g., data collection complete by Week4, script writing complete by Week 5). During the project process, theteacher is also called on to act as an instructor who provides modelsand resources, anticipating learners’ needs by designing relevant in-class learning activities. For example, students are shown short scien-tific documentaries, created either professionally (by the BBC) or bystudents (from a previous pilot course). The ensuing in-class discus-sion draws attention to a range of multimodal design issues: audienceand purpose, generic structure, use of appropriate scientific terms fora nonspecialist audience, use of visuals, use of sound, technical issuesassociated with video recording/editing. Finally, the teacher also actsas mentor, providing students with feedback on their progress, espe-cially any written drafts that they have created, as here, where work-shops are organized to address stages in the project process:workshops on reading (including finding and evaluating information),script writing and presentation, video editing, and lab report writing.

Table 1 summarizes the project process as implemented in thecourse. The two phases (documentary and lab report) receive roughlyequal attention: seven 3-hour tutorials for Phase 1, six for Phase 2. Forthe purposes of this article, I focus on Phase 1, in which students workin teams to construct a multimodal ensemble through digital video.The video project process consists of six overlapping stages: (1)reading, (2) data collection, (3) scripting and storyboarding,2 (4)

1 These topics, as well as other teaching resources used on this project, can be accessed atthe project website: http://www1.english.cityu.edu.hk/acadlit.

2 In film production, storyboard refers to a visual plan of the film, similar to a cartooncomic strip, used to represent the scenes in the multimodal ensemble. Each scene incor-porates a visual depiction, written script/narration, and notes about soundtrack and cam-era action (pan, zoom, etc.).

TESOL QUARTERLY662

TABLE1

Project

Phases/Stagesan

dTeach

ingActivities

Output

Project

stage

Tutorial

focu

sTeach

errole

Studen

tactivity

Englishfor

Science

Project

Phase1:

Digital

video

scientificdocu

men

tary

(Seven

3-hrtutorials)

1A.Reading

Readingstrategies

Criticalliteracies

Readingworkshop

Planner/organ

izer

Instructor

Men

tor

Analyzeresearch

tools

Researchtopic

1B.Dataco

llection

Planningproject

work

Planner/organ

izer

Form

team

sConduct

study

Film

process

1C.Scripting

Docu

men

tary

genre

Writingworkshop

Planner/organ

izer

Instructor

Men

tor

Analyzesample

docu

men

taries

Write

script/storyboard

Orallypresentscriptin

class

1D.Filming

Docu

men

tary

genre

Planner/organ

izer

Instructor

Analyzesample

docu

men

taries

Film/actoutnarrative

Film

backg

roundscen

ery

1E.Editing

Video

editingworkshop

Planner/organ

izer

Instructor

Men

tor

Learn

editingtools

Editvideo

1F.Sh

aring

Sharingsession

Planner/organ

izer

Assessor

Presentvideo

sin

class

Pee

rreview

video

sPhase2:

Written

lab

report

(Six

3-hr

tutorials)

2A.Pre-writing

Lab

report

genre

Writingworkshop

Instructor

Analyzesample

scientificreports

Write

onsimulatedtopics

Planlabreport

2B.Writing

Out-of-classwork

Planner/organ

izer

Write

labreport

2C.Reviewing

Individual

consultations

Men

tor

Assessor

Review/ed

itlabreport


performing and filming, (5) editing, and (6) sharing. The project iscarefully integrated into lessons and some of the project work takesplace in class, with a reading workshop (1 hour), script-writing andpresentation workshop (3 hours), video workshop (two 1-hour ses-sions), and an in-class sharing session (3 hours). In these workshops,the teacher acts as mentor and instructor, guiding students in further-ing project aims—for example, researching their topic, writing andpresenting their script, and focusing students on the development ofrelevant language skills such as critical reading, writing, and speaking.Technical aspects of video editing are addressed in the two video work-shops, but the amount of class time spent is kept to a minimum. Fur-ther technical support is provided through a resource website3 whichincorporates screencasts for common video editing tasks, and thisallows students to work autonomously on their projects outside of class(Hafner & Miller, 2011). The final video is shared online through aclass YouTube account and embedded in a course blog (students pro-vide informed consent and upload the videos themselves), with stu-dents giving online peer feedback after the sharing session.

In constructing the videos, students typically record video and takepictures, which are subsequently edited together with soundtrack,audio narration, and other visual and sound effects. A range of digitalliteracies is embedded in the larger project process, including criticalreading of internet texts; design, recording, and editing of digitalvideo; and sharing, commenting, and associated online interactions inthe course blog. Similarly, multimodal meaning making is an impor-tant component of various stages in the process, particularly in theprocesses of scripting, storyboarding, performing, recording, and edit-ing. As students design their videos, they must make choices aboutcombination of modes, such as text on the screen, oral narration,image, sound, and gesture. Thus the video project allows a focus onmultimodality and new media, engaging students creatively with thepopular science documentary genre. At the same time, it acts as abridge to Phase 2 of the project, in which students draw on the samestudy to complete an academic writing task.

DATA SOURCES AND ANALYTICAL METHODS

This study draws on data from a research project investigating theuse of digital video on the course in question for one cohort of 67 stu-dents over a period of one semester (13 weeks). I was the course coor-dinator as well as the principal investigator on this research project,

3 www1.english.cityu.edu.hk/acadlit/index.php?q=node/3

TESOL QUARTERLY664

with the two other members of the teaching team as co-investigators.4

The research design adopted a qualitative interpretive paradigm(Davis, 1995; Richards, 2003), aiming to provide a rich data set inorder to gain insights into the complex digital video project process.Multiple perspectives were elicited by drawing data from a studentquestionnaire (with open and closed items), students’ weekly com-ments on a course blog (where they were asked to reflect on the pro-ject process), semistructured focus group interviews with both studentsand subject lecturers, students’ digital videos, and students’ writtenreports. For the purposes of the present study, the primary data con-sist of the digital videos that students created, supported by data fromthe course blog and interviews with students. Sixty-two students postedto the course blog over the course of the semester, and 21 studentstook part in interviews. Standard procedures of ethical research prac-tice were adopted, with all students giving their informed consent toparticipate in the study. Students’ names are replaced by pseudonymsthroughout the data analysis.

An initial analysis of the 20 videos that were created by students(working in groups of three to four) revealed a range of quality in thestudents’ work, though it should be noted that all groups were success-ful in completing a video which documented their project in a well-organized way.5 The better videos combined an effective presentationin the L2 (well scripted and well narrated at a lively pace) with appro-priately selected visuals and music. In contrast, a number of problemareas were observed in the weaker videos, including difficulties in pro-nunciation and intonation in the narration, inappropriate choice ofmusic creating an incongruent mood (e.g., overly dark or dramatic) orovershadowing the narration, problems of focus in the script (e.g.,lengthy and repetitive descriptions of procedures), and occasionaltechnical issues (such as poor sound quality in the videos). A persis-tent issue, even in some of the better videos, was that of pacing. Itappears to be challenging for students to create videos that unfold atan effective pace—not too fast, not too slow—with respect to both nar-ration and moving images.

For the purposes of this article, three student videos have beenselected for detailed analysis. This analysis is intended to demonstratethe potential of the video project activity, by identifying examples ofstudent videos that adopt a range of rhetorical strategies and discour-sal identities in their multimodal ensembles. The procedure used forselection was for each of the teachers on the course (including myself)

4 I am grateful to Lindsay Miller and Connie Ng for their essential contributions and par-ticipation in this course.

5 The videos analyzed can be viewed on the course blog: http://en2251.edublogs.org.


to nominate the more effective videos from their class for consider-ation, on the basis of their organization and content, use of multime-dia, and use of language. The videos selected for analysis are thereforenot necessarily typical of all the videos created. In arriving at the finalselection, the aim was not primarily to select cases representative ofthe whole group, but rather cases that demonstrated the variety ofstrategies that L2 students can and do employ when attempting multi-modal compositions that integrate sound and image as well as use ofthe second or foreign language. As Janesick (2003, p. 70) notes, “thevalue of the case study is in its uniqueness.” As we shall see, the threedocumentaries which have been selected for further analysis and illus-tration demonstrate the possibilities for multimodal expression affor-ded by the digital video medium, as well as the range of rhetoricalstrategies, identities, and linguistic registers that students draw on inthe videos in an effort to appeal to their audience.

In order to analyze the selected videos, a system of multimodal tran-scription was devised, drawing on Baldry and Thibault (2006) andJewitt (2006). The basic unit of analysis was the shot, which can bedefined as a continuously filmed stretch of video, without cuts. Shotsare easy to identify reliably but may present more than one visualframe, or configuration of visual resources, as when a shot involves acamera pan or zoom. In such cases, it was necessary to subdivide theshots into visual frames which were separately analysed. Each shot–visual frame combination was transcribed, paying attention to

• speech—spoken narration (voiceover or on-screen narrator);

• soundtrack;

• music or sound effects;

• setting, participants;

• where the shot is located, what or who it depicts;

• action, gesture, gaze—what the participants do, how they usegaze and gesture (this category was also used to transcribe writ-ten text appearing on screen, e.g., titles and subtitles); and

• camera position, perspective—camera distance (e.g., long,medium, close shot), horizontal angle (e.g., direct, oblique),vertical angle (e.g., low, medium, high).

Not every detail of the multimodal ensemble was transcribed. Forexample, tone of voice and facial expression were not captured whenthe speech was transcribed. Nevertheless, the process of transcription,proceeding through the text visual frame by visual frame and parsingeach into its component parts, forces the analyst to notice how thevarious elements combine to make up the whole as well as how these ele-ments may be strategically deployed to achieve a particular rhetorical

TESOL QUARTERLY666

effect on the audience. The process of transcription was facilitated byvideo transcription software called ELAN (Max Planck Institute forPsycholinguistics, 2002). This software provides an interface that makesit easy to associate visual information from the video with the transcribedinformation, as illustrated in Figure 1.

STUDENTS’ PERCEPTIONS OF THE MULTIMODALCOMPOSITION TASK

Students’ perceptions of the multimodal composition task weredescribed in their weekly reflections on progress posted to the courseblog as well as in interviews conducted after the video project was com-pleted. In Week 2 of the course, students were asked to identify possi-ble challenges of the video project and many pointed to their own(lack of) prior experience with the construction of multimedia pro-jects. Although this information was not directly solicited, of the 52students who participated in the online discussion that week, 34 men-tioned prior experience. Of these, 29 indicated that they came intothe course with little or no experience of creating multimedia projectsin general, and scientific documentaries in particular. A typical com-ment reads: “This is my first time to produce a scientific documentaryand I have no previous experience of making any type of videos” (blog

FIGURE 1. ELAN transcription software


post, Annie). The few students who did report some kind of priorexperience had usually acquired it in the course of completing videoprojects at secondary school or for another university course.

Students perceived two other main sources of rhetorical challengerelated to (1) audience attention and (2) multimodal orchestration.First, students were aware of the expectation that, with their permis-sion, their work would be uploaded to YouTube and publicly sharedwith a wide and unspecified Internet audience. As a result, many stu-dents felt motivated to produce videos capable of attracting the atten-tion of this audience. Commenting on the public availability of thedocumentaries, one student said, “I’m pretty concerned that we needto do it as good as we can” (focus group, Terry). Students felt thatmultimodal presentation techniques could be used to creatively getattention, for example, by providing an attention-getting opening totheir documentary or a memorable closing. The following comment isillustrative:

The first impression of audience is the critical point to determine thesuccess of a good documentary since if the audience’s attention cannotbe attracted at the beginning, they will have no interest to continue towatch the video even the information is rich and constructive. I agreewith Jimmy. He said that visual stimulation would be the easiest way tomake them remember the video. (Blog post, Joyce)

Although students perceived alternative modes as a useful resourceto attract the attention of the audience, some also pointed to a dangerin over-relying on multimedia. As one student noted, “Using excesstechnique may also annoy the audiences” (blog post, Bo Lai). Studentsperceived language to be an equally important resource, with somementioning the importance of a good script:

The other challenge is how to write an interesting script for narrationin the documentary. Wording used is really important to attract theattention of the audience. The script has to be funny, at the same time,informative and easy to understand. (Blog post, Kitty)

Although students perceived multimodal representation as aresource for getting attention, they also noted associated challenges.Students had to think carefully about how to combine multiple modessimultaneously, as demonstrated in the following comment:

I think the most challenging thing is how to give an attractive andinteresting present because we use lots of method involved in ourvideo. For example, use pictures, use a narrator, stand in front of thecamera for speaking and use music or many, many elements weinvolved in order to give a whole product to make it more interesting.(Focus group, Dan)

TESOL QUARTERLY668

Students generally demonstrated an appreciation of the range ofmultimodal semiotic resources available to them. In Week 3, they wereasked to evaluate sample scientific documentaries (one professional,one by students) and answer the question: “What do you think workedwell in the documentaries that we viewed?” In their blog commentsstudents noted how different modes complement one another. Inaddition to the importance of a good script, they identified a widerange of semiotic resources, including moving images and animation,charts and tables for scientific data, subtitles, different camera anglesand lighting, background music, sound effects, interesting locations,interesting participants, and facial expression. Taking into accountthese observations, many students suggested possible strategies to usein their own documentaries.

STUDENTS’ MULTIMODAL ENSEMBLES: THREE CASES

The students’ comments reveal that they perceived a rhetoricalchallenge in getting the attention of the YouTube audience and thatthey felt this should be met using effective visuals and an effectivescript. The three cases presented here illustrate the wide range of rhe-torical devices employed by students as well as the way that theserhetorical devices were realized in multiple modes. The three cases,discoursal identities or student roles adopted, and rhetorical appealsemployed are summarized in Table 2. Considering the descriptiveinformation (length, number of shots, number of words in script)reveals some variation. Length varies from 8 minutes 42 seconds to16 minutes 19 seconds, number of shots from 67 to 163, number ofwords in script from 930 to 1,167 (with this much verbal language,clearly language was an important mode in all three cases). As Table 2shows, each case uses a different rhetorical hook to catch theaudience’s attention, setting up various producer roles, or discoursal

TABLE 2

Discoursal Identities and Rhetorical Appeals in the Three Cases

Case Rhetorical appeal Discoursal identity

1. Investigation of a startling fact(8 min 16 s, 67 shots, script930 words)

Did you realize there is ablind spot in your eye?

Student as “scientist”

2. Investigation of a social issue(9 min 49 s, 89 shots, script935 words)

Why does the cafeteria foodtaste so bad? Is it only thetaste, or is it the smell as well?

Student as “reporter”

3. Investigation of a personal issue(16 min 19 s, 163 shots, script1,167 words)

Why can’t I taste this orangejuice? Is there somethingwrong with me?

Student as “traveler”on a “journey ofexperiment”


identities, and a range of writer–reader relationships. Attempts athumour and creativity are evident, especially in the second and thirdcases.

Case 1: Investigation of a Startling Fact

In the first documentary,6 on the Blind as a Bat topic, the studentsframed it as a serious scientific inquiry. One of these students notedthat “the aim of showing a documentary is not for entertainment butfor telling others that fact or phenomenon that people have found ordiscovered” (blog post, Jimmy). In order to attract the attention of theaudience, the students present their topic as an investigation of a star-tling fact, pointing out that although we might feel that our vision is100% complete, each eye actually has a blind spot and objects fallinginto this blind spot cannot be perceived. In the documentary, the stu-dents adopt the role of expert scientists explaining their study to a lay,uninformed audience. This presentation of self as scientist is achievedthrough a combination of visuals, script, and soundtrack. In generalterms, the narrator role is backgrounded and a sense of distancebetween the video producers and their audience is achieved. Thisbackgrounding is accomplished by remaining “offstage”; except forone short extract (see below), the students do not narrate on camerabut use voiceovers instead. The script frequently employs a formal sci-entific register and the soundtrack is a mellow, slow-moving electricguitar playing through the whole documentary. This choice of back-ground music aids in creating a nontechnical frame for delivery of thescientific message, reducing any sense of intimidation that viewersmight have with respect to a scientific presentation.

The opening hook is presented both visually and through thescripted narration, as illustrated in Figure 2, which shows a number ofkey visual frames from Shot 3 and the accompanying narrative. Here,the students introduce the startling idea that what we see may not betrue and invite us to imagine that a tennis ball could suddenly disap-pear in front of us “naturally.” The idea is reinforced by the visualimage which shows one of the students throwing a ball toward thecamera; the ball disappears and reappears as it comes hurtling towardthe viewer (16.0 s). This visual image engages the viewer throughstrong elements of interpersonal interaction. Although the image ofthe student is a medium-long shot, with corresponding far social dis-tance implied, the image is a “demand” that engages the viewer (Kress& van Leeuwen, 2006), showing the student looking directly at the

6 http://youtu.be/mL55njba7IU

TESOL QUARTERLY670

viewer and the ball flying straight toward the viewer. This choice ofcomposition, employing a direct (front-on) camera angle rather thanan oblique (side-on) one, powerfully involves the viewer as part of thescene depicted. At the same time, the action of the ball disappearingconfronts viewers with a sequence of images that they know cannot bereal. Of particular interest here is the way that the students have com-bined visual and verbal resources to heighten engagement: The inter-personal interaction of the moving images is complemented by anengaging use of language with direct address to the audience throughthe use of imperatives (“imagine,” “welcome”) and the second personsingular pronoun you (see Figure 2). In this, the students demonstratetheir knowledge of pragmatically appropriate language as well as visualresources which can augment or reinforce message content.

In order to present background and theory in their documentary,the students created an animation, partly illustrated in Figure 3 (notethat they taught themselves the technical skills needed to do this). Theanimation draws on a series of analytical (diagrammatic) images, show-ing a cross-section of the eye, a close-up representation of rods andcones, and the connection between eye and brain. There is a clearconcurrence between image and text here: In the first frame, the imagedepicts light moving from the picture to the eye, refracting throughthe lens of the eye, and ending up on the retina. The animation zoomsin on the eye to portray nerves, which flash as if stimulated. The

FIGURE 2. Opening (shot 3)


animation then zooms out and the image depicts information movingto the brain, with arrows appearing at the eye and forming a path tothe brain. This sequence of diagrams visually evokes an objective, scien-tific voice which is partly matched by the scientific register adopted inthe script (see Figure 3), drawing on the passive voice (e.g., “an imageis then formed”) as well as formal technical lexis (e.g., “generate nerveimpulse”). Here we see how the structure and content of the imagesand the structure and content (i.e., grammar and lexis) of the textwork together to create a presentation of self as expert scientist. At thesame time the animation of the images is visually appealing, thus serv-ing to enrich the presentation of scientific information in the text andmake it more engaging to a general audience. In this way, the studentsdemonstrate effective use of their L2 in conjunction with digitalresources to communicate through digital media.

It is in the discussion of results that the students adopt the scientistrole in the most explicit way. In Shot 52 (Figure 4), one of the stu-dents appears dressed in his lab coat and explains the results to view-ers, the only instance of on-screen narration in the wholedocumentary. The students have carefully drawn on a range of multi-modal resources to assume the role of expert scientist. First, they havechosen a setting and various props, equipment, and tools associatedwith the university scientist. The student, portrayed at medium-closedistance from an oblique angle, is wearing a white laboratory coat andis seated at the front of a university classroom typing at a computer.The student uses gaze to cue the involvement of the audience. At thebeginning of the shot, his gaze is fixed on the screen, reflecting hisinvolvement in his “work” on the computer. Then his gaze shifts tothe viewers and as the visual image shifts from offer to demand (engag-ing the audience by looking directly at them), we are invited to listen

FIGURE 3. Background and theory (shot 10, visual frames 10b-10d)

TESOL QUARTERLY672

to what he has to say. Finally, after he has delivered his interpretation,his gaze shifts away and he resumes his “work.” The scientific imagecultivated through the visual images is matched by a correspondingscientific register in the verbal text (see Figure 4), demonstrating thestudents’ successful integration of verbal and nonverbal communica-tion modes.

Case 2: Investigation of a Social Issue

In the second documentary,7 on the Taste Me if You Can topic, thestudents have created a video that evokes the experience of watching alight-hearted current affairs program on TV. In order to catch theattention of the audience, they present their video as an investigationof a social issue. In the opening, they point out that students haverecently been complaining about the food in the university cafeteria,and they set out to investigate why the food is so bad by conducting anexperiment into smell and taste. The narrator humorously asks, “Is justthe taste of the food that makes students complain? I don’t think so.Maybe the smell of the food is also an important part” (1 min 59 s).

One of the student producers of this documentary observed that “itis very important to deliver our messages in a easier and funny way”(blog post, Bo Lai), and so it is perhaps no surprise to see this light-hearted approach and attempt at humor. The students position them-selves as “reporters” and adopt a more prominent narrator role, withan on-screen narrator who appears at various points throughout thedocumentary. With the adoption of this on-screen narrator comes agreater sense of viewer involvement (compared to Case 1), with view-ers invited to identify with the narrator when he addresses them. Once

FIGURE 4. Discussion (shot 52, visual frames 52a-52c)

7 http://youtu.be/8-1FAV1_VTg


again, this particular presentation of self is achieved through a rangeof multimodal resources used in combination, and it includes not onlycreative use of visuals and soundtrack but also a creative rhetoricalhook in the form of the text above to catch the attention of the audi-ence. As in the first example, the students show their ability to crea-tively marry visual with verbal pragmatics in their L2.

The opening eight shots of this documentary illustrate how studentshave drawn on multimodal representation strategies. These eight shotsare spread across six different visual frames over about 30 seconds asshown in Figure 5. The text, visuals, and sound are all reminiscent ofa TV channel and TV show, so that we have the impression that weare watching the beginning of a program on the “CityU ScientificChannel.” Shot 1 shows a television test screen, with the initials CUSC(standing for CityU Scientific Channel) in a cluster (Baldry & Thibault,2006) in the top center. Shot 2 is a title page which incorporates infor-mation about language and weather (top left cluster), TV channel(top right cluster: “CUSC”; as well as the bottom cluster which is scroll-ing: “Scientific Channel”), and the “Next Programme” announcement(middle cluster). Shot 3 is an advertisement for the upcoming pro-gram, including the title and an illustration. Shot 4 is the scientificchannel logo, which includes an animated image and the text“Scientific Channel.” Shots 5 to 7 are the program intro and includethree shots of the university cafeteria as well as scrolling credits (e.g.,“Fieldworker & Researcher: Jason”). Finally, in Shot 8 we see thereporter at the scene, eating food in the cafeteria. Throughout thisopening sequence we hear three successive musical jingles, accompany-ing Shots 1 to 3, Shot 4, and Shots 5 to 7, respectively. This accompanying

FIGURE 5. Opening (shots 1-8)

TESOL QUARTERLY674

soundtrack evokes genres of television advertisement and programlead-in in a compelling way. Thus, the students blend their knowledgeof science with their knowledge of communicative conventions andmedia genres—specifically, certain types of TV program—to build aneffective audience-oriented message in their L2.

From a semiotic point of view, these eight shots are very rich; however,a full analysis of the multimodal content is beyond the scope of this arti-cle. What is immediately striking about the images, text, and soundtrackused is their interdiscursivity: The students have appropriated the multi-modal resources of television for the purposes of their documentary. Asone of their classmates commented, “The scientific channel produced isvery much like the one we watch on TV! It’s wonderful!” (blog post,Annie). This appropriation of discursive resources serves a strategic pur-pose, which is to present the documentary in a “funny” way, allowing thestudents to playfully and humorously adopt the role of television repor-ter (the image in Shot 3 contributes to this sense of playfulness, as dothe “bloopers” at the end of the original video). It is also worth empha-sizing the impressive use of sound effects in this documentary, whichhave been very carefully chosen to give the impression of watching TV.The documentary draws on numerous short musical/sound effects,which emphasize, for example, the appearance of a logo or other visualelement. Finally, the rhetorical device that the students have chosen pro-vides a visual way for them to structure their video, following a televisionschema (advertisements, program intro, program content, closing cred-its) that viewers are familiar with. In particular, Shot 4 (the logo) reap-pears at different stages of the documentary in different colors and withdifferent accompanying text, providing visual coherence by introducingdifferent sections of the documentary (e.g., Introduction, ExperimentalMaterial, Procedure). Here, the students demonstrate an ability toreinforce and augment their L2 composition and presentation by draw-ing on visual and aural resources in an effective way.

Case 3: Investigation of a Personal Issue

In the third documentary,8 also on the Taste Me if You Can topic,the students’ video evokes a drama which tells a personal story. Inorder to catch the attention of the audience, the students presenttheir topic as an investigation of a personal issue, and they use anoverarching narrative to unify the documentary. Near the beginningof the video we see two students meeting outside the cafeteria. One(Lee) has a bad cold and is unable to taste his orange juice. He is

8 http://youtu.be/L7LnnpRtCJw


about to rush to the university clinic to see a doctor when his friend(Richard) stops him and they decide to investigate the issue on theirown. The following is a transcript of their role-played conversation:

R: You look so bad. What’s going on?

L: Yes, I got a cold.

R: Oh, you are sick. Do you need to drink something?

L: Oh, yeah, thank you! Oh, Richard, is it orange juice?

R: Yeah, absolutely!

L: But it is only sweet.

R: It should be sour too.

L: I can’t taste any sour.

R: So strange!

L: There is something wrong with my tongue. I better go to the clinic.

R: There may be some underlying reason. Your tongue may be nothing wrong.

L: Oh, really?

R: Yes. Let’s investigate!

As the transcript illustrates, the choice to represent their “investiga-tion” as a personal narrative at times leads the students to adopt a con-versational register, including the use of informal lexis (“yeah,absolutely”), ellipsis (“so strange”) and contractions (“what’s,” “can’t,”“let’s”). The documentary follows the students as they go to the libraryto do research and form their hypothesis, collect data as part of anexperiment about smell and taste, and finally conclude that there isnothing wrong with Lee’s taste buds. Integrated with the overarchingnarrative are more conventional structural elements of introduction,methods, results, discussion, and conclusion also found in the otherdocumentaries, and which require a more formal register than thatadopted in the role-play. In attempting to manage this blend of narra-tive and scientific report, the students create a script which shiftsbetween the different pragmatic demands of the language of conversa-tion and the language of science.

Out of the three cases, this is the boldest in terms of the way that itdeviates from the expectation to document a simple scientific study.In an interview, one of the students mentioned that their intentionwas to present themselves in an original way, different to a scientificstudy, or as he put it, “present it in a more attractive way than what

TESOL QUARTERLY676

the research do” (focus group, Lee). In this documentary, the narratorrole is prominent, in the style of a participatory documentary withdocumentary makers participating actively in the narrative. Viewers areinvited to identify with the protagonists and follow their story. In thisdocumentary the students adopt the role of “traveler” on, as they putit in their script, a “journey of experiment.” The students again drawon multiple modes, incorporating pragmatically appropriate uses ofthe L2, in order to present themselves in this way.

Figure 6 shows Shots 55–65 of the documentary, a montage thatfollows the two students as they move from the cafeteria through thecampus to the library, where they search for books. In Shot 65, we seethe two students studying the books they have found and finding theanswer to their question. Lee says, “Look, Richard, I found it! Yousee?” and the camera zooms in on the text, before the video cuts toan explanation of the theory behind the students’ experiment. Thesoundtrack is upbeat pop music, an extract from “That’s Not MyName” by the Ting Tings. Use of this music helps to frame the mes-sage as positive and youth oriented, making the documentary moreaccessible and less intimidating for its wide, nonspecialist audience. Atthis point in the video, the story is told entirely through these visualsand accompanying soundtrack. An impressive range of visuals is usedin a short space of time, with different kinds of shots, camera angles,and distances, as well as panning, zooming, and fast motion used in anumber of places. The visuals have a homemade quality to them, shotwith a handheld camera and a little bumpy at times, and this qualityfits well with the theme of two students on an adventure. The livelysoundtrack contrasts with the stereotypically dull activity of two stu-dents doing research in a library, with this divergence between imageand sound creating a potentially ironic sense of excitement (because itis counter to common expectations of library research).

Finally, it is interesting to note how the students make use of thedocumentary genre to convey impressions about the research process.Many of the activities visually represented, such as the process of doingresearch in the library, would not normally form part of a scientificreport and as such could not be effectively expressed in the writtenmode. As another example, the students showed how their attempts tofind volunteers for their experiment frequently resulted in rejectionsby other students on campus. Here the use of multiple modes in anonacademic genre allows the students to express meanings and adoptidentities that are not available in conventional academic writing. Therhetorical strategies used were well received by their classmates, whocommented, “They really got the sense of humour” (blog post, Gale)and “Video produced by Gao, Richard, Lee is very funny and creative”(blog post, Joyce).


DISCUSSION

The videos that have been selected for analysis here are drawn froma limited data set and must be interpreted with this limitation in mind.Although the videos under analysis are those that were evaluated byteachers as effective in the rhetorical strategies adopted, they neverthe-less illustrate more generally the rhetorical challenges that the studentsfaced in this digital video project and how such challenges can poten-tially be resolved. The students doing this task had to make choicesabout how to present their work multimodally, the kind of rhetorical

FIGURE 6. Opening (shots 55-65)

TESOL QUARTERLY678

hook to adopt, and the kind of discoursal identities to perform. Thedata provide insights into possible strategic choices—in terms of linguis-tic and multimodal semiotic resources—in order to illustrate optionsavailable to L2 students doing such multimodal compositions.

The findings demonstrate that, in attempting the digital video task,students perceived rhetorical challenges related to both the audienceand the multimodal orchestration of the documentary. In the threecases analyzed students carefully considered their audience and crafteda range of rhetorical hooks in order to attract attention. In this, thecases selected were typical of the efforts of other groups as well: All stu-dents were concerned about designing videos that would “stand outfrom the crowd” and attract the attention of the YouTube audience.This desire to be original and creative arose in part because the projectengaged students with an authentic, online audience on YouTube,thereby creating a powerful rhetorical exigence (Bitzer, 1968), or set ofcommunicative demands, which students generally responded positivelyto. In part, students’ positive response was also shaped through thequality of the input that they received on the course. As noted earlier,students were provided with sample scientific documentaries for analy-sis, but these were not presented as models to be slavishly followed.Rather, teachers presented the various semiotic choices made as oneparticular constellation selected from a range of options. In addition,students were encouraged to be creative in their own documentaries(they were told they would be given extra credit) and responded bothby drawing on the multimodal affordances of digital video and by craft-ing narratives which they hoped would compel their audience.

The case studies show how, in designing their videos, differentgroups made different assumptions about their relationship to theaudience. In particular, Case 1 shows less interest in entertaining,more interest in educating through presentation of scientific knowl-edge. These different assumptions led groups to draw on differentsemiotic resources, including various documentary subgenres, withassociated differences in visual representations, registers, and identitiesin the discourse. A key variable here is the role of the documentarymaker, an issue that was explored in class. Students were introducedto three kinds of documentaries: (1) observational documentary,which has no narrator, lets the action play out naturally in front of thecamera, and refrains from adopting an explicit point of view; (2)expository documentary, which includes a narrator and/or voiceover,tells a story, and has a clear point of view; and (3) participatory docu-mentary, in which the narrator becomes the focal point of the film,conveying an extremely one-sided point of view. Once again, teacherspresented these approaches as options which students could creativelycombine. Here, Cases 1 and 2 were more expository in nature,


whereas Case 3 was more participatory, following the experiences ofthe students very closely. This had implications for the semioticchoices made, with Case 3 adopting a more informal register at times.

Related to the students’ conception of their relationship to the audi-ence, the analysis shows how students adopted a range of discoursalidentities, presenting themselves as scientist, investigative journalist,and traveler on a “journey of experiment.” These discoursal identitiesimplicate different purposes—to educate, to investigate, to entertain—which must be achieved through appropriate use of the L2 and othersemiotic resources like image and sound. As a result students engagedwith a range of genres, interdiscursively appropriating resources frompopular science demonstration, television news report/film, and per-sonal narrative in their videos. The creative mixing of genres that isobserved leads to clear differences in register, most obvious if one com-pares the scientific register of the language use in Case 1 (see especiallyFigures 2 and 3), with the conversational register of Case 3 (especiallythe role-play). We see similar differences in choice of visuals and soundeffects, for example, the extent to which these evoke television genres(Case 2). In the course, the various discoursal identities that studentsimagined for themselves were supported by teachers, who providedinput on students’ scripts and storyboards, particularly issues of organi-zational structure, linguistic appropriacy, and oral presentation tech-nique. Because of this inclusive approach, it appears that the use of thepopular genre of scientific documentary, expressed through the med-ium of digital video and shared through YouTube, legitimated identitieswhich would not be available in more academic genres.

At the same time as the video project opened up many interesting,alternative modes of expression for students, it also provided studentswith many opportunities for spoken and written language practice.The findings show that language plays a very important role in stu-dents’ multimodal ensembles, with students dedicating much effort towriting their script and presenting/recording their narration. Thesetwo moments in particular (writing and presenting) are opportunitiesfor teachers to intervene and provide scaffolded language input(Donato, 1994) and feedback on students’ projects. In effect, theproject offers an authentic communicative context, namely, thecommunication of scientific knowledge to a (real) lay audience, withthe student positioned as expert storyteller and documentary maker.In constructing their documentaries, students inevitably notice short-comings in their communicative competence, which they must over-come in order to succeed in creating an effective video.

The findings also show that students engaged actively with issues ofmultimodal meaning making, reflecting on multimodality and creatingmultimodal ensembles to suit their rhetorical purpose. Although the

TESOL QUARTERLY680

videos described in this study were generally successful, they still leaveroom for improvement. For example, the presentation of results in Case1 (not shown here) relies too heavily on text on the screen where visu-als such as graphs and charts would be preferable, a point which sciencelecturers viewing this video have made. Similarly, students occasionallyhave difficulty when presenting their methods in an appropriatelydetailed fashion: they may leave out important details or repeat them-selves unnecessarily. At times, the choice of visuals may be questionable,as when students remix images from popular culture, like cartoons oradvertisements. These observations suggest a need to address with stu-dents how scientific material can be appropriately presented in a docu-mentary, in terms of both selection of content and visual presentation.As the course has matured, more material related to multimodal aspectsof documentary making have gradually been added. It is necessary todiscuss explicitly the meanings that are conveyed by students’ multi-modal choices, for example, choice of location, degree of intimacyimplied by distance of shot, personal appeals implied by camera angle,and the meanings of basic effects such as slow motion.

An effort is made in the course to provide students with support fortechnical issues related to recording and editing their videos, but it isthe students themselves who have been most innovative at identifyingand resolving such issues. For example, recording sound is often prob-lematic, and although microphones have been purchased for studentsto plug into video cameras, these may not always be reliable. A muchbetter solution that students have come up with is to use their smart-phone as a mobile sound recorder and edit sound and video togetherafter the fact. This technique also allows them to effectively capturesound while filming from some distance, even in noisy areas. The strat-egy can now be suggested to subsequent cohorts of students.

From the point of view of course design, one way to look at the digi-tal video project is as a bridge to the subsequent academic task of writ-ing a lab report. This point is made in the literature on digitalstorytelling (Kajder & Swenson, 2004) but has not been investigatedbefore now. Here, the digital video is the first step in the larger Eng-lish for science project, which begins with the documentary and leadsto the construction of a written lab report for a specialist audience.This means that students are first called upon to work with a mediumand genre—digital video popular science documentary—that theylikely have been frequently exposed to and understand intuitively.Furthermore, engaging with scientific materials and presenting themto a nonspecialist audience is not something that they are typicallyrequired to do for their major courses and, as a result, the video pro-ject helps them to see such materials in new ways, discovering a widerrange of scientific genres and how these genres interconnect. In addi-


tion, the range of identities that are permissible in such genres is argu-ably less restrictive than those in academic genres. Once the digitalvideo is complete, students are then tasked with re-presenting theirfindings for a specialist audience, and it is only at this later stage thatthey must grapple with issues of identity in academic writing. In thisway, the project stages identity development, language, and literacydevelopment in manageable chunks, moving from familiar to unfamil-iar and ultimately increasing access to the more constrained genres ofacademic writing.

CONCLUSION

This study outlines an approach to the use of digital media in lan-guage education, in which digital literacy practices are embeddedalongside more traditional literacy practices, as one element of thecourse design. In the course described, digital literacy practices areincorporated in the form of the digital video project, in a way that (1)fits well with students’ expectations, because creating and sharing adigital video is an authentic everyday practice that students can relateto, and (2) fits well with the aims of the course, because the projectinvolves students in a form of scientific communication between spe-cialists and nonspecialists. Although the approach has worked well inthe reported context, the study is limited to observations of one coursein one institution at one point in time and must be interpreted withthis limitation in mind. It is to be hoped that the three cases analyzedprovide insights into the potential of digital video to provide studentswith opportunities to engage in language learning as well as to prac-tice the important 21st century skill of orchestrating semiotic resourcesin various modes in order to make meaning through multimodalensembles. It is helpful to bear in mind the words of Schetzer andWarschauer (2000, p. 172), who point out that “literacy is a shiftingtarget, and we have to prepare our students for their future ratherthan our past.” In addition, as language teachers, we should take noteof the potential for project-based learning that incorporates digitalmedia to expand opportunities for L2 learners to acquire and to dem-onstrate communicative expertise.

ACKNOWLEDGMENTS

This research was supported by a Teaching Development Grant funded by theHong Kong University Grants Committee (TDG 6000302, City University of HongKong: “Oral Presentations of Academic Projects: Developing MultiliteraciesThrough English for Science”). I would like to thank the co-investigators, Lindsay

TESOL QUARTERLY682

Miller, Richard Kong, and Connie Ng, for their contributions to the project. Iwould also like to thank John Flowerdew, Rodney Jones, and Martha Penningtonfor their comments on drafts of this article.

THE AUTHOR

Christoph A. Hafner is an assistant professor in the Department of English at theCity University of Hong Kong. His research interests include specialized discourse,digital literacies, and educational technology. He is coauthor of Understanding Digi-tal Literacies: A Practical Introduction, published by Routledge in 2012.

REFERENCES

Baldry, A., & Thibault, P. J. (2006). Multimodal transcription and text analysis. Lon-don, England: Equinox.

Barton, D. (1994). Literacy: An introduction to the ecology of written language. Oxford,England: Blackwell.

Barton, D., & Hamilton, M. (1998). Local literacies: Reading and writing in one com-munity. London, England: Routledge.

Beckett, G. H. (2006). Project-based second and foreign language education. InG. H. Beckett & P. C. Miller (Eds.), Project-based second and foreign language educa-tion: Past, present, and future (pp. 1–15). Greenwich, CT: IAP

Bhatia, V. K. (2004). Worlds of written discourse: A genre-based view. London, England:Continuum.

Bitzer, L. F. (1968). The rhetorical situation. Philosophy and Rhetoric, 1(1), 1–14.Canagarajah, A. S. (2006). TESOL at forty: What are the issues? TESOL Quarterly,

40, 9–34. doi:10.2307/40264509Cope, B., & Kalantzis, M. (2000). Multiliteracies: Literacy learning and the design of

social futures. London, England: Routledge.Davis, K. A. (1995). Qualitative theory and methods in applied linguistics research.

TESOL Quarterly, 29, 427–453. doi:10.2307/3588070Donato, R. (1994). Collective scaffolding in second language learning. In J. P.

Lantolf & G. Appel (Eds.), Vygotskian approaches to second language research (pp.33–56). Norwood, NJ: Ablex.

Gee, J. P. (2004). Situated language and learning: A critique of traditional schooling.New York, NY: Routledge.

Gee, J. P. (2005). An introduction to discourse analysis: Theory and method (2nd ed.).London, England: Routledge.

Gee, J. P. (2008). Social linguistics and literacies: Ideology in discourses (3rd ed.).London, England: Routledge.

Gibson, J. J. (1979). The ecological approach to visual perception. Boston, MA: Hough-ton Mifflin.

Hafner, C. A., & Miller, L. (2011). Fostering learner autonomy in English for sci-ence: A collaborative digital video project in a technological learning environ-ment. Language Learning and Technology, 15(3), 68–86.

Hedge, T. (1993). Key concepts in ELT: Project work. ELT Journal, 47, 275–277.doi:10.1093/elt/47.3.275

Hull, G. A., & Katz, M. L. (2006). Crafting an agentive self: Case studies of digitalstorytelling. Research in the Teaching of English, 41(1), 43–81.


Hyland, K. (1998). Hedging in scientific research articles. Amsterdam, the Nether-lands: John Benjamins.

Hyland, K. (2002). Authority and invisibility: Authorial identity in academic writ-ing. Journal of Pragmatics, 34, 1091–1112. doi:10.1016/S0378-2166(02)00035-8

Ivani�c, R. (1998). Writing and identity: The discoursal construction of identity in aca-demic writing. Amsterdam, the Netherlands: John Benjamins.

Janesick, V. J. (2003). The choreography of qualitative research design: Minuets,improvisations, and crystallization. In N. Denzin & Y. S. Lincoln (Eds.), Strate-gies of qualitative inquiry (2nd ed., pp. 46–79). Thousand Oaks, CA: Sage.

Jewitt, C. (2006). Technology, literacy and learning: A multimodal approach. London,England: Routledge.

Jones, R. H., & Hafner, C. A. (2012). Understanding digital literacies: A practical intro-duction. London, England: Routledge.

Kajder, S., & Swenson, J. A. (2004). Digital images in the language arts classroom.Learning and Leading With Technology, 31(8), 18–46.

Kress, G. (2000). Multimodality: Challenges to thinking about language. TESOLQuarterly, 34, 337–340. doi:10.2307/3587959

Kress, G. (2003). Literacy in the new media age. London, England: Routledge.Kress, G. (2010). Multimodality: A social semiotic approach to contemporary communica-

tion. London, England: Routledge.Kress, G., & Jewitt, C. (2003). Introduction. In C. Jewitt & G. Kress (Eds.), Multi-

modal literacy (pp. 1–18). New York, NY: Peter Lang.Kress, G., & van Leeuwen, T. (2006). Reading images: The grammar of visual design

(2nd ed.). London, England: Routledge.Lankshear, C., & Knobel, M. (2003). New literacies: Changing knowledge and classroom

learning. Buckingham, England: Open University Press.Lankshear, C., & Knobel, M. (Eds.). (2008). Digital literacies: Concepts, policies and

practices. New York, NY: Peter Lang.Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation.

Cambridge, England: Cambridge University Press.Lemke, J. L. (1998). Multiplying meaning: Visual and verbal semiotics in scien-

tific text. In J. R. Martin & R. Veel (Eds.), Reading science: Critical and func-tional perspectives on discourses of science (pp. 87–113). London, England:Routledge.

Max Planck Institute for Psycholinguistics. (2002). ELAN [Computer software]. Nijme-gen, the Netherlands: Author. Retrieved from http://www.lat-mpi.eu/tools/elan/

Molle, D., & Prior, P. (2008). Multimodal genre systems in EAP writing pedagogy:Reflecting on a needs analysis. TESOL Quarterly, 42, 541–566. doi:10.1002/j.1545-7249.2008.tb00148.x

Nelson, M. (2006). Mode, meaning and synaesthesia in multimedia L2 writing.Language Learning & Technology, 10(2), 56–76.

New London Group. (1996). A pedagogy of multiliteracies: Designing socialfutures. Harvard Educational Review, 66(1), 60–92.

Prior, P. (2005). Moving multimodality beyond the binaries: A response to Gun-ther Kress’ “Gains and Losses.” Computers and Composition, 22(1), 23–30. doi:10.1016/j.compcom.2004.12.007

Richards, K. (2003). Qualitative inquiry in TESOL. Basingstoke, England: PalgraveMacmillan.

Schetzer, H., & Warschauer, M. (2000). An electronic literacy approach to net-work-based language teaching. In M. Warschauer & R. Kern (Eds.), Network-based language teaching: Concepts and practice (pp. 171–185). Cambridge, England:Cambridge University Press.

TESOL QUARTERLY684

Selfe, C. (2007). Multimodal composition. Cresskill, NJ: Hampton Press.Street, B. V. (1984). Literacy in theory and practice. Cambridge, England: Cambridge

University Press.Street, B. V. (2004). Academic literacies and the “new orders”: Implications for

research and practice in student writing in higher education. LATISS: Learningand Teaching in the Social Sciences, 1(1), 9–20. doi:10.1386/ltss.1.1.9/0

Tardy, C. M. (2005). Expressions of disciplinarity and individuality in a multi-modal genre. Computers and Composition, 22, 319–336. doi:10.1016/j.compcom.2005.05.004

Thorne, S. L., & Black, R. W. (2007). Language and literacy development in com-puter-mediated contexts and communities. Annual Review of Applied Linguistics,27, 133–160. doi:10.1017/S0267190508070074

Unsworth, L. (2008). Multiliteracies and metalanguage: Describing image/textrelations as a resource for negotiating multimodal texts. In J. Coiro, M. Knobel,C. Lankshear, & D. J. Leu (Eds.), Handbook of research on new literacies (pp. 377–405). New York, NY: Lawrence Erlbaum.

Vasudevan, L., Schultz, K., & Bateman, J. (2010). Rethinking composing in a digi-tal age: Authoring literate identities through multimodal storytelling. WrittenCommunication, 27, 442–468. doi:10.1177/0741088310378217

Warschauer, M. (1998). Researching technology in TESOL: Determinist, instru-mental, and critical approaches. TESOL Quarterly, 32, 757–761. doi:10.2307/3588010

Warschauer, M. (1999). Electronic literacies: Language, culture, and power in online edu-cation. Mahwah, NJ: Lawrence Erlbaum.

Warschauer, M., Schetzer, H., & Meloni, C. (2000). Internet for English teaching.Alexandria, VA: TESOL.

Yang, Y.-F. (2012). Multimodal composing in digital storytelling. Computers andComposition, 29, 221–238. doi:10.1016/j.compcom.2012.07.001


Documents

Embedding Digital Literacies in English Language Teaching: Students' Digital Video Projects as Multimodal Ensembles