In defence of writing: a social semioticperspective on digital media, literacy andlearningHavard Skaar
From a learning perspective, social semiotics research-ers tend to focus on the liberation latent in themultimedia options available through the new media.It is true that digitalmedia democratise the possibilitiesopen to the general public of a more varied andcomprehensive text production than ever before, bothin and outside school. Participating in this textproduction naturally implies a richer potential forlearning. But digital technology also allows us to optout of, and thus avoid, semiotic work. With this as thestarting point, the present article sets out to highlightthe pedagogical benefits associated with the writtenmode, precisely in an age when the digital media aremaking multimodal forms of expression increasinglyavailable to us all.
Key words: social semiotics, digital media, writing,literacy, learning
Social semiotics provides a theoretical basis for relatingthe use of signs both to learning and to the use of digitaltechnology. In this article, these two aspects are seen asinterconnected: firstly, that learning takes placethrough the semiotic work we performwhenwe createsigns and texts; and secondly, that digital technologyhas established new premises for what we can learnthrough this meaning-creating process. The firstpremise is that text and sign production implieslearning. The second is that digital technology changesthe basic conditions for text production and thus alsofor what we learn from it. How?
After discussing the first two premises in greater detail,I shall return to this question, which can be put moreexplicitly as follows: if computers and digital mediahelp us to produce text, what exactly is it they help uswith? What does this help mean for what we learn?
First premise: we learn from semiotic work
The first premise is to define learning as semiotic work.In this work, which consists in giving what we wish to
express a form that allows it to be communicated toothers, we learn through the resistance we experiencein our encounter with the semiotic resources we utilise.Defined in this way, however, learning cannot bedistinguished from communicating or expressingsomething in general. Social semiotics accordinglygives us a theoretical basis for conceptualising what itmeans to learn something, but at the same timelearning is usually understood in the context of aninstitutional framework. The schools teaching curri-culum, for example, sets out guidelines for what pupilsare expected to learn through their work with differentsubjects. This means that what is to be representedthrough the pupils semiotic work, and thus learned, isa question to be decided through social and politicalprocesses, which a (social) semiotic learning theorycannot explain. But in the lower grades of theNorwegian school system, from where the empiricalexamples in this article are taken, learning to producetext, for example a story, is a goal in its own right. It isalso a goal that the pupils should produce stories withthe help of digital technology. Text production, withand without the use of digital technology, is in otherwords a learning objective in itself. That makes theassociation between semiotic work and learningclearer in my empirical examples than in cases wherethe knowledge requirements in the school curricula aredefined as something other than, and in many caseswholly independent of, skills linked to text production.
Second premise: digital media change oursemiotic work
The second premise, that digital technology and newmedia change the premises for what we learn throughtext production, is often associatedwith the concepts ofmultimodality and literacy.
The multimodal perspective arises from the focus onthe social andmaterial basis for the production of signsthat Michael Halliday presents in Language as a socialSemiotic (1978). Halliday focuses on the material andsocial premises for the use of spoken language.Multimodal social semiotics is a perspective that alsoseeks to incorporate the underlying principles for theuse of other signs than speech, for example dance,
36 In defence of writing
r UKLA 2009. Published by Blackwell Publishing, 9600 Garsington Road, Oxford OX4 2DQ, UK and 350 Main Street, Malden, MA 02148, USA.
facial expression and gestures, or drawing, paintingand sculpture. While OToole (1994) retains a linguisticmodel to explain the use of semiotic resources otherthan the spoken word, Kress and van Leeuwen (1996,2001) are interested in arriving at common semioticprinciples for the creation of meaning in differentexpressive forms or modes. They aim in this way tocreate a multimodal framework for semiotic analysis.Social semiotics becomes multimodal social semiotics.In this context literacy, defined as our ability to read andwrite, is analysed as part of a multimodal designwheremeaning is created through the interplay betweendifferent modes. Digital media can, for example, makeit easier to combine writing and pictures to createmultimodal texts. In the Saussurean tradition, ClaudeLevi-Strauss (1962/1974) called this kind of text-creating work bricolage(see also Chandler, 2005).Social semiotic theorists like Kress, and researchersfrom other disciplines in new literacy studies(Pahland Rowsell, 2006; Street, 1998), have chosen to use theterm design (Cope and Kalantzis, 2000).
Digital technology is naturally no prerequisite formultimodal text production. In a certain sense it maybe claimed that all text is multimodal. The design of thewriting and its positioning on the page of a book, forexample, does not get its meaning through the writtenmode alone: the quality of the paper, its colour andtexture also contribute to the creation of meaning forthe reader. Digital technology has however radicallyaltered our opportunities to create multimodal textsourselves.
This may explain why digital technology and newmedia are key elements in the groundwork done in thelast decade to relate social semiotic theory to multi-modality and literacy. In the introduction to one of hisbooks on literacy, Kress typically points to a revolu-tion in the landscape of communication in an agecharacterised by social, economic, communicationaland technological change (Kress, 2003, p. 9). Incommon with Snyder (1997) he points to a movementfrom page to screen, and from writing to imagewhen discussing literacy in the new media age. Thebasis for his argument is that digital technology andnewmedia have given bothmultimodality andwritinga new role to play and that this has consequences forour text production and what we can learn from it.
What the sign can tell
Social semiotics stresses the social, not the systematic,premises for the production of signs. According toKress the sign is motivated (1993). With this statementhe places himself in opposition to Ferdinand deSaussure (Saussure et al., 1916/1983) who claimedthat the sign is arbitrary. Saussure believed that thesign is arbitrary because there is a random connectionboth between the content and expression of the signand between the sign and its referent outside the sign
system. According to Saussure the sign gets its mean-ing fromwhat the other signs in the sign system are not.Kress, on the other hand, claims that we create the signanew each time we use it. He believes that each use ofthe sign is therefore based on a double metaphor. Firstwe select some aspect of an object we want to saysomething about (it is impossible to say everything andour interests determine what we want to say about theobject), and then we select a sign that represents whatwe are interested in saying about the object (oursemiotic resources determine our choice). Kress uses a3-year-old boys drawing of a car in the form of circleson a sheet of paper as an example of such a doublemetaphor: the boy (on the basis of his interests) allowsthe wheels to represent the car (first metaphor) beforehe (on the basis of his semiotic resources) allows thecircles to represent the wheels (second metaphor)(Kress, 1993, p. 174). Kress presents the relationshipbetween these two metaphors as follows:
the child decides that wheels are criterial in representingthe signified car. That decision then determines whatmay be an apt signifier, in this case circles. Wheels areselected as the criterial aspect of the object to be repre-sented (the signified), and circles are apt signifiers in thesense that they adequately signify, represent or expressthe characteristics of the signified. All signs are formed inthis metaphoric process. All signs are metaphors(Kress,1993, p. 174).
As we see, Kress pursues Saussures bipartition of thesign in his interpretation of the sign users choices. Thefirst metaphor arises because the referent is repre-sented through the choice of a signified which not onlysays something about the car but also what it is aboutthe car that interests the boy. The second metaphorarises because the signified becomes a sign through thechoice of a signifier which not only tells us somethingabout the wheels but also about the boys choice fromamong the semiotic resources at his disposal. Kressasserts that the signified (the boys interest in the carswheels) determines what may be an apt signifier(inother words, the circles he draws on the paper).
What determines our choice of sign?
However, I do not believe that the choice of sign takesplace in a causal chain in which the choice of thesignified determines or leads to the choice of signifier.Saussure asserted that in the sign the signified andsignifier were fused together like two sides of a sheetof paper: it is impossible to make a cut on one sidewithout simultaneously cutting the other. This mu-tuality between signified and signifier holds true evenif we shift focus from langue to parole, which meansthat we relate to the sign in use instead of (as Saussuredid) to the sign in system. The three-year-old boy hadpaper and pencil to hand and chose to draw somecircles, which he then explained by saying: this is a
Literacy Volume 43 Number 1 April 2009 37
r UKLA 2009
car. He has just about succeeded in making some-thing that looks like circles on the sheet of paper. Canthis have been the simplest solution for him? Or theonly choice he had? We do not know. The actualconnection between the boys interests and thepossibilities he had to express them in this concreteexample is undetermined and not really the point hereeither. The point is that the example illustrates that theproduction of a sign is always a result of an interplaybetween what we intend to express and what wesucceed in expressing with the resources at ourdisposal. It is thus not only the signified that affectsour choice of signifier but also the converse. Computertechnology and new media change the possibilitiesopen to the sign-maker to choose a signifier whichexpresses the interest underlying the choice of sig-nified. It is the dynamic between those choices thatresults in the merging of signified and signifier into asign, in other words the interplay between the sign-makers interests and the available semiotic resources,which are alteredwhen the sign-maker sits down at thecomputer. In this way, digital technology influences thesemiotic work invested in sign production and hencealso the learning arising from this work.
Coding and choices as semiotic work
When we choose to express something through theproduction of signs, we encounter resistance. Thisresistance can be related both to what we can expressand how we can express it, i.e. our choice of both signi-fied and signifier. Both aspects involve semiotic work.Digital mediamake this work easier for us by giving usthe possibility to choose text instead of to code it.
The term code has been given various definitions insemiotics, some of them widely comprehensive (seee.g. Barthes, 1957/1997;1970). In this article, code isgiven a narrower definition, as conventional rules forcombinations of written characters. At the sametime, all digital text (pictures, animations, video, etc.)is also based on coding. When texts are digitalised,they are coded in basically the same way, as thenumbers zero and one. Consequently, written text,pictures, animations and video-clips, regardless oftheir scope andmultimodal complexity, are technicallyavailable to us with the same investment of semioticwork. In word-processing software, for example, thecommands copy and paste are used to incorporate athree-minute video-clip or a single word. By giving usthe possibility to choose a number of signs as fullycoded texts, digital technology thereby intervenes inour semiotic work. In turn, therefore, digital technol-ogy also intervenes in the learning that takes placewhen we create text.
Astudy of digitalmedia, literacy and learning
Kress relates different modes to different principlesfor learning and asserts that it is possible to see
learning as the individuals agentive selection from,engagement with and transformation of the worldaccording to their principles(Kress, 2007a,p. 37). Butthe principles we use to transform our experienceinto concepts or other forms of representation are onlyone aspect of the learning process. When digital mediaare associated with semiotic work (understood aslearning) we must in my view, as mentioned in theintroduction, also ask: howmuch semiotic workwill beinvested in this case in recreating the sign? Whataspect of this semiotic work can digital media help uswith? What does this help mean in terms of what welearn?
To answer these questions, I shall take as my point ofdeparture a study I have made of childrens multi-modal text production in a classroom. In the articleDigitalized story-making in the classroom: asocial semiotic perspective on gender, multimodalityand learning (Skaar, 2007) I examine how girls andboys in a fifth-year primary school class use text andimages to create their own stories at the computer. Incombining written text and images the pupils used theclassroom computers to create their stories as multi-modal discourses (Kress and van Leeuwen, 2001).The backdrop of the study is that research points tomarked differences in girls and boys interest forwriting, for telling stories and for using computertechnology. This study has bearing on the questionsabove because it also shows a connection between theuse of signs, learning and digital technology that goesbeyond an indication of gender differences in them-selves.
With regard to self-discourse I found that the girlsgenerally base their story on their own person, theirown body and their own personal experience. Incontrast, the boys tell stories about a superherofulfilling a mission.
The girls and boys multimodal textual choices showthe same pattern. Within the constraints of their digitalskills and capabilities, the girls prefer semioticresources that afford them the best opportunity todescribe feelings and intimate interpersonal relations.In this case this means that they prefer to use writtentext. They do...