Cognitive Effects of Multimedia Learning - San Diego State University

Cognitive Effects of Multimedia Learning

Robert Z. ZhengUniversity of Utah, USA

Hershey • New YorkINFORMATION SCIENCE REFERENCE

Director of Editorial Content: Kristin KlingerDirector of Production: Jennifer NeidigManaging Editor: Jamie SnavelyAssistant Managing Editor: Carole CoulsonTypesetter: Jeff Ash Cover Design: Lisa TosheffPrinted at: Yurchak Printing Inc.

Published in the United States of America by Information Science Reference (an imprint of IGI Global)701 E. Chocolate Avenue, Suite 200Hershey PA 17033Tel: 717-533-8845Fax: 717-533-8661E-mail: [email protected] site: http://www.igi-global.com

and in the United Kingdom byInformation Science Reference (an imprint of IGI Global)3 Henrietta StreetCovent GardenLondon WC2E 8LUTel: 44 20 7240 0856Fax: 44 20 7379 0609Web site: http://www.eurospanbookstore.com

Copyright © 2009 by IGI Global. All rights reserved. No part of this publication may be reproduced, stored or distributed in any form or by any means, electronic or mechanical, including photocopying, without written permission from the publisher.

Product or company names used in this set are for identification purposes only. Inclusion of the names of the products or companies does not indicate a claim of ownership by IGI Global of the trademark or registered trademark.

Library of Congress Cataloging-in-Publication Data

Cognitive effects of multimedia learning / Robert Zheng, editor.

p. cm.

Includes bibliographical references and index.

Summary: "This book identifies the role and function of multimedia in learning through a collection of research studies focusing on cognitive functionality"--Provided by publisher.

ISBN 978-1-60566-158-2 (hardcover) -- ISBN 978-1-60566-159-9 (ebook)

1. Media programs (Education) 2. Learning, Psychology of. I. Zheng, Robert.

LB1028.4.C64 2009

371.33--dc22

2008024195

British Cataloguing in Publication DataA Cataloguing in Publication record for this book is available from the British Library.

All work contributed to this book set is original material. The views expressed in this book are those of the authors, but not necessarily of the publisher.

If a library purchased a print copy of this publication, please go to http://www.igi-global.com/agreement for information on activating the library's complimentary electronic access to this publication.

51

Chapter IVManipulating Multimedia

MaterialsStephen K. Reed

San Diego State University, USA

Copyright © 2009, IGI Global, distributing in print or electronic forms without written permission of IGI Global is prohibited.

ABSTRACT

This chapter discusses a theoretical framework for designing multimedia in which manipulation, rather than perception, of objects plays the predominant role. The framework is based on research by cogni-tive psychologists and on Engelkamp’s (1998) multimodal model of action-based learning. Although the assumptions of Engelkamp’s model should be helpful for instructional design, they are not complete enough to include the additional demands of multimedia learning. These additional demands can result in unintended actions, involve sequences of related actions, and require reflection about domain-specific knowledge. Actions can be performed on either physical or virtual manipulatives, but virtual manipu-latives exist in idealized environments, support continuous transformations of objects, and allow for dynamic linking to other objects, symbols, and data displays. The use of manipulatives in the Building Blocks and Animation Tutor projects provide illustrations.

INTRODUCTION

In his preface to The Cambridge Handbook of Multimedia Learning Mayer (2005) defines multimedia learning as learning from words (spoken or printed text) and pictures (illustrations, photos, maps, graphs, animation, or video). The Cambridge Handbook consists of 35 excellent chapters on many aspects of multimedia learning that emphasize the viewing of pictures. However,

the word “manipulation” does not appear in the index. This does not imply that the manipulation of objects is ignored in the chapters but action receives comparatively little discussion compared to perception.

The purpose of this chapter is to provide a theoretical framework for designing multimedia in which manipulation, rather than perception, of objects plays the predominant role. The term “manipulation” in this chapter refers to the

52

Manipulating Multimedia Materials

movement of an object by a person. The object is typically referred to as a “manipulative” in instruc-tion and although the chapter focuses on virtual manipulatives that exist on a computer screen, it also includes research on physical manipulatives that exist in the environment. Examples include superimposing shapes to estimate relative areas and selecting and combining parts to build an object. Clicking on navigation buttons and chang-ing parameters in simulations are not included as examples of manipulation.

The discussed theoretical framework for using manipulatives is based on research by cognitive psychologists that should be relevant to the design of multimedia instruction. It must be emphasized that the objectives of the laboratory tasks created by cognitive psychologists often differ from the objectives of the instructional software created by instructional designers. However, at this early stage in applying cognitive psychology to instruc-tional design, I decided not to prejudge which findings will be most helpful and so include a variety of results that potentially could influence the effectiveness of manipulatives.

I use Engelkamp’s multimodal model of learning to organize these findings and refer to recent research to illustrate assumptions of his model. I next discuss applications of the model to instruction by considering some differences between the free recall of action phrases that forms the empirical basis of his model and the instructional learning of schematic knowledge. Although instruction may use physical manipula-tives, there are some advantages to using virtual manipulatives that I discuss in the next section. I conclude by summarizing two multimedia projects before proposing future directions.

BACKGROUND

There are few theoretical frameworks for un-derstanding the role that object manipulation plays in instruction. In my article on cognitive

architectures for multimedia learning (Reed, 2006) only one of the six theories incorporated action. Engelkamp’s (1998) multimodal theory was designed to account for the recall of long lists of action phrases such as “saw wood”, “play a flute”, “blow out a candle”, and “water a plant”. The recall of action phrases is a very different task than the ones designed for multimedia learning but the central finding of this research is relevant. That finding – labeled the enactment effect – is that acting out phrases results in better recall than simply reading phrases (Engelkamp, 1998).

The multimodal components of Engelkamp’s theory are illustrated in Figure 1. They consist of a nonverbal input (visual) and output (enactment) system and a verbal input (hearing, reading) and output (speaking, writing) system. All four of these modality-specific components are connected to a conceptual system. Engelkamp (1998) describes the many assumptions of his multimodal theory in his book Memory for Actions. I have listed the major assumptions (and page numbers) in Table 1 (See Appendix) and evaluate them below within the context of recent research on memory and reasoning.

1. Recall of observed actions should differ from that of performed actions because different systems are involved in encoding.

Engelkamp proposes that observations en-code visual information about movement but performance encodes motor information, as is illustrated in Figure 1. One application of this idea to instruction is that observed actions can lead to performed actions such as initially observing an instructor’s dance steps or tennis serve. Subse-quent recall can then be influenced by both visual memories of observing the instructor and motor memories of performing the action.

One implication of this assumption is that a person should be better at recalling verb-ac-tion phrases by enacting them than by verbally encoding them or by observing another person

53


enact them. Steffens (2007) recently reported that although such results are typically found for action phrases, she did not find an advantage of enactment over observation for goal-directed actions. Students were instructed to either “Pack the backpack exactly as I instruct you” (enact-ment), “Watch closely how I pack the backpack” (observation), or “Listen well while I tell you what you need to pack” (verbal learning). The results of two experiments did not reveal an advantage of enactment over observation although both encoding tasks were superior to verbal learning (in which the objects were also visible). Steffins proposed that enactment creates excellent item-specific encoding that is helpful for recalling unrelated phrases but does not create relational encoding that is helpful for recalling goal-directed actions.

2. Although both sensory and motor processes exert a positive influence on retention, each of these influences should be independent of each other.

This assumption mirrors the assumptions of Paivio’s (1986) dual coding theory in which two memory codes (visual and verbal for Paivio) pro-

vide two opportunities for recall if the memory codes are at least partially independent. However, this assumption raises questions about when visual and motor codes are independent because the actor can usually observe her actions or because sensory experiences are often the precursors of action. For example, the coordination of perception and action is the key assumption of the theory of event encod-ing (Hommel, Musseler, Aschersleben, & Prinz, 2001) that integrates perception and action into a common representational framework. Sensory and motor memory codes could therefore be more coordinated than independent and discovering when each occurs is an important theoretical and applied problem.

Research by Schwartz and Black (1999) dem-onstrates that the extent to which visual and motor codes are coordinated depends on the task. They instructed students to tip a glass of “water” (its level indicated by a mark on the glass) until it would pour from the glass. Students did the task with a blindfold but were allowed to readjust the angle after removing the blindfold. Fifteen of the 16 participants correctly increased the angle after viewing it, indicating that their perceptual and motor codes differed. However, when asked to tilt the glass to a specified angle (such as 2

Figure 1. A flow chart of Englekamp’s (1998) multimodal memory theory

54


o’clock), participants did not readjust the angle after viewing it. In this task the motor and visual codes appeared to be coordinated.

3. Planning an action should lead to poorer recall than performing an action because the perfor-mance includes planning.

Engelkamp proposes that only part of the motor information should be available when the action has been planned but not yet performed. Evidence from brain-imaging studies now indi-cates the involvement of motor areas in the brain during the recall of enacted action phrases. In a study by Nilsson activity in the right motor cortex was strongest following encoding by enactment, intermediate following imaginary enactment, and lowest following verbal encoding (Nilsson, Nyberg, Aberg, Persson, & Roland, 2000). The stronger activation of the motor cortex follow-ing enactment should help people remember the action.

However, recent research showed that both actual manipulation and imagined manipulation of toy objects greatly increased memory and comprehension of text when compared to a control group that read the text twice without manipula-tion (Glenberg, Gutierrez, Levin, Japuntich, & Kaschak, 2004). Children in the second grade were initially shown commercially available toys that consisted of either a farm scene (animals, tractor, barn, hay), a house with several rooms and people (mother, father, baby), or a garage scene (gas pumps, tow truck, car wash). The children in the manipulation condition either physically manipulated the toys or imagined manipulat-ing the toys after reading each sentence. The researchers suggested two reasons why manipu-lation is helpful. The first is that manipulation helps young readers map the words to the objects they represent. The second is that manipulation helps children derive inferences by constructing a mental model of the situation described in the story. Children who manipulated the objects did

better on questions about spatial relations that were not explicitly stated but could be inferred from the story.

4. Responses can be controlled either from the conceptual system or directly from the particular input system without going through the conceptual system.

This assumption reminds us that manipula-

tion is not necessarily meaningful for students. In commenting on the mixed results of research that has used manipulatives, Thompson (1994) proposed that instructors must continually ask what they want their students to understand rather than what they want their students to do. Although the material is concrete, the concepts behind the manipulations may not be obvious because of students’ ability to create multiple interpretations of actions.

Manipulation with limited understanding was observed by Moyer (2002) when she recorded how 10 teachers used manipulatives. The teachers had attended a 2-week summer institute on the use of a Middle Grades Mathematics Kit that included base-10 blocks, color tiles, snap cubes, pattern blocks, fractions bars and tangrams. They made subtle distinctions between real math that used rules, procedures, and paper-and-pencil tasks and fun math that used the manipulatives. Unfortu-nately, the fun math typically was done at the end of the period or the end of the week and was not well integrated with the “real” math.

5. Encoding of relational information occurs only in the conceptual system through a process of spreading activation.

The basis for this assumption is the excellent item-specific, rather than relational, encoding produced by enactment (Engelkamp & Seiler, 2003). However, research by Koriat and Pearl-man-Avnion’s (2003) challenges this assumption by showing that the free recall of action phrases

55


can be clustered by either the similarity of move-ments or the similarity of meaning. A person who recalled the phrase “wax the car” might next recall “spread ointment on a wound” based on similar motor movements. Evidence for the creation of conceptual relations would include recalling together phrases that had similar mean-ings. A person who recalled the phrase “wax the car” might next recall “pour oil into the engine” because both phrases refer to cars.

Undergraduates at the University of Haifa were required to either enact each phrase (enactment instruction) or simply say the phrase aloud (verbal instruction). The enactment instructions required students to imagine the object and pantomime the described action as if the object were present. Students in the enactment condition primarily recalled together phrases based on similar move-ments, whereas students in the verbal condition primarily recalled together phrases based on similar meanings. The authors concluded that the different conditions influence the relative salience of different types of memory organization and their relative contributions to recall. However, as found in many other experiments, the enactment of the phrases resulted in better recall than simply reading aloud the phrases.

6. Performing an action makes it difficult to form new associations in the conceptual system because performing an action forces concentration on information relevant to the action.

This assumption provides a word of caution

for instructional designers. Although there is extensive evidence for the enactment effect, too much focus on actions could distract from learn-ing new information if attention is shifted away from concepts. Research by Shockley and Turvey (2006) demonstrated that performing an action can also reduce retrieval of old associations. Their participants were given 30 seconds to retrieve instances from a semantic category such as four-legged animals. Swinging hand-held pendulums

reduced the number of successful retrievals.However, action can also facilitate reasoning

as shown by the finding that gesturing reduced the cognitive demands on working memory when students explained mathematical solutions (Wag-ner, Nusbaum, & Goldin-Meadow, 2004). This is more likely to occur when the gestures and verbal explanation are compatible. But the mismatch between information conveyed by gesture and by speech provided useful diagnostic information, such as indicating when students were consider-ing different solution options as they reasoned about problems.

APPLICATION TO INSTRUCTION

The assumptions of Engelkamp’s multimodal theory form a theoretical foundation for think-ing about the design of multimedia instruction. The enactment effect – the robust finding that acting out action phrases results in better recall than reading phrases – forms the basis for the theory and for the inclusion of manipulatives in instruction.

However, it is important to keep in mind that Engelkamp’s theoretical assumptions were formulated from research on the free recall of action phrases. Instructional use of multimedia, in contrast, typically requires the learning of in-tegrated schematic knowledge to produce a deep understanding of both procedures and concepts (Baroody, Feil, & Johnson, 2007). One conse-quence of using manipulatives to teach schematic knowledge is that students can perform actions that differ from the ones intended by the instruc-tional designer. Another difference between the free-recall and instructional paradigms is that schematic knowledge usually requires the inte-grated learning of action sequences rather than the recall of independent actions. A third difference is that acquisition of schematic structures requires reflecting on actions rather than simply recalling them. A fourth-difference is that instruction in-

56


volves learning domain-specific knowledge rather than forming associations among words.

Unintended actions. One challenge for using manipulatives in instruction is that students’ ac-tions can differ from normative ones. A formative evaluation of one of the modules in the Animation Tutor software (Reed, 2005) illustrates this chal-lenge. The design of the Dimensional Thinking module attempts to correct students’ tendency to inappropriately apply proportional reasoning to area and volume For instance, many students believe that doubling the diameter of a circle will double its area and doubling the diameter of a sphere will double its volume. Attempts to correct such misconceptions with static diagrams have been largely unsuccessful (De Bock, Verschaffel, & Janssens, 2002). Brian Greer, Bob Hoffman, and I therefore designed the Dimensional Think-ing module so students can virtually manipulate diagrams of circles, squares, cubes, and irregular figures to learn when proportional reasoning does and does not apply.

The module begins with a sign in the window of a pizza parlor that shows the prices of pizzas with different diameters. A 12-inch pizza sells for $6.99 and a 20-inch pizza sells for $12.99. It then raises the following question of which is the better value. A proportional reasoning solution of dividing price by diameter would reveal that the smaller pizza sells for $0.58 per inch and the larger pizza sells for $0.65 per inch. Students who use this approach should falsely conclude that the smaller pizza is a better value.

Kien Lim did a formative evaluation of the Dimensional Thinking module by assigning it to 19 freshmen in his science laboratory class at the University of Texas, El Paso. Eleven of the students initially decided that the 12-inch pizza was the better value. Later in the module students were asked to compare the relative sizes of the two pizzas by determining how many smaller pizzas would cover the area of the larger pizza (see Figure 2). They were again asked which was

the better buy. Only three of the eleven students switched from the 12- to the 20-inch pizza and one student switched from the 20- to the 12-inch pizza. The interesting aspect of the results is that those students who still claimed the 12-inch pizza was the better buy had a mean estimate of 2.26 small pizzas to cover the large one. Those students who claimed the 20-inch pizza was the better buy estimated that it would take 3.35 small pizzas to cover the large one. These two means differed significantly and indicate that although students’ answers were influenced by their manipulations, underestimation of relative area was correlated with incorrect decisions.

The formative evaluation revealed that this aspect of the instruction needs to include more guidance about estimating relative area and using relative area to make best-buy decisions. The larger pizza would still be the better buy even if it were only 2.26 times as large because it costs only 1.86 times as much. Allowing students to calculate exact proportions rather than make estimates may help them make better decisions.

Action sequences. Making comparisons by dragging smaller circles over a larger circle is more typical of the use of object manipulation in instruction than is recalling a list of action phrases. This raises a different set of challenges for instructional researchers such as determining whether students can integrate action sequences and avoid interference from performing similar actions. This challenge is particularly timely because of Steffins’ (2007) recent finding that enactment is not superior to observation for recalling goal-directed actions such as packing a backpack.

Edwards’ (1991) research on middle-school children’s learning transformation geometry is a good example of learning sequences of ac-tions. The children used a set of simple Logo commands to slide, rotate, pivot, reflect, flip, and scale geometric forms. After gaining experience with each transformation, they played a match

57


game the encouraged them to supimpose two congruent shapes by using as few transforma-tions as possible.

A study that Jeffrey Johnsen and I performed some years ago illustrates the challenge that stu-dents face in learning sequences of actions (Reed & Johnsen, 1977). The task required solving the missionaries and cannibals problem by moving tokens across a “river”. The intentional group was told to try to remember their moves because they would have to solve the same problem a second time. The incidental learning group did not know they would have to solve the problem twice. Students in the intentional group improved more on their second attempt than students in the incidental group and two subsequent experiments evaluated what these students had learned.

There was no significant difference between the two groups in their ability to recall what move they made at the different problem states or select the best move at a problem state. Problem solv-ers were not very accurate in remembering the details, perhaps because of the similarity of the problem states that differed only in the number of missionaries and cannibals on each side of the river. Instead, students in the intentional group

were better at learning more generic strategies such as moving cannibals across the river during the first third of the sequence and missionaries across the river during the second third of the sequence. Learning action sequences needs to be part of a theoretical framework for the use of manipulatives.

Reflection. The intentional learners may have learned more about effective strategies because they reflected on their actions rather than simply solved the problem. Both action and reflection are important components of Piaget’s theory. Action is important because knowledge for Piaget is fun-damentally operative; it is knowledge of what to do with something under certain possible condi-tions. Piaget (1977) subsequently emphasized the importance of reflection in his book Recherches sur l’abstraction réfléchissante. According to Robert Campbell (2001), who translated the book into English as Studies in Reflecting Abstraction, reflection became an important part of Piaget’s theory rather late in his prolific writing career.

Campbell illustrates Piaget’s use of this con-cept through an example in which Piaget uses poker chips to teach multiplication as repeated addition. For instance, children are asked to place

Figure 2. A screen design that allows for dragging small circles over a large circle

58


three chips in a row, followed by placing another three chips in the same row. According to Piaget, children have to perform two types of abstraction to think about multiplication as repeated addition (2 x 3 = 3 + 3). The first is to recognize how many chips they are adding each time. The second is to keep track of the number of times that they add the same amount. This requires the use of reflecting abstraction to abstract a property of their actions. Reflecting abstraction is required to create new knowledge such as recognizing that adding 2 chips three times produces the same number of chips as adding 3 chips two times (3 x 2 = 2 x 3). Learning from manipulatives requires students to not only remember their actions, but to reflect on the consequences of their actions.

Domain-specific knowledge. Reflecting on actions, however, will not be sufficient if stu-dents lack domain-specific knowledge to guide their reflections. A study that compared learning domain-specific schemas with learning general strategies found that the more specific (schema-based) instruction was superior to general strategy instruction (Jitendra et al., 2007). The schema-based instruction taught third-grade children to solve addition and subtraction word problems by learning problem types such as change, group, and compare (Marshall, 1995). The general-strat-egy instruction taught a four-step procedure to read and understand the problem, plan to solve the problem, solve the problem, and check the solution. The plan step included more specific advice such as using manipulatives (counters) to act out the information in the problem. The schema-based instruction was more effective in improving students’ ability to solve the word problems. However, both strategies were equally effective in improving computational skills, which the investigators attributed to the use of diagrams in the schema-based instruction and the use of manipulatives in the general-strategy instruction.

ADVANTAGES OF MANIPULATIVE SOFTWARE

Piaget and many others have studied the instruc-tional consequences of manipulatives by using physical objects such as poker chips. However, there may be some unique advantages to using virtual manipulatives in computer-based instruc-tion. Three advantages are that computers make it easier to create idealized environments, dy-namically link materials, and produce continuous transformations of objects.

Idealized environments. One advantage of ma-nipulating virtual objects over real objects is that virtual objects exist in idealized environments. For example, there are many advantages of using computer-based laboratory materials including portability, safety, cost-efficiency, and flexible, rapid, and dynamic data displays. As distance learning becomes more wide spread, there will be a greater need for the virtual manipulation of objects.

An example is the use of virtual objects to teach children how to design scientific experi-ments by isolating and testing one variable at a time (Triona & Klahr, 2003). The task required 4th- and 5th-grade students to evaluate how vari-ables such as the length, width, wire size, and weight influence the stretching of a spring. After selecting pairs of springs and weights from a computer display, children saw a video of how far the springs stretched.

Triona and Klahr compared a group of children who trained on the instructional software with a group of children who trained with real springs and weights. Their results showed that children who trained with the virtual materials were as capable in correctly designing experiments as children who trained with the physical materi-als. Following training, both groups were asked to design experiments to evaluate the effects of four variables on the time it would take a ball to roll down a ramp. Only physical materials were

59


used on this transfer task. Again, the group who had trained on virtual springs did as well as the group who had trained on real springs in design-ing experiments with real ramps, even though they had not interacted with physical materials during the training.

Another study compared the effectiveness of constructing and evaluating toy cars in either a real or virtual environment (Klahr, Triona, & Wil-liams, 2007). Seventh and eighth-grade students assembled and tested the cars in order to design a car that would travel the farthest. Computer-based virtual design was again equally effective and it avoided some of the problems encountered when assembling real cars. These included real cars that did not travel straight, had wheels that were too tight, and required a long corridor for testing. The investigators concluded that their findings support the effectiveness of manipulating virtual objects for learning designing experiments. This does not imply that teachers should abandon hands-on science materials, but teachers should not assume that virtual materials are less effective.

Dynamic linking. Another potential advantage of multimedia learning environments is that ac-tions on screen-based objects can be dynamically linked to more abstract information to establish a direct mapping between actions and mathemati-cal structures. As discussed by Kaput (1994), a central problem of mathematics education is to create functional connections between the world of experience and the formal systems of mathemat-ics. Instructional animations developed by Kaput allow students to observe both how changing the speed of an object is reflected in a graph and how physically manipulating the shape of the graph changes the speed of an object. Bowers and Doerr (2001) report findings from a qualitative study, using Kaput’s Simcalc software, that the dynamic linking of two graphs helped prospective teachers better understand the relations among distance, rate, and time. The students could manipulate the relations in one graph to observe how the relations would change in a corresponding graph.

In contrast, a quantitative study by Thompson (1992; see also Thompson, 1994) failed to find that the dynamic linking of base-ten blocks with decimal numbers improved performance in a pre-test-posttest design. However, interviews revealed that the children who used the Blocks Microworld repeatedly made references to actions on symbols as referring to actions on virtual blocks because of the dynamic linking of symbols and blocks. This contrasts with observations of other children whose operations on symbols and wooden blocks were typically thought of as separate activities.

One of the limitations in using manipulatives is that they may be introduced too late. Resnick and Omanson (1987) expressed disappointment in how seldom their students referred to Dienes blocks in a subtraction task, which they attrib-uted to the students’ automated use of symbols. Thompson proposed that students in his blocks group assimilated instruction on decimals into previously learned operations on whole numbers. He argued that “if students memorize a procedure meaninglessly, it is extremely difficult to get them to change it, even with extended, meaningful remediation” (Thompson, 1992, p. 144).

Continuous variation. The manipulation of blocks is typical of tasks using manipulatives in which students perform actions on discrete objects. However, an advantage of virtual manipu-latives is that it is easier to perform actions that produce continuous variation of variables. Figure 3 shows a screen design that Bob Hoffman and I recently created to teach students about variables in algebra word problems. Students are instructed to raise and lower the height of the second bar to vary both the balance and owed interest on the Visa card. Varying this bar changes the height and amount of the Total Interest bar. It also changes values in the equation.

The purpose of this variation and dynamic linking to other objects and numbers is to dem-onstrate that variables can take on different values but only a single value satisfies the constraint that the total interest is $165. The use of symbols (typi-

60


cally letters) to represent unknown quantities in algebra word problems is initially a challenge for students because letters have many other uses in mathematics classes besides the representation of unknown values (Philipp, 1992). The continuous variation of objects that are dynamically linked to variables in equations will hopefully make the concept of a variable less abstract for students.

ILLUSTRATIVE PROJECTS

The Building Blocks project, which is in the final stages of evaluation, provides an example of the effectiveness of instruction designed around the virtual manipulation of objects. It is an exemplary model of multimedia design and evaluation. The Animation Tutor project is in the initial stages of evaluation but illustrates the application of some of the ideas in this chapter to current research.

The Building Blocks project. The Building Blocks curriculum (Clements & Sarama, 2007) demonstrates the effectiveness of manipulatives. It is an NSF-funded curriculum development project

that has created technology-enhanced mathemat-ics materials for children in pre-kindergarten through second grade. The materials are designed to build on children’s intuitive knowledge to help them learn both spatial-geometric concepts and numeric-quantitative concepts. The project’s title reflects both its literal and metaphorical goals. Building blocks are physical and virtual manipulatives that help children form cognitive building blocks by creating, copying, and com-bining discrete objects, numbers, and shapes to represent mathematical ideas.

The numeric concepts include verbal and ob-ject counting; number recognition, comparison, sequencing, and composition; adding, subtract-ing, and place value. Activities such as placing toppings on pizza support acquisition of these concepts. Geometric concepts include shape identification, composition, and construction; turns, measurement, and patterning. Combining different shapes to make pictures supports these concepts. Although both physical and virtual ma-nipulatives are part of the curriculum, the unique advantages of software include linking virtual

Figure 3. A screen design that allows for the continuous variation of a variable

61


manipulatives to numbers, providing feedback, and guiding children along learning trajectories by moving backward or forward depending on performance.

A summative evaluation of the Building Blocks curriculum demonstrated its effectiveness in im-proving mathematical skills (Clements & Sarama, 2007). The experimental teachers inserted the Building Blocks activities at appropriate points during the day while the comparison teachers used the school’s typical mathematics activities that included most of the same concepts covered in the experimental curriculum. The investiga-tors measured the performance of all children at the beginning and end of the school year. Results showed that the experimental group scores in-creased significantly more than the comparison group scores for both number (effect size = .85) and geometric (effect size = 1.47) concepts.

These large effect sizes are a result of the ex-tensive research (both within and outside of the project) of how children acquire numerical and geometric concepts. The summative evaluation provides empirical support for the effectiveness of computer software that uses manipulatives to improve mathematical reasoning. The Building Blocks project should serve as a role model for the design of future research projects.

The Animation Tutor project. The Animation Tutor project is another NSF-funded curriculum development project that uses computer graphics to support mathematical reasoning. It consists of eight modules that apply high-school level mathematics to topics such as population growth, chemical kinetics, safe driving distances, task completion times, average speed, and personal finance (Reed & Hoffman, in press).

Bob Hoffman, Albert Corbett, and I are con-ducting a study to determine whether the kinds of graphics created for the Animation Tutor will help prepare students for solving problems on the Algebra Cognitive Tutor developed at Carnegie Mellon University. The Algebra Cognitive Tutor provides effective feedback to help students learn

procedures for solving algebra problems (Ritter, Anderson, Koedinger, & Corbett, 2007). Our goal is to determine whether object manipulation will reduce the amount of required tutoring by creating more effective worked examples.

High school students who have been using the Algebra Cognitive Tutor will receive one of three different types of worked examples. One example is a static graphics display that can not be manipulated, a second example is a dynamic graphics display that can be manipulated, and a third example is a verbal control that organizes quantities in a table rather than in a bar graph. Figure 3 shows the static graphics display for one of the algebra word problems. As discussed previously, the dynamic graphics display will en-able to students to raise and lower the unknown quantity to see how it changes total interest in the graphics display and in the equation.

Before receiving instruction on algebra prob-lems students will be instructed on arithmetic word problems, such as the following:

You have a MasterCard with a balance of $532 at a 21% interest rate. You also have a Visa credit card with a balance of $841 at a 16% interest rate. How much money are you paying in total annual interest?

Students in the dynamics graphics group will construct the bar representing total interest from the two bars on the left side of the equation. Ma-nipulation requires that they click on the amount of interest in the first bar and drag a copy to the right side of the equation. Clicking will create a red border around the owed interest in the first bar and around the dragged copy. It will also highlight in red that part of the equation (0.21 x $532) that mathematically represents this amount. Students will then click on the owed interest in the second bar to drag and then stack this amount on top of the first dragged copy. Blue borders and corre-sponding numbers in the equation (0.16 x $841) will be used to dynamically link these quantities within the graphics and to the equation.

62


Each worked example will be followed by an equivalent test problem on the Algebra Cognitive Tutor in which students construct a table to repre-sent quantities and then construct an equation for solving the problem. The Algebra Cognitive Tutor provides constructive feedback that will enable us to determine whether the three different kinds of worked examples influence the amount and type of required feedback. We will also examine whether there are performance differences during a delayed paper-and pencil test that includes all the problems.

Our planned research takes advantage of the strengths of virtual manipulatives. The continuous variation of bar graphs for the algebra problems and the dynamic linking of actions on objects to equations will hopefully encourage students to relate symbols to quantities. However, to be ef-fective, virtual manipulation will need scaffolding to provide constructive feedback on unintended actions, coordinate action sequences, encourage reflection, and provide domain-specific knowl-edge. Providing such support should enable us to fulfill the potential of manipulating multimedia materials.

CONCLUSION

The manipulation of multimedia materials offers a promising method of instruction. However, we still lack a theoretical framework for understand-ing when and how object manipulation facili-tates learning. Engelkamp’s multimodal model provides a beginning of such a framework. This chapter examines its theoretical assumptions listed in Table 1 and discusses their application to multimedia learning. The first three assump-tions that performing actions creates additional memory codes provide a potential explanation for instructional improvement. However, the subsequent three assumptions that performing actions can bypass the conceptual system provide

a potential explanation for the ineffectiveness of actions.

Although Engelkamp’s assumptions are rel-evant to instructional design, they are not com-plete enough to include the additional demands of multimedia learning. These demands can result in unintended actions, involve sequences of related actions, and require reflection about domain-spe-cific knowledge for successful learning. Actions can be performed on either physical or virtual ma-nipulatives but virtual manipulatives have some advantages. They exist in idealized environments, support continuous transformations of objects, and allow for dynamic linking to other objects, symbols, and data displays. The Building Blocks and Animation Tutor programs illustrate the use of virtual manipulatives in instruction.

FUTURE RESEARCH DIRECTIONS

Much more research and development are required to fulfill the promise of multimedia materials. One issue concerns how to optimally blend directed and discovery-based instruction. Some theorists have argued that students need directed instruction based on worked examples (Kirschner, Sweller, & Clark, 2006). Other theorists have argued that students need the opportunity for carefully-scaf-folded inquiry learning (Hmelo-Silver, Duncan, & Chinn, 2007). The appropriate blend of directed and inquiry learning needs to be based on forma-tive evaluations. For example, our initial forma-tive evaluations of the best-buy pizza problem discussed previously revealed that more guidance is required to improve decisions.

Another research issue is to explore how to best coordinate the use of virtual and physical manipulatives. As argued in this chapter, virtual manipulatives have advantages over physical manipulatives but students may also require experience with real manipulatives. Virtual reality environments are now enabling cogni-

63


tive scientists to study situations that combine physical manipulatives with multimedia-produced environments. University of Iowa researchers are studying children’s ability to bike across busy virtual intersections by having them peddle a sta-tionary bike that is partially surrounded by large multimedia screens (Plumert, Kearney, & Cremer, 2007). University of Massachusetts researchers are studying young drivers’ ability to attend to relevant information in virtual environments by driving a stationary car in those environments (Pollatsek, Fisher, & Pradhan, 2006). Such virtual reality environments will provide new opportuni-ties for research and training.

Another future challenge is to place effective multimedia instruction in the schools. I argue in my book Thinking Visually (Reed, in press) that multimedia programs that could support spatial reasoning in mathematics and science education will not be in widespread use by the year 2020. My pessimistic prediction is based on the tre-mendous hurdles required to scale up successful design for widespread use in schools (Goldman, 2005). It also is based on the paucity of research-proven, multimedia programs in mathematics and science education that could be scaled up. I hope my prediction will contribute to creating a stronger commitment for creating and distributing instructional multimedia that has the potential to make dramatic improvements in learning.

ACKNOWLEDGMENT

I would like to thank Kirsten Butcher for her helpful comments on a draft of this chapter. This research was supported by NSF under Award Number SBE-0354420 to Carnegie Mellon Uni-versity. Any opinions, findings, conclusions or recommendations expressed herein are those of the author(s) and do not necessarily reflect the views of the National Science Foundation or Carnegie Mellon University.

REFERENCES

Baroody, A. J., Feil, Y., & Johnson, A. R. (2007). An alternative reconceptualization of procedural and conceptual knowledge. Journal for Research in Mathematics Education, 38, 115-131.

Bowers, J. S., & Doerr, H. M. (2001). An analysis of prospective teachers’ dual roles in understand-ing the mathematics of change: Eliciting growth with technology. Journal of Mathematics Teacher Education, 4, 115-137.

Campbell, R. L. (Ed.). (2001). Studies in reflecting abstraction. Hove, England: Psychology Press.

Clements, D. H., & Sarama, J. (2007). Effects of a preschool mathematics curriculum: Sum-mative research on the Building Blocks project. Journal for Research in Mathematics Education, 38, 136-163.

De Bock, D., Verschaffel, L., & Janssens, D. (2002). The effects of different problem presenta-tions and formulations on the illusion of linear-ity in secondary school students. Mathematical Thinking and Learning, 4, 65-89.

Edwards, L. D. (1991). Children’s learning in a computer microworld for transformation ge-ometry. Journal for Research in Mathematics Education, 22, 122-137.

Engelkamp, J. (1998). Memory for actions. Hove, England: Psychology Press.

Engelkamp, J., & Seiler, K. H. (2003). Gains and losses in action memory. The Quarterly Journal of Experimental Psychology, 56A, 829-848.

Glenberg, A. M., Gutierrez, T., Levin, J. R., Ja-puntich, S., & Kaschak, M. P. (2004). Activity and imagined activity can enhance young children’s reading comprehension. Journal of Educational Psychology, 96, 424-436.

Goldman, S. R. (2005). Designing for scalable educational improvement. In C. Dede, J. P.

64


Honan & L. C. Peters (Eds.), Scaling up success: Lessons learned from technology-based educa-tional improvement (pp. 67-96). San Francisco: Jossey-Bass.

Hmelo-Silver, C. E., Duncan, R. V., & Chinn, C. A. (2007). Scaffolding and achievement in prob-lem-based and inquiry learning: A response to Kirschner, Sweller, and Clark (2006). Educational Psychologist, 42, 99-107.

Hommel, B., Musseler, J., Aschersleben, G., & Prinz, W. (2001). The theory of event coding (TEC): A framework for perception and action planning. Behavioral & Brain Sciences, 24, 849-937.

Jitendra, A. K., Griffin, C. C., Haria, P., Leh, J., Adams, A., & Kaduvettoor, A. (2007). A com-parison of single and multiple strategy instruction on third-grade students’ mathematical problem solving. Journal for Research in Mathematics Education, 99(1), 115-127.

Kaput, J. J. (1994). The representational roles of technology in connecting mathematics with authentic experience. In R. Bieler, R. W. Scholz, R. Strasser & B. Winkelman (Eds.), Mathematics didactics as a scientific discipline (pp. 379-397). Dordecht, The Netherlands: Kluwer.

Kirschner, P. A., Sweller, J., & Clark, R. C. (2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructiv-ist, discovery, problem-based, experiential, and inquiry-based tearching. Educational Psycholo-gist, 41, 75-86.

Klahr, D., Triona, L. M., & Williams, C. (2007). Hands on what? The relative effectiveness of physical vs. virtual materials in an engineering design project by middle school children. Journal of Research in Science Teaching, 44, 183-203.

Koriat, A., & Pearlman-Avnion, S. (2003). Memo-ry organization of action events and its relationship to memory performance. Journal of Experimental Psychology: General, 132, 435-454.

Marshall, S. P. (1995). Schemas in problem solving. New York: Cambridge University Press.

Mayer, R. E. (Ed.). (2005). The Cambridge Handbook of Multimedia Learning. New York: Cambridge University Press.

Moyer, P. S. (2002). Are we having fun yet? How teachers use manipulatives to teach mathemat-ics. Educational Studies in Mathematics, 47, 175-197.

Nilsson, L.-G., Nyberg, L., T., K., Aberg, C., Persson, J., & Roland, P. E. (2000). Activity in motor areas while remembering action events. Neuroreport, 11, 2199-2201.

Paivio, A. (1986). Mental representations: A dual coding approach. New York: Oxford University Press.

Philipp, R. A. (1992). The many uses of algebraic variables. In B. Moses (Ed.), Algebraic thinking, Grades K-12: Readings from NCTM’s School-based journals and other publications (pp. 157-162). Reston, VA: National Council of Teachers of Mathematics.

Piaget, J. (1977). Recherches sur l’abstraction réfléchissante. Paris: Presses Universitaires de France.

Plumert, J. M., Kearney, J. K., & Cremer, J. F. (2007). Children’s road crossing: A window into perceptual-motor development. Current Direc-tions in Psychological Science, 16, 255-263.

Pollatsek, A., Fisher, D. L., & Pradhan, A. (2006). Identifying and remedying failures of selective attention in younger drivers. Current Directions in Psychological Science, 15, 255-259.

Reed, S. K. (2005). From research to practice and back: The Animation Tutor project. Educational Psychology Review, 17, 55-82.

Reed, S. K. (2006). Cognitive architectures for multimedia learning. Educational Psychologist, 41, 87-98.

65


Reed, S. K. (in press). Thinking Visually. New York: Taylor & Francis.

Reed, S. K., & Hoffman, B. (in press). Animation Tutor (Version 1.0) [computer software]. New York: Taylor & Francis.

Reed, S. K., & Johnsen, J. A. (1977). Memory for problem solutions. In G. H. Bower (Ed.), The psychology of learning and motivation, 11 161-201. New York: Academic Press.

Resnick, L., & Omanson, S. (1987). Learning to understand arithmetic. In R. Glaser (Ed.), Advances in instructional psychology (41-95). Mahwah, NJ: LEA.

Ritter, S., Anderson, J. R., Koedinger, K. R., & Corbett, A. (2007). Cognitive tutor: Applied research in mathematics education. Psychonomic Bulletin & Review, 14, 249-255.

Schwartz, D. L., & Black, T. (1999). Inferences through imagined actions: Knowing by simulated doing. Journal of Experimental Psychology: Learning, Memory and Cognition, 25, 116-136.

Shockley, K., & Turvey, M. T. (2006). Dual-task influences on retrieval from semantic memory and coordination dynamcs. Psychonomic Bulletin & Review, 13, 985-990.

Steffens, M. C. (2007). Memory for goal-directed sequences of actions: Is doing better than seeing? Psychonomic Bulletin & Review, 14, 1194-1198.

Thompson, P. W. (1992). Notations, conventions, and constraints: Contributions to effective use of concrete materials in elementary mathematics. Journal for Research in Mathematics Education, 23, 123-147.

Thompson, P. W. (1994). Concrete materials and teaching for mathematical understanding. Arith-metic Teacher, 40, 556-558.

Triona, L. M., & Klahr, D. (2003). Point and click or grab and heft: Comparing the influence of physical and virtual instructional materials on elementary

school students’ ability to design experiments. Cognition and Instruction, 21, 149-173.

Wagner, S. M., Nusbaum, H., & Goldin-Meadow, S. (2004). Probing the mental representation of gesture: Is handwaving spatial? Journal of Memory and Language, 50, 395-407.

ADDITIONAL READING

Allen, B. S., Otto, R. G., & Hoffman, B. (2004). Media as lived environments: The ecological psychology of educational technology. In D. H. Jonassen (Ed.), Handbook of research on educa-tional communications and technology (second ed., pp. 215-241). Mahwah, NJ: Erlbaum.

Barsalou, L., Simmons, W. K., Barbey, A. K., & Wilson, C. D. (2003). Grounding conceptual knowledge in modality-specific systems. TRENDS in Cognitive Sciences, 7, 84-91.

Feyereisen, P. (2006). Further investigation of the mnemonic effect of gestures: Their meaning matters. European Journal of Cognitive Psychol-ogy, 18, 185-205.

Gibbs, R. W. (2006). Embodiment and cogni-tive science. New York: Cambridge University Press.

Glenberg, A. M., & Kaschak, M. P. (2002). Grounding language in action. Psychonomic Bulletin & Review, 9, 558-565.

Goldin-Meadow, S., & Wagner, S. M. (2005). How our hands help us learn. TRENDS in Cognitive Sciences, 9, 234-241.

Linn, M. C., Lee, H., Tinker, R., Husic, F., & Chiu, J. L. (2006). Teaching and assessing knowledge integration in science. Science, 313, 1049-1050.

Moreno, R., & Mayer, R. E. (2007). Interactive multimodal learning environments. Educational Psychology Review, 19, 309-326.

66


Nemirovsky, R., Tierney, C., & Wright, T. (1998). Body motion and graphing. Cognition and In-struction, 16, 119-172.

Noice, H., & Noice, T. (2006). What studies of actors and acting can tell us about memory and cognitive functioning. Current Directions in Psychological Science, 15, 14-18.

Reed, S. K. (2006). Does unit analysis help students construct equations? Cognition and Instruction, 24, 341-366.

Richland, L. E., Zur, O., & Holyoak, K. J. (2007). Cognitive supports for analogies in the mathemat-ics classroom. Science, 316, 1128-1129.

Rieber, L. P. (2003). Microworlds. In D. Jonas-sen (Ed.), Handbook of research on educational communications and technology (2nd ed., pp. 583-603). Mahwah, NJ: Erlbaum.

Rubin, D. C. (2007). The basic-systems model of episodic memory. Perspectives on Psychological Science, 1, 277-311.

Schwarz, C. V., & White, B. Y. (2005). Metamod-eling knowledge: Developing students’ under-standing of scientific modeling. Cognition and Instruction, 23, 165-205.

Staley, D. J. (2003). Computers, visualization, and history: How new technology will transform our understanding of the past. Armonk, NY: M. E. Sharpe.

Steffens, M. C., Buchner, A., & Wender, K. F. (2003). Quite ordinary retrieval cues may deter-mine free recall of actions. Journal of Memory and Language, 48, 399-415.

Wilson, M. (2002). Six views of embodied cognition. Psychonomic Bulletin & Review, 9, 625-636.

Young, M. (2004). An ecological psychology of instructional design: Learning and thinking by perceiving-acting systems. In D. H. Jonassen (Ed.), Handbook of research on educational communications and technology (second ed., pp. 169-177). Mahwah, NJ: Erlbaum.

APPENDIX

Major Assumptions of Engelkamp’s (1998) Multimodal Theory

1. Recall of observed actions should differ from that of performed actions because different systems are involved in encoding (p. 45). Observations encode visual information about movement but performance encodes motor information (p. 37).

2. Although both sensory and motor processes exert a positive influence on retention, each of these influences should be independent of each other (p. 38).

3. Planning an action should lead to poorer recall than performing an action because the performance includes planning (p. 46). Only part of the motor information should be available when the action has been planned but not yet performed (p. 37).

4. Responses can be controlled either from the conceptual system or directly from the particular input system without going through the conceptual system (p. 35).

5. Encoding of relational information occurs only in the conceptual system through a process of spreading activation (p. 40).

6. Performing an action makes it difficult to form new associations in the conceptual system because performing an action forces concentration on information relevant to the action (p. 41).

Documents

Cognitive Effects of Multimedia Learning - San Diego State University