Lessons Learned

Findings from Ten Formative Assessments

of Educational Initiatives at MIT

(2000-2003)

Lori Breslow, Ph.D.

and

The Staff of the

Teaching and Learning Laboratory

The Context

From 1999 to the present, MIT has undertaken approximately 40 experiments in educational innovation. TLL has assessed:

Subjects

1.00,* 6.001*, 6.002x,* 8.02 (TEAL),* 8.224, HST582J, 18.03, Mission 200X*

Programs

The Undergraduate Exchange with University of Cambridge,* Residence-Based Advising*

Technologies

PIVoT, Cross Media Annotation System (XMAS),* MetaMedia,* PRS

Space

The d’Arbeloff/TEAL classroom (26-152)

*Denotes multi-year or multi-semester assessment

Agenda

• Four lessons about the use of educational technology

• Observations on best practices in design, implementation, and assessment

• Where do we go from here?

Four Lessons about Educational Technology

• Educational technologies have contributed to gains in

learning

• Educational technologies that have met a specific educational need that has been unmet or poorly met by traditional media have been more successful

• Too much new technology, too many technologies, technologies that are poorly integrated into the curriculum have been less successful

• Studying the relationship between educational technologies and the contexts in which they are embedded has yielded important understandings

Technology-Enabled Active Learning Increased Learning Gains

Sou

r ce:

Dor

i, Y

. & B

elch

er, J

. (20

04)

“How

Doe

s T

echn

olog

y-E

nabl

ed

Act

ive

Lea

rni n

g A

ffec

t Und

ergr

adua

te S

tude

nts’

Und

ers t

andi

ng o

f E

l ect

r om

agne

tic

Con

cept

s?”

unde

r re

view

at T

he J

ourn

al o

f the

Lea

rnin

g Sc

ienc

es

Lesson #1

Learning gains <g> = %Correct post-test - % Correct pre-test

100% - %Correct pre-test

0

0.1

0.2

0.3

0.4

0.5

0.6

Low Intermediate High

Le

arn

ing

ga

ins

<g

>.

Lecture Students 2002 TEAL Students 2003

A Problem-Based Module Increased Learning Gains

Source: Greenberg, J. Smith, N. & Newman, J. (2003) “Instructional Module in Fourier Spectral Analysis, Based on Principles of How People Learn,” Journal of Engineering Education

Lesson #1

What Has Worked

• Online lectures to teach students basic concepts• Visualizations to help students see what cannot

be seen• Archives of visual images to help students

strengthen different kinds of literacies• Wireless laptops to learn programming• Remote online laboratories• Technologies that provide instantaneous, more

individualized feedback

Lesson #2

Online Lectures Motivate StudentsLesson #2

Online Lectures Were More Effective in Intro Computer Science

Lesson #2

Source: Newman, J. (2002) “6.001 Report on the Outcomes of Online Learning,” unpublished assessment report

Visualizations Help Students See What Cannot Be Seen

An animation used in Physics II, a course in electromagnetism” (8.02T). Click to see how field lines move. For more information on the Technology-Enabled Active Learning project, go to http://web.mit.edu/8.02t/www.

MetaMedia Creates Visual Archives that Expand Literacy

MetaMedia homepage. For more information on MetaMedia, go to http://metaphor.mit.edu/.

iLab Makes Laboratory Facilities Available Remotely

The Flagpole Project allows students to take readings remotely. For more information on iLab, go to http://icampus.mit.edu/projects/iLab.shtml.

Two Other Effective Applications of Educational Technology

• Using wireless laptops to learn programming in “Introduction to Computers and Engineering Problem Solving” (1.00)

• Improving feedback– To the students through hint and check buttons

used to answer problems embedded in online lectures (6.001)

– To the faculty through a personal response system that records student answers to conceptual questions asked during class (8.02T)

Lesson #2

What Hasn’t Worked• Technologies designed for interaction (e.g.,

discussion boards) have been less successful at MIT

• When students need to learn the technology before they can learn the material, they have two things to master

• Too much technology can be detrimental

• When technology is poorly integrated into the curriculum, it is either not used at all or not used effectively by the students

Lesson #3

Applications That Needed Some Work

• Functionality of – MetaMedia – XMAS– Simulation in “Biomedical Signal Image and Processing”

• Discussion boards in “Exploring Black Holes”• Unchat in Mission 200X• Technology in 8.02T

– PRS– PowerPoint

• Laptops when used in lectures in 1.00

Lesson #3

Edtech and the Learning Environment

• Edtech has been most successful when there are strong connections between it and– Learning objectives– Pedagogies– Assessment

• The same technology will have different effects in different environments

• Edtech exerts its impact by changing the nature of information in the system

Lesson #4

There Are Strong Connections in TEAL

• Objective: to increase students’ conceptual and analytical understanding about the nature of electromagnetic fields

• Technology’s contribution– Visualizations– Desktop experiments– PRS

Lesson #4

But the Same Technology May Have Different Effects . . .

• In different learning environments– PIVoT as used at MIT, RPI, and Wellesley

• In different situations– Laptops in 1.00 during active learning and during lecture

• For different students– For example, less prepared MIT students were significantly

more likely than better prepared MIT students to think PIVoT helped their conceptual understanding

– But at RPI, students with better high school preparation derived greater benefit than those with weaker preparation

Lesson #4

Edtech Changes the Nature of Information in the System

Information Has Five Properties

Form Magnitude Velocity Direction Access

Source: Nystrom, C. (1973) “Towards a Science of Media Ecology: The Formulation of Integrated ConceptualParadigms for the Study of Human Communication Systems,” unpublished doctoral dissertation

Lesson #4

How Edtech Changes Information in the System and the Impact

• If form changes student literacies expand

• If magnitude changes students can be overwhelmed by information

• If accessibility and direction changes relationship between students and instructor shifts

• If velocity changes feedback can be more instantaneous

Lesson #4

Best Practices in Design, Implementation, and . . .

• Design– Begin with learning objectives– Research what is already known or has been

done related to the innovation

• Implementation– Estimate the amount of resources (time,

funding, space, etc.) needed; then increase it– Assume mid-course corrections will be needed

... Assessment

Assessment at MIT works best when:• It is formative• It is collaborative, which means

– Demands on faculty time must be honored– Differences in research in the “hard” and

“soft” sciences must be made explicit• Quantitative and qualitative methodologies

are used

Next Steps• The longer term impact of the earliest projects are

being studied– TEAL– Mission 200X– 6.002x

• The “lessons learned” will be disseminated through a variety of venues at the Institute and in the wider academic community

• The next set of research questions of interest to the faculty and the TLL staff will be prioritized