Although instructional methods are moving in evergreater number to a multimedia base, testing is not. Whatprinciples should be considered in correcting thismisalignment?

Aligning Paper Tests with MultimediaInstruction

Scott L. Howell

Although the “click-and-point” virtual classrooms of today hardly resemblethe brick-and-mortar classrooms of yesterday, one thing seems not to havechanged: the prevalence of paper-based tests. Paper-based tests have beenthe staple of education for centuries and will most likely persist for manyyears to come. This article explores some of the issues surrounding thegrowing chasm between the way students are now taught and how they arestill tested from three perspectives: researcher, student, and teacher.

Should educators be concerned with the widening gap in how teachersteach, using rich multimedia in a multidimensional context, and how teach-ers test, on paper in grayscale colors in a two-dimensional setting? Shoulda new generation of students who communicate, learn, and almost live onkeyboards be subjected to paper-based tests that require them to use lessfamiliar handwriting tools, such as pens and number two lead pencils in anunfamiliar context—not on the computer? I believe educators should beconcerned with the lag that prevails in aligning and changing assessmentpractices to the changed, and changing, instructional practices that nowdepend so much on computer-based multimedia environments.

Reliability and Validity

In assessment parlance, two words prevail in almost every discussion andanalysis: reliability and validity. A test’s reliability is a measure of how con-sistent the test measures each time administered, and its validity is a mea-sure of accuracy in representing the intent or construct of the testinstrument.

9

NEW DIRECTIONS FOR TEACHING AND LEARNING, no. 100, Winter 2004 © 2005 Wiley Periodicals, Inc. 75

76 STRATEGIES FOR EVALUATING STUDENT LEARNING

A casual dart game at home where the goal is to see who can get themost darts closest to the bull’s-eye may be illustrative of these rather abstractconcepts of reliability and validity and helpful in our discussion about papertests in a computer-based instructional context. The proximity of the dartsto the bull’s-eye is the goal of the game or the “construct.” The participantwho successfully clusters the darts tightly will have high reliability or con-sistency; the participant who not only clusters the darts but does so nearestto, or even on, the bull’s-eye is also the most accurate or valid. A test or adart game can have high reliability with low validity, but one with highvalidity will also have high reliability.

Whether in dart play or test taking, all participants expect—indepen-dent of high, average, or low scores—reliable and valid scores that predictwithin some acceptable range of variability the appropriate level of mastery.However, the stakes are much higher in test taking than in dart playing,especially with their legal implications, when the measurements from thesetests determine either directly or indirectly placement, college admission,and even employment.

Just as a dart player anticipates after weeks of practicing to see the samekind of dartboard at the “big” contest, so does the student in either a physi-cal or virtual classroom studying a subject expect to see the same kind ofsubject matter presentation on the exam. If at the casual dart contest thedarts are made of lighter or heavier material, the board resized, and the bull’s-eye situated in the corner rather than the center as found previously, wouldanyone predict the results or measurements to be either reliable or valid onthe day of the contest?

Much like the dart game is advancing to the electronic age with its newdarts and electronically sensitized boards, so is the instructional experiencechanging from using only pencils, pens, and blackboards to the ubiquitoususe of keyboards, computer screens, and software applications. Everyonewould agree that practicing the dart game the “new” way but then showingup at a contest where the game was played the “old” way just would not befair, but is this not exactly what is happening in many of our classrooms(physical and virtual) as students taught one way take tests another way?

What Are the Issues?

What are some of the issues with teaching online and testing on paper fromthe perspectives of a researcher, a student, and a teacher?

Researcher Perspective. This specific question of comparabilitybetween paper- and computer-based test results—assuming rich, multi-media instruction—has not been thoroughly studied. Not only are therefew studies on this topic extant, but it is expected that there will never bemany in the future because of research methods concerns and the com-monsensical need for alignment between teaching and testing methods.Most researchers hesitate to conduct this comparison study because of

ALIGNING PAPER TESTS WITH MULTIMEDIA INSTRUCTION 77

well-founded criticism that has mounted over media comparison studiesin recent years, especially in the field of education. One of the most com-pelling reasons either for not researching this comparison or for discredit-ing the results of any comparability studies conducted is the difficulty ofrandomly selecting student participants in control and treatment groups,especially with all of the practical and ethical concerns associated withinaccurate test results. Nonetheless, one practical researcher has suggestedthat this issue still merits evaluation. Frederick (2003) has said that “as theenvironment changes for the delivery of instruction, it is important toreevaluate the ways in which we assess the learning outcome for studentsusing this new format and develop and apply assessment techniques thatare more consistent with the learning environment” (p. 17).

Student Perspective. The students of today are technologically savvyand sophisticated; they also are not patient with what they call old-fashionedteaching or testing practices. Students are accustomed to keyboarding as theycommunicate with friends using e-mail and instant text messaging; a togglestick is a mere extension of their hand, and they have developed motor skillsto prove it in computer game competitions with self and others; and moststudents have access to a computer either at home or at school where manyof them learn, play, and do their homework. At the time of this writing, Iattended a junior high awards ceremony, and the assistant principal men-tioned as an aside that the ninth-grade student who was receiving an out-standing academic award also keyboarded at a rate of 130 words per minute.

As accustomed as today’s students are to keyboards, toggle sticks, andcomputers, they are also as unaccustomed to using number two pencils tofill in test bubble sheets using optical character recognition technology andpens of yesteryear to write long (or even short) papers and essays. Althoughmany students have refined the motor skills of “click, point, and drag” witha mouse, keyboard, or toggle stick and can experience little fatigue doingso, it does not take long for cramps, physical fatigue, and psychological dis-comfort to impair a student who is unaccustomed to writing by hand forextended periods of time with pencils and pens. Two researchers fromBoston College have concluded that “recent research shows that writtentests taken on paper severely underestimate the performance of studentsaccustomed to working on computers. The situation is analogous to testingthe accounting skills of modern accountants, but restricting them to the useof an abacus for calculations” (Russell and Haney, 2000, p. 2).

Teacher Perspective. Even though teachers may be involved as muchas a fourth of their time doing some form of assessment, most have neverreceived formal instruction on good test-writing practices. Training in suchpractices would help teachers improve their tests, but even then, little of thetraining would emphasize the need to incorporate the same multimediaused throughout the course in their tests. Furthermore, tests are often theone neglected part of the instruction process because developing good tests(and even poor tests) is a difficult and time-consuming process.

78 STRATEGIES FOR EVALUATING STUDENT LEARNING

In recent years, school administrators have told teachers that they will“jump aboard” in the classroom (physical or virtual) with any of a numberof technological initiatives that have swept their schools, districts, and uni-versities. Many publishers have packaged textbooks and instructors’ manu-als with CD-ROMS, videos, and supporting Web sites to help “sell” thetextbooks and enrich the instruction; yet, the exam banks and tests stay the same two-dimensional, static, grayscale items that they have always been.Some teachers have embraced these new technological and instructionalmethods willingly and have even taught themselves new skills necessary tokeep up with—not ahead of—their own students’ technological expectations;other teachers would if they could but lack the necessary training, support,and time necessary to acquire new skills; and still others refuse to abandonthe habits and practices of many years of instruction, knowing that retire-ment cannot be far away. The unevenness that exists among the teachers inthis regard also exists in different regions of the country for a complexity ofreasons, from cultural to economic to political to others.

Even though there may not be parity across teachers’ integration oftechnology, multimedia, animation, simulation, color, audio, and video intothe curriculum, there is no question that this integration into instructionhas occurred in the classrooms of today much more quickly than has theintegration of the same into assessment. That this alignment between howteachers teach and how they test needs to occur is clear. What is not as clearis what the barriers are to change and how best to remove them!

Conclusion

Whereas the multimedia-enhanced instruction and the virtual classroomsof today no longer resemble the classrooms of even ten years ago, regret-tably, the one instructional element that has stayed the same is the tests.Chalkboards have given way to computer projection screens, keyboardinghas replaced handwriting, and multimedia simulations and animations nowenhance many learning activities, but still tests are categorically unchangedin their paper-based format.

I suggest that the very pillars and fundamentals of sound test con-struction and delivery—reliability and validity—are threatened by this mis-alignment between teaching and testing methods. Researchers, students,and instructors alike agree, although from different perspectives, that urgentchanges are needed to update and align the antiquated paper-based tests ofthe past to the rich multimedia- and computer-based teaching environmentsof today and tomorrow.

ALIGNING PAPER TESTS WITH MULTIMEDIA INSTRUCTION 79

References

Frederick, P. “The Need for Alternative Authentic Assessments in Online LearningEnvironments.” Journal of Instruction Delivery Systems, 2003, 16(1), 17–20.

Russell, M., and Haney, W. “Bridging the Gap Between Testing and Technology inSchools.” Education Policy Analysis, 2000, 8(19). Retrieved June 7, 2004, fromhttp://epaa.asu.edu/epaa/v8n19.html.

SCOTT L. HOWELL is adjunct professor of instructional psychology and technol-ogy at Brigham Young University in Provo, Utah.