Evaluation and Research for Evaluation and Research for Technology: Technology:

Not Just Playing AroundNot Just Playing AroundByBy

Eva L. Baker Eva L. Baker andand

Harold F. O’Neil, Jr.Harold F. O’Neil, Jr.

AbstractAbstract

• Reading deals with:Reading deals with:– Challenges of technology-based training and Challenges of technology-based training and

educationeducation– Strategies to integrate evaluation into Strategies to integrate evaluation into

technology-based systems for education and technology-based systems for education and trainingtraining

IntroductionIntroduction

• Definition of instructional training Definition of instructional training technology:technology:– ““A vehicle which generates an essentially A vehicle which generates an essentially

reproducible sequence of instructional events reproducible sequence of instructional events and accepts responsibility efficiently and accepts responsibility efficiently accomplishing a specified change” accomplishing a specified change” (Lumsdaine, 1964).(Lumsdaine, 1964).

Challenges of technologyChallenges of technology

• Two sectors of Two sectors of technology use:technology use:– Education in schoolsEducation in schools– Training of military Training of military

personnelpersonnel

EducationEducation

• Educational technology is more broadly Educational technology is more broadly conceivedconceived– IncludesIncludes

• Implementations to improve instructionImplementations to improve instruction• Approaches used to train teachersApproaches used to train teachers

MilitaryMilitary

• Readiness depends Readiness depends on technologyon technology– Weapons, monitoring Weapons, monitoring

devices, vehiclesdevices, vehicles

Goal of AuthorsGoal of Authors

• Inform researchers who are focused on Inform researchers who are focused on only one of these areasonly one of these areas– (Either school or military education)(Either school or military education)

• Prove the believe that these two sectors Prove the believe that these two sectors can learn from one anothercan learn from one another

Technology Use in EducationTechnology Use in Education

• The legacy of The legacy of technology over the technology over the last 50 years is that it last 50 years is that it has been poised to has been poised to make an impact, but make an impact, but almost never doesalmost never does

Why?Why?

• Educational goals are open to Educational goals are open to interpretation (flexible)interpretation (flexible)– Results in a mismatch of instructional strategy Results in a mismatch of instructional strategy

and measured outcomesand measured outcomes• Can also be mismatched due to the capacity of its Can also be mismatched due to the capacity of its

use and the available goals and use and the available goals and outcome outcome measuresmeasures

• Possibly because of classroom-flexibilityPossibly because of classroom-flexibility and and confusion maximized by bureaucracy (state, confusion maximized by bureaucracy (state, district and school)district and school)

Teacher Knows BestTeacher Knows Best

• Teachers have little authority to select Teachers have little authority to select instructional interventionsinstructional interventions– Usually someone outside the classroomUsually someone outside the classroom

• ““Ssolution” is meant to be incorporated into Ssolution” is meant to be incorporated into teachers’ practicesteachers’ practices

– Teachers don’t automatically know how to do this so Teachers don’t automatically know how to do this so technology becomes marginalized (hardly ever technology becomes marginalized (hardly ever integrated into the curriculum)integrated into the curriculum)

MeasurementsMeasurements

• Attempts at measurement have struggled Attempts at measurement have struggled with the outcomes to which technology is with the outcomes to which technology is supposed to contributesupposed to contribute– The actual implementation of any target The actual implementation of any target

interventionintervention– The challenge to disentangle the contributions The challenge to disentangle the contributions

of technology from the differences in:of technology from the differences in:• TeachingTeaching• Teacher commitment andTeacher commitment and• Other context influencesOther context influences

SummarySummary

• Technology Technology implementation has implementation has enormous challenges enormous challenges to address:to address:– Thinned out in its Thinned out in its

goals and authoritygoals and authority– Financial marginFinancial margin– Legitimacy of Legitimacy of

educational functionseducational functions– Personnel are Personnel are

frequently challengedfrequently challenged

Summary (cont.)Summary (cont.)

• Educators are consumers of technology Educators are consumers of technology created by others to address problems created by others to address problems thought to be relevant to learningthought to be relevant to learning– Designers create systems for broad Designers create systems for broad

implementation due to commercialismimplementation due to commercialism• Creates poor fit with individual teacher needsCreates poor fit with individual teacher needs

– Teacher does not see technology as an integral part of Teacher does not see technology as an integral part of his workhis work

Military Research and DevelopmentMilitary Research and Development

• Contributions of R&DContributions of R&D– Advances in testingAdvances in testing– Application of research-based knowledgeApplication of research-based knowledge– Systematizing of instructional design and Systematizing of instructional design and

developmentdevelopment– Advances in cognitive science and technologyAdvances in cognitive science and technology

SummarySummary

• Military R&D has supported advances in:Military R&D has supported advances in:

• Adaptive testingAdaptive testing

• Simulation developmentSimulation development

• Tactical use of technology tools andTactical use of technology tools and

• Embedded training in equipmentEmbedded training in equipment– Rather than being adopted wholesale, these Rather than being adopted wholesale, these

more focused efforts are played outmore focused efforts are played out

• Military’s investment in R&D has paid offMilitary’s investment in R&D has paid off

Proposed Model of Instructional Proposed Model of Instructional DesignDesign

• Incorporates testing and assessment as Incorporates testing and assessment as the force behind decision makingthe force behind decision making

• Involves modeling and graphical displays Involves modeling and graphical displays to enhance data analysis and reportingto enhance data analysis and reporting

• Acknowledges, supports, and builds on Acknowledges, supports, and builds on the skills and preferences of the learnersthe skills and preferences of the learners

• Creates submodels, objects, and designs Creates submodels, objects, and designs that are effective and generalizable to a that are effective and generalizable to a variety of trainings and situationsvariety of trainings and situations

Test DevelopmentTest Development• Two main optionsTwo main options

– First requires the design and First requires the design and validation of outcome validation of outcome measures that match the kind measures that match the kind of experience given by of experience given by technology interventiontechnology intervention

• Requires additional time and costRequires additional time and cost– But likely to show program But likely to show program

impactimpact

– Second uses available Second uses available measures, such as measures, such as standardized testsstandardized tests

• (fill-in-the-blank, multiple-choice)(fill-in-the-blank, multiple-choice)– Less likely to show program Less likely to show program

impactimpact– Costs far less and will be rapidly Costs far less and will be rapidly

implementedimplemented

Fundamental IdeaFundamental Idea

• Treat test design like other systems design Treat test design like other systems design effortsefforts

• Start with domain-independent Start with domain-independent components based on families of cognitive components based on families of cognitive demanddemand

Understanding Technical Understanding Technical Soundness of Tests and Soundness of Tests and

AssessmentsAssessments• The know-how and how-to of authoring The know-how and how-to of authoring

systems will help both military and civilian systems will help both military and civilian testing communities to move beyond their testing communities to move beyond their persistent focus on format in their persistent focus on format in their conceptions of tests for distance learningconceptions of tests for distance learning

TestbedsTestbeds

• Testbeds used for the evaluation of instructional Testbeds used for the evaluation of instructional programs that use technologyprograms that use technology– Would remove key limitationsWould remove key limitations of research and evaluation of research and evaluation

• To investigate, one would need, at minimum:To investigate, one would need, at minimum:– Range of courses and topics to embed studiesRange of courses and topics to embed studies– Agreement of an institution to conduct multiple versions of Agreement of an institution to conduct multiple versions of

a coursea course– Permit random assignment of studentsPermit random assignment of students– Consent of studentsConsent of students– Regular access for the experimentersRegular access for the experimenters– Strategy for instrumenting and collecting student Strategy for instrumenting and collecting student

responsesresponses• Prior, during and following instructional interventionPrior, during and following instructional intervention

Challenges

• The construction of a culture among the The construction of a culture among the course designers, subject matter experts, course designers, subject matter experts, and researchers that emphasizesand researchers that emphasizes– CollaborationCollaboration– Shared accessShared access– Commitment to qualityCommitment to quality

SummarySummary

• Two approaches were sketched for the Two approaches were sketched for the conduct of more productive efforts in the conduct of more productive efforts in the future:future:– Testbeds for technology researchTestbeds for technology research– Formative evaluation and summative Formative evaluation and summative

evaluation conducted distributively with evaluation conducted distributively with appropriate software supportappropriate software support

ReferencesReferences

Baker, E.L., & O’Neil, Jr. H.F. (2003)Baker, E.L., & O’Neil, Jr. H.F. (2003)

Evaluation and research for technology: Evaluation and research for technology:

not just playing around. not just playing around. Evaluation and Evaluation and

Program Planning, 26, 169-176.Program Planning, 26, 169-176.