Evaluating Educational Technology

Brian McNurlen & Chris Migotsky

University of Illinois

at Urbana-Champaign

Perspective & Context

“Like every shining new gift, sensory devices may dazzle one into all kinds of wishful thinking…Audio-visual materials must be understood in their relationship to teaching as a whole and to the learning process as a whole.”

Edgar Dale, 1946

Hold Your Horses!

Student Preference for Use of IT in Courses

0

5

10

15

20

25

30

35

40

45

No IT Limited IT Moderate IT Extensive IT Exclusive IT

Preference

Res

po

nd

ent

Per

cen

tag

e

Who Should Evaluate?

• Instructors who want to try something new with educational technology

• Instructors who want to decide if what they currently do is effective

• Classroom or online (or both)

• Others?

Why Evaluate?

• Systematic documentation of your teaching

• What is working?

• How can it be improved?

• Other reasons?

What is the Focus of the Evaluation?

• What is the product or process that needs to be resolved, improved?

• Is it something that can be addressed as an instructor, with limited resources?

• What counts as “success”?

Activity #1

• Select a program, product, or service to be evaluated

• Define how the program/product/service fits into your course (goals/objectives)

• Discuss with partner

• Share with group

Special Issues Associated with Educational Technology

• Continuity with course objectives

• Access issues

• Technical support

• Time

• Costs

• Ongoing technology development

Defining Key Evaluation Terms

• Evaluand

• Formative vs. summative

• Evaluation vs. assessment

• Evaluation vs research

• Logic model

• Resources for further study– Evaluation Thesaurus (Scriven)– Web sites on last slides

Evaluation Process

1. Define evaluand and connection to course goals

2. Methodology & data collection

3. Data analysis

4. Conclusion & implications

1. Define Evaluand and its Fit with Course

• Boundaries

• What, how, & why…logic model

• Intended and unintended consequences

2. Methodology & Data Collection

• Institutional Review Board (IRB)• Qualitative

– Interviews– Observations/VT– Focus Groups– Quick writes

• Quantitative– Surveys– Tests– Participation, attendance, server hits, etc

• Pilot test & timing

Activity #2

• Outline an evaluation plan– Evaluation questions– Tools or methods you’re going to use– Timeline– Expected outcomes

• Keep it simple– What can be accomplished in a semester?– What can be accomplished with limited resources and

staff?

• Share in small groups

Clicker study: Evaluation Questions

• Will implementing this clicker program affect how students perform academically?

• Will implementing this clicker program affect the overall student feedback scores for my course?

Design

• Comparison groups– Two different lecture sections

• Independent variable – Use of clickers, along with instructional plan– Other section just uses the instructional plan

• Dependent variable – Some measure of academic performance,

perhaps a cumulative final

Timeline

1. July– Select clicker technology, training2. August-- TA training3. Late August– Student training, pre-test4. Sept.-December-- 16 weeks of treatment5. Reliability observations-- October, December6. December-- post-test7. April-- follow-up reliability observation

3. Data Analysis

• Results…what does it mean?– Level of analysis– Practical significance– Outliers/extremes– Baseline/benchmark data– Response rates

4. Conclusion & Implications

• What do your results tell you?– Make changes– Sharing your results (IRB?)– Follow-up evaluation– Share with next generation of students

Evaluating Ed Tech: Common Issues

• Too much data

• Evaluating something that could be done by other means, or is impossible to do without the technology

• Technology is a constantly moving target so results will be quickly outdated

• Any other issues?

Evaluation Resources

• Teaching Center or Survey Center on your campus• Western Michigan University Evaluation Center

http://www.wmich.edu/evalctr/• Key Evaluation Checklist

http://www.wmich.edu/evalctr/checklists/checklistmenu.htm#models• An Educator's Guide to Evaluating The Use of Technology in

Schools and Classrooms

http://www.ed.gov/pubs/EdTechGuide/index.html• Evaluation Springboard

http://www.evaluationspringboard.org/• Survey Monkey

http://www.surveymonkey.com

Teaching & Technology Resources

• Principles of Good Practicehttp://fc.byu.edu/tpages/tchlrn/7princip.htm

http://www.tltgroup.org/programs/seven.html

• The Technology Sourcehttp://technologysource.org/article/

toward_more_effective_instructional_uses_of_technology/

• ECARwww.educause.edu/ecar

Teaching & Technology Resources

• An analysis of technology enhancements in a large lecture course.http://www.educause.edu/ir/library/pdf/

eqm0335.pdf

• Planning and Sustaining Evaluation of Instructional Technology Support Programshttp://www.educause.edu/ir/library/powerpoint/

EDU04110.pps

Questions?

• Brian McNurlen(217) 244-5724

mcnurlen@uiuc.edu

• Chris Migotsky(217) 333-3490

migotsky@uiuc.edu