Evaluating the Effectiveness Evaluating the Effectiveness of Online Learning Programsof Online Learning ProgramsITEC 860 Fall 2003

Aniruddh Mukerji

EvaluationEvaluationThe systematic investigation of the worth or merit of an object

Evaluation in the ISD ModelEvaluation in the ISD ModelFormative Evaluation/Internal Evaluation Summative Evaluation/External Evaluation

The E in ADDIEThe E in ADDIE 2

Kirkpatrick’s Model for Kirkpatrick’s Model for Summative EvaluationSummative Evaluation

3

Level One: Reaction Level Two: Learning Level Three: Transfer Level Four: Results

Level One: ReactionLevel One: Reaction

Measures learner’s satisfaction with the course.

Survey at the end of a Online training program asking questions about what learners liked and did not like about the program

What?What? How?How?

4

Level One: ReactionLevel One: Reaction

Could go beyond the ‘smile sheet’ Add questions like:1. The relevance of the objectives

2. The ability of the course to maintain interest

3. Amount and appropriateness of interactive exercises

4. The ease of navigation

5. The perceived value and transferability of the training to the workplace

5

Level Two: LearningLevel Two: Learning

Measures how much of the content a learner has mastered based on the course objectives

Learning can be measured by pre- and post tests, making sure that test items or questions are truly written to the learning objectives

What?What? How?How?

6

Level Three: TransferLevel Three: Transfer

Measures the amount of knowledge, skills and attitude learners are able to apply on the job several weeks or months after the course

Observation surveys, sometimes called behavioral scorecard can be completed by the learner, the learner’s manager/supervisor or customers

What?What? How?How?

7

Level Four: ResultsLevel Four: Results

Measures the financial improvement to the organization’s bottom line several months or a year after the course

Generally applies to training that seeks to overcome a business problem caused by lack of knowledge or skill.

Isolate a representative control group within a larger student population and then rollout training, complete the evaluation and compare the results against a business evaluation of the non-trained group.

What?What? How?How?

8

Level Four: ResultsLevel Four: Results

Sales Training: Measure change in sales volume, customer retention, length of sales cycle

Technical Training: Measure reduction in calls to help desk; reduced time to complete reports, forms or tasks; or improved use of software or systems

Quality Training: Measure reduction in number of defects Safety training: Measure reduction in number or severity

of accidents Management training: Measure increase in engagement

levels of direct reports 9

Finally…Finally…

Evaluation of training programs, both traditional and online, need to be conducted both before, during, and after training to truly assess the effectiveness of training.

Like traditional classroom programs, an online learning program can be evaluated using Kirkpatrick’s four levels

The number of levels you choose will depend on what your client or funder demands, what your budget allows and what your timeline permits. Most people choose to evaluate at the reaction and learning levels. 10

Other related topicsOther related topics

Program Evaluation A systematic investigation of the worth of an ongoing or

continuing distance education activity. Teaching and Learning at a Distance: Chapter 12 focuses

on approaches to evaluation for the purpose of improving distance education and determining the worth of distance education activities

ResourcesResources

e-Learning Centre: Evaluating e-Learning

http://www.e-learningcentre.co.uk/eclipse/Resources/evaluating.htm

  Distance-Educator.com> Site Channels>Evaluators

http://www.distance-educator.com/de_ezine/categories.php?op=newindex&catid=23

Return on Investment - An Evaluative Framework

http://coe.sdsu.edu/eet/Articles/k4levels/index.htm

12

SourcesSources

BOOKS Technology Based Training by Kevin Kruse and Jason Keil (2000) Web-Based Training, 2nd Edition by Margaret Driscoll (2002) Teaching and Learning at a distance by Simonson, Smaldino,

Albright and Zvacek (2003) ONLINE Evaluation of the Effectiveness of Training Programs by

Debra M. Farmer @ http://research.umbc.edu/~hodell/602rp7.htm 

Kirkpatrick's Four Levels of Evaluation @

http://coe.sdsu.edu/eet/Articles/k4levels/index.htm