Measuring ROI of Training

1

Measuring Measuring ROI of ROI of

TrainingTraining

2

AgendaAgenda

1. Measuring the Effectiveness of Training

Program

2. Measuring ‘Return on Investment’ of Training

3. Enhancing the Effectiveness and ROI of

Training

3

Measuring the Effectiveness of Measuring the Effectiveness of Training ProgramTraining Program

4

Training Need Analysis

Training Objectives

Training Delivery

Training Evaluation

Training Process Training Process

What are the training needs for this person and/or job?

Objective should be measurable and observable

Techniques include on-the-job-training, action learning, etc.

Measure reaction, learning, behavior, and results

5

The Four Levels of EvaluationThe Four Levels of Evaluation

Level 1 - Reaction

Level 2 - Learning

Level 3 – Behavior Application

Level 4 – Business Impact

Four Levels Four Levels of Training of Training

EffectivenessEffectiveness

6

Test the trainees to determine if

they learned the principles,

skills, and facts they were to

learn.

Evaluate trainees’ reactions to

the program. Did they like the

program? Did they think it

worthwhile?

Level 1 - Level 1 -

ReactionReaction

Level 2 - Level 2 -

LearningLearning


7

What final results were achieved in terms of the

training objectives previously set? Did the number

of customer complaints about employee drop? Did

the reject rate improve? Was turnover reduced,

and so forth.

Ask whether the trainees’ behavior on the job

changed because of the training program. For

example, are employees in the store’s complaint

department more courteous toward disgruntled

customers than previously?

Level 3 – Level 3 –

Behavior Behavior

ApplicationApplication

Level 4 – Level 4 –

Business Business

ImpactImpact


8

Value of Value of InformationInformation

Frequency of UseFrequency of Use Difficulty of Difficulty of AssessmentAssessment

Level

I. Reaction

II. Learning

III. Behavior

IV. Results


Least valuable

Most valuable

Frequent

Infrequent

Easy

Difficult

9

Level 1 - ReactionLevel 1 - Reaction

Evaluate trainees’

reactions to the program:

Did they like the program?

Did they like the

facilitators?

Did they like the training

accommodation and

facilities?

Level 1 - Level 1 - ReactionReaction

10

Guidelines for Evaluating ReactionGuidelines for Evaluating Reaction

1. Determine what you want to find out

2. Design a form that will quantify reactions

3. Encourage written comments and suggestions

4. Get 100 percent immediate response

5. Get honest response

6. Develop acceptable standards

7. Measure reactions against standards, and take

appropriate action

8. Communicate reaction as appropriate

11

Please give us your frank reactions and comments. They will help us to evaluate

this program and improve future programs.

Program :

Facilitator :

1. How do you rate the subject (interest, benefit, etc)

a. Excellent

b. Very Good

c. Good

d. Fair

e. Poor

2. How do you rate the facilitator? (knowledge, ability to deliver and communicate?)

a. Excellent

b. Very Good

c. Good

d. Fair

e. Poor

3. How do you rate the facilities? (comfort, convinience, etc)

a. Excellent

b. Very Good

c. Good

d. Fair

e. Poor

4. What would have improved the program?

………………………………………………………………………………… ………………………………………………………………………………… …………………………………………………………………………………

Sample of Reaction Form

12

Level 2 - LearningLevel 2 - Learning

Measuring learning means

determining one or more of

the following :

• What knowledge was

learned?

• What skills were

developed or improved?

• What attitudes were

changed?

Level 2 - Level 2 - LearningLearning

13

Guidelines for Evaluating LearningGuidelines for Evaluating Learning

1. Use a control group if practical

2. Evaluate knowledge, skills and/or attitudes both

before and after the program

3. Use a paper-and-pencil test to measure knowledge

4. Use a performance test to measure skills

5. Get 100 percent immediate response

6. Use the results of evaluation to take appropriate

action

14


Experimental Group Control Group

Pre Test Score 45 46

Post Test Score 55 48

Gain 10 2

Pretest and Posttest Scores on Change Management Training

Example :

15

Level 3 – Behavior ApplicationLevel 3 – Behavior Application

• The frequency of

application of new

skills/knowledge/

attitudes (on the job)

• The effectiveness of the

skills/knowledge/

attitudes (as applied on

the job)

Level 3 – Level 3 – Behavior Behavior ApplicationApplication

16



2. Allow time for behavior change and application to take

place

3. Evaluate both before and after the program if practical

4. Survey and/or interview one or more of the following :

trainees, their immediate supervisor, their subordinates,

and others who often observe their behavior

5. Get 100 percent response or a sampling

6. Repeat the evaluation at appropriate times

7. Consider cost versus benefits

17

Example of Survey to Measure Behavior Application

Understanding and Motivating

1. Trying to understand my subordinates 5 4 3 2 1

2. Listening to my subordinates 5 4 3 2 1

3. Praising good work 5 4 3 2 1

4. Talking with subordinates about

their family and personal interests

5. Asking my subordinates for their ideas 5 4 3 2 1

6. Applying "Management by Walking Around" 5 4 3 2 1

Time and energy spent after the prgram compared to time and energy spent before

the program

5 4 3 2 1

Instruction: The objective of this questionnaire is to determine the extent to which those who attended the recent program on Leadership have applied the principles and techniques that they learned there to the job.

Circle the answer that you consider appropriate for each question.5 = Much more 4 = More 3 = Same 2 = Less 1 = Much less

18

Level 4 – Business ResultsLevel 4 – Business Results

• Indicate the extent to which you

think this program has

influenced each of these

measures in your work unit,

department, or business unit:• Productivity• Quality• Customer Response Time• Cost Control• Employee Satisfaction• Customer Satisfaction• Other

Level 4 – Level 4 – Business Business ResultsResults

19



2. Allow time for results to be achieved

3. Measure both before and after the program if practical

4. Repeat the evaluation at appropriate times

5. Consider cost versus benefits

6. Be satisfied with evidence if proof is not possible

20

Performance IndicatorsPerformance Indicators

HARD DATA INDICATORS

• Downtime duration

• Number of defect products

• Sales volume

• Production unit

• Customer satisfaction index

• Response time to orders

• Number of accidents at work

• Others

21

Performance IndicatorsPerformance Indicators

SOFT DATA INDICATORS (intangible impacts)

• Job satisfaction

• Conducive working relationship

• Effective communication

• Stress rate

• Quality in decision-making

22

Example : Measuring Training ResultsExample : Measuring Training Results

0.00

50.00

100.00

150.00

200.00Program : TQM TQM

TrainingTraining

Results after 3 3

months of months of

training,training, number

of defects

dropped to 80

units/day

120 units

80 units

Before trainingBefore training After trainingAfter training

23

0.00

10.00

20.00

30.00

40.00

50.00

Example : Measuring Training ResultsExample : Measuring Training Results

Program :

Sales TrainingSales Training

Results after 3 3

months trainingmonths training,

number of sales

per salesman

increase to 30

units/month.

20 units

30 units

Before trainingBefore training After trainingAfter training

24

Measuring Measuring Return on Investment of Return on Investment of

TrainingTraining

25

Level 5 : Return on Investment of TrainingLevel 5 : Return on Investment of Training

Level 1 - Reaction

Level 2 - Learning

Level 3 – Behavior Application

Level 4 – Business Impact

Level 5 – Return on Investment of

Training

26

Criteria for Selecting Programs for Levels Criteria for Selecting Programs for Levels

4 and 5 Evaluation 4 and 5 Evaluation

• Importance of the program in meeting the

organization’s goals

• Cost of the program

• Visibility of the program

• Size of the target audience

• Extent of management interest

27

Benefits of ROI of TrainingBenefits of ROI of Training

• Measure contribution

• Set priorities

• Focus on results

• Alter management perceptions of training

28

ROI of Training Model ROI of Training Model

Collect Collect DataData

Isolate the Isolate the Effects of Effects of TrainingTraining

Convert Data Convert Data to Monetary to Monetary

ValuesValues

Calculate ROI Calculate ROI of Trainingof Training

Tabulate Tabulate Program Program

CostsCosts

Identify Identify Intangible Intangible BenefitsBenefits

29

Net Program BenefitsNet Program Benefits

Program CostsProgram CostsX 100X 100

Return on Investment FormulaReturn on Investment Formula

ROI =ROI =

ExampleExample : :

• Costs per program (25 participants)Costs per program (25 participants) $ 88,500$ 88,500

• Benefits per program (1st year)Benefits per program (1st year) $230,625$230,625

$ 230,625 – 88,500$ 230,625 – 88,500

$ 88,500$ 88,500ROI =ROI = X 100X 100

ROI = 161 %ROI = 161 %

30

Collecting DataCollecting Data

• Identify appropriate

performance indicators

• Develop a collection plan

Collect Collect DataData

31

Example of Performance IndicatorsExample of Performance Indicators

• Units produced• Items sold• Work backlog• New accounts opened• Productivity• Inventory turnover• Etc.

OutputOutput

• Equipment downtime• Overtime• Time to project

completion• Processing time• Repair time• Lost time days• Etc

TimeTime

32

Example of Hard IndicatorsExample of Hard Indicators

• Unit costs• Variable costs• Overhead costs• Operating costs• Number of cost

reduction• Etc.

CostCost

• Scrap• Waste• Rejects• Error rates• Rework• Product defects• Product failure

QualityQuality

33

Example of Performance ResultsExample of Performance Results

• Some performance results after training Some performance results after training

program:program:

• Scrap was reduced from 11 % to 7.4 %

• Absenteeism was reduce from 7 % to 3.25 %

• The annual turnover rate was reduced from 30

% to 16 %

• Lost time accidents were reduced 95 %

34

Isolating the Effects of Training Isolating the Effects of Training

Methods to Methods to Isolate the Isolate the Effects of Effects of TrainingTraining

Using Using Control GroupControl Group

Trend Trend LinesLines

Participants Participants EstimateEstimate

Supervisors of Supervisors of Participants Participants

EstimateEstimate

35

Using Using Control Control GroupGroup

• A control group arrangement can be

used to isolate training impact.

• With this strategy, one group

receives training, while another,

similar group does not receive

training.

• The difference in the performance of

the two groups is attributed to the

training program.


36

Trend Trend LinesLines

• Trend lines are used to project the

values of specific output variables if

training had not been undertaken.

• The projection is compared to the

actual data after training, and the

difference represents the estimate

impact of training.


37

Trend Lines AnalysisTrend Lines Analysis

Jan Feb Mar Apr Jul Jun May Aug

At the beginning of May, a Sales training Program session was

held

Vo

lum

e o

f S

ales

The

difference

represents

the estimate

impact of

training.

Trend Projection

Actual sales performance

38

Participants and Participants and Supervisors of Supervisors of

Participants Participants Estimate of Estimate of

Training’s ImpactTraining’s Impact

• This method rests on the assumption that

participants (and their supervisors) are

capable of estimating how much a

performance improvement is related to the

training program.

• Because their actions have produced the

improvement, participants (and their

supervisors) may have very accurate input

on the issue.

• They should know how much of the

change was caused by applying what they

have learned in the program.


39

Participants and Participants and Supervisors of Supervisors of

Participants Participants Estimate of Estimate of

Training’s ImpactTraining’s Impact

• Typical Questions to Estimate :Typical Questions to Estimate :

• What percent this improvement

can be attributed to the application

of skills/techniques/knowledge

gained in the training program?

• What confidence do you have in

this estimate, expresses as a

percent?

• What other factors contributed to

this improvement in performance?


40

1 Training Program 50% 70%

2 Change in Procedures 10% 80%

3 Adjustment in Standards 10% 50%

4 Revision to Incentive Plan 20% 90%

5 Increased Management Attention 10% 50%

6 Other - -

Percent Improvement

Caused by

Confidence Expressed as

a PercentFactors Which Influenced

ImprovementNo.


Example of a Participant’s EstimationExample of a Participant’s Estimation

The confidence percentage is multiplied by the estimate (50 % x 70 %) to produce a usable training a usable training factor value of 35 %factor value of 35 %

41

1 Training Program 50% 70%

2 Change in Procedures 10% 80%

3 Adjustment in Standards 10% 50%

4 Revision to Incentive Plan 20% 90%

5 Increased Management Attention 10% 50%

6 Other - -

Percent Improvement

Caused by

Confidence Expressed as

a PercentFactors Which Influenced

ImprovementNo.


Example of a Participant’s EstimationExample of a Participant’s Estimation

• The confidence percentage is multiplied by the estimate (50 % x 70 %) to

produce a usable training factor value of 35 %• This adjusted percentage is then multiplied by the actual amount of

improvement (post-program minus pre-program value) to isolate the portion

attributed to training• The adjusted improvement is now ready for conversion to monetary values,

and used in the return on investment

42

Converting Data to Monetary ValuesConverting Data to Monetary Values

Converting Converting Data to Data to

Monetary Monetary ValuesValues

• Steps to Convert Data to Monetary Steps to Convert Data to Monetary

ValuesValues

1. Focus on a unit of improvement

2. Determine a value of each unit

3. Calculate the change in performance

data

4. Determine an annual amount of change

5. Calculate the annual value of

improvement

43

Steps to Convert Data to Monetary ValuesSteps to Convert Data to Monetary Values

An example to illustrate the steps to convert data to monetary values

1. Focus on unit improvement

• One grievance reaching step two in the

four-step grievance resolution process

StepsSteps IllustrationIllustration

2. Determine a value of each

unit

• Using internal experts, the cost of an

average grievance was estimated to be $

6,500 when considering time and direct

costs (V = $ 6,500)

44


3. Calculate the change in

performance data

• Six months after the program was completed,

total grievances per month reaching step two

declined by ten. • Seven of the then grievance reductions were

related to the program as determined by

supervisors (isolating the effects of training)


4. Determine an annual amount for the change

• Using the six month value, seven per month

(grievance reductions), yields an annual

improvement of 84 (7 x 12 months)

45


5. Calculate the annual value of improvement

• Annual value = 84 x $ 6,500 = $ 546,000


46

Tabulating Tabulating cost of the cost of the programprogram

• Tabulating the costs involves

monitoring or developing all of the

related costs of the program

targeted for the ROI calculation.

Tabulating Cost of the ProgramTabulating Cost of the Program

47


• Cost components that should be

included are :

• The cost to design and develop

the program

• The cost of all program materials

provided to each participant

• The cost for facilitator

• The cost of the facilities of the

training program


48


• Cost components that should be

included are :

• Travel, lodging, and meal costs

for the participants

• Salaries, plus employee benefits

of the participants who attend

the training

• Administrative and overhead

costs of the training function,

allocated in some convenient

way


49

An Example to Illustrate ROI analysisAn Example to Illustrate ROI analysis

Weeks after training Trained Group Control Group

1 US$ 9,723 9,698

2 9,978 9,720

3 10,424 9,812

13 13,690 11,572

14 11,491 9,683

15 11,044 10,092

Average for weeks 12,075 10,449

13, 14, 15

Post Training Data

Average Weekly Sales

50


Data US$

Average weekly sales

Trained Group 12,075

Control Group 10,449

Increase 1,626

Profit Contribution from Training 2 % 32.50

(Training Impacts)

Total weekly improvement 1,495

(32.5 x 46 participants)

Total annual benefit 71,760

($ 1,495 x 48 weeks)

Annualized Program Benefits

Note : 46 participants were still in job after 3 months

51


Cost US$

Facilitation fees 11,250

Program materials : $ 35 x 46 1,610

Meals : 3 days x $ 28 x 46 1,288

Participant salaries plus benefit (35 %) 12,442

Coordination and Evaluation 2,500

Total Cost 29,090

Cost Summary : An Illustration

ROI (%):

$ 71,760 - $ 29,090

$ 29,090

146 %

x 100=

=

52

Enhancing Training Enhancing Training EffectivenessEffectiveness

53

How Effective is Your Training Program?How Effective is Your Training Program?

Broad and Newstrom (1992) report studies have

shown less than 30% of what is30% of what is actually taught actually taught

transfers to the job in a way that enhances transfers to the job in a way that enhances

performanceperformance.

Source : Broad, M., & Newstrom, J. W. (1992). Transfer of training: Action packedstrategies to ensure high payoff from training investments. Reading, MA: Addison-Wesley.

54

Source of Barriers to TrainingSource of Barriers to Training Transfer Transfer

• Lack of reinforcement on the job

• Interference from immediate (work) environment

• Nonsupportive organizational culture

• Trainees’ perception of impractical training programs

• Separation from the inspiration or support of the trainer

55

The Transfer PartnershipThe Transfer Partnership

Trainee Trainee recognizes need for new skills

Trainee Trainer Trainer designs and/or delivers learning experiences

Trainee Trainer Manager supports learning and application on the job

Manager

56

The Transfer Matrix The Transfer Matrix

Before During After

Manager

Trainer

Trainee

Time PeriodsR

ole

Pla

yers

57

ManagerManager

Before TrainingBefore Training

• Build transfer of training into supervisory

performance standards

• Collect baseline performance data

• Involve supervisors and trainees in needs

analysis process

• Involve trainees in program planning

• Brief trainees on the importance of the training

(course objective, content, process, and

application on the job)

• Review instructional content and materials

• Plan to participate in training sessions

• Encourage trainees attendance at all sessions

58

TrainerTrainer


• Align the training plan with the organization's

strategic plan

• Systematically design instruction

• Provide proactive opportunities

• Design a peer coaching component for the

program and its follow-up activities

59

TraineeTrainee


• Provide input into program planning

• Actively explore training options

• Participate in advance activities

60

ManagerManager

During TrainingDuring Training

• Prevent interruptions

• Transfer work assignment to others

• Monitor attendance and attention to training

• Recognize trainee participation

• Participate in transfer action planning

• Review information on employee in training

• Plan assessment of transfer of new skills to the

job

61

TrainerTrainer


• Develop application-oriented objectives

• Answer the “WIIFM” question

• Manage the unlearning process

• Provide realistic work-related tasks

• Give individualized feedback

• Provide job performance aid

62

TraineeTrainee


• Maintain an ideas and application notebook

• Participate actively

• Form support groups

• Plan for applications

• Create behavioral contracts

63

ManagerManager

After TrainingAfter Training

• Plan trainees’ reentry

• Provide opportunities to practice new skills

• Have trainees participate in transfer-related

decisions

• Reduce job pressures initially

• Give positive reinforcement

• Schedule trainee briefings for co-workers

• Set mutual expectations for improvement

• Arrange proactive (refresher) sessions

• Provide and support the use of job aids

64

TrainerTrainer


• Provide follow-up support

• Conduct evaluation surveys and provide

feedback

• Develop and administer recognition system

• Provide refresher/problem-solving sessions

65

Trainee Trainee


• Practice self-management

• Review training content and learned skills

• Develop a mentoring relationship

• Maintain contact with training ‘buddies’

66

Recommended Further ReadingsRecommended Further Readings

1. Donald Kirkpatrick, Evaluating Training Programs : The Four Levels,

Berrett-Koehler Publishers

2. Jack J. Phillips and Patricia Phillips, In Action : Measuring Return On

Investment, American Society for Training & Development

3. Mary Broad and John Newstrom, Transfer of training: Action Packed

Strategies to Ensure High Payoff from Training Investments, Addison-

Wesley.

67

End of MaterialEnd of Material

Documents

Measuring ROI of Training