ROI training 98

Arriffin Mansor

1

Why ROI Identify a feasible ROI Process Use ROI to measure the effectiveness of

training

2

Training budgets are increasing ROI is the ultimate level of evaluation Competitive pressures on costs and

productivity Top executives are requiring ROI information

3

Commitment of Training Expenditure as a Percentage of Payroll:◦ US – 1% - 4% of Payroll◦ Europe – 2.5% to 3% of Payroll◦ Asia – 4% to 8% of Payroll

4

Importance : 5 – Critically important and 1 is Unimportant

Agreement with Trend: 5 – Very much progress and 1 – No progress

Level of Agreement

Level of Importance

1. Training Costs are monitored more accurately to manage

resources and demonstrate accountability

3.92 4.83

2. Measuring the return on investment in training is growing in

use

4.02 4.71

3. Systematic evaluation processes measure the success of training

4.57 4.69

4. Needs assessment and analysis is receiving more emphasis

3.76 4.64

5. Training staff and line management are forming partnerships to achieve common goals

3.95 4.57

6. Training is linked to strategic direction of the organisation

3.96 4.48

7. The learning organisation concept is being adopted. 4.09 4.47

8. Training delivery is changing rapidly 4.26 4.39

9. Training is shifting to a performance improvement role 4.25 4.37

10. The technology of training is developing rapidly. 4.68 4.32

5

ROI measures the contribution of a program/solution designed in relation to the costs or capital involved

ROI = Value of Benefits – Cost of Training Cost of Training

6

Balanced Scorecard (Drs Kaplan and Norton),

Kirkpatrick Four-Level Framework (Kirkpatrick, 1975),

Jack Phillips’s Five Levels of Evaluation

7

Balanced Scorecard (BSC) is a management framework used for implementing organisational strategy by linking the objectives, initiatives and measures of a business.

It integrates traditional financial measures with other key performance indicators including customer perspectives, internal business processes and organisational development and innovation.

8

Level 1 Reaction Evaluation Level 2 Learning Evaluation Level 3 Transfer of Learning

Evaluation Level 4 Results Evaluation This model is also adopted by Singapore Skills

Development Fund (SDF) for companies’ Training Effectiveness Reports to SDF for funding purposes.

9

Level Measurement Focus1. Reaction & Measures participant satisfaction

with Planned Action the program and captures

plannedactions.

2. Learning Measures changes in knowledge, skills and attitudes.

3. Application Measures changes in on-the-jobbehavior.

4. Business Impact Measures changes in business impact

variables – any measurable results?

5. Return on Compares program benefits to the costs

Investment10

Singapore Airlines Singapore Technology Motorola Compaq Companies who submit to SDF for funding

11

Improve program/Process Discontinue/Expand Programs Approve projects (If Pilot) Rank projects Like a front end analysis – to decide whether

worthwhile to proceed on the training project. Develop Data Base of Program Results Inform/Educate Management Inform/Educate target Groups Build Skills with Staff

12

13

Collecting Post Program

Data

Isolating the Effects of

the Program

Converting Data to

Monetary Value

Calculating the Return on

Investment

Identifying Intangible Benefits

Tabulating Program

Costs

Evaluating Timing

Evaluating Levels

Evaluation Purposes

Evaluation Instruments

Significant InfluencesPolicy StatementProcedures & GuidelinesStaff SkillsManagement SupportTechnical SupportOrganisational Culture

Interactive Selling Skills – 3 days course, 48 participants

Retail Merchandise Company You may want to pay attention – you will be

tested on your understanding! – Level 2 Evaluation

14

15


Data


the Program

Converting Data to

Monetary Value


Investment


Tabulating Program

Costs

Evaluating Timing

Evaluating Levels

Evaluation Purposes

Evaluation Instruments


If management don’t believe in it – don’t do it

Control Groups

Increased Sales/Profits*Performance Monitoring*Questionnaire*Follow-up Session

Questionnaire -3 mths later

Surveys,Performance records

Selected Data:◦ Success with Objectives 4.3◦ Relevance of Material 4.4◦ Usefulness of Program 4.5◦ Exercises/Skill Practices 3.9◦ Overall Instructor Skill 4.1

16

All Participants Demonstrated That They Could Use The Skills Successfully

17

Selected Data:Strongly Agree Neither DisagreeStronglyAgree Agree/ Disagree

Disagree

I utilise the 78% 22% 0% 0% 0%Skills taughtIn the program

With Each Every Several At Least At Least customer 3rd Times Once Daily Once

customer each day Weekly

Frequency ofUse of skills 52% 26% 18% 4% 0%

18

Post Training Data

Weeks After Training Training Groups Control Groups

1 $9,723 $9,6982 $9,978 $9,7203 $10,424 $9,81213 $13,690 $11,57214 $11,491 $9,68315 $11,044 $10,092

Average for Weeks $12,075 $10,44913, 14, 15

19

46 participants were still in job after 3 months Ave Wkly Sales

◦ Trained Groups $12,075◦ Untrained Groups $10,449

Increase $1,626 Profit Contribution 2% $32.50 Total Weekly Improvement (x46) $1,495 Total Annual benefits (x48wks) $71,760

20

48 participants in 3 courses Facilitation Fees: 3 courses @ $3750 $11,250 Program Materials: 48 @ $35/ppt $1,680 Meals/Refreshments: 3 days @$28/ppt $4,032 Facilities: 9 days @ $120 $1,080 Participants Salaries Plus Benefits’ (35%) $12,442 Coordination/Evaluation $2,500 Total Costs $32,984

21

ROI =(Value of benefits –cost of training)/Cost of training

Benefit Cost Ratio 71,760 = 2.2:132,984

ROI (%) = 38,776 = 118% 32,984

22

23


Data


the Program

Converting Data to

Monetary Value


Investment


Tabulating Program

Costs

Evaluating Timing

Evaluating Levels

Evaluation PurposesEvaluation

Instruments


Time spent on ROI Cost of conducting the measurement Complexity of variables in ROI Accuracy in measurements Credibility Lack of Skills to measure If staff does not see the need for ROI, it will

usually fail Without support from management, ROI process

will usually fail.

24

No need to evaluate all courses. Egs of courses to be evaluated:Level % Courses1. Participant satisfaction 100%2. Learning 70%3. On-the-job Applications 50%4. Results 10%5. Return on Investment 5%

25

• Coca ColaCoca Cola – 8 half-day on supervisory skills workshops – 1447% ROI, Benefit/Cost Ratio 15:1

• Yellow Freight SystemYellow Freight System – Performance Appraisal Course – 1115% ROI, Benefit/Coast ratio 12:1

• Litton Industries (Avionics)Litton Industries (Avionics) – Self Directed Work Team course – Productivity increased 30%, Scrap rate reduction 50%, 700% ROI

• Multi-Marques, Inc (Bakery)Multi-Marques, Inc (Bakery) – 15 hr Supervisory Skills Training – 215% ROI, Benefit/Cost Ratio 3.2:1

26

Let’s discuss

27

28

Renewed interest

ManagementWants to see results

Budgetincrease

Budget Levels Off

MinimumTraining Level

Budget Cuts(No Results)

RenewedInterest in Training

..And the Cycle Continues

A Training Executive met up with an Instructional Designer to discuss about the design of an e-learning programme for 5000 participants worldwide. This is the first time that the company would be using e-learning. The initial investment in the training is huge. The CEO is very excited about the programme and has high expectations of this training. The Training Executive suggested to forecast/measure the ROI.

As an instructional designer, would you consider the suggestion by the Training Executive? Why?

As the Training Executive, what would you like to achieve from measuring ROI?

29

Cost of Technology - Initial cost of implementing e-learning

Newness of e-learning (is it effective & efficient as F2F learning?)

E-learning is not a proven process in many organisation – need to show value now than later

30

In many e-learning projects, the client wants to know the projected payback from the project.

To venture into the expensive development process without having some sense of the payback is undesired by many clients.

Consequently, there is tremendous pressure to forecast ROI even if it is not very accurate.

31

A major consulting firm ◦ CBT course for 7000 consultants in 50

countries. Cost of CBT training program = $106/student. Previous F2F program cost is $760/student. Life of program was 5 years & savings = $4.5m

◦ What does this means to U? When requesting monetary investment for e-

learning, be prepared to evaluate effectiveness Conduct cost-benefits analysis to assess cost

savings for organisation

32

Based on interviews with 300 managers in UK◦ Managers discouraged staff from CBT◦ Managers recognised potential benefits of CBT◦ 44% were unsure where CBT was less

expensive than classroom training◦ 66% felt CBT would isolate staff◦ What does this means to U?

Evaluate e-learning to get management support Determine whether e-learning meet organisational

efforts & the identified needs

33

Based on Yr 2000 ASTD State if the Industry Report (USA)◦ Projections for 2001 – significant majority of

organisations expect to be using multimedia (91%), CD-ROM (87%), CBT (81%) and intranets (77%)

◦ What does this mean to U? ROI will provide data to help you decide to continue to

stop existing training programmes ROI will help you to obtain support from the “bean

counters”

34

Isolate the effects by getting before and after training performance data.

Performance data of with and without training groups

35

Identify key output for the job to be analyzed

Identify the input variable for the output with cause effect relationship

Compute the KPI by dividing output with the input. It is a ratio.

Obtain before and targeted KPIs and insert the actual KPIs in the diagram.

Separate the value of the output based on volume and efficiency.

36

Example of input output dimension for Sales

• Output is net profits in a week RM20,000• Input is weekly sales figure RM100,000• The KPI is 0.20 profit per ringgit sales or return

on sales• Insert the previous performance KPI which

may be 0.15 profit per ringgit sales• The performance gap between the old and the

new position.

37

After Profit 0.20 x 100,000 = 20,000 Before Profit 0.15 x 100,000 = 15,000 Performance Improvement = 5,000 Efficiency improvement Volume improvement

38

39

Increase production units Increase value of sales Cost savings

◦ Less wastage◦ Less time◦ Less labour◦ Less materials◦ Less overheads

40

Costs of Trainers Opportunity costs of trainees Designing and Planning costs of training

program Administration cost of training softwares

41

Calculate the total cost of training or intervention

Identify the input and output variable that determine performance.

The relationship must be meaningful which has high cause effect relationship

Isolate the impact of training with the following test, with and without, before and after.

42