Embed Size (px)
Citation preview
Effectively Evaluating Training
ABC 2016 Conference Denver Colorado
You Like Me You Really Like Me!
Goddess Nemesis
Talking Points
• Brinkerhoff’s Success Case Method • Kirkpatrick’s 4 Levels of Evaluation • Phillip’s 5 Levels of Evaluation • Myths about Measuring Training’s Impact
Identify Problem
Analyze and Learn
Possible Solutions
Implementation
Observe
Brinkerhoff’s Success Case Method
Brinkerhoff’s Success Case Method
Find most and
least successful trainees in the
class
Conduct some
selected in-depth
interviews from each group
Documented “Success Stories”
Identify “Weakest
Link” factors
Kirkpatrick’s 4 Levels of Evaluation
• Level 1: Reaction • Level 2: Learning • Level 3: Behavior • Level 4: Results
Phillip’s 5 Levels of Evaluation
• Level 1: Reaction • Level 2: Learning • Level 3: Application • Level 4: Impact • Level 5: ROI (return on investment)
“IF YOU THINK TRAINING IS EXPENSIVE, TRY IGNORANCE.” Peter Drucker
Level 1 Evaluations
Kirkpatrick’s Level 1 - Reaction
Phillips Level 1 - Reaction
• How did the class like the training?
• How was the setting?
• How was the food? • Is the class better
than being set on fire?
Level 2 Evaluations
Kirkpatrick’s Level 2 - Learning
Phillips Level 2 - Learning
• What did they learn?
• Knowledge before?
• Knowledge after? • Pretest • Post test
Level 3 Evaluations
Kirkpatrick’s Level 3 - Behavior
Phillips Level 3 - Application
• How they will apply what was learned?
• Changing SOPs
Level 4 Evaluations
Kirkpatrick’s Level 4 - Results
Phillips Level 4 - Impact
• How performance was changed due to training?
• Performance Records
Level 5 Evaluations
Phillips Level 5 - ROI
• Was the cost of training worthwhile?
• Performance Records
• Financial Records
Typical Evaluations
• Level 1 Commonly Done • Level 2 Maybe Required for CEUs • Level 3 Depends on Trainee's Management • Level 4 Long Term Management Commitment
Level 1 Evaluations
Reaction – Did learners like the process? Liking Training ≠ Knowledge Transfer Hating Training ≠ Knowledge Transfer
Level 1 in Action
• Your Opinions, Please! In a word, describe this workshop? • Intent Solicit feedback about the course.
Smile Sheet!
Session # 1505 Speaker Sean Scuras, Tetra Tech
QUESTIONS & RATINGS
A B C D F
What is your overall rating of this presentation? 1
What rating would you give the Speaker's knowledge of the subject matter?
1
What rating would you give the clarity of the presentation?
1
TOTALS 3
Smile Sheet!
Level 2 Evaluations
Kirkpatrick’s Level 2 - Learning
Phillips Level 2 - Learning
• Knowledge before? • Knowledge after? • Pretest • Post test • Performance
Records
Level 2 Evaluations
• Learning - What did the participants learn?
Perceived Learning ≠ Knowledge Transfer
• Requires Testing
Level 2 in Action
• Use a post class exam • Emphasize and repeat key points. • Use the post test exam as a pretest exam. • Compute the post test-pretest gain.
Smile Sheet! How much did you know about this subject before taking this workshop?
Nothing Some A Lot 1 2 3 4 5
How much do you know about this subject after participating in this workshop?
Nothing Some A Lot 1 2 3 4 5
• NOT Level 2 Evaluation !!
• The question does not assess actual learning, assesses perceived learning!
Testing
Level 3 Evaluations
Kirkpatrick’s Level 3 - Behavior
Phillips Level 3 - Application
• How they will apply what was learned?
• Changing SOPs
Level 3 Evaluations
• Behavior - Did they apply what they learn?
Knowledge Transfer ≠ Real World Application
• Requires Desire and “Climate”
Behavior Changes
Personal Desire • The person must:
• Have a desire to change. • Know what to do and how to do it. • Work in the right climate. • Be rewarded for changing
Behavior Changes
Management Climates • Preventing – Forbidden to change. • Discouraging – No desire to change current
way. • Neutral – New ideas are ignored. • Encouraging – Receptive to new ideas. • Requiring – Change is mandated.
Level 4 Evaluations
Kirkpatrick’s Level 4 - Results
Phillips Level 4 - Impact
• How performance was changed due to training?
• Performance Records
Level 4 Evaluations
• Impact of education and training on the organization.
• Quantifying actions that occurred as a result of training.
Level 4 Evaluation Information
• How did the training save costs • Did work output increase • Was there a change in the quality of work • Did the work condition improve • Did the trained individual impact the work
group. • Is there evidence that the organization
has changed.
Level 4 Evaluations
• Measuring before and after • Allow time for change to take place • Repeat at appropriate times • Consider cost vs. benefits of Level 4. • Remember, other factors can affect results • Be satisfied with Evidence if Proof is not
possible.
Level 5 Evaluations
Phillips Level 5 - ROI
• Was the cost of training worthwhile?
• Performance Records
• Financial Records
Level 5 Evaluations
ROI(%) = 𝑁𝑁𝑁𝑁𝑁𝑁 𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃 𝐵𝐵𝑁𝑁𝐵𝐵𝑁𝑁𝐵𝐵𝐵𝐵𝑁𝑁𝐵𝐵
𝑇𝑇𝑃𝑃𝑃𝑃𝐵𝐵𝐵𝐵𝐵𝐵𝐵𝐵𝑃𝑃 𝐶𝐶𝑃𝑃𝐵𝐵𝑁𝑁𝐵𝐵 x 100%
Level 5 Evaluations
• Step 1. Collect pre-training data • Step 2. Collect Post-training data • Step 3. Isolating the effects of training • Step 4. Convert data to monetary benefits • Step 5. Tabulate training costs • Step 6. Calculate the rate of investment
Level 5 Example
1 Measurement Reduce effluent violations
2 Value each unit Average cost of violation Example - $1000
3 Change in performance
6 months after training the number of violations dropped by 10 with 6 attributed to the training classes
4 Annual reduction For this example - 10 will be used
5 Calculate the Value Total value (savings) due to training is 10 x $1,000 = $10,000
6 Rate of Return Class costs - $1,500 ROI = 10,000/1,500 = 667%
Level 5 Example
1 Measurement Reduce effluent violations
2 Value each unit Average cost of violation Example - $100
3 Change in performance
6 months after training the number of violations dropped by 10 with 6 attributed to the training classes
4 Annual reduction For this example - 10 will be used
5 Calculate the Value Total value (savings) due to training is 10 x $1,000 = $1,000
6 Rate of Return Class costs - $1,500 ROI = 1,000/1,500 = 66.7%
Other Level 5 Considerations
Employee Satisfaction • Attitude Survey Data • Organizational Commitment • Climate Survey Data • Employee Complaints • Grievances • Discrimination Complaints • Stress Reduction
Employee Issues • Employee Turnover • Employee Absenteeism • Employee Tardiness • Employee Transfers
Customer Service • Customer Satisfaction
Surveys • Customer Complaints • Customer Response Time
Operational Effectiveness • Teamwork • Cooperation • Conflict • Decisiveness • Communication
2 Myths about Measuring Training
• Level 1 evaluations provide no useful information
• Self-reported data is biased and lack value
Smile Sheet!
1 Level Evaluations
• Trainers are some of the most caring people on the planet. Nothing makes them happier than to see people enjoy their training and learn the knowledge and skills that they are after.
• Teaching . Unfortunately, as a profession, we are quite self-centered.
Your Smile Sheet
• Often are Trainer Centric • “The facility was...” • “The Instructor…” • “The exercises were…” • “The materials were …” • “The course content was…” • “Which of the topics covered were…” • “Which of the methodologies were…”
Smile Sheet’s Impact
Smile Sheet → Smart Sheet
TOPIC Recycle Rate: Too Much of a Good Thing
STATEMENTS & RATINGS
Agree Neutral Disagree
This presentation contained valuable information for my agency. 1
The materials and visual aids contained valuable information I can use. 1
Comments A very good presenter with great actual experience!
Smile Sheet → Smart Sheet Evaluation Category Trainer-centered Learner-centered
Program Objectives
The program objectives were clearly defined.
The program objectives were covered by the instructor.
The material was the right level of complexity for my background.
I understood the learning objectives.
I was able to relate each of the learning objectives to the learning I achieved.
I was appropriately challenged by the material.
Course Materials
The course materials were well organized.
The course materials complemented the course content.
I found the course materials easy to navigate.
I felt that the course materials will be essential for my success.
Smile Sheet → Smart Sheet Evaluation Category Trainer-centered Learner-centered
Content Relevant The material was relevant to my needs.
I will be able to immediately apply what I learned.
Instructor Knowledge
The Instructor demonstrated a good understanding of the material.
The Instructor shared his/her experiences in regards to the content.
My learning was enhanced by the knowledge of the Instructor.
My learning was enhanced by the experiences shared by the Instructor.
Instructor Delivery
The Instructor effectively delivered the program material.
The Instructor did a good job of generating participant interaction..
I was well engaged during the session.
It was easy for me to get actively involved during the session.
Smile Sheet → Smart Sheet Evaluation Category Trainer-centered Learner-centered
Instructor Delivery (continued)
The Instructor used a good variety of instructional methods.
The pace of the program was good.
The duration of the session was good.
It was easy for me to get actively involved during the session.
I was comfortable with the pace of the program.
I was comfortable with the duration of the session.
Instructor Style
The Instructor managed the program well.
The Instructor allowed for questions during the program.
The exercises and activities were useful.
I was well engaged during the session.
I was given ample opportunity to get answers to my questions.
I was given ample opportunity to practice the skills I am asked to learn.
Smile Sheet → Smart Sheet Evaluation Category Trainer-centered Learner-centered
Program evaluation
The test was a fair representation of the program content.
The role-plays or simulations were a fair representation of the program content.
I was given ample opportunity to demonstrate my knowledge.
I was given ample opportunity to demonstrate my skills.
Breaks The breaks were spaced at the right times during the session.
I felt refreshed after the breaks.
Smile Sheet → Smart Sheet Evaluation Category Trainer-centered Learner-centered
Facility
The lighting was adequate.
The temperature was comfortable.
The coffee was hot
I found the room atmosphere to be comfortable.
I was pleased with the room set-up.
I experienced minimal distractions during the session.
Smile Sheet → Smart Sheet
• “I understood the objectives for the course.” (L2) • “I am clear about what is expected of me as a
result of going through this training.” (L3) • “I will be able to apply on the job what I learned • “I do not anticipate any barriers to applying what
I learned.” • “I anticipate that I will eventually see positive
results.” (L4)
2 Myths about Measuring Training
• Level 1 evaluations provide no useful information
• Self-reported data is biased and lack value
Self-reported data
• Biased and lack value • SDWA • CWA
Wisdom of the Crowds
• Large groups of people are collective smarter than an elite few!
• Large groups are better at • Solving problems • Innovation • Wise Decisions?
Wisdom of the Crowds
• Who wants to be a Millionaire? • Odds on Lifelines
50% / 50% - Phone a friend - Ask the audience -
50% 65% 93%
Are canceled do to the instructor just not caring anymore
Quick Class Evaluation
