Five Myths about Measuring Training’s Impactmasiecontent.s3.amazonaws.com/content/L15/Session... · Training’s Impact ... CEB is the world’s leading member-based advisory company

© 2015 CEB. All rights reserved Delivered: 3 November 2015

Shawn Overcast Managing Consultant CEB | Metrics That Matter Managing Consultant

CEB Metrics That Matter®

Five Myths about Measuring Training’s Impact Why They’ll Get You into Trouble


The World’s Leading Learning Analytics Software

Key metrics

SmartSheet evaluations

Optimization roadmap

>1 billion data points

Integrated in reporting

Filterable

Scrap learning

Business results

ROI

Built-In Expertise

Learning Impact Benchmarks

Automated Insights

from

© 2015 CEB. All rights reserved Delivered: 3 November 2015 1

30+ Years of Experience

110+ Countries Represented

6,000+ Participating Organizations

300,000+ Business Professionals

Who We Are CEB is the world’s leading member-based advisory company. We have a unique view into

what matters—and what works—when capitalizing on drivers of business performance. With 30 years of experience working with top companies to share, analyze, and apply proven

practices, we begin with great outcomes and reverse engineer to help you unlock your full potential.

As a result, our members achieve outsized returns by more effectively optimizing talent investments, creating new sources of efficiency, reducing risk, and enabling and accelerating

growth.

90% of the Fortune 500

85% of the FTSE 100

76% of the Dow Jones Asian Titans

www.cebglobal.com © 2015 CEB. All rights reserved.

Best Practices & Decision Support Talent Management

Leadership Councils Market Insights Talent Acquisition Talent Development Talent Strategy & Analytics


1  Level 1 Evaluation Results Provide Valuable Information 2  Self-Reported Data are Biased and Lack Value 3  Statistical Significance Testing is Necessary 4  Customization is More Valuable than Standardization 5  Evaluation Begins After Training is Deployed

Five Myths: Insights & Tips to Counter


#1: Level 1 Evaluation Results Provide Valuable Information



Change Your Paradigm. An evaluation can do more than Level 1.

ACTION: Level 1 is No Longer Level 1

Smile Sheet Smart Sheet

If you are going to bother the learners to ask them about their reactions to training, then ask them the really important questions. Ask more than Level 1…

•  Did you learn new knowledge and skills?

•  Will you be able to apply them? •  Will your performance improve

due to training? •  Will you have managerial

support? •  Will your improvement impact

the performance of the organization?


Change Your Paradigm An evaluation can do more than Level 1.

ACTION: Level 1 is No Longer Level 1

Reaction Prediction

Change the purpose. Don’t confirm that they liked training …

Use the data provided by leaners to predict performance improvement!

I did learn I will apply I will improve So will the company


#2: Self-Reported Data are Biased and Lack Value


It’s just someone’s

opinion

My boss needs highly accurate

statistical results, not opinion data Where is the

scientific rigor?


Various Levels of Value

Type of Data Value Drawbacks

Anecdotal •  Rich qualitative information •  Details about value

•  Not easy to summarize •  Not statistically reliable

Self-report •  Quantitative and qualitative •  Easy to gather on surveys •  Can standardize to allow

analysis across courses •  Can create predictive

models

•  Requires rigor to develop valid and reliable instruments

•  Opinion-based and biased

Business / Operations data

•  Non-biased •  Shows actual business

performance •  Outcome measures

specific to individual courses/programs

•  Difficult to extract from systems •  Requires business analyst •  Takes months/years to observe

the outcome •  Inconsistent data quality,

accuracy comprehensiveness


Rely on the Wisdom of Crowds •  People who know their roles and their skills provide accurate

estimates of how much their performance will improve Trust but verify •  Whether you build your own data collection instruments or use

those created by others, apply appropriate rigor to determine if the survey tools are valid and reliable

ACTION: Crowds and Rigor


#3: Statistical Significance Testing is Necessary


If it’s not sig, it doesn’t matter.

Is there a sig relationship or

not?

‘Burden of Proof’?


What is Statistical Significance?

Statistical significance testing is a way of determining if the observed data occurred by chance.

Lear

ner

Per

form

ance

Impr

ovem

ent

Low Manager Support

High Manager Support

x x

x x

x x

x x

x x

x

x x

x x

x

x x

x x

x

Is there a relationship between two or more variables?

Are there differences between groups?

Lear

ner P

erfo

rman

ce Im

prov

emen

t

Scrap Learning

x

x

x

x x

x x

x

x

x

x

x

x

x

x

y = mx + b + e


!  Determine for your teams and stakeholders what differences are meaningful—5%, 10% 15%? What difference is practical and would spur action?

!  Be cautious: Large data sets can create false positives (small differences are sig) and small data sets can create false negatives (large differences that are not sig)

!  The CEO wants to know “What is the impact of training?” !  Educate the C-suite that timely, valid, reliable smart sheet data are

equally as valuable as statistically significant results from impact studies !  Impact studies—time consuming, costly, and resource intensive !  Scalable Smart Sheets—reliable, timely indicators of program quality

ACTION: Balance Statistical and Practical Significance


#4: Customization is More Valuable than Standardization


Content varies per course

Stakeholders often demand questions about certain aspects of the course

Evaluation questions should vary as well

Custom questions provide deep insights for stakeholders


•  The use of a core set of questions for every course •  Some variations are necessary for learning methodology (e.g., ILT,

SPWB, Virtual ILT) and possibly business unit

ACTION: Incorporate Standards

•  Simplicity in design; reduce time and effort •  Allows for benchmarking and comparison of results across courses

•  Governance: Reduce or eliminate library of evaluation forms •  Prevent the deletion of core questions

•  Allow for additional questions in an “Additional Questions” section on a limited basis (e.g., only for strategic, visible and costly programs)

Standardization is …

Benefits are …


#5: Evaluation Begins After Training is Deployed



Planning is essential to good evaluation.

1.  When forecasting the value of a training program before it is designed

2.  When selecting training methods and tools that will maximize learning while minimizing costs

3.  While demonstrating the effectiveness of training after it has been deployed

4.  While making decisions about future versions of the program, especially in comparison to other programs in the curriculum

ACTION: Ongoing Evaluation


Key Takeaways


Thank You

Shawn Overcast

727-452-2478

Danny Brown Account Manager, CEB 321-474-9220 [email protected]

Managing Consultant, CEB

Documents

Five Myths about Measuring Training’s Impactmasiecontent.s3.amazonaws.com/content/L15/Session... · Training’s Impact ... CEB is the world’s leading member-based advisory company