Embed Size (px)
DESCRIPTION
So you say your program caused results? How do you know? Training and development programs are an important contributor to organization success - as are marketing, inventory management, IT, and human resources among others. To ensure you present results credibly and explain how you know the results are due to the efforts of your program, you need to isolate the effects of the program. This webinar will describe the most commonly used approaches to isolating the effects of your programs from other influences. During the webinar you will learn: • How the isolation step fits into the business alignment process. • Why isolating program effects ensures you report credible business impact and ROI results. • Three of the most commonly used approaches to isolate program effects.
Citation preview
Training Magazine Network presents
Guest Speaker:
Patti Phillips, Ph.D.
President & CEOROI Institute
Isolating Programs Effects:
The Ultimate Link Between Your
Program and Results
Recorded WebinarThis program was presented for Training Magazine Network members as a live webinar.
We invite you to view the full recorded webinar.
Click Here to View RecordingClick Here to View Recording
Download Handouts / Join Discussions
The speaker invites you to join their discussion group at TrainingMagNetwork.com .
Materials are posted for free download in the discussion group.
Click Here to Join GroupClick Here to Join Group
Register
If you aren’t already a member of Training Magazine Network, you’ll register along the way. Registration is quick & free. Members are privacy-protected. Solicitation by vendors is prohibited.
Click to RegisterClick to RegisterTrainingMagNetwork.com is powered by
VFTNetworks social learning platform
Questions? Click here to ask.
What is Training Magazine Network?
TrainingMagNetwork.com is a social learning community of over 14,000 global learning professionals and thought-leaders.
Members have free access to:• Webinars & online workshops
•Presentations, white papers & resources posted by speakers and peers
•Networking with fellow learning professionals
• Free tools to help in their work Social learning system & collaborative learning platform Goal-setting/action-planning software for learners Track Measurable impacts of training Web 2.0 online coaching system Instant publishing & learning content organization tools Downloadable, easily editable eLearning Flash micro-games
About Patti Phillips
Dr. Patti P. Phillips is president of the ROI Institute, Inc., the leading source of ROI competency building, implementation support, networking, and research. She is also chair and CEO of The Chelsea Group, Inc., an international consulting organization supporting organizations and their efforts to build accountability into their training, human resources, and performance improvement programs with a primary focus on building accountability in public sector organizations. She helps organizations implement the ROI methodology in countries around the world-including South Africa, Singapore, Japan, New Zealand, Australia, Italy, Turkey, France, Germany, Canada, and the United States. She has authored a number of publications on the subject of accountability and ROI, including Show Me the Money (Berrett-Koehler, 2007); The Value of Learning (Pfeiffer, 2007); Return on Investment Basics (ASTD, 2005); Proving the Value of HR: How and Why to Measure ROI (SHRM, 2005); Make Training Evaluation Work (ASTD, 2004) and more.
Patti Phillips, [email protected]
Isolating Program EffectsThe Ultimate Link Between Your Programs and
Results
Objectives
• Describe how the isolation steps fits into the business alignment process
• Explain why isolating program effects ensures credible business impact and ROI
• Apply three most commonly used approaches to isolate the effects of the program
What causes results?
A. Your programsB. Other internal factorsC. Other external factorsD. I can’t explain itE. All of the above
What causes results?
A. Your programsB. Other internal factorsC. Other external factorsD. I can’t explain itE. All of the above
Levels of Evaluation
Measurement Focus Typical Measures
The ROI calculation is simple.
BCR =
ROI =
Program Benefits
Program Costs
Net Program Benefits
Program CostsX 100
Try it!
$750,000
$425,000BCR =
$750,000 - $425,000
$425,000ROI = X 100
The ROI Calculation
$750,000
$425,000BCR =
$750,000 - $425,000
$425,000ROI = X 100
= 1.76:1
= 76%
The levels serve three purposes!
Evaluation
Objectives
Needs
ProgramBusiness Alignment and Forecasting The ROI Methodology
Business Alignment
Learning Needs
Preference Needs
Evaluation
Reaction
Learning
Application
Impact
ROI
Reaction Objectives
Learning Objectives
Application Objectives Performance Needs
Impact ObjectivesBusiness Needs
Payoff Needs ROI Objectives
End HereStart Here
5
4
3
2
5
4
3
2
1 1
Initial Analysis
Input Needs 0 Input Objectives Input 0
Develop EvaluationPlans and
Baseline Data
Develop EvaluationPlans and
Baseline Data
DevelopObjectivesOf Solution
DevelopObjectivesOf Solution
Stage 1Evaluation Planning
Stage 2 Data Collection
Collect DataDuring SolutionImplementation
Collect DataDuring SolutionImplementation
Level 1
Level 2
Collect DataAfter Solution
Implementation
Collect DataAfter Solution
Implementation
Level 3
Level 4
Evaluation Process
CaptureCosts
Of Solution
CaptureCosts
Of Solution
Isolate theEffects ofSolution
Isolate theEffects ofSolution
Convert Data to Monetary
Value
Convert Data to Monetary
Value
Generate Impact Study
Report
Generate Impact Study
Report
Stage 4 Communicate
ResultsStage 3
Data Analysis
Calculatethe Return On
Investment
Calculatethe Return On
Investment
Level 5
Identify IntangiblesMeasures
Identify IntangiblesMeasures
Intangible Benefits
How do you know it was your
program?
• At least one method must be used to isolate the effects of the solution.
• When collecting and analyzing data, use only the most credible sources.
• When analyzing data, choose the most conservative alternative for calculations.
• If no data are available, it is assumed that little or no improvement has occurred.
• Estimates of improvements should be adjusted for the potential error of the estimate.
Principles to Remember
• Use of a control group arrangement• Trend line analysis of performance data• Use of forecasting methods of performance data• Participant’s estimate of program impact (percent)• Supervisor’s estimate of program impact (percent)• Manager’s estimate of program impact• Use of expert/previous studies• Calculate/estimate the impact of other factors• Customer input
Methods to Isolate Program Effects
Financial Services, Inc. • Organization: Financial Services (800 branches) • Issue: Turnover Rate (48% - 63%)• Solution: Comprehensive selection system
–Recruiting strategies– Interviewing guidelines–Evaluation guidelines– Individual feedback
• Isolation Technique: Control Group–1/3 of branches are trained in new selection system–2/3 of branches are not trained in new selection system
• Post implementation data collection: 6 months
Control Group Design
Control Group (2/3)
ExperimentalGroup (1/3)
M1
M1 Solution
M2
M2
Poll: What are the major problems?
A. Length of time to evaluationB. ContaminationC. Too many groups involvedD. All of the aboveE. None of the above
Keys to successful control group implementation include:
• Match control and experimental groups as closely as possible– Randomly select participants when possible– Select a limited a number of specific criteria*– Gain client agreement on criteria
• Avoid contamination– Limit the number of participants– Limit the time between the pre-program measures and post
program measures– Limit the number of people who know there is a performance
comparison between the control and experimental groups
• Take advantage of pilot programs and naturally occurring control groups
*This is a practical approach; the scientific approach requires some type of randomization that considers all variables.
What if we have no pre-program measure?
Control Group
ExperimentalGroup Measurement
Measurement
Solution
Micro Electronics, Inc. • Organization: Electronics Components
Manufacturer• Issue: Reject Rate
–Emphasis on quality has created downward trend in reject rates
–One work unit wanted to improve this rate
• Solution: Continuous Process Improvement• Isolation Technique: Trend Line Analysis
2%
1%
J F M A M J J A S O N D J
REJECT RATE
1.85% Pre Program Six-Month Average
Projected Average — Using Pre Data as a Base1.45%
.7% Post Program Six-MonthAverage
CPI Program Conducted
MONTHS
Micro Electronics, Inc.
What is the difference in reject rates from the program?
A. .5%B. .78%C. .75%D. 1.15%
Two conditions must be met for the trend line technique to successful:
1. The trend that has developed prior to the solution is expected to continue if the solution had not been implemented.
2. No other “new” influences entered the process after the solution was implemented.
National Bank • Organization: Financial Services• Issue: Variety of Product Lines• Solution: Sales Culture
–Sales Training– Incentives–Management Reinforcement
• Isolation Technique: Estimation–Data collected via questionnaire to branch managers–Branch employees provided input during meeting
• Sales Training• Incentives• Management Reinforcement• Market Fluctuations
Four influencing factors
• Described the task and the process.
• Explained why the information was needed and how it will be used.
• Had employees discuss the linkage between each factor and the specific output measure.
• Provided employees with any additional information needed to estimate the contribution of each factor.
During carefully organized meeting, branch manager:
• Asked employees to identify any other factors that may have contributed to the increase.
• Obtained the actual estimate of the contribution of each factor. The total must be 100%. Several consensus-reaching tools were offered.
• Obtained the confidence level from each employee for the estimate for each factor (100%=certainty; 0%=no confidence).
• The values are averaged for each factor.
Results for One BranchMonthly increase in credit card accounts: 175
(fact)
Contributing Factors
Consensus
Impact (%)
Average Confidence (%)
Sales Training 32% 83%
Incentives 41% 87%
Management Reinforcement
14% 62%
Market Fluctuations
11% 91%
Other _________
2% 91%
100%
Poll: How many credit card accounts are due to the sales training program?
A. 46.48B. 56C. 71.75D. 14.43
Monthly increase in credit card accounts: 175 (fact)
Impact (%) Confidence (%)
Sales Training 175 X
32% = 56
X 83% = 46.48
The calculation is:
46.48 credit card accounts are due to the sales training.
Impact (%) Est. Number Confidence (%) Uncertainty (%)
175 X 32% = 56 X 83% 17%
Here is what happens:
The branch employees estimate that 56 new credit card accounts are due to sales training; but, they are only 83% confident in their estimate -- this means they are 17% uncertain. This allows for a margin of error of:
56 x 17% = 9.52 (+/-)
Given that there is a margin of error:
65.52
56
46.48
+ 9.52
- 9.52
Given that there is a margin of error:
65.52
56
46.48
+ 9.52
- 9.52
Poll: Why do we go with the low number?
A. At least one method must be used to isolate the effects of the solution.
B. When collecting and analyzing data, use only the most credible sources.
C. When analyzing data, choose the most conservative alternative for calculations.
D. If no data are available, it is assumed that little or no improvement has occurred.
E. Estimates of improvements should be adjusted for the potential error of the estimate.
Keys to successful use of estimations:• Explain the task up front.
–Describe the rules–Explain the importance of the process–Share the series of questions with participants at the outset
of solution implementation then again during the evaluation process
• Focus on one impact measure at a time–Share the first measure and show improvement– Identify the different factors that have contributed to
performance–Discus the linkage between each factor and the measure
• Allocate the improvement due to the program• Provide confidence estimates
Results are accepted based on credibility.
Reputation of the source Reputation of the source Source of dataSource of data Source of studySource of study
BiasesBiases Motives of researcherMotives of researcher Personal bias of audiencePersonal bias of audience
Methodology of the studyMethodology of the study Assumptions made in the analysisAssumptions made in the analysis Realism of the outcome dataRealism of the outcome data Type of dataType of data
Scope of analysisScope of analysis
Poll: Of the three techniques introduced today, which one(s) will work for you?
A. Control GroupB. Trend Line AnalysisC. EstimationsD. All of them, depending on the solution and
measuresE. None of them, what are my other options?
Let us hear from you!For a copy of the ROI Methodology™ process model, email your name and mailing address to:
Patti Phillips, [email protected]
Isolating Program EffectsThe Ultimate Link Between Your Programs and
Results
Objectives
• Describe how the isolation steps fits into the business alignment process
• Explain why isolating program effects ensures credible business impact and ROI
• Apply three most commonly used approaches to isolate the effects of the program
What causes results?
A. Your programsB. Other internal factorsC. Other external factorsD. I can’t explain itE. All of the above
What causes results?
A. Your programsB. Other internal factorsC. Other external factorsD. I can’t explain itE. All of the above
Levels of Evaluation
Measurement Focus Typical Measures
The ROI calculation is simple.
BCR =
ROI =
Program Benefits
Program Costs
Net Program Benefits
Program CostsX 100
Try it!
$750,000
$425,000BCR =
$750,000 - $425,000
$425,000ROI = X 100
The ROI Calculation
$750,000
$425,000BCR =
$750,000 - $425,000
$425,000ROI = X 100
= 1.76:1
= 76%
The levels serve three purposes!
Evaluation
Objectives
Needs
ProgramBusiness Alignment and Forecasting The ROI Methodology
Business Alignment
Learning Needs
Preference Needs
Evaluation
Reaction
Learning
Application
Impact
ROI
Reaction Objectives
Learning Objectives
Application Objectives Performance Needs
Impact ObjectivesBusiness Needs
Payoff Needs ROI Objectives
End HereStart Here
5
4
3
2
5
4
3
2
1 1
Initial Analysis
Input Needs 0 Input Objectives Input 0
Develop EvaluationPlans and
Baseline Data
Develop EvaluationPlans and
Baseline Data
DevelopObjectivesOf Solution
DevelopObjectivesOf Solution
Stage 1Evaluation Planning
Stage 2 Data Collection
Collect DataDuring SolutionImplementation
Collect DataDuring SolutionImplementation
Level 1
Level 2
Collect DataAfter Solution
Implementation
Collect DataAfter Solution
Implementation
Level 3
Level 4
Evaluation Process
CaptureCosts
Of Solution
CaptureCosts
Of Solution
Isolate theEffects ofSolution
Isolate theEffects ofSolution
Convert Data to Monetary
Value
Convert Data to Monetary
Value
Generate Impact Study
Report
Generate Impact Study
Report
Stage 4 Communicate
ResultsStage 3
Data Analysis
Calculatethe Return On
Investment
Calculatethe Return On
Investment
Level 5
Identify IntangiblesMeasures
Identify IntangiblesMeasures
Intangible Benefits
How do you know it was your
program?
• At least one method must be used to isolate the effects of the solution.
• When collecting and analyzing data, use only the most credible sources.
• When analyzing data, choose the most conservative alternative for calculations.
• If no data are available, it is assumed that little or no improvement has occurred.
• Estimates of improvements should be adjusted for the potential error of the estimate.
Principles to Remember
• Use of a control group arrangement• Trend line analysis of performance data• Use of forecasting methods of performance data• Participant’s estimate of program impact (percent)• Supervisor’s estimate of program impact (percent)• Manager’s estimate of program impact• Use of expert/previous studies• Calculate/estimate the impact of other factors• Customer input
Methods to Isolate Program Effects
Financial Services, Inc. • Organization: Financial Services (800 branches) • Issue: Turnover Rate (48% - 63%)• Solution: Comprehensive selection system
–Recruiting strategies– Interviewing guidelines–Evaluation guidelines– Individual feedback
• Isolation Technique: Control Group–1/3 of branches are trained in new selection system–2/3 of branches are not trained in new selection system
• Post implementation data collection: 6 months
Control Group Design
Control Group (2/3)
ExperimentalGroup (1/3)
M1
M1 Solution
M2
M2
Poll: What are the major problems?
A. Length of time to evaluationB. ContaminationC. Too many groups involvedD. All of the aboveE. None of the above
Keys to successful control group implementation include:
• Match control and experimental groups as closely as possible– Randomly select participants when possible– Select a limited a number of specific criteria*– Gain client agreement on criteria
• Avoid contamination– Limit the number of participants– Limit the time between the pre-program measures and post
program measures– Limit the number of people who know there is a performance
comparison between the control and experimental groups
• Take advantage of pilot programs and naturally occurring control groups
*This is a practical approach; the scientific approach requires some type of randomization that considers all variables.
What if we have no pre-program measure?
Control Group
ExperimentalGroup Measurement
Measurement
Solution
Micro Electronics, Inc. • Organization: Electronics Components
Manufacturer• Issue: Reject Rate
–Emphasis on quality has created downward trend in reject rates
–One work unit wanted to improve this rate
• Solution: Continuous Process Improvement• Isolation Technique: Trend Line Analysis
2%
1%
J F M A M J J A S O N D J
REJECT RATE
1.85% Pre Program Six-Month Average
Projected Average — Using Pre Data as a Base1.45%
.7% Post Program Six-MonthAverage
CPI Program Conducted
MONTHS
Micro Electronics, Inc.
What is the difference in reject rates from the program?
A. .5%B. .78%C. .75%D. 1.15%
Two conditions must be met for the trend line technique to successful:
1. The trend that has developed prior to the solution is expected to continue if the solution had not been implemented.
2. No other “new” influences entered the process after the solution was implemented.
National Bank • Organization: Financial Services• Issue: Variety of Product Lines• Solution: Sales Culture
–Sales Training– Incentives–Management Reinforcement
• Isolation Technique: Estimation–Data collected via questionnaire to branch managers–Branch employees provided input during meeting
• Sales Training• Incentives• Management Reinforcement• Market Fluctuations
Four influencing factors
• Described the task and the process.
• Explained why the information was needed and how it will be used.
• Had employees discuss the linkage between each factor and the specific output measure.
• Provided employees with any additional information needed to estimate the contribution of each factor.
During carefully organized meeting, branch manager:
• Asked employees to identify any other factors that may have contributed to the increase.
• Obtained the actual estimate of the contribution of each factor. The total must be 100%. Several consensus-reaching tools were offered.
• Obtained the confidence level from each employee for the estimate for each factor (100%=certainty; 0%=no confidence).
• The values are averaged for each factor.
Results for One BranchMonthly increase in credit card accounts: 175
(fact)
Contributing Factors
Consensus
Impact (%)
Average Confidence (%)
Sales Training 32% 83%
Incentives 41% 87%
Management Reinforcement
14% 62%
Market Fluctuations
11% 91%
Other _________
2% 91%
100%
Poll: How many credit card accounts are due to the sales training program?
A. 46.48B. 56C. 71.75D. 14.43
Monthly increase in credit card accounts: 175 (fact)
Impact (%) Confidence (%)
Sales Training 175 X
32% = 56
X 83% = 46.48
The calculation is:
46.48 credit card accounts are due to the sales training.
Impact (%) Est. Number Confidence (%) Uncertainty (%)
175 X 32% = 56 X 83% 17%
Here is what happens:
The branch employees estimate that 56 new credit card accounts are due to sales training; but, they are only 83% confident in their estimate -- this means they are 17% uncertain. This allows for a margin of error of:
56 x 17% = 9.52 (+/-)
Given that there is a margin of error:
65.52
56
46.48
+ 9.52
- 9.52
Given that there is a margin of error:
65.52
56
46.48
+ 9.52
- 9.52
Poll: Why do we go with the low number?
A. At least one method must be used to isolate the effects of the solution.
B. When collecting and analyzing data, use only the most credible sources.
C. When analyzing data, choose the most conservative alternative for calculations.
D. If no data are available, it is assumed that little or no improvement has occurred.
E. Estimates of improvements should be adjusted for the potential error of the estimate.
Keys to successful use of estimations:• Explain the task up front.
–Describe the rules–Explain the importance of the process–Share the series of questions with participants at the outset
of solution implementation then again during the evaluation process
• Focus on one impact measure at a time–Share the first measure and show improvement– Identify the different factors that have contributed to
performance–Discus the linkage between each factor and the measure
• Allocate the improvement due to the program• Provide confidence estimates
Results are accepted based on credibility.
Reputation of the source Reputation of the source Source of dataSource of data Source of studySource of study
BiasesBiases Motives of researcherMotives of researcher Personal bias of audiencePersonal bias of audience
Methodology of the studyMethodology of the study Assumptions made in the analysisAssumptions made in the analysis Realism of the outcome dataRealism of the outcome data Type of dataType of data
Scope of analysisScope of analysis
Poll: Of the three techniques introduced today, which one(s) will work for you?
A. Control GroupB. Trend Line AnalysisC. EstimationsD. All of them, depending on the solution and
measuresE. None of them, what are my other options?
Let us hear from you!For a copy of the ROI Methodology™ process model, email your name and mailing address to:
