Upload
others
View
17
Download
0
Embed Size (px)
Citation preview
Pa# Phillips, Ph.D. pa#@roiins1tute.net
ROI FOR PEOPLE WHO (THINK) THEY HATE ROI!
Objec1ves
Calculate ROI, given monetary benefits and program costs
Name the five levels of evaluation Name the ten steps in the ROI
Methodology Explain the need for the 12 guiding
principles
Do you agree with the following?
Top 5 Reasons People (Think) They Hate ROI
1. ROI formula is too complicated
2. ROI process generates only one type of data 3. ROI cannot be applied to soU skills 4. ROI will be requested for every program
5. ROI consumes too many resources
The ROI calcula1on is simple.
BCR =
ROI =
Program Benefits Program Costs
Net Program Benefits Program Costs
X 100
Try it!
$750,000 $425,000 BCR =
$750,000 - $425,000 $425,000
ROI = X 100
The ROI Calcula1on
$750,000 $425,000 BCR =
$750,000 - $425,000 $425,000
ROI = X 100
= 1.76:1
= 76%
What Makes a Good ROI?
1. Set the value at the same level as other investments – 15%
2. Set slightly above other investments – 25%
3. Set at break even – 0% 4. Set at client
expecta1ons
While we don’t strive for a nega1ve ROI, nega1ve is not
always bad!
An Evaluation Framework
Case Applications and Practice
A Process Model
Implementation
To make ROI work, five pieces of the puzzle come together.
Operating Standards and
Philosophy
Levels of Evaluation Measurement Focus
0. Inputs and Indicators The input into the project in terms of scope, volume, efficiencies, costs Participants, Hours, Costs, Timing
1. Reaction & Perceived Value
Reaction to the project or program, including the perceived value
Relevance, Importance, Usefulness, Appropriateness, Intent to use, Motivation to take action
2. Learning & Confidence
Learning to use the content and materials, including the confidence to use what was learned
Skills, Knowledge, Capacity, Competencies, Confidence, Contacts
3. Application & Implementation
Use of content and materials in the work environment, including progress with actual items and implementation
Extent of use, Task completion, Frequency of use, Actions completed, Success with use, Barriers to use, Enablers to use
4. Impact and Consequences
The consequences of the use of the content and materials expressed as business impact measures
Productivity, Revenue, Quality, Time, Efficiency, Customer Satisfaction, Employee Engagement
Typical Measures
5. ROI Comparison of monetary benefits from program to program costs
Benefit-Cost Ratio (BCR), ROI%, Payback Period
The levels serve three purposes!
Evalua1on
Objec1ves
Needs
The Alignment Process
Start Here
End Here
5 ROI Objec1ves 5
4 Impact Objec1ves 4
3 Applica1on Objec1ves 3
2 Learning Objec1ves 2
1 Reac1on Objec1ves 1
Project
Payoff Needs
Business Needs
Job Performance Needs
Learning Needs
Preference Needs
Impact
Learning
Reac1on
Ini1al Analysis
Measurement and Evalua1on
Applica1on
Business Alignment and Forecas1ng The ROI Process Model
ROI Methodology
Develop Evaluation Plans and
Baseline Data
Develop Objectives Of Solution
Stage 1 Evaluation Planning
Stage 2 Data Collection
Collect Data During Solution Implementation
Level 1
Level 2
Collect Data After Solution
Implementation
Level 3
Level 4
Capture Costs
Of Solution
Isolate the Effects of Solution
Convert Data to Monetary
Value
Stage 4 Communicate
Results Stage 3
Data Analysis
Calculate the Return On
Investment
Level 5
Identify Intangibles Measures
Intangible Benefits
14
Operating Standards 1. Report the complete story
2. Conserve resources
3. Use the most credible sources
4. Choose the most conservative
alternatives
5. Isolate the effects of the program
6. No data no improvement
7. Adjust estimates for error
8. Throw out the extreme and unsupported
9. Use first year benefits for short-term
programs
10. Include fully-loaded costs
11. Report intangible benefits
12. Communicate results to all stakeholders
Reac1on
Learning
Applica1on
Impact
ROI
Isolate the Effects of the Program
Intangible Benefits aaaaaaaaaaaaaaaaaaaaaaaaaaaaa
Inputs
Only a select few programs are evaluated to ROI. Consider the following: • Life cycle of the program • Linkage of program to opera1onal goals and issues • Importance of program to strategic objec1ves
• Top administrator interest in the evalua1on • Cost of the program • Visibility of the program • Size of target audience • Investment of 1me required
Southeast Corridor Bank Program
Skills Based Pay Systems
Business Impact Employee Turnover Staffing Level Customer and Job Sa1sfac1on Product Sales Cross Selling
Sprint/Nextel Program
All-‐Inclusive Workforce Program (AIW)
Business Impact Akri1on Rate Employee Sa1sfac1on Communica1on Coopera1on & Teamwork Diversity Mix
Health Systems Program
Sexual Harassment Preven1on Workshop
Business Impact Turnover Reduc1on Complaint Reduc1on Job Sa1sfac1on Absenteeism Stress Reduc1on Recrui1ng
Verizon Communica1ons Program
Customer Service Training
Business Impact (Target) Customer Complaint Escala1on
• Follow-‐Up Surveys • Follow-‐Up Ques1onnaires • Observa1on On the Job • Interviews with Par1cipants • Follow-‐Up Focus Groups • Program Assignments • Ac1on Planning • Performance Contrac1ng • Project Follow-‐Up Session • Performance Monitoring
Level 3
Level 4
Collec1ng Post Program Data
Factors to consider when selec1ng
Data Collec*on Method Timing of Data Collec*on
• Type of data • Time – Par1cipant / Supervisor
• Costs • Accuracy – Validity / Reliability
• U1lity • Culture / Philosophy
• Availability of data • Ideal 1me for behavior change (Level 3)
• Ideal 1me for business impact (Level 4)
• Convenience of data collec1on
• Constraints on data collec1on
• Use of a control group arrangement
• Trend line analysis of performance data
• Use of forecas1ng methods of performance data
• Par1cipant’s es1mate of program impact (percent)
• Supervisor’s es1mate of program impact (percent)
• Manager’s es1mate of program impact
• Use of expert/previous studies
• Calculate/es1mate the impact of other factors
• Customer input
Methods to Isolate Program Effects
Control Group Method Design What is the difference in improvement?
Experimental Group M1 Program M2
Control Group M1 M2
Control Group Method Design What is the difference?
Control Group
Experimental Group M1
M1
Program
Control Group Method Design Does exposure to the measure make a difference?
Experimental
Control
Program
Experimental (No pre-measure)
M1 M2
M1 M2
Program M1
2%
1%
J F M A M J J A S O N D J
ERROR RATE
1.85% Pre Program Average
1.45% Projected Average
MONTHS
CPI Program Conducted
.7% Post Program Average
Example of Es1ma1on Monthly increase in credit card accounts: 175 (fact)
Contributing Factors
Consensus Impact (%)
Average Confidence (%)
Sales Training 32% 83%
Incentives 41% 87%
Management Reinforcement
14% 62%
Market Fluctuations 11% 91%
Other _________ 2% 91% 100%
Es1ma1ons Influence Fact % Contrib. Est. Impact Confidence Adjusted
Impact
Sales Training
175 32% 56 83% 46.48
Fact: 175 New Credit Card Accounts
% Contribution: 32%
Est. Impact: 56
Uncertainty 17%
Margin of Error: +/- 9.52
65.52
56
46.48
Data are converted by:
• Converting output to contribution – standard value • Converting the cost of quality – standard value • Converting employee’s time – standard value • Using historical costs • Using internal and external experts • Using data from external databases • Linking with other measures • Using participants’ estimates • Using supervisors’ and managers’ estimates • Using staff estimates
Example: Cost of One Turnover from External Database
* Value obtained from industry-‐related study (external data)
• $70, 000/annually
Salary of Middle Manager
• 150% of annual Salary
Value of Turnover*
• $105,000
Cost of Turnover
Cost of A Sexual Harassment Complaint
$852,000 35
5 Steps to Data Conversion
Step 1: Focus on a unit of measure
Step 2: Determine the value (V) of each unit
Step 3: Calculate the change in performance (∆P)
Step 4: Determine the annual amount of change (A∆P)
Step 5: Calculate the total annual value of the improvement (A∆P x V)
Example Using Internal Experts
Step 1: One grievance
Step 2: V = $6,500 (from Director of nursing and HR Experts)
Step 3: ∆P = average of 7 out of 10 grievances prevented per month
Step 4: Annual ∆P =
Step 5: A∆P x V =
Example Using Internal Experts
Step 1: One grievance
Step 2: V = $6,500 (from Director of nursing and HR Experts)
Step 3: ∆P = average of 7 out of 10 grievances prevented per month
Step 4: Annual ∆P = 7 x 12 = 84
Step 5: A∆P x V = 84 x $6,500 = $546,000
Example Using Standard Values
Step 1: One sale
Step 2: V = 30% profit margin
Step 3: ∆P = $20,000 revenue per month
Step 4: Annual ∆P =
Step 5: A∆P x V =
Example Using Standard Values
Step 1: One sale
Step 2: V = 30% profit margin
Step 3: ∆P = $20,000 revenue per month
Step 4: Annual ∆P = $20,000 x 12 = $240,000
Step 5: A∆P x V = $240,000 x .30 = $72,000
Why does the ROI Methodology work?
• Balanced set of measures • Step-‐by-‐step process • Bridges evalua1on disciplines • Balances research and reality • Flexible • Credible
Next Steps
Assess your readiness for ROI
Iden1fy stakeholders and their data needs
Determine the purpose of your evalua1on prac1ce
Iden1fy programs suitable for ROI
Develop capability in the ROI Methodology
Pa# Phillips, Ph.D. pa#@roiins1tute.net
ROI FOR PEOPLE WHO (THINK) THEY HATE ROI!