2009 Roi Trainer 1 Rev1

ROI Trainer 2009 ROI Methodology - ROI on coaching 1

ROI Methodology™

Introduction to the methodologyPresentation of main elements

Simple guide on how to get started


The ROI Process Generates Six Types of Data

• Reaction to a project or program• Learning the skills/knowledge/information to make the

program successful• Application/Implementation progress with the program• Business impact related to the project or program• Return on Investment• Intangible benefits …and includes a technique to isolate the effects of the program


ROI by the Numbers

• Process refined over a 25-year period• 3,000 impact studies are conducted each year• More than 300 case studies have been published on ROI• More than 3,000 individuals are certified to implement the ROI

methodology• Over 30 ROI books have been developed by ROI Institute authors• A professional network has been formed to share information• The ROI methodology has been adopted by over 3,000 organizations

in manufacturing, service, non-profit, and government settings in over 40 countries


Applications

• Learning and Development• Career Development • Competency Systems• Diversity Programs• E-Learning• Executive Coaching• Gainsharing• Meetings and Events• Leadership Development• Organization Development

• Orientation Systems• Recruiting Strategies• Safety & Health Programs• Self-Directed Teams• Skill-Based/Knowledge-Based

Compensation• Technology Implementation • Quality Programs• Wellness/Fitness Initiatives


Definition of Results-Based Programs

• Programs are linked to business needs

• Assessment of performance issues and effectiveness

• Assessment of learning needs

• Objectives are defined at higher levels

• Expectations are created with stakeholders


Definition of Results-Based Programs

(continued)

• The application issue is addressed early and often

• The level of evaluation is established for each program

• Partnerships are developed with key managers

• The evaluation is completed, capturing a variety of data

• Results are communicated to the appropriate stakeholders


Criteria for an EffectiveROI Process

What are the characteristics of an effective ROI process that would work within your organization?


An Evaluation Framework

Case Applications and Practice

A Process Model

Implementation

Operating Standards and

Philosophy

Pieces of the Results-Based Puzzle


ROI is Reported In the Context of the Five-Level Framework

Level

1. Reaction & Planned Action

2. Learning

3. Application

4. Business Impact

5. Return on Investment

Measurement Focus

Measures participant satisfaction with the program and captures planned action

Measures changes in knowledge, skills, and attitudes

Measures changes in on-the-job behavior

Measures changes in mission critical outcomes

Compares program benefits to the costs


Chain of Impact Tells a Complete Story

Reaction & Planned Action

Learning & Confidence

Application & Implementation

Isolate the Effects of the Program

Business Impact

ROI

Intangible Benefits


All Programs Are Not Evaluated at All Levels

Evaluation Purposes

Program Need

Program Profile

Stakeholder Needs

Evaluation Purposes

Program Need

Program Profile

Stakeholder Needs

Level 5

Level 4

Level 3

Level 2

Level 1

Changes in Key

Business Measures

Changes inPerformance

Need for Skillsor Knowledge

Preferences

Cost-BenefitComparison

5-10%

10-20%

30%

40-60%

90-100%


ProjectBusiness Alignment and Forecasting The ROI Process Model

V Model

Learning Needs

Preference Needs

Measurement and Evaluation

Reaction

Learning

Application

Impact

ROI

Reaction Objectives

Learning Objectives

Application Objectives Performance Needs

Impact ObjectivesBusiness Needs

Payoff Needs ROI Objectives

End HereStart Here

5

4

3

2

5

4

3

2

1 1

Initial Analysis

3.6


What Can You Do with the ROI Process?

• Payoff of the ROI Methodology• Align programs to business needs• Show contributions of selected programs• Earn respect of senior management/administrators• Build staff morale• Justify/defend budgets• Improve support for programs• Enhance design and implementation processes• Identify inefficient programs that need to be redesigned or

eliminated• Identify successful programs that can be implemented in other areas• Earn a “seat at the table”


Criteria for Selecting Programs

For Level 4 And 5 Evaluation

• Expected life cycle of the program

• The importance of the program in meeting the organization’s goals

• The connection of the program to strategic objectives

• Cost of the program

• Visibility of the program

• The size of the target audience

• Extent of management interest in the programs’ accountability

2.11


Traditional View Emerging ViewExpenses are considered costs Expenditures are viewed as a source

of value

Function is perceived as a support staff

Function is perceived as a strategic partner

Involved in setting HR budget Top executives involved in budget

Metrics focus on cost and activities Metrics focus on results

Metrics created and maintained by HR alone

Top executives involved in metrics design and use

Human Capital Perspectives


Human Capital PerspectivesTraditional View Emerging View

Little effort to understand the ROI in HC

ROI has become an important tool

Measurement focuses on the data at hand

Measurement focuses on the data needed

Measurement is based on what others measure

Measurement is based on organization needs

Programs initiated without a business need

Programs linked to specific business needs

Reporting is input-focused Reporting is output-focused


Baseline Data

Ask the Client:

• What do you (the client) want to change?• What will it look like when it has changed?• How does it look now?• What direct measures reflect the change?

(Output, Quality, Cost, Time)• What indirect measures reflect the change?• Who can provide information about the relationship between

training and the business measures?• What other factors will influence these business measures?• What solutions have you tried?


Baseline Data (continued)

• Determine if baseline data is available from organization records.

• If organization does not have baseline data, determine if it can be estimated by participants or others.• Determine the timing for the baseline data collection.• Collect baseline data as appropriate.


Factors to Consider When Developing an Evaluation Strategy

• Location of participants• Duration of program• The importance of program in meeting organizational objectives• Relative investment in the program• The reason for program’s existence• Ability of participants to be involved in evaluation• The level of management interest and involvement in the process• The content and nature of the program• Interest in evaluation by senior management• The availability of business results measures


Examples from coaching:Objectives for coaching -

ReactionAfter participating in the coaching program, the participant will:• Perceive coaching to be relevant to the job• Perceive coaching to be important to job performance at the present time• Perceive coaching to be value added in terms of time and funds invested• Rate the coach as effective• Recommend this program to other executives


Objectives for coaching – Learning

After completing the coaching program, the participants should improve their understanding of or skills for:

• Uncovering individual strengths and weaknesses• Translating feedback into action plans• Involving team members in projects and goals• Communicating effectively• Collaborating with colleagues• Improving personal effectiveness• Enhancing leadership skills


Objectives for coaching - Application

Six months after completing the coaching program, participants should do the following:

• Complete the action plan• Adjust the plan accordingly as needed for changes in the environment• Show improvements in• Uncovering individual strengths and weaknesses• Translating feedback into action plans• Involving team members in projects and goals• Communicating effectively• Enhancing leadership skills


Objectives for coaching - Impact

Six months after completing the coaching program, participants should improve at least three specific measures from the following areas:

• Sales growth• Productivity/Operational efficiency• Direct cost reduction• Retention of key staff members• Customer satisfaction


Objectives for coaching - ROI

The ROI of the coaching program should at least be 25%


Methods of Collecting Data

• Follow-Up surveys• Follow-Up questionnaires• Observation on the job• Follow-up interviews• Follow-Up focus groups• Program assignments• Action planning• Performance contracting• Program Follow-Up session• Performance records

Level 3 Level 4


Evaluation Framework and Key Questions

Levels of Evaluation Key Questions AnsweredLevel 1: Reaction and Planned Action

Was the program relevant to participants’ jobs and mission?

Was the program important to participants’ job/mission success?

Did the program provide new information?

Do participants intend to use what they learned?

Would participants recommend it to others?

Is there room for improvement with facilitation, materials, and the learning environment?

28



Levels of Evaluation Key Questions AnsweredLevel 2: Learning and Confidence

Do participants know what they are supposed to do with what they learned?

Do participants know how to apply what they learned?

Are participants confident to apply what they learned?

Did participants gain new knowledge, change their attitude, increase awareness?

29



Levels of Evaluation Key Questions AnsweredLevel 3: Application and Implementation

How effectively are participants applying what they learned?

How frequently are they applying what they learned?

If they are applying what they learned, what is supporting them?

If they are not applying what they learned, why not?

30



Levels of Evaluation Key Questions Answered

Level 4: Business Impact So what?To what extent does participant application of what they learned improve the measures the program was intended to improve?How did the program impact output, quality, cost, time, customer satisfaction, employee satisfaction, work habits?What were the consequences of participants’ application of knowledge and skills acquired during the program, process, intervention, change?How do we know it was the program that improved these measures?

31



Levels of Evaluation Key Questions AnsweredLevel 5: ROI Do the monetary benefits of

the improvement in business impact measures outweigh the cost of the program?

32


Impact Questions for Follow-Up Evaluation

• How did you apply what you learned in this program?• What was the impact of these efforts in your work unit?• What measures were changed in your work unit?• How much did they change?• What is the monetary value of the changes?• How did you arrive at this value?• What percent of this improvement was actually caused by the

training?• What level of confidence do you place on this value? (Expressed as a

percentage)


Techniques to Isolate the Effects of the Program

• Use of a control group arrangement• Trend line analysis of performance data• Use of forecasting methods• Participant’s estimate of the program’s impact (percent)• Supervisor’s estimate of the program’s impact (percent)• Management’s estimate of the program’s impact (percent)• Use of experts/previous studies• Calculating/Estimating the impact of other factors• Use of customer input


Common Intangibles

• Brand Awareness• Job satisfaction• Organizational

commitment• Work climate• Employee complaints• Employee awards• Employee stress

reduction• Employee innovation• Employee networking• Employee collaboration• Employee partnering

• Customer satisfaction /dissatisfaction

• Reputation• Image• Customer complaints• Customer response

time• Customer loyalty• Teamwork• Cooperation• Conflict• Decisiveness• Communication


Sources for Intangible Benefits

• Identified during needs assessments

• Identification during early planning (with no intention to convert to monetary value)

• Clients identify intangibles

• Participants identify intangibles

• Managers identify intangibles

• Customers identify intangibles

• Intangibles are often identified when attempts at conversion are aborted


Issues with Intangibles

• May be the most important data set• Are not converted to money by definition• Are usually not subjected to “isolating”• Must be systematically addressed• Must be reported “credibly”


Twelve Guiding Principles Applied During the ROI

Process1. When conducting a higher-level evaluation, collect data at lower levels2. When planning a higher-level evaluation, the previous level of

evaluation is not required to be comprehensive3. When collecting and analyzing data, use only the most credible sources4. When analyzing data, select the most conservative alternative for

calculations5. Use at least one method is to isolate the effects of a project6. If no improvement data are available for a population or from a specific

source, assume that little or no improvement has occurred


Guiding Principles Applied During the ROI Process

7. Adjust estimates of improvement for potential errors of estimation.8. Avoid use of extreme data items and unsupported claims when

calculating ROI.9. Use only the first year of annual benefits in ROI analysis of short-

term solutions.10. Fully load all costs of a solution, project, or program when analyzing

ROI.11. Intangible measures are defined as measures that are purposely

not converted to monetary values.12. Communicate the results of ROI Methodology to all key

stakeholders.

Spiritual

2009 Roi Trainer 1 Rev1