Embed Size (px)
DESCRIPTION
April 19, 2012 ASTD-SHRM Joint Meeting, Jackson, MS. John Kmiec, PhD, CRP
Citation preview
Putting Program Evaluation to Work for You
A Learning Intervention for Work Engagement
PolyWrighton, USA
John Kmiec, PhD, CRP The University of Southern Mississippi
[email protected] 228.365.2559
Overview
Why do we need evaluation? What are your biggest problems with evaluation? How can evaluation planning help? What are the components of effective evaluation? Can you give us an example? (PolyWrighton case study) Where can we find additional resources?
The purpose of this discussion is to explore the elements of effective program evaluation
Obtain feedback from stakeholders Verify that the methods specified were used Determine whether the solution worked Establish if sufficient impact was made Help make adjustments during the program Maintain some control over the project Account for resources and results Help plan future actions
Why do we need evaluation?
In today’s globally competitive changing market and constant technological advancement, training is a given.
Doing training well – getting results from learning investments – is a must, not a choice. Robert Brinkerhoff
What is your biggest problem with evaluation? Too many theories and models Models are too complex Lack of understanding of evaluation Evaluation is considered a post-program activity Lack of support from key stakeholders Improper use of evaluation data Other
Shows in advance how you will know when you have accomplished your objectives Reassures stakeholders that their financial commitment is being well spent Identifies what is going well and what is not
How can evaluation planning help?
How can evaluation planning help?
Who… Will be doing the evaluation?
What… Is the criteria for evaluation?Is the process for data analysis?Test instruments/questionnaires will be used?Is and validity/reliability of these instruments?
When… Will the evaluation be conducted?Will evaluation reports be produced?
Where… Will adjustments be made if needed?
How… Was the evaluator chosen?Will you determine if objectives were met and specified methods were used?Will the data be gathered? Analyzed? Disseminated?
It h
elps
you
defi
ne th
e…
What are the components of effective evaluation? An evaluation framework A data collection plan A data analysis plan
What are the components of effective evaluation? An evaluation framework
Illustrates program objectives Shows the chain of impact Provides Structure Aids communication Facilitates Planning
0. Inputs & Indicators: Measures inputs, including indicators of volume and scope
1. Reaction & Planned Action: Measures satisfaction with the program and planned actions
2. Learning: Measures changes in participants’ knowledge, skills, abilities and attitudes
3. Application: Measures changes in behavior4. Impact: Measures changes in critical outcomes5. ROI: Compares program benefits to costs
Phillips ROI Methodology®
The Levels of Evaluation Framework
Starts with a proper needs assessmentwhere
Needs drive program objectivesand
Program objectives drive measurement and evaluation, continuous improvement and results!
ProgramPhillips ROI
Methodology®
Evaluation Framework
What are the components of effective evaluation? A data collection plan shows
Broad program objectives (Why do I ask?) Measures/Data (What do I ask?) Data Collection Methods/Instruments (How do I ask?) Data Sources (Who do I ask?) Timing (When do I ask?) Responsibilities (Who asks?)
What are the components of effective evaluation? A data analysis plan shows
Methods for isolating effects of the program Ways for converting results to dollar values Program cost categories Intangible benefits Communication targets
Can You Give Us an Example?
A Learning Intervention for Work Engagement
PolyWrighton, USA
Case Study
Background• PolyWrighton manufactures high quality, lightweight plastics used
to package food, beverage and personal care products• Complex, massive, hazardous machinery used to process tons of
toxic, flammable chemicals under high heat and pressure • Rigorous hygienic, chemical and environmental safety standards
and specifications leave very little room for error• Raw materials are very expensive and frequently in short supply,
making reduction of product waste and rework imperative
Background• Constant monitoring and testing, state-of-the-art technology and a
highly skilled workforce are critical to quality and safety• Bankruptcy led to purchase by an out-of-state investment group• All but one site was shut down to reduce costs – all operations
consolidated into the single most modern facility in Mississippi• One year mandate to improve business performance imposed
Opportunities• Product Waste and Rework: $272,850/month ($3.27M/year)
– Waste costs $245K for every 1% OR $600/minute generated– Rework costs $35K for every 1% of rework per total product produced
• Stressed employees under the gun to cut costs and improve quality and efficiency – perceived threat of losing their high paying jobs
• Perceptions of employee satisfaction and work engagement were, at best, mediocre – improvement needed to move forward
• Few immediate managers possessed sufficient training, development or experience in leadership, coaching and supervision
Opportunities
Companies with world-class engagement have 3.9 times the Earnings per Share growth rate compared with organizations with lower engagement in
their same industry. (Source: http://www.gallup.com)
Performance Ability + Motivation + Opportunity
Training, education and experience
Confidence (I can) and commitment (I will)
What the organization provides to employees
=
• Immediate managers play a pivotal role in creating motivational work environments that positively impact employee satisfaction
• Employee satisfaction is related to work engagement; psychological state characterized by absorption, dedication, vigor
• Increased work engagement is associated with improved job performance, business impact and profitability
InterventionPurpose: To prepare immediate managers to more effectively create
and sustain motivational work environments that positively impact work engagement and organizational performance.
InterventionPurpose: To prepare immediate managers to more effectively create and sustain motivational work environments that positively impact work engagement and organizational performance.
5-Skills of Savvy Managers (Flagello & Dugas, 2009): 1. Self-managing refers to the practice of continuous self-improvement
through purposeful self-observation and -monitoring, self-assessment, goal-setting, and conscious action
2. Reflecting is the simple practice of quietly contemplating, thinking, and/or observing…without judgment
3. Acting consciously is the practice of deliberately and intentionally selecting from feasible options decisions that are better aligned with desired outcomes
4. Collaborating is working with the full involvement of people in order to better align efforts, add value and generate results
5. Evolving is a personal, life-long commitment to the deliberate and continuous pursuit of learning, development and professional growth
InterventionPurpose: To prepare immediate managers to more effectively
create and sustain motivational work environments that positively impact work engagement and organizational performance.• Impact Measures:
• Work Engagement: Test/Control Group Arrangement – 32 line employees from Production unit (test)– 31 line employees from Maintenance unit (control)– Utrecht Work Engagement Scale (Schaufeli & Bakker, 2003)
• Production business unit controllable Waste and Rework – Waste costs $245K for every 1% of waste per total product
produced OR $600/minute generated– Rework costs $35K for every 1% of rework per total
product produced
0. Inputs & Indicators: Measures inputs, including indicators of volume and scope
1. Reaction & Planned Action: Measures satisfaction with the program and planned actions
2. Learning: Measures changes in participants’ knowledge, skills, abilities and attitudes
3. Application: Measures changes in behavior4. Impact: Measures changes in critical outcomes5. ROI: Compares program benefits to costs
Phillips ROI Methodology®
The Levels of Evaluation Framework
Evaluation
NeedsProgram Needs• Product Waste ($245K/1%) and Rework ($35K/1%) costs
$272,850/month ($3.27M/year)• Stressed work environment barrier to Work Engagement• Immediate managers unskilled and inexperiencedEvaluation Needs• Effectiveness of learning intervention?• Increase Work Engagement?• Reduce Product Waste and Rework?• Rollout Decision?
Program ObjectivesReaction (1)• Relevance, importance and intent to use on the jobLearning (2)• Successful application of 5 self-coaching skills• Can create and sustain motivational environmentOn-the-Job Application (3)• Effectively, continuously apply 5 self-coaching skills• Create and sustain motivational work environmentBusiness Impact (4)• Reduce % Product Waste and Rework • Increase Work EngagementROI (5): 15% Target ROI
Data Collection PlanReaction (1)• Reaction questionnaires taken at the end of each of seven sessionsLearning (2)• Pre- and post program skill self-assessment profiles• Facilitator assessment of participant discussions, responses to
questions, and completed assignments during the program• Utrecht Work Engagement Scale at program start, middle and endOn-the-Job Application (3)• Skill self-assessment profiles (3-months)• Utrecht Work Engagement Scale (3-months)Business Impact (4)• Percent Product Waste and Rework• Utrecht Work Engagement Scale (6-months)
Data/ROI Analysis PlanData Items Isolation of Effects Monetary
ConversionPercent Waste Trend analysis
Participant and management estimates
Percentage of waste times $245,000
Percent Rework Trend analysis
Participant and management estimates
Percentage of rework times $35,000
Work Engagement Control Group Not converted
Results
Reaction (1)• Content relevance (4.16/5.0, 4.0)• Importance (4.07/5.0, 4.0) • Intent to use (4.23/5.0)Learning (2)• Pre program skill self-assessment profile mean (105.0/125)• Post program skill self-assessment profile mean (107.6/125)• Utrecht Work Engagement Scale: Test group higher than control
group by the end of the 90-day learning interventionOn-the-Job Application (3)• Skill self-assessment profile mean (112.6/125) at 3-months• Utrecht Work Engagement Scale: Test group higher than control
ResultsBusiness Impact (4)
$367,500 Projected Trend - $119,350 Post Average = $248,150 per month $248,150 X 12 = $2,977,800 per year less cost for Waste and Rework
0.50 Impact Estimate X 0.85 Confidence X $2,977,800 = $1,265,565
ROI (5)
Net Program Benefit/Program Costs X 100
($1,265,565 - $253,761)/$253,761 = 3.99 X 100 = 399%
Intangibles• Increased Work Engagement• Employee Satisfaction• Improved Teamwork and Communications • Better Decision-making
ResourcesMeasuring ROI in Learning & Development: Case Studies
from Global Organizations. Editors: Patricia Pulliam Phillips, Ph.D. and Jack J. Phillips, Ph.D. ASTD Press, Alexandria, VA , 2012.
The Savvy Manager: 5 skills that Drive Optimal Performance. Jane R. Flagello, Ph.D. and Sandra B. Dugas, Ph.D. ASTD Press, Alexandria, VA, 2009.
The Value of Learning: How Organizations Capture Value and ROI and Translate them into Support, Improvement, and Funds. Patricia Pulliam Phillips, Ph.D., Jack J. Phillips, Ph.D. Pfeiffer, San Francisco, CA, 2007.
ResourcesThe ROI Institute is the leading global resource on research, training,
and networking for practitioners of the Phillips ROI Methodology™Web: http://roiinstitute.net/ Email: [email protected]: 205-678-8101 Fax: 205-678-8102
The Jack and Patti Phillips Workplace Learning and Performance Institute, located on The University of Southern Mississippi Gulf Park Campus, serves as a research and outreach center for best practices in workplace learning and performance. Web: http://www.usm.edu/gulfcoast/ Email: [email protected]: 228-214-3517
ResourcesThe University of Southern Mississippi offers the following programs
on the USM Gulf Park Campus in executive format, designed for working professionals:Professional Development Certificate in Training and
DevelopmentMaster of Science in Workforce Training and DevelopmentDoctor of Philosophy in Human Capital Development
Web: http://www.usm.edu/gulfcoast/ Email: [email protected]: 228-214-3517
Conclusion
Why do we need evaluation? What are your biggest problems with evaluation? How can evaluation planning help? What are the components of effective evaluation? Can you give us an example? (PolyWrighton case study) Where can we find additional resources?
The purpose of this discussion was to explore the elements of effective program evaluation
Putting Program Evaluation to Work for You
A Learning Intervention for Work Engagement
PolyWrighton, USA
John Kmiec, PhD, CRP The University of Southern Mississippi
[email protected] 228.365.2559
