The why behind the what:
A process evaluation of Healthy@Work
Dr Theresa DohertyDr Fiona Cocker
Seminar Aims
• Introduce a methodology for process evaluation
• Apply this to the H@W program• Identify key evaluation questions• Demonstrate what existing program data can
answer• Identify what other data we need • Explain how we are going to collect it
Defining Evaluation
• The systematic acquisition and assessment of information to provide useful feedback about some object
• ‘The process of determining the worth, merit, or value of things …or the result of that process.’ (Scriven 1991)
• ‘The systematic collection of information about the activities, characteristics, and/or outcomes of programs to make judgements about the program, improve program effectiveness, and/or inform decisions about future programming.’ (Patton 1997)
Purpose of Evaluation
‘Evaluation determines the merit, worth, or value of things. The evaluation process identifies relevant values or standards that apply to what is being evaluated, performs empirical investigation (often) using techniques from the social sciences, and then integrates conclusions with the standards into an overall evaluation or set of evaluations.’ (Scriven, 1991)
Evaluation vs. Research
• The main difference between research and evaluation is that research is conducted with the intent to generalize the findings from a sample to a larger population. (Priest 2001)
• Evaluation focuses on an internal situation with no intent to generalize the results to other settings and situations.
• Research generalizes, evaluation particularizes.
Same or Different ?
‘Evaluation and research are closely related and should be synergistic but they serve different
purposes.’ (Fain)
Evaluation Research Particularises GeneralisesSeeks to improve Seeks to proveInforms decision making
Draws conclusions
Asks ‘so what?’ Asks ‘what’s so?’How well it works How it worksWhat is valuable What ‘is’ (Matheison)
Meaningful Evaluation
The more interesting endeavour for evaluation:– how to do rigorous and robust
evaluation
We need a plan so we can be confident:– Conclusions are sound; – Recommendations will improve program
delivery and outcomes
Evaluation Plan Checklist
1. Program clarification2. Purpose of evaluation3. Stakeholder assessment4. Program logic underpinning this program5. Scope of evaluation6. Data collection7. Data analysis8. Timeline for evaluation9. Report and dissemination 10. Budget requirements
Program Logic Approach
• Clarify the purpose of a project– what change do you want to bring
about?• Plan how to bring about this change• Allow yourself to explore and test the
assumptions underlying your approach• Test how well your approach is working
and what, if anything, needs changing• Find ways of demonstrating your approach
is working
The Logic Model
What are our results?
INPUTS OUTPUTS OUTCOMESProgram
investments
Short-term
What we
invest
Activities, outputs
What we do
or produc
e
Target
Who we
want to
reach
Medium-term
Long-term
Program Logic Approach
• Provides a program description that guides our evaluation process
• Helps match the evaluation to the program
• Helps identify what and when to measure– outputs and outcomes
• Helps identify key data needs• Makes evaluation an integral part of
program planning and implementation
University of Wisconsin-Extension, Program Development and Evaluation
Logic Model & Evaluation
Types of Evaluation
Process evaluation
Good ideas do not always yield good results
• Process evaluation looks at how a program was implemented
• It assesses the extent to which planned activities were carried out
• Ideally this happens AS a a program is being implemented NOT retrospectively
• It alerts us to changing conditions that may impact on the program
• It allows us to make adjustments to implementation to keep on track OR
• Rethink whether the program design is the right one
pH@W Process Evaluation Project
• Process evaluation was not in the original pH@W project plan
• No specific funding allocated BUT
• Investigators decided it was an important part of understanding Healthy@Work
Purpose of the Evaluation
• What do we want to know?• How was Healthy@Work
implemented across the Tasmanian State Service?– What happened?– What happened, in what
circumstances which led to particular outcomes?
– In light of these findings, would we implement a workplace H&WB program differently?
The Issue of Scope
What is achievable given time and resource limitations?• How was the H@W program conceptualised
and developed?• To what extent did TSS Agencies engaged with
and implement H@W as designed?• What are the implications for sustainable
health promoting workplaces?
H@W Program Logic
H@W Process Evaluation Program Logic - Mark 1
INPUTS OUTPUTS TARGET ST OUTCOMES H@W Funding H@W Project advisory group H@W program Manager H@W Project officer Partnerships with agencies Partnerships with service providers Governance structures Partnering H@W (DPAC, DHHs, UTAS, MRI)
DPAC Ministerial directive Healthy@work website Healthy@work online
survey H&WB Grant program ? marketing and
promotion to agencies AGENCIES X12 Response to MD Develop H&WB governance structures
Develop H&WB implementation plans
Promote H@W to employees
Identify potential H&WB strategies/interventions
H@W coordinators employed
H@W grant
submissions
Heads of Agencies TSS Employees Managers/employers Employees Service providers
MD implemented across TSS H of A engage with H@W Employees engage with H@W Agency H&WB profiles Awareness and knowledge of H&WB Access H&WB programs /information Employees engage with H&WB programs Contracts/agreements with H&WB service providers H@W grants
H@W Process Evaluation Program Logic -Mark 3
INPUTS OUTPUTS TARGET ST OUTCOMES MT OUTCOMES LT OUTCOMES
Ministerial direction H@W Funding H@W Comms and marketing Strategy H@W Project advisory group H@W program staff H@W website H@W online survey H@W Grant program Partnerships with agencies Partnerships with service providers Governance structures Partnering H@W(DPAC, DHHs, UTAS, MRI)
Agency response to MD • Use the H@W
guidelines
• Develop H&WB governance structures
• Develop H&WB implementation plans
• Allocate resources for H@W coordinators
• Promote H@W to employees
Used t he H@W website
Participate din the online survey
Identify H&WB interventions
Apply for H@W grants
STATE SERVICE AGENCIES HEAD of
AGENCY MANAGERS SS EMPLOYEES
SERVICE PROVIDERS
H@W is implemented across TSS H of A engage with H@W Employees engage with H@W Awareness and knowledge of agency H&WB profiles Awareness and knowledge of H&WB risks Access to H&WB programs /information Employee engagement with H&WB programs TSS Employees are motivated to change health risk behaviours Contracts/agreements with H&WB service providers in place H@W grants secured
TSS workplaces have H@WB policies TSS Workplace have health promoting infrastructure /programs in place TSS Employees have health promoting behaviours
Sustainable H&WB programs Health promoting TSS workplaces Healthy TSS workforce
FOCUS of PROCESS EVALUATION
Process Evaluation Activities
1. Review and analyse H@W files – the ‘formal’ story
2. Interview key informants – ‘informal’ story
3. Synthesise these data – deeper understanding of H@W
4. Report the findings – what are the implications of this ‘new’ knowledge?
5. Disseminate the findings – who are the stakeholders?
What do we want to know – Example outputs
Outputs cont.(from your logic plan)
What you produce/what you do
What do we want to know?
Performance indicatorWhat can you count/how will
you know?
How will we
find out?How will we collect the
dataMD implemented To what extents did
agencies follow the guidelines?
# who conducted a needs assessment# who set up a committee# who engaged management support# who developed a H&WB plan
Audit data
H&WB activities in place
What sort of H&WB programs did agencies implement?
#r /type of programs implemented
Audit data
Agency H&WB coordinators (HWBC) allocated
• Did agency create a new position?
• Were H&WB responsibilities added to an existing position?
• Did agency allocate specific resources to this position?
• Where in the org structure does this position fit?
# H&WBC positions created # H&WBC positions filled # agencies with a H&WBC
Agency recordsKey informants
What do we want to know - Example outcomes
ST Outcomes (logic plan)
What you achieve
What do we want to know?
Performance indicatorWhat can you count/how will you
know?
How will we find out?
How will we collect the
dataMD is implemented across TSS
How much progress had agencies made toward implementing a H&WB program in place as prescribed by the MD Did agencies experience barriers to implementation?
# agencies with an approved H@WB plan # of agencies who identified barriers
Agency recordsKey informants
H of A engage with H@W
How did H of A support the H@W initiative?
Funding allocated to H&WB activitiesChanges to organisational structure to accommodate H&WBH of A membership of H&WB governance structures
Key informantsMinutes of meetings
Employees engage with H&WB programs
Did H&WB program meet employees’ needs?Did any employee groups not participate?Did employees experience financial barriers to participation
Participation rates in programsEmployee satisfaction with H&WB activitiesEmployee participation on H&WB committees/ working groupsEmployee identified barriers to participation
Key informantsAgency records
Existing Data Sources
• Annual audit data• Healthy@Work agency reports • Grants program – applications, progress/final
reports• H@W project review and closure report
What they reveal• Baseline H&WB capacity • The sort of activities implemented• Where the grant money was allocated
How will we find out?
Data sources available• Annual audit data• Healthy@Work agency reports • Grants program – applications, progress/final
reports• H@W project review and closure report
What they reveal• Baseline H&WB capacity • The sort of activities implemented• Where the grant money was allocated
Performance Indicator:Needs Assessment
Audit Year
Top Three Priorities Bottom Three Priorities
2010 NutritionSedentary behaviourStress/Psych distress
SmokingHydrationAlcohol
2011 NutritionStress/Psych distressPhysical activity
SmokingHydrationAlcohol
2012 NutritionPhysical activityStress/Psych distress
AlcoholOtherSmoking
• All agencies completed a needs assessment (2010-2012)• Were the organisation’s H&WB issues/needs identified?
Performance Indicator:Needs Assessment
Audit Year
Top Three Motivators
2008/09 1. Increase health and wellbeing of staff2. Increase engagement / morale of staff3. Recruitment / retention strategy
2010 1. Increase health and wellbeing of staff2. Increase engagement / morale of staff3. Ministerial direction
2011 1. Increase health and wellbeing of staff2. Increase engagement / morale of staff3. Increase productivity
2012 1. Increase health and wellbeing of staff2. Injury prevention3. Staff demand
Performance Indicator: Needs Assessment
Audit Year
Top Three Barriers
2008/09 1. Lack of human resources 2. Difficulty agreeing on the program
content/structure 3. Never thought about it
2010 1. Unable to justify costs 2. Insufficient financial resources3. Employee time constraints
2011 1. Employee time constraints 2. Insufficient financial resources3. Lack of human resources
2012 1. Employee time constraints 2. Lack of employee interest 3. Unable to justify costs against other priorities
Performance Indicator:Committee/staffing support?
Resource Allocation 2008/09(14
Agencies, Authorities)
2010(14
Agencies, Authorities)
2011(14
Agencies, Authorities)
2012(19
Agencies, Authorities
)Position/s - program part of normal role 5 9 12 13No allocated position - Volunteer(s) given time to work on project
1 1 1 1
No allocated position. Volunteer/s works on project and normal duties
2 0 1 1
A steering committee/working group given time to work on project
1 5 11 14
A steering committee/working group works on project and normal duties
6 0 1 0
Examples of responsible positions:2008 – Senior HR officer (1/2 hr/week); Senior HR consultant (2 hr/week)2010 – H@W Policy & Program Officer (23 hr/week)2011 – H&WB co-ordinator (up to 20hr/week)2012 – OH&S Manager (7hr/week); HR co-ordinator (4hr/week)
Performance Indicator:Engaged Management?
Examples of Senior Management Engagement
2010(14 Agencies, Authorities)
• Executive Group reviews and approves proposals for H&WB initiatives
• Approves allocation of resources for Agency • Approved & participate in the program• Are very supportive of health & wellness in the workplace
2011(14 Agencies, Authorities)
• Fully endorse & participate in program• Allow the program to be conducted in work time• Encourage all staff to get involved• Committed to having a budget for H&WB initiatives
2012(19 Agencies, Authorities)
• Provision of information, resource allocation and reporting on progress
• Developed the framework• Oversee program content• Have a key role in supporting staff to actively engage in
H&WB activities• Funding support for various programs
Performance indicator:H&WB plan?
H&WB Plan
2010(14
Agencies, Authorities
)
2011(14
Agencies, Authorities
)
2012(19
Agencies, Authorities
)
No 8 3 4Yes 6 11 15
Agency Evaluation of H&WB Program
Audit Year
Most commonly cited method of evaluation
2010 1. Number of participants taking part2. Employee satisfaction with program3. Number of initiatives/activities undertaken
2011 1. Number of participants taking part2. Employee satisfaction with program3. Employee satisfaction with organisation
2012 1. Number of participants taking part2. Employee satisfaction with program3. Number of initiatives/activities undertaken
Still to come
Further examination of:• Annual audit data – Analysis by
agency
Examination of:• Healthy@Work agency reports • Grants program – applications,
reports• H@W project review and closure
report
H@W project review and closure report
• Project performance: – project vision, measured outcomes, outputs, budget
• Lessons learnt: – what worked well, what could be improved, challenges – recommendations to guide future TSS H&WB activities
• Highlights and innovations:– Organisation change and development– Agencies with a H&WB program (3 in 2010, 14 in
2012)– All Agencies report H&WB activities (14 in 2009, 21 in
2010)– Working in partnerships: DHHS, Workcover, UTAS,
Menzies Research Institute Tasmania (pH@W), Healthy Workers (NPAPH)
H@W grants program
• Number of grant rounds
• Who applied for money, in which round?
• What type of project?
• Who was successful?
• How much money did they get?
• Progress/final reports provided
• Any sustainable outcomes?
Purposive Data Collection
• What happened in the implementation of H@W is only part of the story…
• The context in which a program operates can be important data for evaluation
• Semi structured interviews with key informants is one method for collecting this data
What we would like to knowexamples
1. How did you (your agency) gain management support?2. What sort of a coordination mechanism was used to
manage H@W?3. How much did employees engage with H@W? 4. How useful/relevant was the information on the H@W
website?5. How did your organisation review and update the
H&WB Program?6. How did you promote , market, communicate the
H&WB Program to employees?7. Has your agency continued to resource workplace
H&WB?8. Base on the H@W experience what factors do you
believe influence employee participation in workplace H&WB programs?
Data Analysis
Analysis• Qualitative and quantitative methods and descriptive statistics
where appropriate
Interpretation• Interpretavist methods to analyse content of interviews and
documents• What understandings or conclusions can be gained?• What does it mean for the program/project?• How does this inform the evaluation purpose?• Reveal the complexities of implementing a large scale, multi
agency workplace based health promotion initiative • Identify enablers and barriers to successful implementation and
implications for transferability to non state sector workplaces• Inform program design and implementation of future WHP
strategies
Who is this evaluation for ?
Target Audience Dissemination Format
Responsible Person/Author
pH@W partners Written report Evaluators/Process Evaluation Working Group (PEWG)
Key stakeholders Summary reportPresentation
Evaluators/PEWG
Stakeholders Presentation Evaluators/PEWG
Academic peers (UTAS, MRIT)
Peer-reviewed journal articles
Individual authors
Primary audience: pH@W partnersOther interested parties: SSMO, Tasmanian State Service Agencies, TSS employees, key informants
Still to come
• Ethics application prepared for key informant interviews
• Interviews planned for June/July 2013
Importance of Process and Implementation Evaluation
• Uses scientific rigor to understand the etiology of gaps between expected results and observed outcomes– Information can be gathered during the implementation of a
program or trial • Explores the relationship between process and
outcomes• Generates evidence to guide future implementation
and improve program outcomes JOEM, Volume 55, Number 5, May 2013