The why behind the what:

A process evaluation of Healthy@Work

Dr Theresa DohertyDr Fiona Cocker

Seminar Aims

• Introduce a methodology for process evaluation

• Apply this to the H@W program• Identify key evaluation questions• Demonstrate what existing program data can

answer• Identify what other data we need • Explain how we are going to collect it

Defining Evaluation

• The systematic acquisition and assessment of information to provide useful feedback about some object

• ‘The process of determining the worth, merit, or value of things …or the result of that process.’ (Scriven 1991)

• ‘The systematic collection of information about the activities, characteristics, and/or outcomes of programs to make judgements about the program, improve program effectiveness, and/or inform decisions about future programming.’ (Patton 1997)

Purpose of Evaluation

‘Evaluation determines the merit, worth, or value of things. The evaluation process identifies relevant values or standards that apply to what is being evaluated, performs empirical investigation (often) using techniques from the social sciences, and then integrates conclusions with the standards into an overall evaluation or set of evaluations.’ (Scriven, 1991)

Evaluation vs. Research

• The main difference between research and evaluation is that research is conducted with the intent to generalize the findings from a sample to a larger population. (Priest 2001)  

• Evaluation focuses on an internal situation with no intent to generalize the results to other settings and situations. 

• Research generalizes, evaluation particularizes.

Same or Different ?

‘Evaluation and research are closely related and should be synergistic but they serve different

purposes.’ (Fain)

Evaluation Research Particularises GeneralisesSeeks to improve Seeks to proveInforms decision making

Draws conclusions

Asks ‘so what?’ Asks ‘what’s so?’How well it works How it worksWhat is valuable What ‘is’ (Matheison)

Meaningful Evaluation

The more interesting endeavour for evaluation:– how to do rigorous and robust

evaluation

We need a plan so we can be confident:– Conclusions are sound; – Recommendations will improve program

delivery and outcomes

Evaluation Plan Checklist

1. Program clarification2. Purpose of evaluation3. Stakeholder assessment4. Program logic underpinning this program5. Scope of evaluation6. Data collection7. Data analysis8. Timeline for evaluation9. Report and dissemination 10. Budget requirements

Program Logic Approach

• Clarify the purpose of a project– what change do you want to bring

about?• Plan how to bring about this change• Allow yourself to explore and test the

assumptions underlying your approach• Test how well your approach is working

and what, if anything, needs changing• Find ways of demonstrating your approach

is working

The Logic Model

What are our results?

INPUTS OUTPUTS OUTCOMESProgram

investments

Short-term

What we

invest

Activities, outputs

What we do

or produc

e

Target

Who we

want to

reach

Medium-term

Long-term

Program Logic Approach

• Provides a program description that guides our evaluation process

• Helps match the evaluation to the program

• Helps identify what and when to measure– outputs and outcomes

• Helps identify key data needs• Makes evaluation an integral part of

program planning and implementation

University of Wisconsin-Extension, Program Development and Evaluation

Logic Model & Evaluation

Types of Evaluation

Process evaluation

Good ideas do not always yield good results

• Process evaluation looks at how a program was implemented

• It assesses the extent to which planned activities were carried out

• Ideally this happens AS a a program is being implemented NOT retrospectively

• It alerts us to changing conditions that may impact on the program

• It allows us to make adjustments to implementation to keep on track OR

• Rethink whether the program design is the right one

pH@W Process Evaluation Project

• Process evaluation was not in the original pH@W project plan

• No specific funding allocated BUT

• Investigators decided it was an important part of understanding Healthy@Work

Purpose of the Evaluation

• What do we want to know?• How was Healthy@Work

implemented across the Tasmanian State Service?– What happened?– What happened, in what

circumstances which led to particular outcomes?

– In light of these findings, would we implement a workplace H&WB program differently?

The Issue of Scope

What is achievable given time and resource limitations?• How was the H@W program conceptualised

and developed?• To what extent did TSS Agencies engaged with

and implement H@W as designed?• What are the implications for sustainable

health promoting workplaces?

H@W Program Logic

H@W Process Evaluation Program Logic - Mark 1

INPUTS OUTPUTS TARGET ST OUTCOMES H@W Funding H@W Project advisory group H@W program Manager H@W Project officer Partnerships with agencies Partnerships with service providers Governance structures Partnering H@W (DPAC, DHHs, UTAS, MRI)

DPAC Ministerial directive Healthy@work website Healthy@work online

survey H&WB Grant program ? marketing and

promotion to agencies AGENCIES X12 Response to MD Develop H&WB governance structures

Develop H&WB implementation plans

Promote H@W to employees

Identify potential H&WB strategies/interventions

H@W coordinators employed

H@W grant

submissions

Heads of Agencies TSS Employees Managers/employers Employees Service providers

MD implemented across TSS H of A engage with H@W Employees engage with H@W Agency H&WB profiles Awareness and knowledge of H&WB Access H&WB programs /information Employees engage with H&WB programs Contracts/agreements with H&WB service providers H@W grants

H@W Process Evaluation Program Logic -Mark 3

INPUTS OUTPUTS TARGET ST OUTCOMES MT OUTCOMES LT OUTCOMES

Ministerial direction H@W Funding H@W Comms and marketing Strategy H@W Project advisory group H@W program staff H@W website H@W online survey H@W Grant program Partnerships with agencies Partnerships with service providers Governance structures Partnering H@W(DPAC, DHHs, UTAS, MRI) 

  Agency response to MD • Use the H@W

guidelines

• Develop H&WB governance structures

• Develop H&WB implementation plans

• Allocate resources for H@W coordinators

• Promote H@W to employees

Used t he H@W website

Participate din the online survey

Identify H&WB interventions

Apply for H@W grants

   

 STATE SERVICE AGENCIES   HEAD of

AGENCY MANAGERS SS EMPLOYEES

SERVICE PROVIDERS

  H@W is implemented across TSS H of A engage with H@W Employees engage with H@W Awareness and knowledge of agency H&WB profiles Awareness and knowledge of H&WB risks Access to H&WB programs /information Employee engagement with H&WB programs TSS Employees are motivated to change health risk behaviours Contracts/agreements with H&WB service providers in place H@W grants secured

       TSS workplaces have H@WB policies     TSS Workplace have health promoting infrastructure /programs in place    TSS Employees have health promoting behaviours 

        

Sustainable H&WB programs   Health promoting TSS workplaces     Healthy TSS workforce

  FOCUS of PROCESS EVALUATION    

Process Evaluation Activities

1. Review and analyse H@W files – the ‘formal’ story

2. Interview key informants – ‘informal’ story

3. Synthesise these data – deeper understanding of H@W

4. Report the findings – what are the implications of this ‘new’ knowledge?

5. Disseminate the findings – who are the stakeholders?

What do we want to know – Example outputs

Outputs cont.(from your logic plan)

What you produce/what you do

What do we want to know?

 

Performance indicatorWhat can you count/how will

you know?

How will we

find out?How will we collect the

dataMD implemented To what extents did

agencies follow the guidelines?

# who conducted a needs assessment# who set up a committee# who engaged management support# who developed a H&WB plan

Audit data

H&WB activities in place

What sort of H&WB programs did agencies implement?

#r /type of programs implemented   

Audit data

Agency H&WB coordinators (HWBC) allocated

• Did agency create a new position?

• Were H&WB responsibilities added to an existing position?

• Did agency allocate specific resources to this position?

• Where in the org structure does this position fit?

# H&WBC positions created # H&WBC positions filled # agencies with a H&WBC

Agency recordsKey informants

What do we want to know - Example outcomes

ST Outcomes (logic plan)

What you achieve

What do we want to know?

 

Performance indicatorWhat can you count/how will you

know?

How will we find out?

How will we collect the

dataMD is implemented across TSS 

How much progress had agencies made toward implementing a H&WB program in place as prescribed by the MD Did agencies experience barriers to implementation? 

# agencies with an approved H@WB plan  # of agencies who identified barriers     

Agency recordsKey informants 

H of A engage with H@W 

How did H of A support the H@W initiative? 

Funding allocated to H&WB activitiesChanges to organisational structure to accommodate H&WBH of A membership of H&WB governance structures  

Key informantsMinutes of meetings

Employees engage with H&WB programs 

Did H&WB program meet employees’ needs?Did any employee groups not participate?Did employees experience financial barriers to participation

Participation rates in programsEmployee satisfaction with H&WB activitiesEmployee participation on H&WB committees/ working groupsEmployee identified barriers to participation  

Key informantsAgency records

Existing Data Sources

• Annual audit data• Healthy@Work agency reports • Grants program – applications, progress/final

reports• H@W project review and closure report

What they reveal• Baseline H&WB capacity • The sort of activities implemented• Where the grant money was allocated

How will we find out?

Data sources available• Annual audit data• Healthy@Work agency reports • Grants program – applications, progress/final

reports• H@W project review and closure report

What they reveal• Baseline H&WB capacity • The sort of activities implemented• Where the grant money was allocated

Performance Indicator:Needs Assessment

Audit Year

Top Three Priorities Bottom Three Priorities

2010 NutritionSedentary behaviourStress/Psych distress

SmokingHydrationAlcohol

2011 NutritionStress/Psych distressPhysical activity

SmokingHydrationAlcohol

2012 NutritionPhysical activityStress/Psych distress

AlcoholOtherSmoking

• All agencies completed a needs assessment (2010-2012)• Were the organisation’s H&WB issues/needs identified?

Performance Indicator:Needs Assessment

Audit Year

Top Three Motivators

2008/09 1. Increase health and wellbeing of staff2. Increase engagement / morale of staff3. Recruitment / retention strategy

2010 1. Increase health and wellbeing of staff2. Increase engagement / morale of staff3. Ministerial direction

2011 1. Increase health and wellbeing of staff2. Increase engagement / morale of staff3. Increase productivity

2012 1. Increase health and wellbeing of staff2. Injury prevention3. Staff demand

Performance Indicator: Needs Assessment

Audit Year

Top Three Barriers

2008/09 1. Lack of human resources 2. Difficulty agreeing on the program

content/structure 3. Never thought about it

2010 1. Unable to justify costs 2. Insufficient financial resources3. Employee time constraints

2011 1. Employee time constraints 2. Insufficient financial resources3. Lack of human resources

2012 1. Employee time constraints 2. Lack of employee interest 3. Unable to justify costs against other priorities

Performance Indicator:Committee/staffing support?

Resource Allocation 2008/09(14

Agencies, Authorities)

2010(14

Agencies, Authorities)

2011(14

Agencies, Authorities)

2012(19

Agencies, Authorities

)Position/s - program part of normal role 5 9 12 13No allocated position - Volunteer(s) given time to work on project

1 1 1 1

No allocated position. Volunteer/s works on project and normal duties

2 0 1 1

A steering committee/working group given time to work on project

1 5 11 14

A steering committee/working group works on project and normal duties

6 0 1 0

Examples of responsible positions:2008 – Senior HR officer (1/2 hr/week); Senior HR consultant (2 hr/week)2010 – H@W Policy & Program Officer (23 hr/week)2011 – H&WB co-ordinator (up to 20hr/week)2012 – OH&S Manager (7hr/week); HR co-ordinator (4hr/week)

Performance Indicator:Engaged Management?

Examples of Senior Management Engagement

2010(14 Agencies, Authorities)

• Executive Group reviews and approves proposals for H&WB initiatives

• Approves allocation of resources for Agency • Approved & participate in the program• Are very supportive of health & wellness in the workplace

2011(14 Agencies, Authorities)

• Fully endorse & participate in program• Allow the program to be conducted in work time• Encourage all staff to get involved• Committed to having a budget for H&WB initiatives

2012(19 Agencies, Authorities)

• Provision of information, resource allocation and reporting on progress

• Developed the framework• Oversee program content• Have a key role in supporting staff to actively engage in

H&WB activities• Funding support for various programs

Performance indicator:H&WB plan?

H&WB Plan

2010(14

Agencies, Authorities

)

2011(14

Agencies, Authorities

)

2012(19

Agencies, Authorities

)

No 8 3 4Yes 6 11 15

Agency Evaluation of H&WB Program

Audit Year

Most commonly cited method of evaluation

2010 1. Number of participants taking part2. Employee satisfaction with program3. Number of initiatives/activities undertaken

2011 1. Number of participants taking part2. Employee satisfaction with program3. Employee satisfaction with organisation

2012 1. Number of participants taking part2. Employee satisfaction with program3. Number of initiatives/activities undertaken

Still to come

Further examination of:• Annual audit data – Analysis by

agency

Examination of:• Healthy@Work agency reports • Grants program – applications,

reports• H@W project review and closure

report

H@W project review and closure report

• Project performance: – project vision, measured outcomes, outputs, budget

• Lessons learnt: – what worked well, what could be improved, challenges – recommendations to guide future TSS H&WB activities

• Highlights and innovations:– Organisation change and development– Agencies with a H&WB program (3 in 2010, 14 in

2012)– All Agencies report H&WB activities (14 in 2009, 21 in

2010)– Working in partnerships: DHHS, Workcover, UTAS,

Menzies Research Institute Tasmania (pH@W), Healthy Workers (NPAPH)

H@W grants program

• Number of grant rounds

• Who applied for money, in which round?

• What type of project?

• Who was successful?

• How much money did they get?

• Progress/final reports provided

• Any sustainable outcomes?

Purposive Data Collection

• What happened in the implementation of H@W is only part of the story…

• The context in which a program operates can be important data for evaluation

• Semi structured interviews with key informants is one method for collecting this data

What we would like to knowexamples

1. How did you (your agency) gain management support?2. What sort of a coordination mechanism was used to

manage H@W?3. How much did employees engage with H@W? 4. How useful/relevant was the information on the H@W

website?5. How did your organisation review and update the

H&WB Program?6. How did you promote , market, communicate the

H&WB Program to employees?7. Has your agency continued to resource workplace

H&WB?8. Base on the H@W experience what factors do you

believe influence employee participation in workplace H&WB programs?

Data Analysis

Analysis• Qualitative and quantitative methods and descriptive statistics

where appropriate

Interpretation• Interpretavist methods to analyse content of interviews and

documents• What understandings or conclusions can be gained?• What does it mean for the program/project?• How does this inform the evaluation purpose?• Reveal the complexities of implementing a large scale, multi

agency workplace based health promotion initiative • Identify enablers and barriers to successful implementation and

implications for transferability to non state sector workplaces• Inform program design and implementation of future WHP

strategies

Who is this evaluation for ?

Target Audience Dissemination Format

Responsible Person/Author

pH@W partners Written report Evaluators/Process Evaluation Working Group (PEWG)

Key stakeholders Summary reportPresentation

Evaluators/PEWG

Stakeholders Presentation Evaluators/PEWG

Academic peers (UTAS, MRIT)

Peer-reviewed journal articles

Individual authors

Primary audience: pH@W partnersOther interested parties: SSMO, Tasmanian State Service Agencies, TSS employees, key informants

Still to come

• Ethics application prepared for key informant interviews

• Interviews planned for June/July 2013

Importance of Process and Implementation Evaluation

• Uses scientific rigor to understand the etiology of gaps between expected results and observed outcomes– Information can be gathered during the implementation of a

program or trial • Explores the relationship between process and

outcomes• Generates evidence to guide future implementation

and improve program outcomes JOEM, Volume 55, Number 5, May 2013