Evaluation: what is it?

Anita Morrison

What is evaluation?

Evaluation…

– …is the process of determining the merit, worth, or value of something, or the product of that process. (Scriven)

– uses a range of research methods to systematically investigate the effectiveness of interventions, implementation and processes, and to determine their merit, worth or value in terms of improving the social and economic conditions of different stakeholders (Government Social Research)

Easier as a set of questions?……….

• What is the need we are trying to address? • What specifically needs to change? • How will we know if change has taken place? • What will we actually do? • How will we make sure we're doing it as planned? • How successful have we been and what have we learned? • What now needs to change?

LEAPhttp://leap.scdc.org.uk/

Clarity of aims and objectives for any intervention – crucial!

• Identify needs or problem you are addressing• Clarify inputs (staff, equipment, volunteers etc)• Decide on activities / outputs • Nail the outcomes you want to achieve

Tools to do this

Weaver’s triangle

A Basic Results Chain

6steve.montague@pmn.net

7. End results 7. What is our impact on ‘ends’?

6. Practice and behavior change 6. Do we influence [behavioural] change?

5. Knowledge, attitude, skill and / or aspirations changes

5. What do people learn? Do we address their needs?

4. Reactions 4. Are clients satisfied? How do people learn about us?

3. Engagement / involvement 3. Who do we reach? Who uses / participates?

2. Activities and outputs 2. What do we offer? How do we deliver?

1. Inputs 1. How much does our program cost? ($, HR etc)

Program (Results) Chain of Events(Theory of Action) Key Questions

Source: Adapted from Claude Bennett 1979. Taken from Michael Quinn Patton, Utilization-Focused Evaluation: The New Century Text, Thousand Oaks, California, 1997, p 235.

Indirect Influence

Direct Influence

Control

WHY?

WHAT?

WHO?

HOW?

Quick exercise:Input, output or outcome?

• Public information campaign• Decline in HIV incidence• Peer educators• Increase in condom use• Quality of care improvements• Trained staff for sexual health clinics

Approaches to evaluation

Focus of evaluation

Key evaluation questions Evaluation approach

Delivery Have we implemented the intervention / service effectively? How can we improve?

Process

Objectives Have we achieved our outcomes? What has worked / not worked?

Outcome

Problems/needs Have we made an impact on the problems and needs that provided the rationale for intervention? What has worked / not worked?

Impact

Process evaluationPurpose:

– to assess programme design/ coverage/ delivery

Key questions:– have problems/needs been diagnosed correctly?– has the intervention been implemented effectively?– are the intended beneficiaries being reached?– what lessons can we learn to improve intervention?

Common sources of information:– Administrative records/management information– Observation– Surveys: beneficiaries, staff, partners– Case study research

Evaluation of Free Personal Care (2006-2007)

Aim: to assess the operation of free personal care across Scotland and identify possible improvements

Methods:

• Interviews with national organisations• Survey of 32 LAs• Telephone survey of 1000 members of the public• Postal survey of unpaid carers• 6 case studies • Stakeholder workshop

Outcome evaluation

Purpose:– assess degree to which objectives / intended outcomes

achieved

Key questions:

– have intended beneficiaries actually benefited?– have we achieved our intended objectives/outcomes?– have we produced any unintended effects

(beneficial/adverse)?– to what extent are the observed changes due to the

intervention (and would not have occurred without it)?

Central Heating Programme: Assessing Impacts on Health

(2002-2006)

Aim: assess whether the Scottish Executive Central Heating Programme has had a positive impact on recipients health

How was it done?

• COMPARATIVE: The evaluation compared the experiences of a group of 1,281 households which received heating under the Programme with those of a 'comparison' group of 1,084 households not included in the CHP

• LONGITUDINAL - collected data on respondents' experiences at multiple points in time.

First Interview; postal questionnaire; Final interview

CONCLUSION:

CHP was associated with reduction of problems with cold, damp and mould, with higher levels of heating in the home, with feeling warmer in winter, and with greater level of satisfaction with heating. It was not clearly or consistently associated with direct impacts on the cardio-respiratory health of recipients, nor with reductions in NHS service use. It is possible that these direct health impacts will become visible over a longer period of time.

Exercise

Resources

Evaluation Support Scotland http://www.evaluationsupportscotland.org.uk/index.asp

Learning, Evaluation and Planning http://leap.scdc.org.uk/

Barnardoshttp://www.barnardos.org.uk/theevidenceguide