Workshop 6 - How do you measure Outcomes?

Workshop 6 - How do you measure

Outcomes?

www.yhtphn.co.uk/win

“if you don’t measure it, people will know you are not serious about delivering it!!”

(Duxbury 2005)


How do you measure outcomes

• Listen to others and value the diversity of opinions in the group

• Be constructive• Value the differences; there are no rights or

wrongs, no good or bad• Be open and honest• Keep to agreed time, especially start and finish• Mobile phones!

(extracted from, Working with Groups. General Improvement Skills. Modernisation Agency)


Ground Rules

• appreciation of evaluation methods• skills to evaluate their practice or any changes to

practices that have been implemented

• This is not a session on research techniques.


Programme Itinerary

• What is evaluation?• Why evaluate?• Including evaluation within Project Planning • Involving stakeholders in evaluation• Methods • End report and dissemination• Putting evaluation into practice• Complex Interventions Evaluation• Evaluation SOS


Contents

• The means of systematically determining the extent to which aplanned intervention or programme achieves predetermined objectives(World Health Organisation)

• a way of:o assessing whether objectives have been achievedo assessing the outcomes of projects o learning from successes and mistakes


Definition of evaluation

• Have the aims been met?• Was it cost effective?• Funding• Accessibility for target population• Measuring impact/outcomes • Process


Why Evaluate

Consider the following:• Evaluation should be an ongoing part of planning and

development – not an afterthought• Evaluation should be part of learning and sharing

successes and difficulties• Evaluation should involve all stakeholders and be

relevant to them• Who is going to conduct each stage of the evaluation• What type of evaluation is needed ie internal/external• Are there any ethical issues


Including Evaluation within Project Planning - 1

• What do you want to know?• How are you going to measure this?• Clarify what information already exists?• What information do you need to collect?• What do you need to be able to measure this?• How will we learn from our experiences en route?• Are there milestones we need to reach as part of the

evaluation of the project?


Including Evaluation within Project Planning - 2

• What ethical issues should you be considering?o Data sensitivity o Confidentialityo Informed consent o Storage of informationo Insure there is no coercion o Respect


Ethical issues

• A stakeholder is anyone who is affected by or who can influence the impact

• Primary stakeholders: people who are directly affected by the initiative (either benefiting from it or adversely affected by it)

• Secondary stakeholders: all others who have a stake or interest in the initiative


Involving Stakeholders within Evaluation - 1

Stakeholders are:• People who use the service/s • People who live in the area• Staff and volunteers• Partner agencies• Funding bodies• Local and national policy or decision-makers

• Do you think all stakeholders have the same level of influence or importance ?….


Involving Stakeholders within Evaluation - 2

Example of a basic stakeholder analysis• High importance High importance• High influence Low influence

Service users on steering group

Vols

Local Services

Staff

Service users

Local community

Funders

National Gov’

Vols

Local ServicesLocal community

Funders

National Gov’

Local Gov’

Low importanceHigh influence

Low importanceLow influence

• You have been working with a community organisation that have been delivering support and advice to mums with post natal depression. This involves a wide range of social, mental health and other advice.

• Your funding is coming to an end. You are seeking further funding, but need to conduct an evaluation of what is currently being delivered


Scenario 1

• You are involved in a multi disciplinary falls clinic for older people. This involves providing advice and support to people identified at risk of falling.

• Your manager has asked ‘if it is working’


Scenario 2

• Who• What power do / might they have• Who is most important• What might they want to know from an

evaluation• What level of involvement should they

have?


Exercise 1- Identify the potential stakeholders


Coffee

Methodse.g. What type of data?

Qualitative Quantitative

Particularly useful for…. • exploratory work – where issues not clearly understood or preliminary investigations

• issues which need detailed understanding of individual circumstances

• can identify new ground• complex issues or processes• sensitive issues• map the range of behaviour and

experience

• to describe patterns and trends

• measure extent• measure location• measure differences

Data collection emphasises

Open-ended questions Can be closed questions

The types of tools you might use include e.g.

• In depth interviews, semi-structured interviews, unstructured interviews, focus groups, observations, case study

Structured interviews, survey, postal questionnaire


Methodse.g. has the project improved access to services for its target group?

Evaluation questions Possible measures/methods

Do more people access services?

• take-up rates• treatment records• self report through questionnaires• survey of attendance levels

Are people more aware of services?

• questionnaire/survey of target group• interviews with target group

Do people feel more able to use services?

• focus groups on views of service users• individual interviews with service users & non

service users• questionnaires

Do professionals feel more people are accessing their services?

• telephone interview with professional• questionnaire to professionals


Methodse.g. has the project increased community involvement in services?Evaluation questions Possible measures/methods

Are more people having a say in the way services are run?

• interviews• focus groups• questionnaires• number of people on advisory

board/steering groups levels

Do people feel more involved in the way services are run?

• focus groups• interviews • questionnaires• pictures• drama

Do people have power and influence in this project?

• attendance rates• influence on decisions• membership of steering group/advisory

board


• For your scenario • What might be the key evaluation

questions?• Thinking of your stakeholders – what types

of data / information might you be interested in seeking out

• What methods can you use to do this• Who might be responsible for doing this

and at what stage• How might you disseminate your findings


Exercise 2

• Who is the evaluation for?• How effective was the process?• Have the planned outcomes been achieved?• Is it cost effective?• Who is the audience for the end product?• What role do service users have within the evaluation?• Who is going to do the work?• What is the timescale?• What format is the end product e.g. a report.


The evaluation planning checklist

• “A Health indicator is a characteristic of an individual, population, or environment which is subject to measurement (directly or indirectly) and can be used to describe one or more aspects of the health of an individual or population (quality, quantity and time)” (WHO 1998)

• Some evidence is easier to collect than others which can make it tempting to collect what is collectable regardless of how useful/meaningful it may be.


Indicators

• “A change in the health status of an individual, group or population which is attributable to a planned intervention or series of interventions, regardless of whether such an intervention was intended to change health status” (WHO 1998)

• Crucially important that you consider which outcomes you may want able to measure.

• A hierarchy of outcomes.


Outcomes

Consider • What outcomes do you need to measure?• What are the most important and are they

measurable?• Attribution - Can you be certain that the

outcome is as a result of your intervention?• For complex/long term interventions it is

unlikely to show impact quickly


Outcome measures / indicators

• For your scenario• Pick the indicators and outcomes that

you might choose to evaluate your intervention


Exercise 3

• Public Health’s interventions are often complex with multiple components and influences

• Boundary-less – many different activities undertaken by many stakeholders in the name of the intervention

• Complex social change depends on many environmental factors often beyond reach of intervention


In summary

• Choosing Health - strategy for public health. It is focused on improving lifestyles – with a view to reducing the incidence of chronic diseases and improving life expectancy.

• eg Smoke-free agenda• Discuss


Example of a complex intervention Is Choosing Health


Evaluation SOS

Documents

Workshop 6 - How do you measure Outcomes?