EVALUATINGPARTICIPATIONWORK

EVALUATIONIN ANUTSHELL

Acknowledgements

This resource has been madepossible thanks to the support ofmany individuals and organisations.In particular, we would like to say abig thank you to the following:

Author Amanda Mainey

The Research, Evidence andEvaluation Department, NCB

Designer Stephen Brown

The Participation Works team andPaulina Filippou in particular fortheir input and support in producingthis resource.

The organisations and individualswho tested and commented on thetoolkit in its pilot phase.

Produced by NCB on behalf ofParticipation Works

December 2008

Participation Works is a consortium made up of

the the British Youth Council (BYC), Children’s

Rights Alliance for England (CRAE), National

Children’s Bureau (NCB), National Council for

Voluntary Youth Services (NCVYS), The National

Youth Agency (NYA) and Save the Children -

England.

Evaluationinanutshell

1

Plan

Evaluation in a nutshell

Work out how much time and what skills you have to do the job and make a plan with that in mind.

Develop a handful of simple evaluation questions – what does your evaluation need to show –

what questions do you need answered? What do your funders want to know? What did you set

out to achieve and how will you know you’ve done it? What would help do it again?

For example:

• What do young people think of our service?

• What helps and hinders young people’s involvement in our service?

• What impact has the programme had on the local community?

• What impact has the programme had on participating young people?

• How could we improve the programme offered to young people?

InvestigateMonitor what’s going on – what goes out and what comes in.

Record what you and others are putting in to the work (e.g. how much time, expenses) and what

outputs you’re getting (e.g. how many sessions delivered, how many people attended, who were they).

You may already have some forms or systems in place – check to see if they give you enough of

the right information. You can add brief questions to other forms you collect information with e.g.

registration forms, or you can ask people to fill a form in specially.

Get some feedback from some of the people involved – it doesn’t have to be everyone, but just

make sure you get some feedback from the whole range of people involved – staff, volunteers,

young people whether involved or not, wider community.

Make sure you get informed consent from anyone taking part in your evaluation, including from

carers of children and young people aged less than 16 years. See Section 4 of The Guide.

Evaluationinanutshell

2

How do I get feedback from people?

Use a questionnaire to get some basic feedback about people’s experiences

of the project and what difference they think it has made.

• What did they do?

• What did they think of it?

• What worked well?

• What didn’t work well?

• What would they change next time?

• What difference has it made?

• Who are they (age, sex, ethnicity, disabilities)?

Include closed and open-ended questions so you get a mix of quantitative

information (which you can count) and qualitative information (which gives you

more detail).

See Section 5.3 in The Guide and sample questionnaires in the Toolkit for more ideas.

Want more in-depth information? Interview someone! See Section 5.4 in

The Guide.

You can interview the same people who answered a questionnaire and ask them

things in more detail, or interview a different group of people. For instance, you

might interview participating young people and ask people in the local community

to fill out a questionnaire about what impact the project had on them. Or you

might ask people to fill out a questionnaire at the beginning of a project, and then

interview them at the end.

See ‘the Question Bank’ in the Toolkit for ideas of the types of questions to ask in

interviews.

Take notes during the interviews, or record your interviews and make notes or

transcribe them afterwards. But ask permission first!

Have time to wait for the evaluation findings and want to know whether

your work makes a difference in the long-term? Go back to the same

people later and ask them more questions about how they’ve put things

into practice in their lives or organisations.

Involve children and young people in the evaluation – you can do this in lots

of different ways, such as planning the evaluation, checking methods are

appropriate for young people, drafting interview questions, or writing summaries

of findings etc. See Section 3 and throughout The Guide.

Evaluationinanutshell

3

Think about your project aims and outcomes - when designing questionnaires

or interview topics, think about what impact your work has had. Ask questions

relating to what you are trying to change or achieve e.g. to what extent do you feel

you have been able to make a positive contribution to your community?

Look at Every Child Matters or Youth Matters – your project outcomes might relate

to these key issues for those who work with young people. This will place your

project in the wider context and may be what your funders expect too.

Consider how to show what the outcomes of your work are for everyone involved,

including young people, workers and the wider community. These can be short,

medium or long-term outcomes and shouldn’t be confused with outputs – the

more concrete things you’ve produced or achieved.

Is it an outcome or an output?

Some sample outcomes are:

• Older people will feel less isolated

• Young people more aware of local opportunities for leisure activities

• Young people will have improved skills for independence

• Increased awareness of support services

• What has changed and for whom as a result of your work

Some sample outputs are:

• Seven fortnightly meetings held for local young people

• 14 young people trained in writing submissions

• A written submission to local government on youth issues

• Set up a young person’s working group

Analyse itAllow lots of time to make sense of the information you’ve collected! It can take ages just to get it

into a state where you can compare what people have said.

Number your questionnaires or interview transcripts/notes so you can refer to them and find them

easily later if you need to.

Next you can do averages and maybe some cross-tabs (how many of the people who did ‘x’ also

did ‘y’). For example, this is where you might show that of the 14 young people who went on the

training course, eight have joined the local youth council. Just be careful about saying that one

thing caused another, unless you can prove this is true!

Involve children and young people – take some of your findings back to young people to discuss,

or ask them to write up summaries of your findings for other young people to read.

See Section 6 in The Guide.

For interviews or qualitative data, start by reading through your notes or

transcripts and get a feel for common themes in what people have said.

Decide what the key points are to sort your information with – these may

follow your interview questions or they might be common themes that

people have raised themselves.

Put your key points into a grid with the main themes or questions

across the top and each person’s views and comments in one row.

This way you can see similarities and differences in what each

respondent has said about your key themes. Make notes where

there are useful quotations to show points people make – it makes

it easier to find them later to back up your findings.

Or you might find it easier to cut and paste parts of interviews into different Word

documents, headed up with key themes or questions. How you go about it is not

the most important thing, as long as you are consistent and thorough.

For questionnaire or quantitative data, start with summarising/counting

how many people have given the same response to a question, e.g. how

many said they had enjoyed the experience? How many young people

wanted to do more as a result? How many felt that it had allowed them to

increase their skills? How many under 24 year-olds answered the

questionnaire? Give these as percentages too if your numbers are large

enough (say over 20 respondents).Evaluation

inanutshell

4

Evaluationinanutshell

5

Write it upNow is time to make sense of everything you’ve found. You probably need to start with a written

report – though how long or detailed it is may be determined by funders’ or organisational

requirements.

There are no golden rules – but an evaluation report usually looks a bit like this:

Go back to your evaluation questions and answer them! Use all the information you have to do

this – include all groups’ perspectives, information from all different methods – so it’s based on

evidence rather than a hunch!

Be selective in what you include in your report – you don’t need to report everything you

discovered – just all that is relevant to your evaluation questions.

When using quotations make sure they can’t identify anyone. Be aware that even the way you

describe someone could risk the anonymity or confidentiality you’ve promised. In a small group or

community it might be simple for others to identify someone from a description like ‘18 year old

mother of two’.

Use plain language, short sentences and avoid jargon.

Ask a colleague to read your report to see if it makes sense.

See Section 7 in The Guide.

Executive summary

Background/introduction description of service/project

Evaluation Methods how you did it

Findings usually broken into several shorter sections, by themes

or key areas, for example:

Respondents

Information sharing

Satisfaction and dissatisfaction with services

Improved participation

Professional involvement

Implications for future services

Conclusions and recommendations

Appendices including copies of questionnaires

Evaluationinanutshell

6

Tell everyoneYoung people, staff, volunteers, community members, funding bodies,management, partner organisations - lots of people will have been involved inthe project and the evaluation. Tell them what it found!

Make it appropriate – write summaries for young people or ask young people to draft them for you!

Take it out to the wider world! Depending on your project (and whether it’s ok

to make it public) you might find others outside your circle would be interested in

what you did or found. Consider conferences, staff meetings, management groups,

other agencies or organisations.

Need to know more?Read the relevant sections of The Guide and the Toolkit for more ideas andinformation about evaluating your work. If you want more information aboutparticular areas look at the list of resources at the back of The Guide.

Participation Works8 Wakley Street, London EC1V 7QEwww.participationworks.org.ukEnquiry Line: 0845 603 6725

Participation Works is based at the National Children’s Bureau

Registered charity number 258825

Participation Works enables organisations to effectively involve children and youngpeople in the development, delivery and evaluation of services that affect their lives.