Embed Size (px)
Citation preview
Evaluation:
Find out what visitors think
Improve what we do
Nicola Bell, MAculture ~ evaluation ~ [email protected]
What is Evaluation?
Evaluation is the systematic collection of
information about the content, characteristics,
and outcomes of a programme to make
judgements about it, improve its effectiveness,
and/or inform decisions about future activities.
(Kirby, P., and Bryson, S. (2002) Measuring the Magic? Evaluating and Researching Young People’s Participation in Public Decision Making, London: Carnegie Young People's Initiative)
Consulting with young people - literature reviewhttp://www.creative-partnerships.com/data/files/consulting-young-people-13.pdf
What is evaluation?
It is Quality Control, to find out:
Is it good?
How can we make it better?
Why do we want to carry out an evaluation?
To find out how to develop a project
To find out if the project was successful
To find out if the money was well spent
To get money for future activities
To find out what non-visitors think
If you don’t evaluate, you might:
waste time
waste money
produce something that you can't change
lose interest from your target audience
lose an opportunity to learn something
useful
lose an opportunity to tell people about the
good work you are doing
lose funding
When planning an evaluation, remember:
Plan the evaluation at the beginning of the project
Allow enough time to collect and analyse the data
Allow enough money for the evaluation
Consider using a freelance evaluator who can bring an independent view to the project
Who is the evaluation for?
Museum staff
Project partners
Project participants
Funders
Local or national government, future partner organisations
Visitors
Types of evaluation
Front end - e.g. planning new exhibit
Prototyping - e.g. for a new interactive
Baseline - e.g. visitors’ knowledge at the beginning
Formative - e.g. how the programme is developing
Summative - e.g. was the programme effective?
Usually evaluations only cover a short time; longitudinal studies (over several years) would be good
Methodology:
Link the evaluation to the project’s aims and objectives
Aims and objectives must be made clear at the start
Aim = what you are going to do? What changes will happen? (to participants, to the museum, to staff)
Objectives = how you are going to do it?
Objectives should be SMART
SMART objectives:
S - specific
M - measurable
A - agreed
R - realistic
T - time-based
Outputs, Outcomes and Impact
Logic Framework model:
Inputs > activities > outputs > outcomes > impacts
Examples:
Inputs Activities Outputs Outcomes Impacts
MoneyStaff timeIdeas
ActivitiesEventsExhibitions
Numbers of visitors at eventsNumber of events
Changes which happen:People learn skillsPeople have more confidence
Changes which happen beyond the outcomes: Changes for visitorsChanges for the museum
Outcomes can be:
Positive Negative Expected Unexpected
They can happen:
During the project At the end of the project
What sort of data do you need to collect?
Quantitative data:
How much?
How many?
Qualitative data:
What was it like?
Opinions and views
Decide how many people you need to gather data from:
(assuming 3% error)
Population size
90% confidence level
95% confidence level
99% confidence level
10 10 10 10
50 47 48 49
100 88 91 95
500 301 340 393
1000 431 516 648
5000 657 879 1347
http://www.custominsight.com/articles/random-sampling.asp
Sampling
The sample should be representative of the whole population (e.g. everyone who attended the exhibition)
Random sampling: e.g. every 5th or 15th person who leaves the exhibition
Convenience sampling: e.g. people who are available
Snowball sampling: e.g. for people who are hard to reach - ask some people, then ask them to suggest other people (e.g. their friends) to interview
Make sure that the data collection method is suitable for the people and the information
Triangulation = using several different methods, to get strong evidencehttp://www.creative-partnerships.com/data/files/consulting-young-people-13.pdf
Make sure that you have appropriate permissions for photography, audio recording, interviewing children
Methods of collecting data:
Mediated (with a member of staff):
Questionnaires (with interviewer) Interviews Focus groups Observation or tracking Participant observation Video Personal Meaning Mapping (concept
maps)
Need not (necessarily) be mediated:
Questionnaires (self-completion) Diaries, journals, scrapbooks Comments books Comments cards Drawings Mind maps On-line surveys (e.g.
www.surveymonkey.com) Blogs SMS, Facebook, Twitter
What do you know, think, feel about:
Etnografski
Muzej
Zagreb
Concept map: visitors write what they know etc about the topic, before the visit the exhibition. Afterwards, they write (in a different colour) what they now know, think, feel etc. Analyse comments using e.g. Generic Learning Outcomes.
Make an Evaluation Plan
Who are the participants? What is the activity? Who is the evaluation for? What are the aims and objectives? Inputs, activities, outputs, outcomes,
impacts? Baseline, formative, summative? Which methods? for each part of the
projectfor each group of people(participants, staff, artists
etc)How will you present the evaluation?
Analysing the data
The analysis must reflect the evaluation aims and objectives
Allow plenty of time (and therefore money) for analysis
A written report is usually necessary for managers, funders and other stakeholders
The report should give recommendations for the future
The report can be in alternative formats …
Alternative formats:
Website, blog
DVD or video
Performance
Presentations
Set of reference cards
Book
What will you do with the evaluation when it is finished?
Present it in a suitable format
Tell the relevant people about it
Ensure that recommendations are acted upon
Don’t leave it on the shelf!
Best Practice in Educational and Cultural Action
http://ceca.icom.museum/node/208
Conception of the project
Achievement
Evaluation
Remediation / Improvement
Inspiring Learning for All
http://www.inspiringlearningforall.gov.uk/
Generic Learning Outcomes
Generic Social Outcomes
Ways of measuring hard-to-measure things
Shows to other organisations what museums do, to benefit their visitors
Generic Learning Outcomes
Generic Social Outcomes
Working with community groups
Youth groups
Services for disadvantaged people
Services for people with disabilities
Communities of place
Communities of interest
Work with partner organisations, who already have a good relationship with local people
Why talk to teenagers and children?
United Nations Convention on the Rights of the Child, article 12:
“assure to the child who is capable of forming his or her own views the right to express those views freely in all matters affecting the child, the views of the child being given due weight in accordance with the age and maturity of the child.”
Every child should be able to express their views and influence decisions affecting them. “Child” is anyone under 18 years old, so includes teenagers.
Why talk to teenagers and children?
They have views in their own right - they are not just part of a family, part of a school, or on the way to being an adult
It develops a relationship between the teenagers and the museum
They can learn skills, e.g. decision making, interviewing, analysing data
They can have a positive influence on their community
It can increase their confidence and social skills
Ways of involving teenagers in the museum:
Volunteer tour guides
Provide training for them as volunteers in shop and café - good skills for jobs
Young people’s forum
Curating exhibitions
Creating interpretation (e.g. film, guide) for an exhibition
Projects to develop skills (e.g. reading, writing, art)
Young people as evaluators
It is better if young people are involved in planning and doing the evaluation
They can learn, and grow as a person
Everyone (any age) brings their own knowledge and world-view
Everyone makes their own meaning
UNICEF guide to participatory evaluation with teenagers:
http://www.artemis-services.com/downloads/tools-for-participatory-evaluation.pdf
Maslow’s Hierarchy of Needs
Outcomes Star: a way to measure things like increase in self-confidence
http://www.outcomesstar.org.uk/
Do…
• Make it fun!• Explain the process and what influence they can have • Provide a welcoming place• Provide food which young people like• Respect young people’s views; confidentiality, privacy• Celebrate achievements• Treat young people as individuals, not as a group• Use e.g. artists to help with creative evaluation• Be aware of needs of people with physical / learning disabilities• Tell young people the outcomes of the project• Provide help with transport to the museum
Don't…
• Make assumptions - about what they know (or don’t know), want or can do• Promise too much• Forget about child protection (especially in mixed age groups)• Do adult things e.g. boring meetings• Consult too much - people will get bored• Do it too quickly – good consultation takes time
