What makes a good evaluation? - University of Sheffield/file/MasterclassEvaluation.p… · Types of evaluation • Formative: to support your activity, e.g. identifying gaps or barriers,

What makes a good evaluation?

Write your answer on a post-it

Public Engagement Masterclass 30: Some

ideas for evaluating your public engagement activity

Fran Marshall, Research & Evaluation Manager Dr Helen Featherstone, University of Bath

27/01/2016 © The University of Sheffield

Outline of session

• Evaluation aims, planning and methods – Fran

• Some examples of evaluation - Helen

Introduction to the project and Public Engagement and Impact Team

• Introduction to the Public Engagement and Impact Team

• The masterclass series

• What we can do to help?


Masterclass series Masterclass title Date & time

20: Some ideas about how to evaluate your public engagement event Thursday 22 January 2015

21: What to think about when organising your public engagement event Tuesday 24 February 2015

22: Working in partnership with others

Wednesday 25 March 2015

23: Coordinating large scale public engagement, e.g. faculty based events, film or lecture series etc

Wednesday 29 April 2015

24: Public Engagement or Widening Participation/ Outreach? Things to think

about when organising events for children and young people

Wednesday 27 May 2015

25: Grant writing and impact Wednesday 24 June 2015

26: Writing a risk assessment for your event/ event planning Wednesday 29 July 2015

27: Engaging the public as researchers and different types of engagement Wednesday 30 September 2015

28: How to communicate your event

Wednesday 28 October 2015

29: Storytelling in public engagement Wednesday 25 November 2015

30: Some ideas about how to evaluate your public engagement event Wednesday 16 December 2015


• The University’s Good Research and Innovation Practices policy

www.sheffield.ac.uk/ris/grip

• Minimum accepted practices in public engagement

• Higher practices to aspire to


What makes a good evaluation?

Write your answer on a post-it

Who is here?

A. Academic staff

B. Non-academic staff

C. Post-graduate student

D. other


Academ

ic st

aff

Non-aca

demic

staff

Post-g

raduate

student

other

14% 14%

38%34%

Which faculty are you from?

A. Arts & Humanities

B. Engineering

C. Medicine Dentistry and Health

D. Science

E. Social Sciences

F. Cross faculty


Arts &

Hum

anities

Engineerin

g

Medici

ne Dentis

try a

nd H...

Science

Social S

cience

s

Cross

facu

lty

10% 10%13%

37%

3%

27%

How much do you know about planning evaluation?

A. A lot

B. Quite a lot

C. A little

D. Don’t know anything


A lot

Quite a

lot

A little

Don’t kn

ow a

nythin

g

0%

32%

55%

13%

How much do you know about evaluation methods and when to use them?

A. A lot

B. Quite a lot

C. A little



A lot

Quite a

lot

A little

Don’t kn

ow a

nythin

g

9%

34%

53%

3%

How much do you know about different kinds of evaluation?

A. A lot

B. Quite a lot

C. A little



A lot

Quite a

lot

A little

Don’t kn

ow a

nythin

g

6%

32%

55%

6%

What is Public Engagement?

• “Public engagement describes the myriad of ways in which the activity and benefits of higher education and research can be shared with the public. Engagement is by definition a two-way process, involving interaction and listening, with the goal of generating mutual benefit”

National Coordinating Centre for Public Engagement

What is impact?

In relation to the Research Excellence Framework and Pathways to Impact:

• A change created by your research outside of academia

• From a small change in one person’s attitude to complete revolution in the way of life of millions worldwide

• Undertaking evaluation can help you demonstrate the impact of your project


Where to start with evaluation…


What is evaluation? • About the audience – show of hands

• Who has organised a public event before?

• Who has devised an evaluation?


Monitoring or evaluation?

• Monitoring is counting

• Audience numbers, demographic information

• Evaluation is about critical reflection

• Can be: formative, about a process or summative


When do we evaluate?

• Ideally before, during and after an activity

• In reality often done at the end - OK for single event but not for longer piece of research

• Includes looking at the quality of content, delivery and impact on participants


Types of evaluation

• Formative: to support your activity, e.g. identifying gaps or barriers, gathering information to identify the need for your activity. Testing ideas and concepts – what is going to be most effective? What works best?

• Process evaluation: gathering information about how you can do it better next time

• Summative: i.e. a summary – final impact of your activity

• Is it a single evaluation or do you want base line and comparative data?


Evaluation Tree


What do you want to know?


Start at the bottom

• Aims and objectives of event

• What kind of evaluation is it?

• What do you want to know?


What do we need to evaluate? • Simple answer is the

public engagement activity you are doing

• More complex answer is what can we evaluate?

• What do you want to know v what can the audience tell you?


Developing an evaluation plan


Evaluation plan • Who do you want to evaluate?

• Who are your public?

• What resources are available for evaluation?

• What type of event is it?

• What questions should you ask?


Who do you want to evaluate? • Audience

• Event leaders (University staff)

• Partner

• Volunteers


Who are your ‘public’?

• Who are you expecting to attend your event?

• Design the evaluation activity around your audience


Resources • Who will do your evaluation on the day?

• How much time do you have to analyse the data?

• What do you want to do with the data?

• How much money do you have for evaluation?


Example costs

• Postcards - £35 for 1000

• A1 poster - £17

• Software for tablet survey - $49 per month

• Comment book - £10

• Google forms – free

• Surveymonkey - £25 per month

• Photographer - £75 per hour

• Plus pens, post-its etc


What type of event is it?


• Keep it simple and clear

• Should link to what you want to know

• Have a bottom line

• Ask the right questions

• Remember the unexpected

• Reflect on what you found out


What questions should you ask?


Asking questions

• Nothing is perfect – decide what fits best

• Keep it relevant

• Make it fun

• Don’t make it too onerous

• Put yourself in the participants' shoes

Evaluation questions exercise

1. How much do you earn?

2. Would you come to this venue again?

3. Do you have any problems with your boss?

4. To what extent do you agree - the Earth is like a spaceship with very limited room and resources?

5. Do you think URMS is a important method for calculating the fEC of the PI’s time in an ESRC grant application?


Methods of collecting data


Choose the method that can answer your questions and fits with your event


Voting

Comments


Any comments on the comments cards?


Some comments from FotM

• So much to see! The 83 project is a real eye opener.

• Thermal imaging was fun and carnival. Was an immersive floor

• Great – very inspirational

• Thanks you so much for a wonderful performance. Really enjoyed it. Was great to see the power of music and dance and drama in action and witness the talent in this group! Also great to meet the Professors and staff behind it x

BUT

• I no want it anymore

• Bu bu bu bang!!!!

• My kittens enjoyed the exhibits (I brought him in my bag)

• I like the glass bowl guitar recorder be … is good

• Poor chat

• Peas

• The postit notes

• Carrots?

• The king of the cats

• Phew it smells in this building


Social media and media coverage

• Google analytics for website traffic – help available through CiCS

• Storify: to share curated social media stories

• Twitonomy for twitter analytics

• Media reach


• Focus groups

• Interviews



http://www.youtube.com/watch?v=8RlE8hp4WdI

Films and videos

Feedback video from masterclass 1


“As a participant, I think it did work and reinforced (in other ways) some of the key issues of that early workshop, so doing the reflection/evaluation in that format was a useful process for me too”.



Some creative techniques


Some creative techniques

Surveys and questionnaires – paper or online


Arts & Humanities 9 14%

Medicine, Dentistry & Health

10 15%

Science 11 17%

Social Science 21 32%

Professional Services 8 12%

Other 7 11%

Any comments on the surveys?


Observation and photography


• Try to record your event with photos

• Observe your audience’s reaction • Count number of visitors • See what kind of engagement is

going on

After your evaluation


Data analysis

• How will you analyse the data?

• Anything written on paper takes time

• Online surveys can be good for automated data analysis

• Make sure you think about the analysis at the start


Example of output from iPad survey Why did you decide to come the event today?

I thought it would be enjoyable 15

Just passing/ looked interesting 10

I came with a friend 9

I know someone who is involved 8

To hear the speaker 4

To learn something new 3

other 8

Grand Total 57


Summary from postcard feedback How much new information did you learn at the event?

A lot of new information 416

A small amount of new information 130

No new information 14

(blank) 19

Grand Total 579


Reports • Assistance with planning

• Provides evidence

• Demonstrates value for money

• Reporting to funders

• Learning that can be shared with others and inform future activities

• Influence research


Example report • Background to event

• Comments about how it was curated

• Budget

• Audience numbers

• Marketing

• Media coverage


Example report • Comment about types of events and which

departments are involved

• Feedback from audience

• Feedback from event leaders

• Comments about what worked and what could be improved


Link your findings back to the beginning


Example of evaluation


Festival of the Mind 2014

• Visitor numbers 26,000

• Visitor comments

• Website stats: Page visit, unique visitors, page views, pages viewed per visit, average visit duration, visitors using a mobile/tablet, non-mobile visitors, new or returning visitors and language settings

• Feedback from academics and partners involved


Festival of the Mind evaluation

Visitors

• 579 postcards

• 57 surveys completed using iPads

• Online survey for pre-booked events (335 responses)

• Comment books

• Post-it note comments

• Economic survey for 1 event by external company


Festival of the Mind evaluation

Event leaders and partners

• Online survey

• Small amount of interviews and event observation undertaken by an external company


Feedback on social media

• Twitter followers: 3127 for main FOTM account with an additional accounts related to projects

• Facebook 1348 likes

• Blogs – over twenty different blogs set up or featured FOTM in the week of the festival.


Dr Helen Featherstone, University of Bath


Where can I go for help?

• Public Engagement and Impact Team

• Digital Engagement Team – social media

• CiCS – audience response system training, technical help and creative media


Guidance

http://www.publicengagement.ac.uk/how/guides/evaluation/resources

http://www.rcuk.ac.uk/Publications/policy/Pages/Evaluation.aspx

http://www.publicengagement.ac.uk/how/guides/evaluation/examples

https://www.sheffield.ac.uk/life - evaluation reports


Evaluation

What have you learnt?

How much have you learnt about planning evaluation?

A. A lot of new information

B. A small amount of information

C. No new information

D. Don’t know


A lot o

f new

info

rmatio

n

A small a

mount o

f info

r...

No new

info

rmatio

n

Don’t kn

ow

50%

0%0%

50%

How much have you learnt about evaluation tools and when to use them?




D. Don’t know 27/01/2016 © The University of Sheffield

A lot of n

ew inform

ation

A small amoun

t of infor...

No new in

form

ation

Don’t know

42%

0%

8%

50%

How much have you learnt about different kinds of evaluation?




D. Don’t know 27/01/2016 © The University of Sheffield

A lot o

f new

info

rmatio

n

A small a

mount o

f info

r...

No new

info

rmatio

n

Don’t kn

ow

57%

0%

9%

35%

Opportunities for engagement – what is coming up?

• Life festival (April 2016)

• Festival of the Mind (September 2016)

• Off the Shelf (October 2016)

• Festival of Social Science (November 2016)

• Doc/Fest (March and June 2016)


For further information contact:

[email protected]

@UniShefEngage

#GetEngaged


Questions


To Discover And Understand.

Documents

What makes a good evaluation? - University of Sheffield/file/MasterclassEvaluation.p… · Types of evaluation • Formative: to support your activity, e.g. identifying gaps or barriers,