Embed Size (px)
Citation preview
Festival-‐based Public Engagement
Dr Eric Jensen Assistant Professor of Sociology University of Warwick [email protected]
1
Seminar Overview Today we will discuss… • Key characteris-cs of fes-vals • Student volunteering in fes-vals • Fes-val Evalua-on
– Feedback and Self-‐report – Cambridge Science Fes-val Evalua-on example
– Evalua-on Design – Survey Design
3 Ques'ons for you!
1. How would you define ‘public engagement’? (one sentence)
2. How would you define a ‘fes-val’? (one sentence)
3. If you went to a public engagement fes-val run by a university as a visitor, what would you be expec-ng from the experience? (one sentence)
Speaker Background • Lecturing on research methods, MSc in
Science, Media & Public Policy at Warwick. • Research on public engagement practice,
impacts (e.g. festivals, London Zoo, Natural History Museum, etc.)
• ISOTOPE project (Informing Science Outreach & Public Engagement)
Informing science engagement prac-ce through theory and research
• NESTA-‐funded ac-on research project (2006-‐2009)
• Prac--oner-‐led, academically edited website with informa-onal resources.
Informing Science Outreach and Public Engagement
ISOTOPE website (isotope.open.ac.uk)
Introducing public engagement • ‘Public engagement’ can be seen as an umbrella term within which ‘public communica-on’, ‘public consulta-on’ and ‘public par-cipa-on’ all fall (Rowe & Frewer, 2005).
• However, Rowe and Frewer (2000, p. 254) dis-nguish between – public par-cipa-on exercises, where “informa-on of some sort flows from the public to the exercise sponsors”,
– communica-on exercises, where informa-on flows “solely from ‘sponsors’ to the public”
OVER TO YOU!
• What do you see as the defining characteris-c of a ‘Fes-val’ for public engagement?
• What do you see as the inherent challenges and promise of Fes-val-‐based Public Engagement?
Discuss in groups of about 3 using your preliminary defini7ons as a point of discussion.
Introducing Fes'val-‐based public engagement
• At one level, fes-vals are events, with many typical prac-cal tasks that have to be worked out, including – Venue selec-on – Health and safety management – Educa-onal content: for young people – Educa-onal content: for adults. – Event formats: programming talks, panel discussions, hands-‐on ac-vity and other events
– Use of online media to promote and enhance fes-vals – Design and branding for fes-vals – Ticke-ng and public informa-on for fes-val visitors
Introducing Fes'val-‐based public engagement
• educa-onal fes-vals as enjoyment-‐oriented sites for engaging publics with new ideas, knowledge and research.
• Fes-vals defined by temporary / transient nature. – This transience has both posi-ve and nega-ve aspects.
The Issue of Transience • Positive:
– investment may be made in a level of activity which would be hard to sustain for a longer period.
– People willing to try new things and encounter new ideas.
• Negative: – Challenge of making a lasting impact.
• Can address by linking into year-‐round institutions and activities, which can embed the gains from the festival.
University Student Volunteers and Festival-‐based Public Engagement:
Research Results
Dr Eric Jensen Assistant Professor of Sociology
University of Warwick [email protected]
12
Research Results: Overview • Student volunteers survey:
– Fes-vals seen as posi-ve experience by vast majority of student volunteers
– Perceived as valuable for skills development
• Fes'val organisers survey: – Student volunteers fulfil crucial roles in spaces between paid staff’s capabili-es and responsibili-es
– Enthusiasm enhances experience for both organisers and visi-ng publics
Student Sample Table 1: Student Survey Sample Distribution by Festival Type
Festival Type Percent
Science, technology or nature (e.g. Cheltenham Science Festival) 69%
Performing arts (e.g. Edinburgh Festival Fringe), or other music, theatre, dance etc. festival
28%
Children’s / family (e.g. Belfast Children’s Festival) 5%
Visual arts (e.g. Glasgow International Festival of Visual Art) 1%
Film (e.g. Encounters Film Festival, Bristol) 1%
Literary / books (e.g. Hay Festival) -‐
Other 6%
Note: Total percentage exceeds 100% because respondents could select multiple categories.
Organiser Sample Table 1: Organiser Survey Sample Distribution by Festival Type
Festival Type Percent
Performing arts (e.g. Edinburgh Festival Fringe), or other music, theatre, dance etc. festival
61.7%
Science, technology or nature (e.g. Cheltenham Science Festival) 42.6%
Children’s / family (e.g. Belfast Children’s Festival) 36.2%
Visual arts (e.g. Glasgow International Festival of Visual Art) 34%
Literary / books (e.g. Hay Festival) 23.4%
Film (e.g. Encounters Film Festival, Bristol) 17%
Other 12.8%
Note: Total percentage exceeds 100% because respondents could select multiple categories.
Student Volunteers: Recruitment • University communica-on networks are key hub for recrui-ng student workers and volunteers – e.g. through e-‐mail lists or student socie-es.
– Word of mouth (friends
Student Volunteers Survey • Student volunteer respondents highly suppor-ve of fes-val-‐based public engagement: – 92% would volunteer in future
Student Volunteers Survey • Majority (75%) believe fes-val volunteering will help them in their future study or career (12% did not).
Student Volunteers Survey: Motivations
• Students interested in volunteering for: – “Skills / career development”; furthering their experience and future career possibili-es.
– ‘Public engagement’ goals of engaging publics with their favoured subjects.
Mo'va'ons: Example Quota'on “To be useful in science you need to be able to communicate your topic effec-vely and to all age ranges with all educa-onal backgrounds. When the opportunity to volunteer at the science fes-val came up I thought it would be a great opportunity to work at my communica-on skills and also experience public engagement first hand”. (F, Postgraduate, Russell Group)
Student Volunteers Survey • Most common type of role (47%) was ‘educa-onal’, followed by ‘workshop or ac-vity leader’ (28%) and ‘steward’ (26%).
Student Volunteers Survey • 51% of respondents interacted with 1-‐99 fes-val aiendees, 14% reported interac-ng with 100-‐499 aiendees, about 10% with 500+ fes-val visitors.
• 25% reported having no contact with fes-val visitors at all.
• Interac-ng with visitors was iden-fied as a source of sa-sfac-on for student volunteers
• Therefore, organisers should consider arranging for volunteers to spend -me interac-ng with visitors.
Student Volunteers Survey • Student volunteers felt they made a “small contribu-on” individually, but were an important part of the fes-val’s “overall effec-veness”.
Student Volunteers: Example Quota'on
“Volunteers at the fes-val where I worked proved to be the face of the fes-val, as they are present at every single event as the front line of staff represen-ng the organisa-on. The volunteers had to do work that other staff members would not have had -me to do, but if those tasks had been neglected, the fes-val would not have been half as successful”. (F, Postgraduate, post-‐1992)
Student Volunteers Survey • Respondents highlighted importance of good training and guidance from fes-vals.
• Good training and guidance is crucial: – to ensure volunteers’ effec-veness – to ensure volunteers have sa-sfactory experience that builds new skills they can take forward into employment and other sekngs.
• Need for training to expand as few respondents had received detailed prac-cal training or guidance for their fes-val roles.
Festival Organisers Survey • Fes-val organisers rely on having at least some paid staff, however majority of respondents employ five or fewer people.
Festival Organisers Survey • While majority of organisers in sample work with universi-es (66%), a substan-al minority do not.
• Given benefits of working with universi-es (and their students), these fes-vals should be a key focus for building new links
Festival Organisers Survey • Festival organisers reported that the enthusiasm and expertise of volunteering students and staff comprised
the most valuable aspect of engaging
with universities in delivering their
festivals.
Festival Organisers Survey • Aligning with the results of the student survey, organisers overwhelmingly emphasised that the value of festival-‐based volunteering for students centred on what could be categorized as “skills development/employability”, as well as for expanding students’ range of professional contacts.
Festival Organisers Survey • Fes-val organisers
indicated that universi-es provide a great deal of support for fes-vals overall
• Most olen universi-es by providing human resources in the form of unpaid student volunteers (70%) or speakers, ar-sts and workshop leaders (61%).
• Universi-es are also more likely to offer fes-vals free venues (57%) than venues for hire (48%).
Festival Organisers Survey
• Least successful aspect of using student volunteers in festivals is high level of training required for each iteration of the festival.
• Several festival organisers have had to adjust their expectations of student volunteers’ prior practical knowledge, starting training at a basic level.
OVER TO YOU!
• What do you think is reasonable to expect of university student volunteers and fes-val organisers in the social exchange of fes-val volunteering?
Discuss in groups of about 3.
University Student Volunteers and Festival-‐based Public Engagement:
Research Results
Dr Eric Jensen Assistant Professor of Sociology
University of Warwick [email protected]
33
FESTIVAL EVALUATION
34
2 Ques'ons for you! 1. How would you define ‘impact’ in context of
fes-val-‐based public engagement? (one sentence)
2. What should an ‘evalua-on’ be achieving in the context of fes-val-‐based public engagement? (one sentence)
Tricky Issues in Festival Evaluation • Educa-onal fes-vals typically encompass a wide range of different kinds of engagement ac-vi-es from which individual members of the public may select or encounter.
• This variegated context raises a number of methodological challenges for evalua-on.
Defining Impact • I define impact in terms questions like:
– What difference have you made in people’s lives?
– What ideas, relationships, interests, motivations have been transformed as a result of your intervention? (and in what ways?)
Defining Impact • OR: ‘impact’ is the difference between the profile
of those you engaged pre-intervention and their profile post-intervention.
• That is, the overall net effects or results of an activity or intervention (intended or unintended). – i.e. what were those engaged like before you
encountered them and what (if anything) changed for them as a result of your encounter?
• Note that changes or ‘impacts’ can be in negative or dysfunctional directions!
Defining Impact • Impacts could include:
– development in learning – attitude and behaviour change – a greater sense of self-efficacy – enhanced curiosity or interest in a subject – improved skills – greater connectedness with others – improved understanding of self and the
broader world / universe – improved confidence or skills, etc.
Defining Impact Evaluation
• The systematic collection and/or analysis of information to provide useful and focused feedback on the effects of an activity or intervention.
Why Evaluate? • To build a better understanding of your public,
(e.g. needs, interests, motivations, language). • To inform your plans and to predict which
engagement or learning methods and content will be most effective.
• To know whether you have achieved your objectives (and why or why not).
• To re-design your approach to be even more effective in future.
Tricky Issues in Festival Evaluation Challenges include: 1. collec-ng data from a transitory visitor
popula-on in a crowded informal context. 2. designing survey ques-ons that can
accommodate feedback on a broad range of public engagement ac-vi-es
3. using untrained individuals working with diverse organisa-ons to collect data
4. analyzing the diversity of feedback on such mul--‐faceted experiences in a way that allows common paierns to emerge.
Tricky Issues in Festival Evaluation • Accurate Overall ARendance Counts:
– Using representa-ve sampling can address this. – Using results from survey-‐based evalua-on can help
• E.g. You might have hard aiendance counts at all sit down events during the fes-val.
• Survey can ask how many total sit down events people aiended. Dividing total hard count by the mean number of events aiended yields unique visitor result.
• Gathering representa've feedback: – Can’t use comment cards as self-‐selec-ng (non-‐representa-ve sample)
– Can use brief data collec-on on site, collec-ng email address to send follow-‐up online form.
Festival Evaluation: Feedback and Self-Report
Context • Although science festivals are becoming
more prevalent, their impacts are under-researched, with very little quality research literature currently in print. – This study aimed to address this issue using
rigorous social scientific methods – Results submitted for publication in a peer
reviewed journal
Methods • A combination of research methods was employed
in this evaluation 1. Focus groups were conducted with attendees to
investigate qualitative dimensions of impact on science festival attendees.
2. An on-site questionnaire (n = 957) – this short one-page questionnaire sought basic information about the demographic characteristics of CSF attendees, as well as their comments and ratings of festival events.
• Comments about the festival were categorized and tallied to reveal patterns.
Results: Questionnaire
• Overall levels of satisfaction were heavily skewed towards the ‘Excellent’ end of the scale. With ‘1’ as ‘Poor’ and ‘5’ as ‘Excellent’, the mean rating was 4.53.
Results: Questionnaire
Table 3 – Generic positive and negative responses
Code Number
Positive 156
Negative 6
Results: Questionnaire Table 4 – Festival Impacts
Code Number
Creating Interest 230
Knowledge 125
Interactivity 23
Results: Questionnaire Table 5 – Festival Delivery
Code Number
Positive 181
Negative 28
Results: Ques'onnaire Event Sa-sfac-on Sta-s-cs
• Statistically significant differences in satisfaction ratings were found based on the variable of ‘gender’, with female respondents significantly more satisfied with festival events than male respondents.
• However, both male and female ratings skewed heavily towards ‘excellent’ (5 on the Likert scale) • Means of 4.47 and 4.58 respectively on 1-5 scale.
• ‘Age’ was also found to be a significant predictor of Cambridge Science Festival satisfaction ratings (F(6, 696) = 2.62, p <.05). Mean scores were highest with the 41-50 year-old age category.
Results: Focus Groups • Focus group results highlighted the
processes of planning and selecting flows through Cambridge Science Festival
• Perceived benefits of ‘live’ as compared to mediated science communication.
• Social appropriation of festival visits. • Other issues related to science
popularisation and the perceived failures of formal education were also raised.
Conclusions: Cambridge Science Festival prior research
• This evaluation research study has pointed to a number of important impacts fostered by the Cambridge Science Festival.
• Some of the complexities of the ways in which festival attendees approach this informal engagement context can be seen in the focus group results, available in the full report.
Cambridge Science Festival 2011 Evaluation Approach
• On-site survey form distributed, collected by organisers and volunteer staff
• Follow-up web survey to collect more detailed individual views
• Focus group
Evaluation Data Collection • Saturation sampling approach (gather data
from as many people as possible: try not to discriminate).
• Focus on people in queues, waiting for lecture to start, etc.).
• Welcome to have a read for yourself when you collect them up. Please supply your data to the festival as a whole though for larger-scale analysis.
• Invite visitors to contribute their feedback. State that we are very interested to learn what they think and we carefully analyse responses to improve the festival each year.
• Discourage joint completion of evaluation forms. (These are designed for individuals to complete).
• If they ask, say the evaluation report will be published on the CSF website a few months after the festival
Evaluation Data Collection
Thank you for your help – your feedback will help us to improve and develop the Festival. Your anonymous responses about the festival will be used for evaluation and research purposes only.
1. Which event have you just attended? ___________________________ Date___________ 2. What was your impression of the event you just attended? (Please tick) Very Good Good Neutral Poor Very Poor No Opinion 3. What comments do you have about the event you just attended? 4. What is your overall impression of the Cambridge Science Festival? (Please tick) Very Good Good Neutral Poor Very Poor No Opinion 5. What was the most successful element of the Festival for you (and why)? 6. What was the least successful element of the Festival for you (and why)? To what extent do you agree or disagree with the following statements? 7. I felt I was able to participate actively in the Science Festival.
Strongly Disagree Disagree Neutral Agree Strongly Agree No Opinion 8. I am interested in further investigating scientific topics I encountered at the Festival.
Strongly Disagree Disagree Neutral Agree Strongly Agree No Opinion 9. What, if anything, do you feel you have gained from taking part in the Festival?
On-site evaluation form: p.1 (pt.2)
Thank you for your help – your feedback will help us to improve and develop the Festival. Your anonymous responses about the festival will be used for evaluation and research purposes only.
1. Which event have you just attended? ___________________________ Date___________ 2. What was your impression of the event you just attended? (Please tick) Very Good Good Neutral Poor Very Poor No Opinion 3. What comments do you have about the event you just attended? 4. What is your overall impression of the Cambridge Science Festival? (Please tick) Very Good Good Neutral Poor Very Poor No Opinion 5. What was the most successful element of the Festival for you (and why)? 6. What was the least successful element of the Festival for you (and why)? To what extent do you agree or disagree with the following statements? 7. I felt I was able to participate actively in the Science Festival.
Strongly Disagree Disagree Neutral Agree Strongly Agree No Opinion 8. I am interested in further investigating scientific topics I encountered at the Festival.
Strongly Disagree Disagree Neutral Agree Strongly Agree No Opinion 9. What, if anything, do you feel you have gained from taking part in the Festival?
On-site evaluation form: p.1 (pt.2)
10. How would you rate your general level of interest in science outside of the Science Festival?
Strongly interested
Interested Neutral Not interested
Strongly interested
No Opinion / Not Sure
11. What is the highest level of education you have completed?
GCSE equivalent or less A-level or equivalent First Degree Postgraduate Degree
12. If you are willing, please tell us your postcode ______________________ 13. Please indicate the age and genders of all people in your party:
0-15 yrs 16 -25 yrs 26-39 yrs 40-64 yrs 65 yrs + No. of females
No. of males
14. Would you describe yourself or anyone in the group you visited the Festival with as disabled? Yes o No o 15. Please indicate the ethnic origins of all people in your party:
Asian or Asian British
Black or Black British
Chinese Mixed White Other
Number of people
On-site evaluation form: p.2 (pt.1)
On-site evaluation form: p.2 (pt.2) 16. How did you find out about the Festival? (Please tick all that apply) Already on mailing list o Poster o Local press o Local interest group o
Work o Library o School o Family / friend o
Online web page o Word of mouth o Social media o
17. Please give your email address to be included in our emailing list for future public events at the University of Cambridge: ………………………………………………………………………………………. 18. Would you be willing to participate in further online evaluation of the Cambridge Science Festival? Yes o No o If yes, please specify contact details if not provided above: ……………………………………………. Your email address will be stored in accordance with the Data Protection Act 1998. We will only contact you 1) regarding the University of Cambridge and Cambridge College public events, if you have indicated you would like to receive updates 2) for evaluation and research purposes, if you have indicated you are willing to participate in further evaluation of the festival. We will not share or transfer the information you have provided for any other purpose. Please hand this form to a steward or send it to: Festivals and Outreach Assistant, Office of External Affairs and Communications, The Pitt Building, Trumpington Street, Cambridge, CB2 1RP
• Closed-ended, scale items intended to gain comparable snapshot of visitors’ views.
• Quite general, open-ended questions designed to allow visitors to provide guidance about what they think is most important to feed back.
Analysing Evaluation Data: first thoughts
Analysing Evaluation Data: first thoughts
• Questions 1, 2 and 3 about the event just attended will be of most immediate interest to presenters and organisers
• Other questions may be of interest once placed within the context of overall trends in festival visitors’ responses. – e.g., Is satisfaction with the festival overall
higher for attendees at particular events?)
Calculating mean satisfaction with event
• Previous year’s mean satisfaction ratings overall are around ‘4.5’. Calculate your event’s mean response by:
1. Apply values to the different response options (‘1’ for very poor to ‘5’ for very good)
2. Sum all values for your sample. 3. Divide by the number of respondents.
Analysing Evaluation Results: Qualitative Survey Responses
• Qualitative responses can be analysed as well, either quantitatively or qualitatively: – Qual.: Organise responses into categories
and themes, identifying representative examples of each category or theme.
– Quant.: Can identify ‘codes’ or categories, then apply these deductively to the different responses to assess the prevalence of a particular category of response.
Impact Evalua'on
Good Impact Evaluation • Is SYSTEMATIC • Tells you how and why particular aspects of
activity are effective – NOT a binary ‘good’/‘bad’ or ‘successful’/‘unsuccessful’ result.
• You don’t learn anything from binary results – A ‘successful’ project can always develop the good
aspects of their practice further – There will be specific aspects of an ‘unsuccessful’
project or method that were ineffective (and should be avoided in future projects)
– Either way, it is important to have some specifics!
Evaluating Impacts: Context
• Full-scale evaluation research unrealistic as a continuous activity for most institutions.
– May need to bring in external expertise – May need to develop additional training / skills in-house
Recommended approach: 1. At Minimum: Engage in Reflective Practice and use Audience Feedback Forms (Sampling!)
Evaluating Impacts: Context
2. At minimum: Specify intended outcomes and specific connections between content and delivery approach and these outcomes. (check against current research / theory and other practitioners’ evaluations)
Evaluating Impacts: Context
§ If possible, formative evaluation before full public rollout of an activity. Ø e.g. focus groups, other pre-testing of ideas
§ If possible, summative evaluation to address 'how' and 'why' an activity worked well / poorly. Ø ‘How’ and ‘why’ hold implications for other
activities and for other practitioners (share!)
Evaluation Research • Evalua-on = sub-‐category of 'social research' (thus all principles of social research apply)
• Dis-nguishing feature of evalua-on: Focus on objec'ves / claimed outcomes (prac55oners must specify these outcomes)
• In order to evaluate them, prac--oner objec-ves should be Specific, Measurable, Achievable, Realis-c and Targeted.
• The evalua-on process begins with concepts / ideas that a prac--oner is aiming to deliver or communicate.
• Evalua-on measures the degree to which these objec-ves (e.g. 'learning') are realized.
Transla'ng Prac''oner Aims into Evalua'on Research Ques'ons
The Evalua'on Process: 1st steps • Vital process of translating abstract / general
ideas / concepts (e.g. scientific literacy) into concrete, measurable variables.
• Easier said than done. • This is called ‘Operationalization’ –
consider: • How would you know that a particular kind of change has happened? • Think about what people would say or do if you were successful in achieving your aims.
Definition: Evaluation Research Design
Process of choosing how to most effectively assess intended outcomes from your activity / intervention.
Evaluation Design: Getting Started • matching goals that motivate activity with
evaluation methods for assessing goals. • Evaluation design all about making
choices. • To make a good choice, you need to know
(1) what your evaluation options are and (2) how to decide between those options.
• It is helpful to think of evaluation research methods as tools that offer a set of strengths/limitations that can be used to accomplish range of goals.
Research Design: Getting Started
Assessing Research Quality
• Allowing for Negative Findings Ø Can your hypotheses be shown to be
wrong with the kind of evidence you are collecting?
• Validity Ø What are you really measuring?
• Reliability
SUMMATIVE EVALUATION:
Sampling
Introduction to Sampling • Sometimes the whole population of interest will be
accessed (e.g. every member of your audience). • But most of the time this would be too difficult or time
consuming. • So we usually study just a sample of the cases that we
are interested in. (e.g. a few members of your audience)
• What is most important in selecting a sample is that it is representative of the population.
• When a sample is representative we can make statements / claims about the population based on the sample.
What is a Representative Sample?
• To be representative, the sample should accurately reflect the whole population of interest.
• We cannot fully know how to select a sample that is representative based on what people look like, etc.
• Therefore the best we can do is be sure that every member of the population has an equal chance of being included in the sample.
• The central principle here is random selection.
A Simple Random Sample
What is a Representative Sample?
• Some random samples are more complex: Ø For example, involving ‘clustering’ or ‘stratifying’.
• At minimum, should use systematic sampling (e.g. every 15th person)
Non-random Samples (less good)
• Types of Non-‐Probability Sample: • Convenience sampling • Snowball sampling • Quota sampling
• Since non-‐probability samples do not involve Equal Probability of Selec-on, cannot make accurate sta-s-cal statements / claims about whole popula-on.
SUMMATIVE EVALUATION: Reviewing the
Toolkit
The Toolkit • Quantitative Evaluation Methods
Are used to answer any counting related question: • How many? What proportion? What
percentage? Survey Research Structured Observation of Audiences / Visitors
• Qualitative Evaluation Methods Any study involving non-counting data (e.g.
words, drawings, etc.) – These can be converted into quantitative
data
Qualitative Research • Qualitative Interviews • Focus Groups
Data Analysis - Must be systematic to avoid tendency to select quotes based on personal bias and preferences. - Can convert qualitative data into quantitative data through content analysis
Qualitative Evaluation Design • Qualitative Evaluation Research typically
starts with observations – i.e. it is INDUCTIVE.
• These observations are then used to generate hypotheses about what is working and why.
• This process leads to evaluation research goals such as discovery and exploration.
Qualitative Evaluation Design • Inductive research purposes aimed at theory-
generation and discovery support an “emergent” approach to research design.
• Very good for formative evaluation to identify what is most likely to be effective with a given audience.
• Very good for exploratory evaluation research, when you don’t know much about audience outcomes.
Measurement • Operational definitions are required for the more
abstract concepts: – A key issue is what will be captured on a particular
measure (i.e. ‘what counts?’)
– Measurement error is an issue. (i.e. error due to measurement approach/tool)
(e.g. important to directly measure relevant variables such as knowledge, e.g. before/after)
OVER TO YOU!
• What do you think we should be evalua-ng in fes-val-‐based public engagement? (e.g. feedback versus impact? What kinds of feedback / impact?)
Discuss in groups of about 3 using your preliminary defini7ons as a point of discussion.
Survey Design Flaws (Avoid!) • Construct Validity: The soundness of the measures as
indicators of the constructs purported to be examined by the investigators
• Non-specific effects: Improvements or changes from effects not specific to the factor or treatment under study
• Novelty effect: General energizing and uplifting effects of a new, exciting experience
• Confounding Variables: Failure to take into account the fact that the experience under study may include more than one component that affects outcome
Survey Design Flaws (Avoid!) continued
• Demand Characteristics: The tendency of participants to alter their responses in accord with what they believe to be the researchers’ hypothesis
• Experimenter expectancy effect: • The tendency of investigators to unintentionally bias the results in accordance with their hypotheses
Survey Design Flaws (Avoid!) continued
• Response Bias: A bias in subject responding due to the test instrument rather than the subjects’ actual beliefs
• Sampling Bias due to non-random sampling: Unintentional sampling of subjects that introduces systematic error or bias into the results
Survey Design Flaws (Avoid!) continued
• Acquiescence Bias: A bias from respondents’ tendency to agree with statements
à Control for this by including reverse wording items on agreement scales
“Put me down for whoever comes
out ahead in your poll”.
Survey Design Issues: Self-Report – what is it good for?
Advantage Offers direct access to respondents’ views Disadvantages Validity issues such as: • Response biases such as social desirability • Lack of (self-)conscious awareness • Attributional biases
Principles of question design • Survey question responses
need to be: Exhaustive – that everyone fits
into one category Exclusive – so that everyone fits
into only one category (unless specifically required to ‘tick as many as apply).
Unambiguous – so that they mean the same to everyone and all responses are comparable.
Questions • Beware of social
desirability bias: Phrase sensitive questions impartially so respondent can answer truthfully without feeling stigmatised
Questions • Ensure you don’t have any double-barrelled
questions (e.g., ‘What interested you in visiting the zoo this year and last year?’)
• Avoid Leading Questions!!! – Leading questions such as “Do you agree that
Durrell is doing important work to save animals from extinction?”
Survey Design Quality
• Allow for Negative Findings
• Validity
• Reliability
Piloting your Survey • First, you can probe in-
depth with pilot respondents about some particular questions
• Second, the survey in its entirety should be administered to pilot respondents.
Piloting your own Questionnaire
• Exercise: Design one survey question + response options related to your project as an individual then try out the questions in small group (3 people) and get feedback (mainly at the first level of pilot survey feedback). – Report back on what kinds of changes were
recommended
Top Tips • Evaluation requires very clear, specific and
measurable objectives Beware of ‘Raising Awareness’ and ‘Inspiring
Interest’! • Quantitative Methods Get the design right at the beginning! (e.g. pilot
testing) • Sampling Equal probability of selection is optimal!
Top Tips Surveys Think carefully about questions and limit self-report! Evaluation Design Avoid positive bias and allow for possibility of negative outcomes.
Conclusion
• Based on all the research I have done, I believe the top impact of fes-val-‐based public engagement to be CURIOSITY
• The key then is to capitalize on this impact by having good systems in place for extending impact beyond the physical and temporal confines of the fes-val.
Further Resources on Public Engagement and Informal Learning Impact Evaluation
EVALUATING IMPACTS OF PUBLIC ENGAGEMENT AND NON-‐FORMAL LEARNING SEMINAR / TRAINING SERIES ONLINE!
107
Festival-‐based Public Engagement
Dr Eric Jensen Assistant Professor of Sociology University of Warwick [email protected]
108
