View
18
Download
1
Category
Preview:
DESCRIPTION
How to Assess Student Learning in Arts Partnerships Part II: Survey Research Revised April 2, 2012 Mary Campbell-Zopf, Ohio Arts Council Mary.Campbell-Zopf@oac.state.oh.us Craig Dreeszen Ph.D., Dreeszen & Associates craig@dreeszen.com. Your Instructors. Our intended learning outcomes. - PowerPoint PPT Presentation
Citation preview
How to Assess Student Learning in Arts Partnerships
Part II: Survey Research
Revised April 2, 2012
Mary Campbell-Zopf, Ohio Arts CouncilMary.Campbell-Zopf@oac.state.oh.us
Craig Dreeszen Ph.D., Dreeszen & Associatescraig@dreeszen.com
Mary Campbell-Zopf is deputy director at the Ohio Arts Council and has been with the agency since 1989. Between 1989 and 2011, Ms. Campbell-Zopf helped secure $12 million for state-level arts, arts education and international programming. She also played a central role in an agency-wide effort to expand the OAC’s International Program through a $1.2 million U.S. Department of Education grant, for which she served as the evaluation manager and participated in professional
development activities with educators and arts administrators in Chile. In 2006, a major milestone in international work was the publishing of The Appreciative Journey: A Guide to Developing International Cultural Exchanges. Campbell-Zopf oversaw the guide’s development and writing and was a major contributor to its content. Ms. Campbell-Zopf has an enduring interest in strategic planning and program evaluation, which led to the publishing of the seven-volume series, Focusing the Light: the Art and Practice of Planning in 2009. Ms. Campbell-Zopf has also been active at the state and national levels in arts education, including serving on numerous state advisory committees for content standards, curriculum, learner assessment, as well as on national advisory committees for the U.S. Department of Education, the National Endowment for the Arts, the National Assembly of State Arts Agencies and the Arts Education Partnership.
Craig Dreeszen directs Dreeszen & Associates, a consulting firm in Northampton Massachusetts. He is a planner, educator, program evaluator, and organizational development consultant. For twelve years he directed the Arts Extension Service at the University of Massachusetts. Since 1986 he has evaluated 30 programs for foundations and state arts agencies and provided strategic
planning support to over 50 public and nonprofit organizations, mostly community based cultural organizations and the agencies that fund and support them.
Dreeszen is a contributing editor of Fundamentals of Arts Management, 5th Edition, author of the chapters, “Program Evaluation,” “Strategic Planning,” and “Board Development.” He wrote the online courses “Strategic Planning” and “Outcome Based Program Evaluation,” the “Learning Partnerships Planning and Evaluation Workbook” and other arts management books and articles. Dreeszen earned his Ph.D. in regional planning and M.A. in organizational development from the University of Massachusetts Amherst.
Your Instructors
Our intended learning outcomes
• Understand the use and limitations of written assessment instruments
• Write effective survey questions
• Understand survey distribution options
• Interpret survey data– Qualitative data analysis– Quantitative data analysis
• Report survey findings
Written Assessment Instruments
• Evaluation forms
• Tests– Pre- and post-tests– Standardized tests
• Surveys
This workshop had relevance for my understanding of seeing art materials as a language. Great relevance, Relevance, little relevance, None at all This workshop prepared me think about ways I might use materials in my classroom. Very prepared, Prepared, Somewhat prepared, Need more instruction Which of the following significant skills were addressed in this workshop? Observation skills, Listening, Evidentiary reasoning, Multiple perspectives
How do you envision yourself using this material in your classroom?
Using Written Assessments
Good reasons to use:
•Efficient•May reach large samples
•Test qualitative conclusions from small samples•Provide quantitative data
•Enables reticent people to participate
May not be appropriate if:
•Uncertain what to ask
•Questions are ambiguous
•Questions are complex
•Respondents not literate
Two Sequencing Options
Start with interviews or focus groups:•Test questions•Learn likely responses•Explore conclusions•Create fixed-response questions•Distribute to larger sample
Start with survey:•Explore ambiguous survey findings in interviews or focus groups•Investigate reasons for survey findings
Survey Steps
04/19/23
• Decide what data is needed
• Design questionnaire• Select sample• Acquire lists• Pretest• Distribute and remind• Analyze and report
Determine Needed Data
• Learning outcomes– To what extent were did
participants learn what was intended
– Unintended outcomes
• Program satisfaction– What liked best– What needs
improvement
• Recommendations• Marketing advice• Advocacy evidence
Write Questions
Open-ended questions•Short answer•Long answer•Prompted
Fixed-response questions•Yes/No•Rating Scale•Choice (one or multiple)•Matrix rated•Ranked
Ask basic demographic questions too
Review Sampling
• Select representative sample of larger population
• Or survey entire population
• Acquire mailing or email lists
Open-ended Questions
Fixed-Response Questions[Replace sample with OAC AP grantee examples]
Practice Writing Questions
• Try writing an open-ended question
• Then, write a fixed-response question and response choices
Paper and Online Surveys
• Mailed surveys– Envelopes– Address labels– Postage– Cover letter– Survey– Return envelope and
postage
• Handout surveys
• Online surveys• SurveyMonkey.com• Zoomerang.com
– Email invitation– Survey– May provide links to
survey results and organization website
– Survey closed notice
Online Survey Features
• Screening questions• Skip logic
• Quick response• Inexpensive• Track responses• Send reminders to
unresponsive email addresses
Test Survey
• Send early draft survey to partners
• Revise• Send next draft to
small sample of your population
• Ask for feedback• Look for ambiguity
or misunderstanding• Revise final survey
Distribute Surveys
Paper surveys•Address and mail•Remind (newsletter, announcements…)
Handout surveys•Distribute and collect
Online surveys•Time right•Email invitation with link to survey URL•Web site links•Social media links•Track responses•Send reminders
Statistical SignificanceResponse Rate
Number of responses divided by number of surveys distributed
90 responses200 surveys sent = 45% response rate
Deduct undeliverable surveys
Margin of error
When sampling populations, larger samples have smaller margin of errorUse online calculator
If you survey 200 students and 90 respond, margin of error = 7.7If 120 respond, margin of error is 5.7
Some apparent results may be due to chance – not statistically significant
No margin of error if entire population sampled, e.g., all participants in your program
Analysis of data
04/19/23
• Reduce large amount of data• Display it in notes, tables, or charts• Make sense of it
– Statistical analysis of numbers– Content analysis of words
Making sense of words
04/19/23
• For each question, read all answers• Look for patterns and clusters• Code similar statements• Count clusters of similar statements• Select typical quotes• Write summary• Do the same for next question
Making sense of numbers
04/19/23
• Count• Average• Note patterns and clusters• Note most frequent responses• Cross tabulate to compare variables• Check statistical significance• Write summary
Report Results with Text
Executive summaryInterpret results
Key-point summaryHeadingsExplanationDataQuotes
Appendix with detailed data and methods
“Findings: The Arts Partnership Program works as intended. The Arts Partnership program furthers the OAC strategic plan, especially Goal 3, to “Help citizens of all ages learn and thrive through the arts.”
Quantitative evidence cited Just over half of AP grantees (55%) cited quantitative evidence to support their evaluation results in the final report.”
Display Results with Graphics
• Data tables• Bar charts• Histograms• Pie charts• Line charts• Scatter plots
Report Results
Report intended learning objectives and actual learning outcomes
Summarize activities
Suggest best practices and needed improvements
Acknowledge funders and partners
Provide back-up data
Sample OLGA report form
Use Results• Draw conclusions
• Report to stakeholders
• Use data to improve programs and account to funders authorizers
Recommended