14
Funded by: NOVEMBER 2007 Storytelling Approaches to Program Evaluation: An Introduction PICTURES WRITING THEATER

Storytelling Approaches to Program Evaluation · STORYTELLING APPROACHES TO PROGRAM EVALUATION: AN INTRODUCTION There is no “one-size-fits-all” approach to program evaluation

  • Upload
    doananh

  • View
    216

  • Download
    0

Embed Size (px)

Citation preview

Evaluatingtheimpactofprogramsaimedatsocialchange

andinvolvingdiverseconstituentsdemandsespecially

innovativeandculturallyappropriateapproaches.

Funded by:

NOVEMBER2007

Storytelling Approachesto Program Evaluation:An Introduction

WeiserLithoWe

iser Litho

Weiser

Litho

WeiserLitho

PICTURES WRITING THEATER

1000 North Alameda StreetLos Angeles, CA 90012800.449.4149www.calendow.org© 2007 The California Endowment

CPA/StorytellingTCE 1019-07

Evaluateyour programsby storytelling.

Everyone has a storyto tell, but many people

need help to gettheir stories out.

STORYTELLING APPROACHES TO PROGRAM EVALUATION: AN INTRODUCTION

Adapted by Sylvia Sukop from an original report by Joseph Tobin and Gustavo E. Fischman.

WeiserLitho

Weiser

Litho

WeiserLitho

We

iser Litho

THEATERWRITINGPICTURES

This document provides apractical introduction to storytelling

approaches being successfully appliedin grassroots program evaluation.

These approaches can be used in almost any setting, with any number

of participants, in any language. They are flexible and can be creatively

tailored to the needs and styles of your particular organization or project.

Table of Contents

Introduction 2

Gathering Stories: Basic Interviewing Approaches 2Story Circle 3Storytelling Interviews 3Oral History Interviews 4Institutional Memory 4

Getting Creative: Documenting Stories in Pictures, Writing and Theater 4Visual Documentation 4Scrapbooking and Story-quilting 5Story Theater 5Case Studies 6Vignettes: Short Stories as Illustration 6

Five Tips for Gathering and Organizing Stories 6

Evaluation and Beyond: Using the Stories you Collect 7Public Relations and Fund Raising 7Policy Advocacy 8

IntroductionTraditional evaluation methodologies–whichrely on formal questionnaires and statisticaldata that can be scientifically analyzed–failto give program participants a chance to telltheir own stories in their own ways.

Storytelling is an ancient tradition thathas, over time and across cultures, servedmany different purposes, from education

and the transmission of values topolitical mobilization and pure

entertainment. It can takemany forms, from oral andwritten narratives to gesture,movement, art, music,movies and more.

Storytelling is a powerful mode ofhuman expression and a sophisticated

form of “meaning-making.” It beginswith a storyteller, a singular experience,a unique point of view.

This is one reason that storytelling hasbecome such a useful tool in programevaluation: It accommodates diverse voicesand perspectives, while making the most ofthe particular resources and ways of learningreadily available in your program. Unlike atraditional evaluation approach that isimposed from outside, the storytellingapproach emerges organically from within yourown organization, projects and participants.

Gathering Stories: BasicInterviewing ApproachesThese basic approaches use a simpleinterview or question-and-answer format.There should always be a clearly designatedinterviewer or facilitator guiding theconversation. Ideally the conversation willbe recorded (audio or video) or detailednotes will be taken. Using these recordingsand notes, you will be able to share thestories you have gathered with others. Before

STORYTELLING APPROACHES TO PROGRAM EVALUATION: AN INTRODUCTION

There is no “one-size-fits-all”approach to program evaluation.Evaluating the impact of programs aimed

at social change and involving diverse

constituents demands especially innovative

and culturally appropriate approaches.

2

Storytelling is proving to bea highly effective, culturallyappropriate approach tograssroots program evaluationthat gives voice to individualand collective experience.

you get started, be sure to read “Tips forGathering and Organizing Stories” (page 6).

Story CircleAt the end of a staff meeting or sessiongo around the group in a circle asking foreach participant to share a story. To getthe conversation going, the intervieweror facilitator can ask an open-endedquestion like, “What have you learnedfrom being part of this program?” or“How has participation in this programchanged your life?”

At the end of the session ask the groupto summarize the most important lessonslearned based on the stories told in the circle,and be sure to record or write these down.You can implement the story circle everythree to six months to assess your programprogress and involve diverse stakeholders.

Storytelling InterviewsEveryone has a story to tell, but manypeople need help to get their stories out.One or more staff members can sit downwith program participants one-on-one orin small groups to gather their stories.The interviewer can begin by explainingthe purpose of the meeting, by sayingsomething like, “Our agency wants tohear from our participants abouthow our programs are helpingthem. Hearing your experiencewill help us know how we are doingand give us ideas for improvement.”

The interviewer shouldask questions to get theperson talking, but notinterrupt unnecessarily.The interviewer can askthe participant to describe:

• The problems he/she is facing thatled him/her to come to the agency;

GATHERING STORIES: BASIC INTERVIEWING APPROACHES 3

Why Storytelling Works• Storytelling values and respects diverse ways of knowing and learning.

It is empowering and participatory, and is based on popular knowledge.

• Stories can be used effectively alongside statistics and surveys.Including stories in your program evaluations puts a face on the facts and figures,and helps you figure out what’s working, what’s not, and why.

• Stories speak to a broad audience.Including your stakeholders’ voices and perspectives can help you communicateto your stakeholders, your funders, and the larger community what you areaccomplishing and why your program is so important.

Including stories inyour program evaluations

puts a face on the factsand figures, and helps youfigure out what’s working,

what’s not and why.

• The activities in whichhe/she has participatedor the services he/she

has been receiving;

• How these servicesare making a difference

in his/her life.

The interviews should be taped and thentranscribed and edited, cutting out commentsthat are off topic and focusing on the mostcompelling stories told by the participants.

Oral History InterviewsIncluding the voices of the people in acommunity who have familiarity with itshistory is a way to gather information onhow it has changed over time. It can help todescribe the historical, social and culturalcontext of your program as well as how thecommunity and the program have evolvedover time. In addition, it can help youidentify community cultural events thatexisted in the past that can becomeresources for the future.

Oral history interviews may also be usedto collect information on the lives of yourstakeholders before you initiate your programactivities and then after your program has

concluded, to see how their livesmay have changed as a result

of their participation.

Institutional MemoryCollecting and cataloguingstories not only from

participants but also fromstaff can be an excellent way

of recording the history and charting theprogress of a program and the growth of anorganization. For example, staff memberscan be asked to tell stories about how theprogram has evolved over time. Thesestories can be recorded and transcribed.This process can provide program staff agrowing archive of stories that can beanalyzed to chart growth in individualprogram participants, staff members, theprogram and the organization as a whole.

Getting Creative:Documenting Storiesin Pictures, Writingand TheaterVisual DocumentationStories can be told in pictures as well aswords. Program staff and participants canbe given cameras and invited to create avisual record of their participation in theprogram. These photos can be organizedinto a montage on a poster or presented ina digital slide show to demonstrate programactivities and growth. This participatoryevaluation approach is called “Photovoice.”It utilizes photographs taken by programstakeholders to enhance need assessments,discussions and reflection, gather data,promote dialogue, conduct participatoryevaluations and communicate results withvarious audiences, including policymakers.Similar approaches can be done with videocameras, and when shared through computers,the approach is called “Digital Storytelling.”With participants’ permission, it is bestto save the stories as digital files. Stories

STORYTELLING APPROACHES TO PROGRAM EVALUATION: AN INTRODUCTION4

See “Tell Us YourStory” HandoutThis handout can be used toguide your interview and evenshared with participants inadvance, if appropriate.

can then be selected to depict specificprogram impacts, successes and challenges,including diverse views and perspectiveson how the program has impacted the livesof involved stakeholders and, possibly, thelives of their communities.

Scrapbooking and Story-quiltingCollecting artifacts and images and placingthem in an album to document programactivities, process and outcomes is a veryold practice that has recently become apopular hobby. Participants and staff can beasked to keep a scrapbook throughout theirparticipation in a program to document theiractivities, challenges, accomplishmentsand growth. These scrapbooks can thenbe used as a form of portfolio assessmentin program evaluation.

Quilting is another folk craft that can beused for telling stories. For example, in aquilt-making activity, each participant cancontribute a square that contains a personalreflection on the program, its process andoutcomes. The participants can discuss howto organize their squares to make a quiltthat tells the story of their individual andcollective experience. Story-quilting canbe used to assess and promote communitypride, by describing how the community hascome together to accomplish activities andmake changes that resulted in improvedcommunity life and a sense of belonging.

Story TheaterTheater has been used effectively in the fieldsof health and mental health services and insocial movements. Theater can dramatizecommunity concerns and demonstratepossible solutions by ordinary people whoare trained neither as playwrights nor actors.Community members are asked to developstories that they turn into scripts andperform, either for themselves or for thepublic. Most often, the script emergesfrom role-play exercises.

Elements of story theater can be adapted intoprogram evaluation. For example, every monthor two, you can ask your program participantsto role-play the process, challenges andaccomplishments, as well as outcomes ofa program. If you tape the role plays (audioor video) over the course of a year, theyprovide a running record of growth.

Story theater can also be anexcellent means to inform thecommunity about your programevaluation results. For example,an organization in Californiacalled Lideres Campesinas(Farmworker Women Leaders)writes short plays in whichthey describe the livesof women who haveexperienced domesticviolence and howtheir lives changed asa result of becomingactive members ofthe organization.

GETTING CREATIVE: DOCUMENTING STORIES IN PICTURES, WRITING AND THEATER 5

Online ResourcesPhotovoice

www.photovoice.com

The Center forDigital Storytelling

www.storycenter.org

Case StudiesCase studies are more in-depth,written stories of the livesof your stakeholders. Theymay also depict the places

where they live, where yourorganization is located, and how

community life and other factorssuch as the environment and the social

determinants of health (e.g., employment,race and housing) interact with the processesand outcomes that emerge as a result ofyour stakeholders’ participation in yourprogram. A case study may include:

• Program participants’ biographicalinformation;

• Their reasons for being involved;

• The services they received or theactivities they participated in; and

• Challenges, successes and outcomes.

Vignettes: Short Storiesas IllustrationStories are used to illustrate andcontextualize the points you make

with facts and numbers. For example,after presenting statistical information

on the activities that you implementedin the past year, you can add a box thatcontains an account told by a staff memberor program participant that describes aparticular event that supports the statisticaldata. Vignettes are strong personal statementsthat give life–a human face–to collected data.

Five Tips for Gatheringand Organizing Stories1. Be consistent and systematic.

Like keeping a diary, you need to have aplace to record your stories; you need toallow time to write them down and youneed to keep at it. Do a little at a timerather than waiting until a lot of timehas passed and trying to remember andrecord weeks or months of stories all atonce. It’s easier to record stories whenthey are fresh in your mind than to goback and re-construct them.

2. Designate Storytellersand Story-collectors.Everyone has a story to tell, but noteveryone is a good storyteller. Identifythe good storytellers among your staffand program participants and encouragethem to tell their stories and also tohelp other, less skilled storytellers, gettheir stories out. You also need toidentify someone who takes primaryresponsibility for story-collecting.By story-collecting, we mean recording,writing and transcribing stories aboutyour program. This takes time and energy.You may want to train a staff memberto be the designated story-collectorfor your agency or you may want torotate the responsibility each monthor for each event.

STORYTELLING APPROACHES TO PROGRAM EVALUATION: AN INTRODUCTION6

Vignettes are strongpersonal statementsthat give life–a humanface–to collected data.

3. Be strategic in your choice of storiesyou include in a program evaluation.Everybody has a story to tell, but thisdoes not mean that every story told toyou by staff members and programparticipants needs to end up in yourreport. Strategically select the storiesthat are most appropriate for the taskat hand—and these may not alwaysbe success stories! Certain stories mayillustrate problems and challengesencountered by your stakeholders;others may demonstrate your program’ssuccessful interventions or barriers tosuccess; and still others may dramatizethe particular needs, assets and resourcesof the communities you serve.

4. Stories do not tell “the whole story.”It is important that you complement thestories you include in your evaluationreport with other sources of information.Stories should be combined with surveys,focus groups, observations and othermethods of evaluation. Providingmultiple forms of data and includingthe perspectives of the full range ofyour participants will enhance thequality of your program evaluationas well as the your stories’ impact.

5. Consider the ethical implications ofsharing personal and institutionalstories. Always ask participants forpermission before recording theirstories. Also get their approval toshare with others the stories they tellyou, explaining that their real namewill not be used in connection with

their stories. It is good practice to havewritten consent (see sample consent formincluded with this document) to assureconfidentiality in the use of personalnarratives; and to apply responsible andethical research practices, ensuring thatthe human rights, dignity and welfareof human subjects are protected.

Evaluation and Beyond:Using the Stories you CollectYour participants’ stories can serve as apowerful tool in service of your organization’sgoals. Think about the particular audiencethat you want to target, from fundersand policymakers to the media andthe general public.

Public Relationsand Fund RaisingPrograms can use storytellingfor publicity, fund raising andrecruitment of new participants,stakeholders, volunteersand staff. The same storiesyou use for evaluationcan be used to explainto potential participants,other community serviceorganizations, potential fundersand the mediawho you are, whatyou do, what youhave accomplished,successes and challenges,and what you need.

EVALUATION AND BEYOND: USING THE STORIES YOU COLLECT 7

It is good practice tohave written consent.

See Sample ConsentForm Included in

this Brochure.

Policy AdvocacyCompelling human

stories will strengthenyour policy advocacy

message. Stories of yourparticipants’ struggles and how

your program helped to mitigate them canbe used to raise awareness of your programimpacts. Stories of their resiliency and ofthe gains they make in your program canbe used not just for funding but alsofor social and systemic change.

How to Disseminate Your StoriesSelect statements from your stories that “speak to” specific audiences and presentthem in the appropriate format for that audience.

• For funders: Write an evaluation report with an executive summary that highlights,in brief, the main stories compiled in your report.

• For policymakers: Write policy briefs that incorporate vignettes of your mostcompelling stories along with salient facts and figures.

• For the media: Write a press release that includes one or more compelling storiesand includes direct quotes from participants.

• For community members and stakeholders: Weave stories and quotes into yourorganization’s publications such as newsletters, brochures and annual reports. Writean article in a popular community newspaper or a community newsletter. Use storytheater to dramatize community concerns and potential solutions.

STORYTELLING APPROACHES TO PROGRAM EVALUATION: AN INTRODUCTION8

Tell Us Your StoryHearing your experience will help us know howwe are doing and give us ideas for improvement.

First, tell us a little about yourself.

Please describe the problem or problems you were facing that first led you to come to us.

Describe some of the activities in which you have participated or the services you havebeen receiving.

Have these activities or services made a difference in your life?

Looking ahead, what are some of the new or continuing challenges that you expect toface? Do you plan to continue participating in our programs?

1

2

3

4

5

Consent FormWe would like to assess the impact of your participation in our activities and services by

making audio or video recordings of your stories. Through this assessment, our organization

will learn what worked and what did not work and why. This learning will help us improve

our programs and services.

We may also use the products of our assessments to share the stories we collect with our

community and other stakeholders or to advocate for our clients’ needs. We may use these

products to publish our accomplishments and to seek support for our ongoing services,

program(s), and/or activities. We may publish information, stories, photographs and artwork

through various media—including but not limited to print, electronic and audio-video

recordings. Examples of these publications may include but are not limited to newsletters,

brochures, reports, Web sites, slideshows, PowerPoint presentations, program photo albums

and/or audio-visual public service announcements.

No media shall be used for exploitation or promotion of activities unrelated to the mission

of our organization, .

Please complete and return this form to our office. In order to register for our program, theform must be completed, signed and returned regardless of your decision.

Office Use Only:

Signature Program Participant or Parent/Guardian

Your Name/Youth’s Name

I GIVE (organization’s name)permission to publish my name/child’s name, image, written work and/or artwork forthe purposes stated above.

I DO NOT GIVE permission to publish my name/child’s name, image, written workand/or artwork for the purposes stated above.

Date of Birth / / (month/date/year) Today’s Date / /

Print Name Program Participant or Parent/Guardian

STORYTELLING APPROACHES TO PROGRAM EVALUATION: AN INTRODUCTION

Adapted by Sylvia Sukop from an original report by Joseph Tobin and Gustavo E. Fischman.

WeiserLitho

Weiser

Litho

WeiserLitho

We

iser Litho

THEATERWRITINGPICTURES

This document provides apractical introduction to storytelling

approaches being successfully appliedin grassroots program evaluation.

These approaches can be used in almost any setting, with any number

of participants, in any language. They are flexible and can be creatively

tailored to the needs and styles of your particular organization or project.

Evaluating the impact of programs aimed at social change

and involving diverse constituents demands especially

innovative and culturally appropriate approaches.

Evaluatingtheimpactofprogramsaimedatsocialchange

andinvolvingdiverseconstituentsdemandsespecially

innovativeandculturallyappropriateapproaches.

Funded by:

NOVEMBER2007

Storytelling Approachesto Program Evaluation:An Introduction

WeiserLithoWe

iser Litho

Weiser

Litho

WeiserLitho

PICTURES WRITING THEATER

1000 North Alameda StreetLos Angeles, CA 90012800.449.4149www.calendow.org© 2007 The California Endowment

CPA/StorytellingTCE 1019-07

Evaluateyour programsby storytelling.

Everyone has a storyto tell, but many people

need help to gettheir stories out.