Upload
anthony-nichols
View
213
Download
0
Embed Size (px)
Citation preview
1
I N S T I T U T E I N S T I T U T E ofof M U S E U MM U S E U M and and L I B R A R Y S E R L I B R A R Y S E R V I C E SV I C E SWashington, DC , www.imls.govWashington, DC , www.imls.gov
Knowing What Audiences Learn:Knowing What Audiences Learn: Outcomes and Program PlanningOutcomes and Program Planning
Association of Children’sAssociation of Children’sMuseumsMuseums
20032003
2
OverviewOverview
We willWe will Distinguish Outcome-based Planning and Distinguish Outcome-based Planning and
Evaluation (“OBE”) from other kindsEvaluation (“OBE”) from other kinds
Talk about choosing outcomesTalk about choosing outcomes Talk about basic elements of a Logic Model (a Talk about basic elements of a Logic Model (a
project or program plan)project or program plan)
Talk about measuring outcomesTalk about measuring outcomes
Review and summarizeReview and summarize
3
Outcomes are achievements or changes inOutcomes are achievements or changes in
Skill –Skill – Painting, basketball Painting, basketball
Knowledge –Knowledge – Zoology, state capitols Zoology, state capitols
Behavior – Behavior – Visits museums, reads dailyVisits museums, reads daily
Attitude –Attitude – I like science, I love animalsI like science, I love animals
What are Outcomes?What are Outcomes?
4
Outcomes can be achievements or Outcomes can be achievements or changes inchanges in
Status –Status – In school, citizenIn school, citizen Life condition –Life condition – Overweight, healthyOverweight, healthy
What are Outcomes?What are Outcomes?
5
What are Outcomes?What are Outcomes?
Ella goes to the Zoo’s summer programElla goes to the Zoo’s summer programandand
(Immediate)(Immediate) Takes more interest in animalsTakes more interest in animals
(Medium)(Medium) Gets better biology gradesGets better biology grades
(Long term)(Long term) Becomes a veterinary technicianBecomes a veterinary technician
6
OutcomesOutcomes
Where do they come from?Where do they come from? Social Services and United WaySocial Services and United Way OMB: (GPRA) 1993 and PARTOMB: (GPRA) 1993 and PART Trends in funding Trends in funding The need to focus on audienceThe need to focus on audience The need to communicate museum valueThe need to communicate museum value
IMLSIMLS
7
Forms of EvaluationForms of Evaluation
Formative EvaluationFormative Evaluation
You want to know which brochure works You want to know which brochure works best to bring school tours to the zoo best to bring school tours to the zoo
Develop two or three prototypes and test Develop two or three prototypes and test on a sample of your target audience; on a sample of your target audience; decide which worked bestdecide which worked best
8
Process EvaluationProcess Evaluation
You want to see how efficiently your You want to see how efficiently your conservation education program is runconservation education program is run
You count the number of participants in You count the number of participants in program components, examine how the program components, examine how the components are delivered, how long they components are delivered, how long they take, and what it coststake, and what it costs
Forms of EvaluationForms of Evaluation
9
Forms of EvaluationForms of Evaluation
Summative EvaluationSummative Evaluation
You want to know if your exhibit program You want to know if your exhibit program made visitors aware of how they can made visitors aware of how they can protect their environmentprotect their environment
You run the program, then interview visitors You run the program, then interview visitors to learn how many might use “green” to learn how many might use “green” products more in futureproducts more in future
10
Forms of EvaluationForms of Evaluation
Impact Evaluation (aggregated Impact Evaluation (aggregated outcomes create impact)outcomes create impact)
You want to know if your programs helped You want to know if your programs helped increase recycling in your areaincrease recycling in your area
You find out the rate of recycling, run the You find out the rate of recycling, run the program, then see if the recycling rate has program, then see if the recycling rate has increasedincreased
11
Forms of EvaluationForms of Evaluation
Outcome-Based Program EvaluationOutcome-Based Program Evaluation
You want to know if your conservation You want to know if your conservation education program changes participants’ education program changes participants’ behaviors behaviors
You identify audience needs, plan services You identify audience needs, plan services to provide participant-oriented outcomes, to provide participant-oriented outcomes, and assess program results on a regular and assess program results on a regular basisbasis
12
What can OBE achieve?What can OBE achieve?
Increase program effectivenessIncrease program effectiveness
Provide a logical, focused framework Provide a logical, focused framework to guide program designto guide program design
Inform decision-making Inform decision-making
Document successes Document successes
Communicate program valueCommunicate program value
13
What are its limitations?What are its limitations?
This is a management toolThis is a management tool OBE may suggest cause and effect OBE may suggest cause and effect ––
it doesn't try to prove it it doesn't try to prove it
OBE settles for contribution, not OBE settles for contribution, not attributionattribution
OBE uses some of the same OBE uses some of the same methods as research, but methods as research, but ……
14
What are its limitations?What are its limitations?
OBE is not the same as RESEARCH:OBE is not the same as RESEARCH: Doesn’t try to compare your programDoesn’t try to compare your program
with another, similar programwith another, similar program
Doesn’t try to compare your methodsDoesn’t try to compare your methods with different methods to create with different methods to create a similar resulta similar result
Accepts “good enough” dataAccepts “good enough” data
15
How to develop an outcome-based How to develop an outcome-based program: program: ExampleExample
What need do you see?What need do you see? Many kids don’t understand basic science principles; Many kids don’t understand basic science principles;
many girls are intimidated by or uninterested in many girls are intimidated by or uninterested in sciencescience
Head Start is an opportunity for early science Head Start is an opportunity for early science experience, but many teachers have no science experience, but many teachers have no science education and don’t know how to make it funeducation and don’t know how to make it fun
Parents in at-risk families often have little science Parents in at-risk families often have little science knowledge, limited child-development skills, and knowledge, limited child-development skills, and limited reading skillslimited reading skills
16
Who has the need (target audience)?Who has the need (target audience)? Kids need science experienceKids need science experience
Head Start teachers could provide experience if Head Start teachers could provide experience if they had knowledge and skillsthey had knowledge and skills
Families might encourage science learning if we Families might encourage science learning if we made it easy and funmade it easy and fun
Who could you work with most easily Who could you work with most easily and effectively? and effectively? Head Start teachersHead Start teachers
How to develop an outcome-based How to develop an outcome-based program: program: ExampleExample
17
What could your museum do?What could your museum do? Provide science education resources for Head Start Provide science education resources for Head Start
Teachers – learning kits and training to use themTeachers – learning kits and training to use them
Provide take-home science kitsProvide take-home science kits
Partner with public libraries to provide age- and Partner with public libraries to provide age- and education-appropriate books on science subjectseducation-appropriate books on science subjects
MESS – Marvelous Explorations MESS – Marvelous Explorations through Science and Stories!through Science and Stories!
How to develop an outcome-based How to develop an outcome-based program: program: ExampleExample
18
Planning for OutcomesPlanning for Outcomes
Articulates process Articulates process andand outcomes of outcomes of your programyour program
Clarifies each element of your programClarifies each element of your program
Identifies indicators of change to be Identifies indicators of change to be measuredmeasured
Identifies targets for program impactIdentifies targets for program impact
The Logic ModelThe Logic Model
19
The amount of impact desired
When data is collected
The population to be measured
Sources of information about conditions being measured
Observable and measurable behaviors or conditions
Targets Targets (Goals)(Goals)
Data IntervalData Interval(When)(When)
Applied to Applied to (Who)(Who)
Data Data SourcesSourcesIndicatorsIndicators
What goes into outcomes planningWhat goes into outcomes planning
Mission + Influencers PROGRAM PURPOSE
OutputsActivities ServicesInputs Outcomes
Outcomes Logic ModelOutcomes Logic Model
20
InfluencersInfluencers
To change process To add partners
To change responsibilities
Is responsibility equal?
Which services produce outcomes?
Project Partners
Who does the program serve?
Is the program effective?
To fund program To increase funding To promote replication
Funders
Is the program meeting target audience needs?
To improve the program To end program To start another program
Your Organization
How they will How they will use informationuse information
What they wantWhat they wantto knowto know
InfluencerInfluencer
GoalWhenWhoData Source
IndicatorsOutcomesAudienceActivities Services
ProgramPurpose
Mission Influencers
Staff, participants, agencies, funding sources, Staff, participants, agencies, funding sources, competition, community groups, professional competition, community groups, professional associations associations
21
Logic Model: MissionLogic Model: Mission
How do the organization’s mission How do the organization’s mission and the program connect?and the program connect?
Is the link between mission and Is the link between mission and program purpose reasonable?program purpose reasonable?
What action words connect the What action words connect the mission to the program?mission to the program?
GoalWhenWhoData
Source IndicatorsOutcomesAudience
ActivitiesServices
ProgramPurposeMission Influencers
22
We do what, for whom, for what outcome or We do what, for whom, for what outcome or benefit?benefit?
Program PurposeProgram Purpose
The Science Museum, Public Libraries, and School District The Science Museum, Public Libraries, and School District will partner to provide science kits, training, reference will partner to provide science kits, training, reference materials, and take-home kits for Head Start teachers to materials, and take-home kits for Head Start teachers to increase teachers’ ability to make science learning fun, increase teachers’ ability to make science learning fun, increase kids’ science interest and knowledge, and engage increase kids’ science interest and knowledge, and engage parents in science playparents in science play
GoalWhenWhoData
Source IndicatorsOutcomesAudience
ActivitiesServices
ProgramPurpose
Mission Influencers
23
Logic Model: Activities/ServicesLogic Model: Activities/Services
Program Program activitiesactivities make it possible to deliver make it possible to deliver services to program participants services to program participants
(Most “activities” are managerial or administrative)(Most “activities” are managerial or administrative)
Program Program servicesservices engage participants and engage participants and produce outcomesproduce outcomes
Program Program services and activitiesservices and activities are driven by are driven by audience characteristicsaudience characteristics
GoalWhenWhoData
Source IndicatorsOutcomesAudienceActivities
ServicesProgramPurposeMission Influencers
24
ActivitiesActivities
Create kitsCreate kits Design trainingDesign training Workshop LogisticsWorkshop Logistics
ServicesServices Teacher trainingTeacher training MESS kits and booksMESS kits and books Take home kits Take home kits
A Museum /Library/School CollaborationA Museum /Library/School Collaboration
25
Target AudienceTarget Audience Head Start teachers in Alachua Co.Head Start teachers in Alachua Co. 3 to 5 year-olds from at risk families in Alachua3 to 5 year-olds from at risk families in Alachua Families in Alachua Co.Families in Alachua Co.
Knowing if you reaching the intended audience is Knowing if you reaching the intended audience is critical–you must decidecritical–you must decide
What information is critical to know?What information is critical to know? How will I get the information?How will I get the information? What are the confidentiality issues?What are the confidentiality issues?
GoalWhenWhoData
Source IndicatorsOutcomesAudienceActivitiesServices
ProgramPurposeMission Influencers
26
Logic Model: OutcomesLogic Model: Outcomes
OutcomesOutcomes
State how you expect people to benefit State how you expect people to benefit from your programfrom your program
State the intended results of your services State the intended results of your services
Describe changes in skills, attitudes, Describe changes in skills, attitudes, behaviors, knowledgebehaviors, knowledge
GoalWhenWhoData
Source IndicatorsOutcomesAudience
ActivitiesServices
ProgramPurposeMission Influencers
27
SamplesSamples
Outcome 1Outcome 1
Teachers will be more confident helping Teachers will be more confident helping students learn about sciencestudents learn about science
Outcome 2Outcome 2
Teachers will include science experiences Teachers will include science experiences in their classroomsin their classrooms
GoalWhenWhoData
Source IndicatorsOutcomesAudience
ActivitiesServices
ProgramPurposeMission Influencers
28
Logic Model: IndicatorsLogic Model: Indicators
IndicatorsIndicators
Are Are measurablemeasurable conditions or behaviors conditions or behaviors that can show an outcome was achievedthat can show an outcome was achieved
Say what you hope to see or knowSay what you hope to see or know
Are observable evidence of Are observable evidence of accomplishments, changes, or gainsaccomplishments, changes, or gains
““Stand for” the outcomeStand for” the outcome
GoalWhenWhoData
Source IndicatorsOutcomesAudienceActivitiesServices
ProgramPurposeMission Influencers
29
IndicatorsIndicators
The # and % of teachers who say they have at least “some” confidence on a 5-point scale (none, a little, some, a lot, complete confidence)
Teachers will be Teachers will be more confident more confident helping students helping students learn about sciencelearn about science
IndicatorsOutcome 1Outcome 1
GoalWhenWhoData
Source IndicatorsOutcomesAudienceActivitiesServices
ProgramPurposeMission Influencers
30
Logic Model: Data SourcesLogic Model: Data Sources
Data sources are tools and locations for Data sources are tools and locations for information that will show what happened information that will show what happened other forms of informationother forms of information
Pre/Post test scoresPre/Post test scores Program recordsProgram records Assessment reportsAssessment reports Records from other organizationsRecords from other organizations ObservationsObservations
GoalWhenWhoData
Source IndicatorsOutcomesAudience
ActivitiesServices
ProgramPurposeMission Influencers
31
Data SourcesData Sources
Teachersurveys
Data Sources
The # and % of teachers who say they have at least “some” confidence on a 5-point scale
Teachers will Teachers will be more be more confident confident helping helping students learn students learn about scienceabout science
IndicatorsIndicatorsOutcome 1Outcome 1
32
Logic Model: Applied to (Who)Logic Model: Applied to (Who)
Decide if you will measure all Decide if you will measure all participants, completers of the participants, completers of the program, or another subgroup program, or another subgroup
Special characteristics of the target Special characteristics of the target audience can further clarify the group audience can further clarify the group to be measuredto be measured
GoalWhenWhoData
Source IndicatorsOutcomesAudience
ActivitiesServices
ProgramPurposeMission Influencers
33
Applied toApplied to
All teachers N = 445
Applied to
Teacher surveys
Data SourcesData Sources
The # and % of teachers who say they have at least “some” confidence on a 5-point scale
Teachers Teachers will be more will be more confident confident helping helping students students learn about learn about sciencescience
IndicatorsIndicatorsOutcome 1Outcome 1
34
Logic Model: Data Intervals (When)Logic Model: Data Intervals (When)
Outcome information can be collected at specific Outcome information can be collected at specific intervals (end of program, every 6 months)intervals (end of program, every 6 months)
Data can be collected at the end of an activity or Data can be collected at the end of an activity or phase and as follow-up (after 3 workshops, after 2 phase and as follow-up (after 3 workshops, after 2 yearsyears
Data is usually collected at program start and end Data is usually collected at program start and end for comparison when increases in skill, behavior, for comparison when increases in skill, behavior, or knowledge are expectedor knowledge are expected
GoalWhenWhoData
Source IndicatorsOutcomesAudience
ActivitiesServices
ProgramPurposeMission Influencers
35
Data IntervalData Interval
At end of program year 1
Data Interval
All partici- pating teachers
N = 445
Applied toApplied to
Teacher survey
Data Data SourcesSources
The # and % of teachers who say they have at least “some” confidence on a 5-point scale
Teachers will Teachers will be more be more confident confident helping helping students students learn about learn about sciencescience
IndicatorsIndicatorsOutcome 1Outcome 1
36
Logic Model: Targets (Goals)Logic Model: Targets (Goals)
Targets (goals) are chosen expectations Targets (goals) are chosen expectations for outcomes a program hopes to achieve for outcomes a program hopes to achieve usually a percentage and/or numberusually a percentage and/or number
Influencer expectations affect targets, Influencer expectations affect targets, which can also be based on a program’s which can also be based on a program’s past performancepast performance
TargetWhenWhoData
Source IndicatorsOutcomesAudience
ActivitiesServices
ProgramPurposeMission Influencers
37
TargetTarget
At end of program year 1
Data Data IntervalInterval
All partici- pating teachers
N = 445
Applied Applied toto
Teacher surveys
Data Data SourcesSources
The # and % of teachers who say they have at least “some” confidence on a 5-point scale
Teachers Teachers will be will be more more confident confident helping helping students students learn learn about about sciencescience
IndicatorsIndicatorsOutcome 1Outcome 1
50%
Target
38
What should reports say?What should reports say?
We wanted to do what?We wanted to do what? We did what?We did what? So what?So what?
Above allAbove all Reports should meet influencer needs forReports should meet influencer needs for
information and program results information and program results
Reports should guide program staff toReports should guide program staff toimprove outcomes for program participantsimprove outcomes for program participants
39
What will it cost?What will it cost?
Assume 7-10% of program costs for Assume 7-10% of program costs for program evaluation (non-research)program evaluation (non-research)
What will you get?What will you get?
Low cost –Low cost – Know numbers, Know numbers, audience characteristics, and audience characteristics, and customer satisfactioncustomer satisfaction
Low to moderate cost –Low to moderate cost – Know Know changes in audience skills, changes in audience skills, knowledge, behaviors, and attitudesknowledge, behaviors, and attitudes
40
What will you get?What will you get?
Moderate to high cost –Moderate to high cost – Comparison groups can show Comparison groups can show attribution, short-term changes due attribution, short-term changes due to program to program
High cost –High cost – Long-term follow up, Long-term follow up, can attribute long-term changes to can attribute long-term changes to audience due to program services audience due to program services (research)(research)
41
Karen MotylewskiKaren MotylewskiInstitute of Museum and Library ServicesInstitute of Museum and Library Services1100 Pennsylvania Avenue, NW1100 Pennsylvania Avenue, NWWashington, DC 20506Washington, DC 20506
202-606-5551202-606-5551http://www.imls.govhttp://[email protected]@imls.gov
For more informationFor more information