Embed Size (px)
Citation preview
Theory of Change
Theory of Change and Evidence
1:00 PM – 2:45 PM
Wednesday, August 21, 2013
Broadway A/B
Presenters:
Cheryl Leménager – Serve Wisconsin
Sarah Hinzman – Volunteer Iowa
Mary Cannon – Serve Ohio
Theory of Change
AGENDA• Welcome• Pre-test • TOC• Break• Evidence• Q and A• Post-test
2
Theory of Change
Designing Effective Action for Change
How a Theory of Change helps you clarify the cause-and-effect
relationship at the heart of your program
Copyright © 2012 by JBS International, Inc.
Developed by JBS International for the Corporation for National & Community Service
Theory of Change
Learning Objectives
By the end of the module, you will be able to:
• Describe the benefits of a Theory of Change
• Define the three elements needed to construct a Theory of Change
• Identify how the Theory of Change informs the program design
4
Theory of Change
Theory of Change Elements
Community Problem/Need
Specific Intervention
IntendedOutcome
5
Theory of Change
Theory of Change Perspective
the most effective
set of activities for volunteers
and participants
Community Problem/Need
Specific Intervention
IntendedOutcome
6
• Looks at cause and effect relationships
Adapted from www.theoryofchange.org/about/
Theory of Change
Theory of Change Perspective
the most effective
set of activities for volunteers
and participants
Community Problem/Need
Specific Intervention
IntendedOutcome
7
• Identifies specific interventions to achieve the desired result
Adapted from www.theoryofchange.org/about/
Theory of Change
Theory of Change Perspective
the most effective
set of activities for volunteers
and participants
Community Problem/Need
Specific Intervention
IntendedOutcome
8
• Shifts thinking from “what we are doing” to focus on “what we want to achieve”
Adapted from www.theoryofchange.org/about/
Theory of Change
Theory of Change Elements
Community Problem/Need
Specific Intervention
Intended Outcome
Evidence• Guides choice of intervention
• Supports cause-effect relationship
Statistics documenting the
need
9
Theory of Change
Everyday Life Example
• I have strep throat (problem)
• I will take antibiotics (intervention)
• I will get better (outcome)
10
Strep throat Antibiotics Be Healthy
Evidence• Guides choice of intervention
• Supports cause-effect relationship
Theory of Change
Everyday Life Example
But which antibiotic(s) fight strep the best?
(Look at evidence to make the choice)
11
Evidence: Guides choice of intervention
Supports cause-effect relationship
Theory of Change
Example:Riverton Literacy Corps
Statistics on the number of students at below grade level in program’s service area; Research on why reading proficiency by 3rd grade is important.
Children reading below grade level
in 3rd grade
12
Community Problem/need
Theory of Change
Example:Riverton Literacy Corps
Community Problem/need
IntendedOutcome
Statistics on the number of students at below grade level in program’s service area; Research on why reading proficiency by 3rd grade is important.
Children reading below grade level
in 3rd grade
Students are able to read at 3rd grade level (as measured by 3rd grade reading exam)
13
Theory of Change
Example:Riverton Literacy Corps
Community Problem/need
Specific Intervention
IntendedOutcome
Evidence: Research on building block skills leading to reading proficiency. Research on design, frequency,
duration of tutoring sessions.
Statistics on the number of students at below grade level in program’s service area; Research on why reading proficiency by 3rd grade is important.
Children reading below grade level
in 3rd grade
Students are able to read at 3rd grade level (as measured by 3rd grade reading exam)
Individualized tutoring 3 times/week for 20 min on five “building block” literacy skills through reading, writing and verbal communication activities
14
Theory of Change
Theory of Change Elements
Community Problem/Need
Specific Intervention
IntendedOutcome
Evidence• Guides choice of intervention
• Supports cause-effect relationship
Statistics documenting the
problem/need
15
Community Problem/Need is the specific issue your project, with its specific intervention (service activity), is designed to address. What is the extent and severity of this need in the community?
Theory of Change
Theory of Change Elements
16
Statistics
documenting the problem/need
Theory of Change
Community Problem/Need
Data documenting problem/need should answer these questions:
• SCOPE: Who and how many are directly affected? How severe is this?
• SIGNIFICANCE: What makes this a compelling need? Is it likely to become worse? What will happen if we do nothing?
• CAUSE(S): Why does the need exist? How is it perpetuated?
17
Theory of Change
Documenting Community Need
Healthy Futures Program Example• Scope: National Survey of Children and Health
found childhood obesity in State X increased 23% between 2003 and 2007 – the 2nd fastest rate of increase in US (2010). CDC data show nearly one-third of children and teens are obese or overweight in Webb County (2009).
• Significance: • Obese children found to become obese adults• Links between childhood obesity and early
onset of cardiovascular disease, and Type II diabetes
• Webb County rising in state and national averages for cardiovascular disease
• Long term health care costs rising
• Causes: Diet, sedentary lifestyle18
Theory of Change
Intended Outcome
What change are you hoping to make related to the identified need?
Community Need
Specific Intervention
IntendedOutcome
Evidence• Guides choice of intervention
• Supports cause-effect relationship
Statistics documenting the
need
19
Theory of Change
Identifying the IntendedOutcome
Healthy Futures Program Example
Possible outcomes to measure:• Increased knowledge of what is “healthy food”
• More frequent choice of healthy foods to eat
• More frequent involvement in physical activity or exercise
• Improved physical condition
20
Theory of Change
Intervention
An intervention is the specific set of activities in which participants and volunteers will be engaged. What is the best way to achieve the intended outcome?
Community Need
Specific Intervention
IntendedOutcome
Evidence• Guides choice of intervention
• Supports cause-effect relationship
Statistics documenting the
need
21
Theory of Change
Intervention
An intervention is the specific set of activities in which participants and volunteers will be engaged. What is the best way to achieve the intended outcome?
Community Need
Specific Intervention
IntendedOutcome
Evidence• Guides choice of intervention
• Supports cause-effect relationship
Statistics documenting the
need
22
Theory of Change
Intervention
Describe the design and dosage of your intervention (service activity):
• Design (who does what with whom?)
• Dosage• Frequency (how many sessions a week?)• Intensity (length of each session)• Duration (how many total weeks of sessions?)
23
Theory of Change
Intervention Example
Healthy Futures Program Example• Design: national service participants implement
the Shape Up curriculum with economically disadvantaged urban girls ages 14-16 to increase physical activity (30 minutes/session) and educate them on healthy eating
• Frequency: twice a week afterschool
• Intensity: 60 minutes per session
• Duration: 12 weeks
24
Theory of Change
Testing Your Theory of Change:
IS YOUR THEORY OF CHANGE:
PLAUSIBLE: Does the logic of the model seem correct: “if we do these things, will we get the results we expect?”
FEASIBLE: Are resources sufficient to implement the chosen intervention?
MEANINGFUL: Are intended outcomes important? Is the magnitude of expected change worth the effort?
25
Theory of Change
Summary of Key Points
• A theory of change identifies cause/effect
• The three elements of a theory of change; community problem/need, intervention, intended outcome are supported by data and evidence
• Data documenting community need should show scope, significance, and causes
26
Theory of Change
Additional Resources
CNCS Priorities and Performance Measures: http://www.nationalserviceresources.gov/npm/home
Program-Specific Notices of Funding Opportunities and Application Instructions: http://www.nationalservice.gov
27
Theory of Change
Group exercise
Case Study of Rose Blossom ABC Club
Answer:• Question 2• Question 5• Question 4
28
Theory of Change
Evidence: What It Is and Where to Find It
How evidence helps support
your intervention
Copyright © 2012 by JBS International, Inc.
Developed by JBS International for the Corporation for National & Community Service
Theory of Change
Learning Objectives
By the end of the module, you will be able to:
• Describe evidence-based interventions
• Identify the approved sources of evidence
• Find evidence for your intervention
• Assess the utility of evidence you have located
• Use evidence for your intervention
Theory of Change
Module Overview
• What is evidence?• What are evidence-based interventions?• What are sources of evidence?• What are methods to assess evidence?
Theory of Change
Theory of Change Elements
Community Need
Specific Intervention
Intended Outcome
Evidence• Guides choice of intervention
• Supports cause-effect relationship
Statistics documenting the
need
Theory of Change
Evidence-Based Intervention
Service activities that are supported by research to demonstrate a high likelihood that these activities will address the need, and result in intended changes.
Evidence demonstrates that the proposed intervention (design, dosage) is likely to lead to the outcome.
Theory of Change
Everyday Life Example of a Theory of Change
Problem/NeedStrep Throat
InterventionAntibiotics
OutcomeHealthy
Evidence Penicillin Amoxicillin Tetracycline
Theory of Change
EVIDENCE: Information that supports your choice of a specific intervention and its ability to produce your intended outcome.
It is NOT enough to just say “we believe” our intervention is “likely” to be successful. You need to justify your choice with evidence.
Evidence
Theory of Change
Lessons from the Evidence
Evidence provides a reality check for theory of change elements
• For new programs:• What is the recommended design (specific program
activities) and dosage (frequency, intensity, and duration) to achieve an intended outcome?
• For existing programs:• Is there sufficient evidence for the intervention to
continue its use?
• Based on the evidence, are there modifications to the intervention that should be considered to have greater impact?
• Do you need to choose a new intervention?
Theory of Change
Knowledge check!
37
Theory of Change
Theory of Change
Evidence Source: Your Performance Measurement Data
Theory of Change
Evidence Source: Your Program Evaluation
Results from impact evaluation of your program:
• Is it an impact evaluation?
• Does it document change?
• Does it clearly show that your intervention is what caused the change?
Theory of Change
Evidence Source:Research Studies and Evaluations
• Research studies: Focused on increasing knowledge or understanding of a particular group, problem or issue
• Impact Evaluations: Look at intervention success and outcomes produced
Theory of Change
Assessing Evidence
Criteria include:
• Relevant: Cites comparable intervention with similar beneficiaries and results
• Compelling: Persuasive, shows clear likelihood of success
• Up-to-date: Recently published or most recent available
• Objective: Source with no stake in outcome and published in a peer reviewed journal or by credible organization
Theory of Change
Sources of Strong Evidence
• University or research organizations (national or local)
• Known professionals/ experts in the field
• Similar sounding programs/ descriptions
• Articles that review multiple studies (meta studies)
Evidence Source: ResearchStudies and Evaluations
Theory of Change
Knowledge check!
44
Theory of Change
Web Search Tool
Google Scholar is one web search tool that helps narrow the search for your keywords to professional journals http://scholar.google.com
Tip:
Use quotation marks around key terms - may help narrow the number of search results
Theory of Change
Is the article:• Relevant?• Compelling?• Up-to-date?• Objective?
Search Result Questions
Theory of Change
Knowledge Check!
47
Theory of Change
Evidence Continuum
Preliminary Moderate Strong
• Doesn’t show causality
• Outcome results from performance measurement
• Show causality, compares intervention recipients to non-recipients
• Results from studies/impact evaluations
• Comparison groups: Quasi-experimental Design
• Show causality, compares intervention recipients to non-recipients
• Results from studies/impact evaluations
• Randomly-assigned control groups: Experimental Design
CausalityLow High
Theory of Change
Evidence Continuum
Preliminary Moderate Strong
Performance measurement shows that 75% of girls age 14-16 participating in the Shape Up program made more healthy food choices
A 2005 impact evaluation by internal evaluators (using a quasi-experimental design of girls participating in a similar program to Shape Up) found that after 12 weeks, the girls in the program made 50% more healthy food choices than the comparison group.
A 2010 impact evaluation of the GEMS program by University of MN using experimental design/ random assignment found after 12 weeks, the girls in the experimental group made 50% more healthy food choices than control group girls
Shape Up: afterschool obesity prevention program
Theory of Change
Knowledge Check!
50
Theory of Change
Key Points Summary
• The term “evidence-based intervention” describes service activities that are supported by research to indicate a high likelihood that the intervention will address the need and result in intended outcomes.
• Evidence is information or documentation that a specific intervention (design and dosage) will produce your intended outcome.
Theory of Change
Key Points Summary
•The four sources of evidence that CNCS has identified include:
• Past performance measurement data• Results from a program impact evaluation• Research studies that document the outcomes of similar
programs• Evaluations that document outcomes of similar programs
• When assessing evidence, check to be sure it is relevant, compelling, up-to-date, and objective.
• The strongest evidence clearly proves that the intervention caused the positive change by using an experimental design.
Theory of Change
Group exercise 20 minutes including debrief!
Case Study of the Rose Blossom ABC Club
Answer:• Question 3• Question 6
53
Theory of Change
Final Concerns & Questions
Plus post test!
54
Theory of Change
ResourcesCNCS Priorities and Performance Measures: http://www.nationalservice.gov/resources/npm/home
Program-Specific Notices of Funding Opportunities and Application Instructions: http://www.nationalservice.gov
Additional Resources
