Embed Size (px)
Citation preview
www.3ieimpact.org Lindsey Novak
Theory of Change
International Initiative for Impact Evaluation
Lindsey Novak International Initiative for Impact
Evaluation (3ie)
www.3ieimpact.org Lindsey Novak
Terminology Theory of Change Program Theory Causal Chain Theory-Based Evaluation Logic Model
www.3ieimpact.org Lindsey Novak
Terminology Theory of Change Program Theory Causal Chain Theory-Based Evaluation Logic Model
vs the “Black box” approach
www.3ieimpact.org Lindsey Novak
Terminology Theory of Change Program Theory Causal Chain Theory-Based Evaluation Logic Model
vs the “Black box” approach
Ultimate goal: Answer the “why?” question
www.3ieimpact.org Lindsey Novak
1. What is the program?
Questions
www.3ieimpact.org Lindsey Novak
1. What is the program? 2. What outcomes does the program aim to
achieve?
Questions
www.3ieimpact.org Lindsey Novak
1. What is the program? 2. What outcomes does the program aim to
achieve? 3. What intermediate steps lead to those
outcomes?
Questions
www.3ieimpact.org Lindsey Novak
1. What is the program? 2. What outcomes does the program aim to
achieve? 3. What intermediate steps lead to those
outcomes? 4. What assumptions are associated with
each link in the causal chain?
Questions
www.3ieimpact.org Lindsey Novak
1. What is the program?
Campaign for Clean Water
School-Based Malaria Intervention
www.3ieimpact.org Lindsey Novak
School-Based Malaria Intervention
Malaria Intervention
www.3ieimpact.org Lindsey Novak
2. What outcomes does the program aim to achieve?
School-Based Malaria Intervention
Village of Love
www.3ieimpact.org Lindsey Novak
School-Based Malaria Intervention
Malaria Intervention
I N C R E A S E D K N O W L E D G E
www.3ieimpact.org Lindsey Novak
3. What intermediate steps lead to those outcomes?
School-Based Malaria Intervention
www.3ieimpact.org Lindsey Novak
School-Based Malaria Intervention
Reduced Clinical Attacks
Malaria Intervention Reduced
Asymptomatic Parasitemia
I N C R E A S E D K N O W L E D G E
www.3ieimpact.org Lindsey Novak
School-Based Malaria Intervention
Increased School
Attendance
Reduced Clinical Attacks
Malaria Intervention Reduced
Asymptomatic Parasitemia
I N C R E A S E D K N O W L E D G E
www.3ieimpact.org Lindsey Novak
School-Based Malaria Intervention
Increased School
Attendance
Reduced Clinical Attacks
Malaria Intervention Reduced
Asymptomatic Parasitemia
Reduced Anemia
I N C R E A S E D K N O W L E D G E
www.3ieimpact.org Lindsey Novak
School-Based Malaria Intervention
Increased School
Attendance
Reduced Clinical Attacks
Malaria Intervention Reduced
Asymptomatic Parasitemia
Reduced Anemia
Improved Cognition
Increased Concentration
I N C R E A S E D K N O W L E D G E
www.3ieimpact.org Lindsey Novak
1. What outcomes do you want to achieve? 2. What program will induce those
outcomes? 3. What pathways theoretically lead to those
outcomes? 4. What assumptions are associated with
each link in the causal chain?
School-Based Malaria Intervention
www.3ieimpact.org Lindsey Novak
School-Based Malaria Intervention
Increased School
Attendance
Reduced Clinical Attacks
Malaria Intervention Reduced
Asymptomatic Parasitemia
Reduced Anemia
Improved Cognition
Increased Concentration
I N C R E A S E D K N O W L E D G E
Children adhere to full med regimen
Medicine is effective Students correctly
targeted - RDT devices accurate,
Teachers categorize correctly
Not considered substitution for
preventative measures
www.3ieimpact.org Lindsey Novak
School-Based Malaria Intervention
Increased School
Attendance
Reduced Clinical Attacks
Malaria Intervention Reduced
Asymptomatic Parasitemia
Reduced Anemia
Improved Cognition
Increased Concentration
I N C R E A S E D K N O W L E D G E
Schools function sufficiently well – teachers are present and
teaching, teachers are knowledgeable
Binding constraint is addressed
Child labor doesn’t increase
www.3ieimpact.org Lindsey Novak
• Outcomes and assumptions can be tested by: – Counterfactual analysis
Data Collection
www.3ieimpact.org Lindsey Novak
Increased School
Attendance
Reduced Clinical Attacks
Malaria Intervention Reduced
Asymptomatic Parasitemia
Reduced Anemia
Improved Cognition
Increased Concentration
I N C R E A S E D K N O W L E D G E
Schools function sufficiently well – teachers are present and
teaching, teachers are knowledgeable
Binding constraint is addressed
Child labor doesn’t increase
Medicine is effective Students correctly
targeted - RDT devices accurate,
Teachers categorize correctly
Children adhere to full med regimen
Not considered substitution for
preventative measures
Data Collection
www.3ieimpact.org Lindsey Novak
• Outcomes and assumptions can be tested by: – Counterfactual analysis – Factual analysis
Data Collection
www.3ieimpact.org Lindsey Novak
Increased School
Attendance
Reduced Clinical Attacks
Malaria Intervention Reduced
Asymptomatic Parasitemia
Reduced Anemia
Improved Cognition
Increased Concentration
I N C R E A S E D K N O W L E D G E
Schools function sufficiently well – teachers are present and
teaching, teachers are knowledgeable
Binding constraint is addressed
Child labor doesn’t increase
Medicine is effective Students correctly
targeted - RDT devices accurate,
Teachers categorize correctly
Children adhere to full med regimen
Not considered substitution for
preventative measures
Data Collection
www.3ieimpact.org Lindsey Novak
• Outcomes and assumptions can be tested by: – Counterfactual analysis – Factual analysis
– Some outcomes and assumptions cannot be measured or tested
Data Collection
www.3ieimpact.org Lindsey Novak
Increased School
Attendance
Reduced Clinical Attacks
Malaria Intervention Reduced
Asymptomatic Parasitemia
Reduced Anemia
Improved Cognition
Increased Concentration
I N C R E A S E D K N O W L E D G E
Higher Test
Scores
Test accurately measures knowledge
Medicine is effective
Students correctly targeted
- RDT devices accurate, Teachers
categorize correctly
Children adhere to full med regimen
Not considered substitution for
preventative measures
Binding constraint is addressed
Child labor doesn’t increase
Schools function sufficiently well – teachers are present and
teaching, teachers are knowledgeable
Data Collection
www.3ieimpact.org Lindsey Novak
Criticisms of Theory of Change
• Over-simplify the program
www.3ieimpact.org Lindsey Novak
Campaign Against Child Marriage in Bihar & Jharkhand Breakthrough & Catalyst Management Services
www.3ieimpact.org Lindsey Novak
Criticisms of Theory of Change
• Over-simplify the program • Long-term effects not considered
www.3ieimpact.org Lindsey Novak
I N C R E A S E D K N O W L E D G E
Better Labor Market
Outcomes
Healthier & Smarter
Children Next Generation
Higher Incomes
Long Term Effects
Reduced Poverty
www.3ieimpact.org Lindsey Novak
Criticisms of Theory of Change
• Over-simplify the program • Long-term effects not considered • Unintended consequences not considered
www.3ieimpact.org Lindsey Novak
Increased School
Attendance
Reduced Clinical Attacks
Malaria Intervention Reduced
Asymptomatic Parasitemia
Reduced Anemia
Improved Cognition
Increased Concentration
I N C R E A S E D K N O W L E D G E
Unintended Consequences
Not considered substitution for
preventative measures
Child labor doesn’t increase
www.3ieimpact.org Lindsey Novak
Criticisms of Theory of Change
• Over-simplify the program • Long-term effects not considered • Unintended consequences not considered • Does not consider that program to
outcomes is not uni-directional
www.3ieimpact.org Lindsey Novak
Increased School
Attendance
Reduced Clinical Attacks
Malaria Intervention Reduced
Asymptomatic Parasitemia
Reduced Anemia
Improved Cognition
Increased Concentration
I N C R E A S E D K N O W L E D G E
Feedback
www.3ieimpact.org Lindsey Novak
Criticisms of Theory of Change
• Over-simplify the program • Long-term effects not considered • Unintended consequences not considered • Does not consider that program to
outcomes is not uni-directional • Heterogeneous effects are not considered
www.3ieimpact.org Lindsey Novak
Increased School
Attendance
Reduced Clinical Attacks
Malaria Intervention Reduced
Asymptomatic Parasitemia
Reduced Anemia
Improved Cognition
Increased Concentration
I N C R E A S E D K N O W L E D G E
Heterogeneous Effects Binding
constraint is addressed
Disaggregate by Distance to School
www.3ieimpact.org Lindsey Novak
• What is the best time to create a theory of change? – During the program design – Before data collection – Before data analysis
Timing
www.3ieimpact.org Lindsey Novak
Closing Remarks
• Ultimate goal: Answer the “why?” question • Identifies assumptions and creates
testable hypotheses • Identifies areas of potential weakness in
the program • Can be created for new and existing
programs
