Upload
others
View
0
Download
0
Embed Size (px)
Citation preview
1
Evaluation Café:
A Study of Three
Developmental Evaluations
This presentation is made possible by the support of the American People through the United States Agency for International Development (USAID.) The contents of this presentation are the sole responsibility of DEPA-MERL Consortium and do not necessarily reflect the views of USAID or the United States Government.
March 2020
2
Who we are
3
Before we begin,
• How familiar are you with Developmental Evaluation (DE)?
• How familiar are you with Outcome Harvesting?
• How many are interested in international evaluation work?
4
• Introduction to DEPA-MERL
• Overview of research questions & methods
• Key lessons learned
• Audience questions
Our agenda today,
5
• WDI working with USAID since 2015
• MERLIN Program:
– Innovate on traditional
approaches to monitoring,
evaluation, research and learning
(MERL)
• DEPA-MERL is a consortium under
MERLIN that implements
developmental evaluations in the
USAID context
Introduction to DEPA-MERL
6
• Supports the continuous
adaptation of development
interventions in complex
environments
• Developmental Evaluators are
“embedded” within the program
– Use a variety of M&E
approaches
What is developmental
evaluation?
7
How we applied DE
• Three DE pilots were conducted
– Family Care First in Cambodia
– Sustained Uptake DE, the US Global Development Lab
(Washington, DC)
– Bureau for Food Security (Washington, DC)
• Each DE pilot included:
– 1 full-time, external Developmental Evaluator
– A “DE Administrator” from DEPA-MERL
8
Our research questions
INSERT DE RQ QUESTIONS TABLE
FROM IP AAP REPORT
Research question Methods Data sources
1: How does DE capture, promote,
and enable the utilization of emergent
learnings in support of ongoing
programming in a complex system, in
the USAID context?
Outcome
harvesting
(qualitative)
● Monthly reflection interviews with two Developmental
Evaluators (n=35)
● Substantiation interviews with keys stakeholders at
endline of two DEs (n=26)
● Document reviews, as required
2: What are the barriers and enablers
to implementation of DE in the USAID
context?
Semi-
structured
interviews
(qualitative)
● Monthly reflection interviews with two Developmental
Evaluators (n=35)
● Substantiation interviews with keys stakeholders at
endline of two DEs (n=26)
3: What is the perceived value of
conducting a DE, especially versus a
traditional evaluation approach?
Survey
(quantitative
and
qualitative)
● Value of Developmental Evaluation Survey with DE
stakeholders at endline (n=30)
9
Outcome harvesting approach
10
What was the most
surprising finding from our
research on DE?
Be
n W
hit
e
11
What is the biggest difference
between DE in theory and
DE in practice?
12
If you were approached by
new DE implementer, what
advice would you give them
about how to successfully
launch and oversee a DE?
https://bit.ly/32kJGOA
13
What is the most
controversial idea or topic in
DE right now?
Die
go
PH
14
Resources!
https://bit.ly/2JXCUru https://bit.ly/2VkTbgO
15
Questions?
16
USAID-MERLIN (DEPA-MERL)
Shannon Griswold, [email protected]
WDI
Contact us!
WDI Website
https://bit.ly/2NtuVU7
17
Annex
18
What is developmental evaluation?
• Development Evaluation (DE) is an approach to evaluation that supports the
continuous adaptation of development interventions.
• DE provides evaluative thinking and timely feedback to inform ongoing adaptation as
needs, findings, and insights emerge in complex dynamic situations.
• The DE helps facilitate the process from findings to action in a collaborative process
with the DE stakeholders.
RealizedOutcomes
Intended Outcomes
EmergentOutcomes
UnrealizedOutcomes
ImplementedOutcomes
Source: Henry Mintzberg, Sumatra Ghoshal, and James B. Quinn, The Strategy Process, Prentice Hall, 1998.
What outcomes can you expect from DE?
20
Traditional Evaluation Developmental Evaluation
Purpose Supports improvement,
summative test and
accountability
Supports development of innovation and
adaptation in dynamic environments
Standards Methodological competence
and commitment to rigor,
independence, credibility with
external authorities
Methodological flexibility and
adaptability; systems thinking, creative
and critical thinking balanced; high
tolerance for ambiguity; able to facilitate
rigorous evidence-based perspectives
Options Traditional research and
disciplinary standards of quality
dominate options
Utilization focused: options are chosen in
service to developmental use
Evaluation
Results
Detailed formal reports;
validated best practices,
generalizable across time and
space.
Rapid, real time feedback; diverse, user-
friendly forms of feedback. Evaluation
aims to nurture learning.
How is DE different?
• The DE evaluator works
collaboratively with implementing
teams to conceptualize, design, and
test new approaches in a long-term,
ongoing process of adaptation,
intentional change, and
development.
• The DE evaluator thinks and
engages evaluatively; questions
assumptions; applies evaluation
logic; uses appropriate methods; and
stays empirically grounded— that is,
rigorously gathers, interprets, and
reports data.21
What is a developmental evaluator?
22
Analysis Categories for the Harvested Outcomes (RQ1)
Factor Percent of all enablers* Percent of all barriers*
Skills of the Developmental Evaluator 17% 8%
Data collection and sharing 16% 10%
Data utilization 12% 8%
Integration of the Developmental Evaluator 11% 10%
Leadership (of program being evaluated) 11% 15%
Stakeholder relationships 9% 11%
DE readiness 8% 8%
USAID dynamics 7% 14%
Funding dynamics 2% 6%
Local and international dynamics 2% 4%
* Percentages do not total 100% because only 10 of the 13 factors that were coded across DE pilots for are shown.
23
Barriers and Enablers (RQ2)
24
Value of Developmental Evaluation Survey (RQ3) (1/2)
Takeaway:
In all areas except
DE cost and time
savings, survey
respondents said
the DEs were
better than
traditional
evaluation (n=29)
25
Value of Developmental Evaluation Survey (RQ3) (2/2)
Takeaway: The overwhelming majority of respondents reported positive interactions with
their Developmental Evaluators.
26
Description of codes (1/2)
27
Description of codes (2/2)
28
The DEPA-MERL DE pilots