Monitoring Progress & Evaluating Impact Presented by: Cindy Banyai, Ph.D. Executive Director, efocus Institute IREX Community Solutions Program September 1, 2011 – September 8, 2011
Monitoring Progress & Evaluating Impact Presented by: Cindy Banyai, Ph.D. Executive Director, efocus Institute IREX Community Solutions Program September
Monitoring Progress & Evaluating Impact Presented by: Cindy
Banyai, Ph.D. Executive Director, efocus Institute IREX Community
Solutions Program September 1, 2011 September 8, 2011
Slide 3
Learning Objective Identify and employ key methods and tools
for monitoring progress and evaluating impact 2 2011 IREX CSP,
Cindy Banyai, refocusinstitute.com
Slide 4
Project Management Cycle (PMC) 3 Based on Miyoshi, 2008 2011
IREX CSP, Cindy Banyai, refocusinstitute.com
Slide 5
Monitoring Progress Monitoring - mid-term and terminal
evaluations Examine implementation process -- assess inputs through
outputs Measure performance assess outputs to intermediate outcomes
2011 IREX CSP, Cindy Banyai, refocusinstitute.com 4
Slide 6
Evaluating Impact Impact evaluation measures relationship
between intermediate outcomes and end outcomes Sustainability
evaluation measures long- term impacts, periphery effects of
program 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 5
Slide 7
USAID on Impact Evaluation Measures the change in a development
outcome that is attributable to a defined intervention Based on
models of cause and effect Requires a credible and rigorously
defined counterfactual to control for factors other than the
intervention that might account for observed change Comparisons
between beneficiaries that are randomly assigned to either a
treatment or a control group provide the strongest evidence of a
relationship between the intervention under study and the outcome
measured (USAID 2011) 2011 IREX CSP, Cindy Banyai,
refocusinstitute.com 6
Slide 8
DAC 5 Evaluation Criteria Efficiency - A measure of how
economically resources/inputs (funds, expertise, time, etc.) are
converted to results. Effectiveness - The extent to which the
development interventions objectives were achieved, or are expected
to be achieved, taking into account their relative importance. 2011
IREX CSP, Cindy Banyai, refocusinstitute.com 7
Slide 9
DAC 5 Evaluation Criteria (cont) Impact - Positive and
negative, primary and secondary long-term effects produced by a
development intervention, directly or indirectly, intended or
unintended. Sustainability - The continuation of benefits from a
development intervention after major development assistance has
been completed. The probability of continued long- term benefits.
The resilience to risk of the net benefit flows over time. 2011
IREX CSP, Cindy Banyai, refocusinstitute.com 8
Slide 10
DAC 5 Evaluation Criteria (cont) Relevance - The extent to
which the objectives of a development intervention are consistent
with beneficiaries requirements, country needs, global priorities
and partners and donors policies. (DAC 2002) 2011 IREX CSP, Cindy
Banyai, refocusinstitute.com 9
Slide 11
10 Relationship between Evaluation Criteria & Logic Model
End Outcomes Intermediate Outcomes Outputs Inputs impact
effectiveness efficiency Relevance sustainability Activities
Measuring performance Examining process Assessment of 5 criteria
Measuring performance Examining implementing process Miyoshi
2008
Slide 12
TOOLS FOR MONITORING AND EVALUATION 2011 IREX CSP, Cindy
Banyai, refocusinstitute.com 11
Slide 13
Traditional types of data gathering Surveys Tests Interviews
Observation Documents 2011 IREX CSP, Cindy Banyai,
refocusinstitute.com 12
Slide 14
Participatory types of data gathering Focus groups cooperative
inquiry Training Action research project Photographs Video
Plays/storytelling Metaphor drawing sculpture 13 2011 IREX CSP,
Cindy Banyai, refocusinstitute.com
Slide 15
Quantitative data Need counterfactual - situation a
participating subject would have experienced had he or she not been
exposed to the program Baseline or pre-program data used to assess
impact Predict outcomes that might result from the program (as in
ex ante evaluations) Make before-and-after comparisons (also called
reflexive comparisons) (WB 2010) 2011 IREX CSP, Cindy Banyai,
refocusinstitute.com 14
Slide 16
Quantitative impact evaluation Ex-ante impact evaluation -
measures intended impacts of future policies/programs/projects,
predicts program impacts. Looks at a potentially targeted areas
current situation May involve simulations based on assumptions
about how the economy works Based on structural models of the
economic environment facing potential participants 2011 IREX CSP,
Cindy Banyai, refocusinstitute.com 15
Slide 17
Quantitative impact evaluation Ex-post impact evaluations -
measure actual impacts accrued by beneficiaries attributable to
policy/program/project Examines immediate benefits, reflects
reality Sometimes miss the mechanisms underlying the programs
impact on the population, which structural models aim to capture
(WB 2010) 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 16
Qualitative data 3 main sources of qualitative data open- ended
interviews, observations, documents (Patton 2002) Emerging sources
photographs, video, artistic expressions (Hesse-Beiber and Leavy
2008) Can be coded, transformed into statistics 2011 IREX CSP,
Cindy Banyai, refocusinstitute.com 18
Slide 20
Qualitative research Looks at a phenomenon, situation in-depth
and detail The researcher is the instrument validity relies on her
skill, competence and rigor Internal validity when qualitative data
is corroborated from multiple sources 2011 IREX CSP, Cindy Banyai,
refocusinstitute.com 19
Slide 21
Quantitative vs. Qualitative Need both to provide depth and
understanding of situation Numbers balance narrative accounts
Stories and visuals put a human touch on numbers 2011 IREX CSP,
Cindy Banyai, refocusinstitute.com 20
Slide 22
Determine approach and tools Use these questions to guide your
selection Who is the information for and who will use the findings?
What kinds of information are needed? How is the information to be
used? For what purposes are the evaluation being done? When is the
information needed? What resources are available to conduct the
evaluation? Given the answers to the proceeding questions, which
methods are appropriate? (Patton 2002) 2011 IREX CSP, Cindy Banyai,
refocusinstitute.com 21
Slide 23
Parting thoughts on M&E Collect the data you need. Use the
data you collect. Doing so improves your efficiency and the clarity
of your evaluations, in addition to freeing up time for other
program activities. 2011 IREX CSP, Cindy Banyai,
refocusinstitute.com 22
Slide 24
Thanks for your attention! Please send questions & comments
to photoevaluation@refocusinstitute. com
[email protected]
Slide 25
References Banyai, Cindy. (2010). Community capacity and
governance New approaches to development and evaluation. PhD diss.,
Ritsumeikan Asia Pacific University. Bureau for Policy, Planning,
and Learning United States Agency for International Development
[USAID]. (2011.) Evaluation Policy. Washington, D.C.: USAID.
Development Assistance Committee [DAC] of the Organization for
Economic Cooperation and Development [OECD]. (2002). Glossary of
Terms in Evaluation and Results Based Management. Paris: OECD.
Hesse-Beiber, Sharlene Nagy and Leavy, Patricia. (2008). Handbook
of Emergent Methods. New York: Guilford Press. Japan International
Cooperation Agency - Office of Evaluation and Post Project
Monitoring, Planning and Evaluation Department (2004). JICA
Evaluation Handbook: Practical Methods for Evaluation. Tokyo: Japan
International Cooperation Agency. Khandaker, Shahidur R., Koolwal,
Gayatri B., Samad, Hussain A. [WB] (2010). Handbook on Impact
Evaluation: Quantitative Methods and Practices. Washington, D.C.:
World Bank. Miyoshi, Koichi. (2008). What is Evaluation?. In
Miyoshi, Koichi (Ed). Hyoka-ron wo Manabu Hito no tameni (For
People Learning Evaluation Theory). (pp. 1-16). Tokyo.
Sekaishisosha. Patton, Michael Quinn (2002). Qualitative Research
& Evaluation Methods. Thousand Oaks: Sage Publications, Inc.
2011 IREX CSP, Cindy Banyai, refocusinstitute.com 24