-
8/8/2019 Leveraging Learning in Evaluation
1/32
The views expressed in this presentation are the views of the
author/s and do not necessarily reflect the views or policies of
the AsianDevelopment Bank, or its Board of Governors, or the
governments they represent. ADB does not guarantee the accuracy of
the data includedin this presentation and accepts no responsibility
for any consequence of their use. The countries listed in this
presentation do not imply anyview on ADB's part as to sovereignty
or independent status or necessarily conform to ADB's
terminology.
Leveraging Learning
in Evaluation
Olivier Serrat
2010
-
8/8/2019 Leveraging Learning in Evaluation
2/32
Overview
Introduction to monitoring and evaluation, and itsresults chain
and life cycle
Introduction to independent evaluation
Distinction between evaluation for accountabilityand evaluation
for learning
Considerations in evaluation capacitydevelopment
Description of areas of competence for knowledge
management and learning Overview of tools for evaluation for
learning
-
8/8/2019 Leveraging Learning in Evaluation
3/32
Monitoring is the continuous collection of data andinformation
on specified indicators to assess the
implementation of a development intervention in relationto
activity schedules and expenditure of allocated funds,and progress
and achievements in relation to its
intended outcome. It involves day-to-day follow-up of activities
during
implementation to measure progress and identifydeviations.
It requires routine follow-up to ensure activities areproceeding
as planned and are on schedule.
It needs continuous assessment of activities andresults answers
the question, "what are we doing?"
Monitoring and Evaluation
-
8/8/2019 Leveraging Learning in Evaluation
4/32
Evaluation is the periodic assessment of the
design,implementation, outcome, and impact of a development
intervention. It should assess the relevance andachievement of
the intended outcome, the performance ofimplementation in terms of
effectiveness and efficiency,and the nature, distribution, and
sustainability of impact.
It is a systematic way oflearning from experience toimprove
current activities and promote better planningfor future
action.
It is designed specifically with the intention to
attributechanges to the intervention itself.
It answers the question, "what have we achieved andwhat impact
have we had?"
Monitoring and Evaluation
-
8/8/2019 Leveraging Learning in Evaluation
5/32
The Results Chain Outputs
Outcome
Impact
ActivitiesInputs
Monitoring and Evaluation
-
8/8/2019 Leveraging Learning in Evaluation
6/32
The Results Chain Explained
Needs
Objective I nputs Activities Outputs
Outcome
Impact
Monitoring and Evaluation
Re l e v a n ce Ef f i ci e n cy
E f f e c t i v e n e s s
S u s t a i n a b i l i t y
-
8/8/2019 Leveraging Learning in Evaluation
7/32
-
8/8/2019 Leveraging Learning in Evaluation
8/32
Challenges and Limits to Management
LogicDegree ofControl
Challenge ofMonitoring
and
Evaluation
Impact
What the
development
intervention is
expected to
contribute to
OutcomeWhat the
developmentintervention
can be
expected toachieve and
be
accountablefor
Outputs What is withinthe direct
control of the
developmentintervention'smanagement
Activities
Inputs
De
creasing
Control
Inc
reasing
Difficulty
Monitoring and Evaluation
-
8/8/2019 Leveraging Learning in Evaluation
9/32
-
8/8/2019 Leveraging Learning in Evaluation
10/32
1. Adhere to strategic principles.
2. Sharpen evaluation strategies.
3. Distinguish recommendation typologies.
4. Make better recommendations.5. Report evaluation
findings.
6. Track action. on recommendations.
Increasing Value Added
from Independent Evaluation
-
8/8/2019 Leveraging Learning in Evaluation
11/32
Provides for control of the organization.
Implies lack of utility.
Diverts resources.
Focuses on justification rather than improvement. Distorts
program activities.
May promote incentive to lie, cheat, and distort.
Misplaces accent on control.
Emphasizes results orientation while maintaining
traditional checks on use of inputs andcompliance with
procedures.
Some Criticisms of
Evaluation for Accountability
-
8/8/2019 Leveraging Learning in Evaluation
12/32
Learning is the acquisition of know ledge or skills
through instruction, study, and experience.
Learning is driven by organization, people, know ledge,
and technology working in harmonyurging better and
faster learning, and increasing the relevance of an
organization.
Learning is an integral part of know ledge management
and its ultimate end.
Data WisdomInformation Know ledge
Know WhyKnow HowKnow What
Reductionist Holistic
Learning
Systemic
-
8/8/2019 Leveraging Learning in Evaluation
13/32
Learning from Experience
Evaluation fromlearning canserve as animportantfoundation
block
of a learningorganization.
Researchers
now recognize itas the greatestneed today and
tomorrow.
-
8/8/2019 Leveraging Learning in Evaluation
14/32
Item Evaluation forAccountability
Evaluation for Learning
Basic Aim The basic aim is to find outabout the past.
The basic aim is to improvefuture performance.
Emphasis Emphasis is on the degreeof success or failure.
Emphasis is on the reasonsfor success or failure.
Favored by Parliaments, treasuries,media, pressure groups
Development agencies,developing countries,research
institutions,
consultantsSelection ofTopics
Topics are selected basedon random samples.
Topics are selected fortheir potential lessons.
Status ofEvaluation
Evaluation is an endproduct.
Evaluation is part of theproject cycle.
Evaluation for Accountability and
Evaluation for Learning
-
8/8/2019 Leveraging Learning in Evaluation
15/32
Item Evaluation forAccountability
Evaluation for Learning
Status ofEvaluators
Evaluators should beimpartial and independent.
Evaluators usually includestaff members of the aidagency.
Importance
of Data fromEvaluations
Data are only one
consideration.
Data are highly valued for
the planning andappraising of newdevelopment activities.
Importance
of Feedback
Feedback is relatively
unimportant.
Feedback is vitally
important.
Source: Adapted from Cracknell, B. 2000. Evaluating Development
Aid: Issues, Problems, and Solutions. East Sussex: Sage
Publications.
Evaluation for Accountability and
Evaluation for Learning
-
8/8/2019 Leveraging Learning in Evaluation
16/32
Stage Category
Preparation Failures of intelligence: not knowing enough at the
earlystages of project formulation, resulting in crucial aspects
of
the project context being ignored.
Failures of decision making: drawing false conclusions ormaking
wrong choices from the data that are available, andunderestimating
the importance of key pieces ofinformation.
Implementation Failures of implementation: bad or inadequate
managementof one or more important aspects of the project.
Failures of reaction: inability or unwillingness to modify
theproject in response to new information or changes in
conditions that come to light as the project proceeds.
Evaluation Failures of evaluation: not paying enough attention
to theresults.
Failures of learning: not transferring the lessons into
future
plans and procedures.
Source: Adapted from Nolan, R. 2002. Development Anthropology:
Encounters in the Real World. Boulder, Colorado: Westview
Press.
Types of Learning Failure
-
8/8/2019 Leveraging Learning in Evaluation
17/32
Who Should BeLearning?
What ShouldThey Be Learning?
Field Staff Participation in practice
Effective empowerment
Local-level collaboration with government and othernongovernment
organizations
Gender dimensions of local developmentTechnicalSpecialists
Good practice in their area of expertise
Ways of integrating with other disciplines
How to improve cost-effectivenessHow existing internal and
external policies affectperformance
Operational
Managers
What factors make interventions and projects work well or
badly, for example, funding conditionsHow to be more
cost-effective
How to coordinate internally and externally
Learning in Nongovernment
Organizations: Food for Thought
-
8/8/2019 Leveraging Learning in Evaluation
18/32
Who Should BeLearning?
What ShouldThey Be Learning?
Fund-RaisersandDevelopmentEducationalists
Principles and insights to be used in negotiation
withprofessional donorsNew messages to get across to private
contributorsExamples of impact and what made things work or
fail
Leaders How policy choices and strategies work out in
practice
How to make external relationships more effective
How best to exert influence
What environmental factors have had unforeseen effects andmust
be taken into account
Governors The quality and costs of donors
The degree of stakeholder satisfaction
Consistency between mission, strategy, and impact
Improving social standing and credibility of the
organizationSource: Adapted from Fowler, A. 1997. Striking a
Balance: A Guide to Enhancing the Effectiveness of Non-Governmental
Organizations in InternationalDevelopment. London: Earthscan.
Learning in Nongovernment
Organizations: Food for Thought
-
8/8/2019 Leveraging Learning in Evaluation
19/32
-
8/8/2019 Leveraging Learning in Evaluation
20/32
Evidence PersuasiveArgument
Clear purposeCohesive argumentQuality of evidence
Transparency of evidence underpinning policyrecommendations
(e.g., a single study or asynthesis of available evidence)
Authority Clear purpose
Cohesive argumentQuality of evidence
Transparency of evidence underpinningrecommendations (e.g., a
single study or asynthesis of available evidence)
Context AudienceContextSpecificity
Addresses the specific context (e.g., national,local)
Addresses the needs of target audience (e.g.,social, economic
policy)
ActionableRecommendations
Information linked to specific processesClear and feasible
recommendations on steps tobe taken
Making Evaluation Reports
Effective
-
8/8/2019 Leveraging Learning in Evaluation
21/32
Engagement Presentationof Evidence-Informed
Opinions
Presentation of author's own views about theimplications of
findings
Clear identification of argument components that
are opinion based
ClearLanguageand Writing
Style
Easily understood by educated nonspecialists
Appearanceand Design
Visually engaging
Presentation of information through charts,graphs, and
photographs
Source: Adapted from Jones, N., and C. Walsh. 2008. Policy
Briefs as a Communication Tool for Development Research. Overseas
DevelopmentInstitute Background Note. May. Available:
www.odi.org.uk/publications/background-notes/0805-policy-briefs-as-a-communication-tool.pdf
Making Evaluation Reports
Effective
http://www.odi.org.uk/publications/background-notes/0805-policy-briefs-as-a-communication-tool.pdfhttp://www.odi.org.uk/publications/background-notes/0805-policy-briefs-as-a-communication-tool.pdfhttp://www.odi.org.uk/publications/background-notes/0805-policy-briefs-as-a-communication-tool.pdfhttp://www.odi.org.uk/publications/background-notes/0805-policy-briefs-as-a-communication-tool.pdf
-
8/8/2019 Leveraging Learning in Evaluation
22/32
Capacity is the ability of people, organizations, andsociety to
manage their affairs successfully.
Capacity to undertake effective monitoring andevaluation is a
determining factor of aideffectiveness.
Evaluation capacity development is the process ofreinforcing or
establishing the skills, resources,structures, and commitment to
conduct and usemonitoring and evaluation over time.
Evaluation Capacity Development
-
8/8/2019 Leveraging Learning in Evaluation
23/32
Stronger evaluation capacity will help development agencies
to
Develop as a learning organization
Take ownership of their visions for poverty reduction, if
theevaluation vision is aligned with that
Profit more effectively from formal evaluations
Make self-evaluations an important part of their activities
Focus on quality improvement efforts
Increase the benefits and decrease the costs associated
withtheir operations
Augment their ability to change programming midstream andadapt
in a dynamic, unpredictable environment
Build evaluation equity, if they are then better able to
conduct more of their own self-evaluation, instead of hiringthem
out
Shorten the learning cycle
Evaluation Capacity Development
-
8/8/2019 Leveraging Learning in Evaluation
24/32
In starting to develop evaluation capacity internally,consider
key decisions in
Architecture Locating and structuring evaluationfunctions and
their coordination.
Strengthening Evaluation Demand Ensuringthat there is an
effective and well-manageddemand for evaluations.
Strengthening Evaluation Supply Makingcertain that the skills
and competencies are in placewith appropriate organizational
support.
Institutionalizing Evaluations Buildingevaluation into
policy-making systems.
Evaluation Capacity Development
-
8/8/2019 Leveraging Learning in Evaluation
25/32
Strategy Development
A strategy is a long-term plan of action to achieve aparticular
goal.
Management Techniques
Leadership is the process of working out the right thingsto do.
Management is the process of doing thingsright.
Collaboration Mechanisms
When working with others, efforts sometimes turn out tobe less
than the sum of the parts. Too often, not enoughattention is paid
to facilitating effective collaborative
practices.
Competencies for Know ledge
Management and Learning
-
8/8/2019 Leveraging Learning in Evaluation
26/32
Know ledge Sharing and Learning
Two-way communication that take placesimply and effectively
build knowledge.
Know ledge Capture and Storage
Know ledge leaks in various ways at various times.
Competencies for Know ledge
Management and Learning
-
8/8/2019 Leveraging Learning in Evaluation
27/32
-
8/8/2019 Leveraging Learning in Evaluation
28/32
Tools for Evaluation for Learning
Conducting After-Action Reviews
Posting Research Online
Storytelling Identifying and Sharing Good
Practices
Disseminating Knowledge
Products
Learning from Evaluation
Asking Effective Questions
Embracing Failure
Enriching Policy with Research
Know ledge Sharing andLearning
Conducting Exit Interviews
Monthly Progress Notes
Assessing the Effectiveness ofAssistance in
CapacityDevelopment
Showcasing Knowledge
Harvesting Knowledge
The Critical Incident Technique
Know ledge Capture andStorage
-
8/8/2019 Leveraging Learning in Evaluation
29/32
Level StrategyDevelopment
ManagementTechniques
CollaborationMechanisms
KnowledgeSharing and
Learning
KnowledgeCapture and
Storage
5
4
3
2
1
Organizational Competencies for
Know ledge Management and Learning
-
8/8/2019 Leveraging Learning in Evaluation
30/32
Source: Adapted from ScanteamInternational. 1993. Internal
Learningfrom Evaluations and Reviews. ReportNo. 193. Oslo: Royal
NorwegianMinistry of Foreign Affairs.
Learning from Evaluation
How is learninggenerated in the
organization?
Are users
capable of usingthe information?
Is theinformation
easily accessible?
Are usersinvolved in
generating theinformation?
Is the information
used in decisions?
Is the information channeledto the right people
at the right time?
Is the information relevant
and useful?
How is the need for theinformation generated?
Are users opento the
information?
-
8/8/2019 Leveraging Learning in Evaluation
31/32
Monitoring and evaluation are closely interrelated.
The role of independent evaluation has grown inrecent years. It
is now the focus of efforts to raise itscontribution to development
effectiveness.
Evaluation for accountability should be supplemented
by evaluation for learning. Many shortcomings stem from learning
failures.
Evaluation capacity development is an intuitive area
for investment. Notions of knowledge management have a role to
play
in evaluation.
Key Learning Points
-
8/8/2019 Leveraging Learning in Evaluation
32/32
Olivier D. Serrat
Principal Knowledge Management Specialist
Knowledge Management Center
Regional and Sustainable Development Department
Asian Development [email protected]
www.adb.org/knowledge-management/
Know ledge Management Center
mailto:[email protected]://www.adb.org/knowledge-management/http://www.adb.org/knowledge-management/mailto:[email protected]
