Using the Outcome Mapping concept of outcomes in evaluations

To enrich theories of change and to foster an evaluative culture

John MauremootooRichard Smith

Presentation given at the American Evaluation Association on 14th November 2015 in the session entitled: Applying Outcome Mapping to the evaluation of socially

transformative projects, programs, and policies

Photo: Public Domain

Project locationsMongolia

World Bank & SDC Funded

DEFRA & DFID (UK) funded

India

Indian OceanEU funded

Mongolia

Strengthen the capacity of Civil Society Organisations to undertake social accountability monitoring

People and Pollinators initiative: Enhance pollination services for sustainable agriculture and biodiversity conservation

India

All photos of India from the Centre for Pollination Studies (CC-BY-NC)

Manage coastal zones for environmental sustainability and livelihood generation

Indian Ocean

All Indian Ocean photos by John Mauremootoo (CC-BY)

Theories of Change

Mongolia: Results chain from proposal

India: Logframe from proposal

Indian Ocean: Logframe from proposal

The Use of Outcome Mapping Concepts

Development

Implementation

Closure

Mongolia

Indian Ocean

India

Summative evaluation 3 years after project

closure

Summative evaluation 3 years after project

closure

Introduced 8 months into implementation for

internal planning, monitoring & evaluation

All projects planned and implem

ented using donor-m

andated logframes

Value of using OM Conceptsto enrich Theories of Change

Theory of change – the preconceived map of the journey

Theory in use – what actually happens

Value of using OM Conceptsto enrich Theories of Change

Outcome mapping makes step two explicit by using concepts such as outcomes as behavioural change, boundary partners and contribution not attribution

Outcomes as Behavioural Change

Mongolia

Activities

OutcomesAs indicators of effectiveness & sustainability

An effectiveness and sustainability evaluation framework

Outputs

Outcome Challenge

BoundaryPartner

ProgressMarkers

Project Purpose

Cont

rol

Influ

ence

Inte

rest

Outcomes as Behavioural Change

India

Activities

Inputs

Outcomes as Behavioural Change

Boundary Partners

From Tableatny on Flickr

Individuals, groups or organisations with which the program interacts directly and which the program hopes to influence

Boundary Partners

Mongolia

Limited scope of intervention meant the selection of participants / boundary partners by the implementing body was central to any achievements.

Evaluation data collection was directly from those who participated as trainees / mentees in the intervention.

Boundary Partners

India

The Boundary Partner concept highlighted the importance of site-specific political and administrative structures e.g. very different intervention models in the two project sites (not explicitly accounted for in the project design)

Boundary Partners

Indian Ocean Given the short duration of the projects (<18 months) most of the successful projects were those which built on existing partnerships.

Contribution not Attribution

The intervention may be only one among many factors that contributes to an outcome

Contribution not Attribution

Mongolia

The intervention introduced tools that were previously unused in Mongolia to CSOs with little or no experience of social accountability work therefore the contribution of the intervention could be easily identified.Several direct outcomes led to further outcomes and even impact. In such cases other factors and actors contributed.

Contribution not Attribution

India

Appreciation of the value of partnerships to produce sustainable outcomesValuable for team spirit as everybody understood that they were contributing to results

Contribution not Attribution.

Some projects failed to sustain outcomes nor contribute to impact because of failure to address actors and factors in the enabling environment (outside the control of the project) e.g. policy makers, legislation and the provision of infrastructure.

Indian Ocean

Outcome Mapping Concepts Strengthen the Design or Management of Interventions

OM Concepts Inform Further Support

Mongolia

Evaluation informed what to focus on for a follow up project and made a case for further support

Recommendations for discussion contributed to new project design

India

OM Concepts Adopted

Planning, monitoring and evaluation is now seen as integral to project implementation by project leaders who are looking to incorporate OM concepts into new project development.

Donor Interest Recommended

The project was rated A++ (outcome substantially exceeded) in its final external review and it was recommended that the PME manual (based on an OM-LFA fusion) should be publicised by defra as ‘best practice’

India

Evaluation results inform programme development.

Indian Ocean

Incorporation of lessons learned from evaluation into guidance documents for follow up EU programme – emphasis on partnerships, participation and building upon existing actions.

OM concepts helping us to become better Cartographers

OM helps us to gain a better understanding of the mental maps of those in the project’s ecosystem

Gervase R. Bushe ~ Professor of Leadership and Organization Development. A quote on leadership which could equally be applied to project implementation.

Leadership is a lot about influencing people and you cannot do that if you don't understand their maps.

Acknowledgements

Mongolia: Jeremy Gross (Co-evaluator) Amarbayasgalan Dorj (Co-evaluator)

India: Parthib Basu (Project Leader India & evaluation support)Barbara Smith (Project Leader UK & evaluation support)Soumik Chatterjee (Project Manager & evaluation support)

Indian Ocean: Dunstan Kishekya (Co-evaluator)