Evidence On Trial: weighing the value of

evidence in academic enquiry, policy and

everyday life

CCAFS outcomes and targets

Tonya Schuetz, Philip Thornton, Wiebke Foerch

Durham 11 July 2016

Structure

CCAFS Experience

Background

Theories of Changes

RBM MEL Framework

Issue of Evidence

Credibility of Outcomes

Operational Support

Challenges & opportunities

CGIAR,

a global partnership that unites organizations engaged in research for a food secure future

Background

FTA MAIZE Fish WHEAT PIM A4NH

RICE Livestock RTB DCLAS CCAFS WLE

Strategy and Results Framework 2017-30

Outcome Indicators and Targets

Background

4. Climate information services

and climate-informed safety

nets

1. Policies and institutions for

climate-resilient food systems

2. Climate smart

technologies,

practices, and

portfolios

Background - Flagship Programs

• Large bodies of work (e.g. US$1.5 million)• Source of funds: any kind• Single or Multiple Centers/Partners• Single or Multiple Regions or Global • Key feature: fits into an impact pathway

Projects

Background – Regional Programs

Theory of Change & Impact Pathways – why?

• Bring in focus on people +

outcomes, i.e. behavioral changes,

+ unanticipated changes +

incentives for change

• Ensure that CCAFS plan of work is

targeted at outcomes, requires ‘use

of outputs’ tasks built into each

activity plan

• Strategically encourage

communication and collaboration,

and improved work plans through

harmonizing IPs

• Revisit trajectory of CCAFS

contributions to change and use

them as foresight/ ex-ante impact

assessment – measure progress

towards change, adapted according

to learning

From output-focused supply-driven

(logframe approach) to outcome-

focused demand driven Theories of

Change over a two-year process (starting

from the regions):

• Harmonization achieved through capacity building (beyond trainings: meetings

and frequent virtual communication) in all flagships, regions and partners

• Built with partners & projects

• RBM first trialed on one Flagship

• Iterative planning process for the CRP project portfolio building

Creating our Theories of Change

• Set global flagships

• Regional priorities

• Calls for concepts

• Regional planning meetings

• Detailed project planning

• CCAFS operates across multiple sectors, scales, stakeholders

• Allowing for:

� Structured linear thinking

and planning with

assumptions for how desired

changes can happen

� Complex systems thinking:

flexibility to react according

to lessons and opportunities

arising during implementation

Research Development

Universities,

research

institutes,

bio-medical

facilities,

genetics etc.

UN,

development

NGOs,

philanthropic

foundations,

governments

Solutions

R4D: How were we supposed to do it?

R4D: What it looks like in practice:

Action

Research

Development

Leadership

Learning

Communications

Partnerships

Trust

Combination of ordered & complex systems

MEL theoretical framework

Results-based management

• Accountability for outcomes: logical chain

• Emphasis on systematic, constructive looped learning from past experiences and subsequent adaption -adaptive management

• Three thirds principle

� engaging with partners to decide what needs to be done and how;

� doing the actual research, often in partnership;

� sharing results in appropriate formats and strengthening capacity of next users to utilize the research to achieve outcomes and impact.

Examples

• Desk studies (e.g. current World Bank investment on Climate Smart Agriculture)

• Third-party surveys (deforestation rates in CCAFS regions)

• Other CGIAR/ CRP surveys (e.g. Climate Smart Agriculture technologies adopted in CCAFS sites)

• Formal impact assessment and evaluation methods incl. the ones conducted by independent units

• Project activities (CSA technologies adopted in other sites, GHG emissions in a landscape, …)

• Other (emails, internal government documents, …)

What kind of “evidence” is needed?A

t pro

ject , fla

gsh

ip, a

nd

reg

ion

al le

vel

Ch

an

ges t

hro

ug

h t

ime –

wit

h b

aseli

nes

� A lot of methodological development is needed to make evaluation incl. Impact Assessments to be

adequate for TOC / IP approaches.

• Definition of an outcome: ‘use of the research by non-research partners to develop new, or change, policies and practices’, i.e. beyond new research designs, dissemination, training and need to be 1000’ of farmers not involved in the research

• Evaluation of ‘outcome case studies’ � By Whom? 1/3 Internal (2) from Program Management Unit

(eliminated conflict of interest), 1/3 external (1) with experience in a farmer organization, and 1/3 external (1) with >20 years in R4D

� Criteria for scoring: three criteria with different weights, (a) quality/clarity of write up - 15%, (b) evidence - 20%, and (c) significance of outcome - 65%.

Credibility of Outcomes

How to strengthen the Outcomes?

• Resubmission of outcomes from previous years shows how these improve over time due to further scaling.

• Trend over four years (2012-15) of reported outcomes:• improved quality & relevance for 2015, large jump in no. • increase in % evaluated as ‘not yet outcomes’

• satisfactory no. of very good outcomes reported for 2015• it would seem that a lot of valuable time went into

developing Outcome Case Studies which did not yet meet the outcome criteria.

• seems worthwhile to remind of the criteria.

Contribution to some System Level Outcomes, which link with SDGs

CRP defined outcome indicators:

• for the flagship level 2030 and 2022

• Quantitative capturing of outcome target values combined with

• Qualitative narrative descriptions

What is CCAFS being held responsible for?

2 + 3

15 equitable national/subnational food system

policies enacted that take into consideration climate

smart practices and strategies, informed using

knowledge, tools and approaches derived from CCAFS

science

1 + 2

10 regional/global organisations inform their

equitable institutional investments in climate smart

food systems using CCAFS outputs.

20 equitable

national/subnational

jurisdictions will have

increased institutional

investments in climate

smart food systems

2015 2016 2022 2030

Flagship 4: Policies and institutions for climate-resilient food systems

� Understand ToC

� Meaningful performance expectations for results (outputs, outcomes)

� Measure results, assess contribution of program to observed outcomes

� Report on performance achieved against expectations

� Practical mechanisms and tools to ensure balanced quantitative and qualitative monitoring

A modular MEL system

CCAFS M&E System Modules

Harmoniza-tion of IPs &

ToCs

Indicators & Baselines

Reflective Spaces and

Activities

Reporting

Assessment & Bonus allocation

Research on

Institutional Transfor-mation

Process

• Planning Work & $$• System/ donor Reports

• Synthesis at Flagship level

• Synthesis at Regional level

• Project Synthesis• Project evaluation

• Deliverable ranking• Project annual reporting • Project annual planning

Program Mgmt. Unit

Flagship Program Leaders

Regional Program Leaders

Project Leaders

Project SettingProject Setting PlanningPlanning ReportingReporting Synthesis & EvaluationsSynthesis & Evaluations

Process supported by an Online Platform

Iterative processes and built-in • Looped learning• System for adaptive management• Project evaluation (traffic light) • CRP mapped to outcome targets• Modular MEL

Project Leaders

Wider system

CRP Managers

CCAFS M&E System

Modules

Harmonization of IPs & ToCs

Indicators &

Baselines

Reflective Spaces

and Activities

Reporting

Assessment &

Bonus allocation

Research on

Institutional

Transfor-mation

Dynamic program and project, planning and reporting modules

CoA

Technical features • Open source • Interoperable with

or linkage to other data platforms

• Modular

MEL Support Pack

• To support strategy implementation through online platform for different users

Incentives – readiness of a system

• The current CGIAR system’s incentive framework can be improved to make R4D via a TOC/IP approach work properly, including its financial system.

• A portfolio approach: in which some projects revolve much more around science and others around engagement. Rarely will projects do both. Key is that science and engagement are happening within a regional or global conceptualized and coordinated programmatic manner.

• Appropriate performance assessment required: Evaluation needs to be consistent with different measures required.

Challenges & Opportunities

• Complexity and harmonization – of different levels/scales involved: System, Program, project, countries

• System level – operationalization of the Strategy and Results Framework with a System MEL Framework and Monitoring Plan

• Donor demands – outcome delivery promise aspirational vs. R4D reality

• Balancing act of quantitative and qualitative measuring of outcomes

• Results-based management for R4D - learning from the development sector – yet there are no off-the-shelf solutions

• Accommodating negative outcomes and asking for those explicitly

• ....

Remaining Challenges

The “Evidencers”

(the results agenda)

Vs

“The Complexers”

(complex systems,

emergent

properties, messy)

Oxfam blog, “From Poverty to Power”

Key reference documents

• Schuetz, T, Förch, W, Thornton, P, Vasileiou, I. (accepted). Pathway to Impact: Supporting and evaluating enabling environments for research for development in Juha I. Uitto, Jyotsna Puri and Rob D. van den Berg, eds. Evaluating Climate Change for Sustainable Development. Springer: Dordrecht. Forthcoming in 2016.

• CCAFS Planning and Reporting online Platform, Learning Brief No. 16, Nov. 2015

• CCAFS Reporting and Evaluation in a results-based managementframework, Learning Brief No.15, reporting cycle, Jul. 2015

• Report to CO on RBM Trial, planning cycle, Dec. 2014

• Lessons and Insights from the CCAFS Results-Based Management Trial, RBM projects feedback/ experience, survey summary, Dec. 2014

• Lessons in theory of change from a series of regional workshops, Learning Brief No. 11, Dec. 2014

• CCAFS Theory of Change Facilitation Guide, Dec. 2014

• CCAFS RBM MEL strategy, framework from Jul. 2014