The Use of Evidence in the Monitoring and Evaluation of Programmes in Fragile and Conflict Affected Countries (FCAS)

Matthew Waterfield12 March 2015

O B J E C T I V E & S T R U C T U R E

Structure1. Why there is a need for

better use of evidence in

FCAS?

2. What are the challenges to

using evidence in FCAS?

3. Innovative methods for the

collection and use of

evidence in FCAS

1. WHY THERE IS A NEED FOR BETTER USE OF

EVIDENCE IN FCAS?

Myths used to justify lack of use of evidence

• Need to do something now

• No data available

• Too difficult to collect evidence

• Situation changes so quickly all data is inaccurate

“DFID has committed to spending 30% of Official Development Assistance (ODA) tosupport these (FCAS) countries and tackle the drivers of instability by 2014-15 (up from 22% in 2010)” (Independent Commission on Aid Impact, June 2014)

“At the 2011 4th High Level Forum on AidEffectiveness, the major bilateral and multilateral donors committed to the New Deal for Engagement in Fragile States” (OECD, 2011)

“Building Stability Overseas Strategy committed to a joint understanding of conflict to drive policy and programmes” (HMG, 2011)

Doing a better job with more resources in FCAS is a priority

“Development implies social

change and is thus deeply

political.”

(DFID/GTZ)

“Any project set in a conflict-prone region will inevitably

have

an impact on the peace and conflict environment – positive

or negative, direct or indirect, intentional or unintentional”

(Conflict Sensitivity Consortium)

“Addressing the causes and effects

of fragility is ‘core business’ for DFID

and our partners” (DFID)

Better understanding of context is an imperative

GENEVA DECLARATION ON ARMED VIOLENCE AND DEVELOPMENT (2006)

Integration of armed violence reduction and conflict prevention programmes into development frameworks and strategies

WORLD DEVELOPMENT REPORT: CONFLICT, SECURITY AND DEVELOPMENT (2011)

Importance of understanding the links between conflict and development

Development assistance in fragile states must become more sensitive to conflict dynamics

MDG 2015 REVIEW PROCESS

Need to reduce violent conflict is recognised and

likely to result in a “conflict MDG”

Addressing conflict is central to the development agenda today

Example – Helmand: Priorities of Helmand population

On behalf of:

• Focus on people’s priorities not best method to generate support for government

31%27% 26% 24%

21%19% 19%

16% 14%11% 10%

10%

47%

11%

33% 31%

10%

16%

23%26%

11%2%

18%

10%

Improving the availability of

education services

Reconstruction Improving the availability of

clinics and hospitals

Reducing the extent of

corruption in government

Improving access to

information and news

Increasing effectiveness at delivering justice

Improving the security in my

area

Improving the supply of electricity

Improving the security along roads to the major urban

areas

Tribal disputes Improving the availability of

water for irrigation

Improving the local economy

What would you like the government to focus on over the next 6 months? Three mentions

Wave 8 Aug -12

All waves

Source HMEP

Example – Helmand: What actually impacts on District Government Legitimacy in Helmand

On behalf of:

35

30

26

1614

10 107 6

5

rating of security situation

improved effectiveness at delivering justice

(State Courts)

satisfaction state justice

DG used taxes to improve public

services

improved access to information and

news

ANP acts in interests of local

people

satisfaction state education

satisfaction state healthcare

improved the central bazaar

improved security on roads to urban

area

What recent improvements to/perceived quality of servicesare most likely to improve district government legitimacy? Helmand 11 districts

Respondents' current rating of the security situation is the factor most strongly associated with positive views about the district government, closely followed by reported improvements to the effectiveness of and satisfaction with statutory justice services. Across all districts over the last year, respondents reporting that the security situation was somewhat or very good were 35% more likely to say that the district government listened and acted on behalf of the people rather than in its own interest.

Source HMEP: Waves 5-8

STRONGEROVER TIME

STRONGEROVER TIME

STRONGEROVER TIME

• Improve security and justice• Maintain Justice, health care, access to information & news

2 . W H A T A R E T H E C H A L L E N G E S T O U S I N G E V I D E N C E I N F C A S ?

12

DFID Lebanon UNDP Monitoring and Evaluation

Scope and aim of researchQuality of services

& access to services

Inter-community tension

Perceived capacity & legitimacy of municipality

Likelihood of violence and

instability

Page 13

Summary of Research Method

Prompting Question / Story / Title

9 Sense-making Triads

Attribute & Demographic Questions 2 Evaluation Questions

Research carried out in 3 communities, one in the North, one in Bekaa and one in the South for four DfID funded projects selected on specified criteria.

Baseline Evaluation: (September / October 2014)776 responses collected.

Impact Evaluation: (January 2015)778 Responses collected

Community UNDP Project

Rajam Issa: Sewerage

Saadnayel: Roads / BridgesYouth Facilities

Sarafand Provision of water

Page 14

Lebanon Municipal Support Programme

Rajam Issa

Saadnayel

Sarafand

1. In the story, people …T1

Baseline Evaluation Impact Evaluation

Segment Corner

A 16% A+ 10%

B 42% B+ 25%

C 41% C+ 21%

Note: Segment includes cornerCentre = 1% | N/A = 1%

Segment Corner

A 29% A+ 20%

B 37% B+ 19%

C 30% C+ 17%

Note: Segment includes cornerCentre = 2% | N/A = 4%

Challenges

• Accuracy of data – sampling & validation

• Conflict sensitivity & security

• Remote management

• Real time data needs

Example – Helmand: Validity of HMEP Data HMEP Surveys/Interviews

Quarterly Head of Household Survey• 26,179 Respondents (7 waves)• 11 Districts• 95% Confidence Level• 5% Margin of Error

Female Survey (1 wave)• 384 Respondents• 6 Group One Districts

Qualitative In-depth Interviews • 4 x 80 Men and 3 x 80 Women• 6 Districts• Help us answer the “why?”• Validate quantitative findings

Triangulation of Data• Other PRT Data• RC(SW)• Other Surveys • Regression analysis covering 7 waves of data

Example - Lebanon-Syria Border Programme: Real time data

• Community perceptions of border capability

• Performance assessment of Lebanese Armed Forces (LAF)

18

3. INNOVATIVE METHODS FOR THE COLLECTION AND USE OF EVIDENCE IN FCAS

Innovative Methods

• Adaptive management

• New and big data

• New conflict analysis tools • Resilient Peace

Adaptive Management: Designing, evaluating and refining interventions

- Results-driven research, monitoring and evaluation

- Design, adapt, implement, redesign

- Expert reviews

- Independent validation

21

Syria CP Programme Results Chain

‘A more stable and democratic Syria that better meets the needs and aspirations of its people and presents a

significantly reduced threat to regional and international security and stability’

Nat, Reg, Intl Consequences

Contained

Terrorist, CBW, Proliferation

Managed

Strengthen Inc Opp

Isolation of Assad

Negotiated Pol Solution

Humanitarian & Regional

Limiting Terrorism,

CBW

Day after & Transition

5 Outputs

GO

ALO

BJEC

TIVE

SPI

LLAR

S

PILLAR OUTPUTS

PILLAROUTCOMES

Projects

Violence Reduced – Transition not

Threatened

5 Outcomes

Transition to Moderate post-Assad Authority

Outputs

Outcomes

7 Outputs

4 Outcomes

Outputs

Outcomes

Outputs

Outcomes

6 Outputs

4 Outcomes

55% 1% 7% 33%FUNDING

PROPORTION

Syria Conflict Pool Programme Review

EffectivenessContribution AnalysisAn assessment of programme level effectiveness will assess if and how CP funded interventions have achieved progress towards their pillar level outcomes. Two contribution analyses will be undertaken, focused on Pillars I and VI, which together represent 88% of total CP funded activities in FY14/15.

Review Contribution

Analysis

Assess Alternative

Explanations

Assemble Contribution

Narrative

Assess Existing

Evidence

Analyse Results Chain

Revise & Strengthen

Analysis

- External Studies- Expert Consultations- Beneficiary Feedback- Project Documents- Project Evaluations

- HMG Project Team- Implementing Partner- External Experts

Mapping Relevance and Coherence

Strengthening an Inclusive Opposition

Output 1Output 2Output 3Output 4Output 5

PILLAR

PILLAR OUTPUTS

PILLAROUTCOME

S

Outcome 1

Outcome 2

Outcome 3

Outcome 4

Outcome 5

Project A

Project B

Project C

Project D

Project E

Project F

Project G

Example Syria: How armed groups are perceived

Not Trustworthy

Trustworthy

Ineffective

Effective

Revolutionaries are most common actor for protection. They are trusted and untrusted to the same extent, but are seen to be mostly ineffective

Brigades are mostly trusted and mostly but not exclusively seen as effectiveFree Police were uncommon actors but were seen as trustworthy and effective

FSA and the courts were all seen to be untrusted and ineffective

ResilientPeaceTM: Systems analysis needed to make sense of complex conflict dynamics

Key features of conflict systems:

• Interconnectedness

• Non-linear causality and feedback loops

• Time delay in effects

• Possibility of creating vicious as well as virtuous circles

• Resistance to change

Localised conflicts may have feedback loops and connections to wider conflictsDevelopment interventions can magnify (positive and negative) conflict dynamics

Source: Aktis, ResilientPeace (2014); interpeace, “Peacebuilding How? Systems Analysis of Conflict Dynamics” (2010)

ResilientPeaceTM: Multi-dimensional conflict model

Nested conflicts have feedback loops and are interrelatedLocalised conflicts often not isolated; connected to wider conflict system

Source: Aktis, ResilientPeace (2014)

Localised conflicts, incl crime

Provincial/national level conflicts

Regional level conflicts

Use of proxies

Ideology

Financing

Illegal substances

Arms

Interconnected conflict dynamics in Pakistan

PAK-AFGHANRELATIONSHIP Localised

conflicts e.g. over

resourcesINDO-PAK

RELATIONSHIP

Political-criminal

Nexus

Source: Aktis ResilientPeace™

analysis

State vs TTP State vs other

extremist groups

Role of the state

Sectarian conflict

Corruption

US DRONE STRIKES

FINANCING FOR

EXTREMISTGROUPS

Ethnic separatism

• Conflict risk drivers– Structural or deep seated dynamics that are difficult to

change, e.g. poverty, discrimination, history of conflict, corrupt or predatory state

– Proximate causes, e.g. weapons proliferation, financing of extremist groups, weak security sector

– Immediate triggers, e.g. political crises, price shocks

• Stabilisers– Internal, e.g. legitimate government, macroeconomic

stability, social safety nets– External, e.g. international diplomatic, trade, security

relationships

• Influencers– Political groups, insurgents, state, civil society groups, etc.

and their Interests, sources of influence

ResilientPeace™: Conflict analysis as starting point of conflict sensitivity in programming

• Futures analysis

• Scenario building and testing

• Implications for assistance programme

Source: Aktis, ResilientPeace (2014)

Aktis adaptive management framework Resilient Peace®

Discussion

31

Matthew WaterfieldDirectorAktis Strategy LtdT: +44 (0) 207 873 2428M: +44 (0) 7824 392 733Skype: matthewwaterfieldwww.aktisstrategy.com

Aktis Strategy Ltd is a company registered in England & Wales.Registration Number: 08327972Registered Office: Sterling House, 19/23 High Street, Kidlington, Oxford, OX5 2DH.