John YoungHead of Programme, RAPID, ODI

For Assessing the Impact of Policy Engagement

RIMISP-IFAD Learning Event IFAD, Rome 23rd June 2015

Methods for M&E of Policy Advocacy

Overview

• Me and my background

• Why is it important?

• What do you want to evaluate?

• Some ways of thinking (frameworks)

• Some ways of doing it (tools)

• Some conclusions

Why do M&E?

• To learn about what works• To manage better• To account:– to donors– to recipients

Image source: LQ-CN7499 World Bank

What to evaluate?

• Do the policies deliver better outcomes?

• Do the policies support improved education?

• Do your programmes influence education policy?

• Does your programme do it well?

Monitoring and Evaluation

Agenda Setting

DecisionMaking

Policy Implementation

Policy Formulation

Policy processes are complex

Civil Society

DonorsCabinet

Parliament

Ministries

Private Sector

The Working Paper

• Trends and issues in M&E of policy engagement.

• Frameworks for understanding policy engagement.

• Methods and tools for M&E of policy engagement.

• Case studies.

Image source: LQ-CN7499 World Bank

Current issues

• What is the purpose of M&E for policy engagement?

• How do you measure success?• Attribution vs Contribution?• How can you assess causality?

Tactics:Inputs, activities, outputs

Outcomes: Changes in behavior, relationships, policy

Impact: Changes in state, wellbeing, environment

Sphere of control

Sphere of influence Sphere of concern

Spheres of influence

The RAPID framework

The political context – political and economic structures and processes, culture, institutional pressures, incremental vs radical change etc.

The evidence – credibility, the degree it challenges received wisdom, research approaches and methodology, simplicity of the message, how it is packaged etc

External Influences Socio-economic and cultural influences, donor policies etc

The links between policyand research communities – networks, relationships, power, competing discourses, trust, knowledge etc.

Eight “policy” outcomes

1. Attitudes of stakeholders.

2. Public opinion.

3. Capacity/engagement of other actors.

4. Change in discussions among policy actors.

5. Improvements in policy processes.

6. Change in policy content.

7. Behaviour change for implementation.

8. Systems for supporting delivery of change.

A systematic approach1. Strategy and direction –are you doing

the right thing?

2. Management –are you doing what you planned to do?

3. Outputs – are the outputs appropriate for the audience?

4. Uptake – are people aware of your work?

5. Outcomes and impacts –are you having any impact?

1. Strategy and direction

2. Management

3. Outputs

4. Uptake

5. Outcomes and impacts

Logical frameworks

Goal Indicator MOV

Purpose Indicator MOV Assumptions/Risks

Output 1 Indicator MOV Assumptions/Risks

Output 2 Indicator MOV

Output 3 Indicator MOV

Output 4 Indicator MOV

√√

Scalar models & rubrics

0. No engagement with SDA.

1. Occasional meetings, no participation

2. Some meetings, poor communication

3. Some meetings, limited inputs

4. Regular meetings, opinion considered.

5. …. and can monitor implementation

6. …. and consulted on proposed plans

7. …. and influence plans

8. …. and documented and shared

9. Commitment to action

10. Policy or practice change.

Impact logs

Impact: stories of change

1. Stories of change:• Situation before• Context• Situation after• What changed and why

http://www.mande.co.uk/docs/MSCGuide.pdf

2. Most significant change (Davies) • Stories of change from

different stakeholders• Systematic analysis of significance.

3. Micro-narratives (Snowden)

Stories of change (DFID 10%)

1. What is the story about?2. Is there any other important

background information?3. What was the existing behaviour?4. What did the RPC do that

contributed to the change?5. What was the behaviour change?6. What factors helped make this

change successful?7. What comes next for the key actors?8. Where can we find more information or

supporting evidence?

Stories of change (TTI Evaluation)

http://www.redstonestrategy.com/reports/global-think-tank-network/

Redstone Strategy

Case studies

RAPID Outcome Assessment Assessment

www.odi.org.uk/RAPID/Publications/RAPID_WP_266.html

Conclusions

• What exactly are you trying to do?

• What do you want to know, why and who for?

• What is your theory of change?

• Are you seeing straight?

• Do the changes have anything to do with you?

• Are you changing?

• “Perfect is the enemy of good1”

1 Voltaire, La Bégueule: “Dans ses écrits, un sage Italien Dit que le mieux est l'ennemi du bien".