Monitoring and Evaluation of

Influence

John Young

2

Outline

• Why measure impact?

• What to measure?

• What sort of policy change?

• Planning/measuring in complex environments

• Behaviour change

• Some tools we use

• Other tools

• A systematic approach

• Recommendations

• Resources

2

3

Why should you do M&E?

• To learn• To manage better• To get more funds• To keep funders/clients happy

• (“what is the guarantee that doing this actually helps us?”)

3

4

What should you measure?

“If you don't know where you are going, any road will get you there”

4

5

Monitoring and Evaluation

Agenda Setting

DecisionMaking

Policy Implementation

Policy Formulation

Policy processes are...

Civil Society

DonorsCabinet

Parliament

Ministries

Private Sector

6

RAPID Outcome Mapping Approach

7

Policy objectives

• Discursive: Client-focused services

• Attitudinal: Farmers have good ideas

• Procedural: Participatory approaches to service development

• Content: UU20, UU25. New guidelines

• Behavioural: Approach being applied in practice

7

8

Visibility or substance?

Visibility Substance

Short term ‘relevant’ research Long term research

Focus on solutions for ‘agreed’ problems

Engage with the definition of the problem

Media exposure Lobby, network, horse trading

Briefing papers, Opinion pieces Estimates, costed proposals, policy options

Website, Blogs, Facebook, etc. Academic publications, long reports

Online communities with millions of hits

Communities with the right people

Delegations at high level global conferences

Private meetings at Party conferences and private meetings while planning for the high level conferences

Event focused influence Problem focused influence

Global Go-To-Survey Prospect magazine Think Tank of the year

8

9

The Cynefn Framework

9

10

Focus on behaviour change

Inputs Activities Outputs

Other Actors

Project Team

Outcome ImpactOutcomes ImpactOutcomes Impact

BehaviourChange10

11

Professionalisation of Public Services.Structural Adjustment → collapse of services. Paravet projects emerge.ITDG projects. Privatisation.ITDG Paravet network.Rapid spread in North.KVB letter (January 1998).Multistakeholder WSs → new policies.Still not approved / passed!

Episode Studies

1970s

1980s

1990s

2000s

Professionalisation of Public Services.Structural Adjustment

Privatisation ITDG Paravet network and change of DVS.

KVB letter (January 1998).Multistakeholder WSs → new policies.

ITDG projects – collaborative research.

The Hubl Study

Dr Kajume

International Research

11

12

Outcome Mapping

OUTCOME MAPPING:Building Learning and Reflection into Development ProgramsSarah Earl, Fred Carden, and Terry Smutylo

http://www.idrc.ca/en/ev-9330-201-1-DO_TOPIC.html

12

13

Social Network Analysis

www.odi.org.uk/RAPID/Tools/Toolkits/KM/Social_network_analysis.html13

14

RAPID Outcome Assessment

www.odi.org.uk/RAPID/Publications/RAPID_WP_266.html14

15

15 minute team debrief, conducted in a “rank-free” environment.

After Action Reviews

• What was supposed to happen?

• What actually happened?• Why was there a difference?• What can we learn from this?

15

16

Most Significant Change

1. Collect “stories of change” from different stakeholders

2. Systematic analysis of “significance”.

http://www.mande.co.uk/docs/MSCGuide.pdf16

1717

Impact Logs

1818

ODI Impact Logs

Contact: n.scott@odi.org.uk

19

Others....

• Classical case studies (IDRC, IFPRI)

• Stories of Change (Denning)

• Innovation Histories (CIAT)

• HERG Payback Framework (Hanney)

• Micro-Narratives (Snowden)

• Impact matrices (Davies)

• Peer evaluations (CHSRF)

• Systematic reviews?

• RCTs?

19

2020

A systematic approach

1. Evidence and advice:• Peer-review of outputs• Uptake logs• Outcome mapping

2. Public campaigns and advocacy• Surveys and focus groups• Media tracking logs• Media/public frame analysis

3. Lobbying• Records of meetings• Tracking people• Key informants

http://www.odi.org.uk/resources/details.asp?id=5252&title=monitoring-evaluation-me-policy-influence

21

1. Strategy and direction –are you doing the right thing?

2. Management –are you doing what you planned to do?

3. Outputs – are the outputs appropriate for the audience?

4. Uptake – are people aware of your work?

5. Outcomes and impacts –are you having any impact?

A systematic approach

1. Strategy and direction

2. Management

3. Outputs

4. Uptake

5. Outcomes and impacts

21

2222

3ie PIM Project – Initial ideas

23

Conclusions

Research to influence:• Clear objectives

• Understand the context

• Theory of change

• Iterative / learning approach

Measuring impact• Clear objectives• Theory of change• 5-levels• Multiple methods• Triangulation• Expect the unexpected

23

24

Recommendations

• Strategy: theory of change, impact pathway, peer review, log frame

• Management: appreciative inquiry, logs, AARs, PRINCE2

• Output: logs, peer-review,

• Uptake: logs, webstats, surveys

• Impact: outcome mapping, stories of change, episode studies, peer review

24

25

Resources

• Helping researchers become policy entrepreneurs. How to develop engagement strategies for evidence-based policy-making. John Young and Enrique Mendizabal. ODI Briefing Paper 53. 2009 - http://www.odi.org.uk/resources/details.asp?id=1127&title=become-policy-entrepreneur-roma

• Outcome Mapping: Building Learning and Reflection into Development Programs. Sarah Earl, Fred Carden, and Terry Smutylo IDRC - http://www.idrc.ca/EN/Resources/Publications/Pages/IDRCBookDetails.aspx?PublicationID=121

• Making a difference: M&E of policy research - ODI Working Paper 281. Ingie Hovland. July 2007. http://www.odi.org.ukwww.odi.org.uk/resources/docs/2426.pdf

• A guide to monitoring and evaluating policy influence. Harry JOnes. ODI Background Notes, February 2011 - http://www.odi.org.uk/resources/details.asp?id=5252&title=monitoring-evaluation-me-policy-influence

25