Four key tasks in Four key tasks in impact assessment of impact assessment of complex interventionscomplex interventions

Rethinking Impact: Understanding theComplexity of Poverty and Change

26 - 28 March 2008Cali, Colombia

Professor Patricia RogersRoyal Melbourne Institute of Technology, Australia

Some contextSome context1997 Causal models for evaluation, Harvard Project on 1997 Causal models for evaluation, Harvard Project on

Schooling and ChildrenSchooling and Children

2001 Impact evaluation research project, Department of 2001 Impact evaluation research project, Department of Primary Industries, Victoria, AustraliaPrimary Industries, Victoria, Australia

2006 Qualitative cost benefit analysis of Stronger Families 2006 Qualitative cost benefit analysis of Stronger Families and Communities Strategy, Australiaand Communities Strategy, Australia

2008 NONIE (Network Of Networks on Impact Evaluation) 2008 NONIE (Network Of Networks on Impact Evaluation) – all multilateral banks, all UN agencies, all aid – all multilateral banks, all UN agencies, all aid agencies of OEDC countriesagencies of OEDC countries

2008 Edinburgh Summer School, Health Scotland ‘Getting 2008 Edinburgh Summer School, Health Scotland ‘Getting real about impact’real about impact’

2008 NORAD Conference ‘Evaluating the complex’2008 NORAD Conference ‘Evaluating the complex’

What is impact?What is impact?

……the positive and negative, primary and the positive and negative, primary and secondary long-term effects produced by a secondary long-term effects produced by a developmentdevelopment intervention, directly or intervention, directly or indirectly, intended or unintended. These indirectly, intended or unintended. These effects can be economic, socio-cultural, effects can be economic, socio-cultural, institutional, environmental, technological institutional, environmental, technological or of other typesor of other types..

DAC definitionDAC definition

Different types of impact may need Different types of impact may need different methods different methods

Long-term impacts that will only be evident in future generations OR Short term impacts evident during the life of a project and an evaluation

Transformational impacts which are likely to remain once achieved OR fragile impacts which can be easily undone

‘Smoking gun’ impacts where one intervention is both necessary and sufficient to produce the impact OR …

Different types of intervention may need Different types of intervention may need different methods different methods

Simple interventions that can be tightly specified and standardized intervention (e.g. product, technique)

Complicated interventions that are part of a larger multi-component intervention

Complex, emergent program or policy (e.g. community development, natural resources management, emergency situation)

Different types of impact assessment Different types of impact assessment may need different methods may need different methods

Intended use and intended users:

• Knowledge building for replication and upscaling (by others?)

• Knowledge building for learning and improvement

• Accountability – to whom, for what, how?

Timing:

• Ex-ante

• Built into implementation

• Retrospective

Four key tasks in impact assessmentFour key tasks in impact assessment

A) Decide impacts to be included in assessment (conceptualise valued impacts)

B) Gather evidence of impacts (describe and/or measure actual impacts)

C) Analyse causal attribution or contribution

D) Report synthesis of impact assessment and support use

Each of these tasks requires appropriate methods.

A. Decide impacts to include.A. Decide impacts to include.

Identifying and negotiating impacts to include:Identifying and negotiating impacts to include: Not only stated objectives – also unintended outcomes

(positive and negative) Values about good and bad impacts and about the

distribution of impacts Conceptualising how impact occurs Prioritising information needs Adequate consultation and legitimisation

Some approaches:Some approaches: Program theory (impact pathway) - possibly

developing multiple models of the program, eg Soft Systems, negotiate boundaries (eg Critical Systems Heuristics)

Participatory approaches to values clarification –eg Most Significant Change

B. Gather evidence of impacts.B. Gather evidence of impacts.Balancing accuracy, utility, feasibility and ethics:Balancing accuracy, utility, feasibility and ethics:

Dealing with time lags before impacts are evident Avoiding accidental or systematic distortion of level of

impacts Evidence that is sufficiently comprehensive Making use of existing data as well as additional data Valuing the impacts

Some approaches:Some approaches: Program theory (impact pathway) – identify short-term

results that can indicate longer-term impacts Participatory approaches – engaging community in

evidence gathering to increase reach and engagement Real world evaluation – mixed methods, triangulation,

making maximum use of existing data, strategic sampling, rapid data collection methods

C.C. Analyse causal contribution or attributionAnalyse causal contribution or attributionAvoiding false negatives and false positives:Avoiding false negatives and false positives:

Systematic search for disconfirming evidence and analysis of exceptions Distinguish between theory failure and implementation failure Understanding the contribution of context: implementation environment,

participant characteristics and other interventions

Some approaches:Some approaches: Addressing through design –

eg experimental designs (random assignment) and quasi-experimental designs (construction of comparison group eg propensity scores)

Addressing through data collection – eg participatory Beneficiary Assessment, expert judgement

Addressing through iterative analysis and collection – eg Contribution Analysis, Multiple Levels and Lines of Evidence (MLLE), List of Possible Causes (LOPC) and General Elimination Methodology (GEM), systematic qualitative data analysis, realist analysis of testable hypotheses

D. Report synthesis and support useD. Report synthesis and support use

Providing useful information to intended Providing useful information to intended users:users:

Balancing overall pattern and detail Assisting uptake/translation of evidence

Some approaches:Some approaches: Utilization-focused evaluation - Identification and

involvement of intended users from the start Layered reports (1 page, 5 pages, 25 pages) Scenarios showing different outcomes in different

contexts Workshopping report to support knowledge translation

Following a RecipeFollowing a Recipe A Rocket to the MoonA Rocket to the Moon Raising a ChildRaising a Child Formulae are critical

and necessary

Sending one rocket increases assurance that next will be ok

High level of expertise in many specialized fields + coordination

Rockets similar in critical ways

High degree of certainty of outcome

Formulae have only a limited application

Raising one child gives no assurance of success with the next

Expertise can help but is not sufficient; relationships are key

Every child is unique

Uncertainty of outcome remains

Complicated Complex

•The recipe is essential

•Recipes are tested to assure replicability of later efforts

•No particular expertise; knowing how to cook increases success

•Recipes produce standard products

•Certainty of same results every time

Simple

(Diagram from Zimmerman 2003)(Diagram from Zimmerman 2003)

SIMPLESIMPLE COMPLICATEDCOMPLICATED COMPLEXCOMPLEX

What works?What works? What works in What works in what contexts? what contexts? (implementation (implementation environments environments and participant and participant characteristics)characteristics)

What works What works here and now?here and now?

What do we What do we mean by mean by ‘works’?‘works’?

Types of interventionsTypes of interventionsSimple intervention Simple intervention Complicated or complexComplicated or complex

interventionintervention

Single causal strand. Intervention is sufficient to

produce the impacts

Multiple simultaneous causal strands required to produce the

impacts

Universal mechanism. Intervention is necessary

to produce the impacts

Different causal mechanisms operating in different contexts

Linear causality, proportional impact

Recursive, with feedback loops, leading to disproportionate

impact at critical levels

Agreement and certainty about pre-identified

outcomes

Disagreement and uncertainty about outcomes and emergent

outcomes

Simple causal attributionSimple causal attributionintervention is both necessary and sufficient to intervention is both necessary and sufficient to

produce the impactproduce the impact

Impact

Intervention

Causality in an INUS worldCausality in an INUS worldIntervention is an Insufficient but Necessary part of a Intervention is an Insufficient but Necessary part of a

causal package which is itself Unnecessary (not the only causal package which is itself Unnecessary (not the only way) but Sufficient to produce the impactway) but Sufficient to produce the impact

Impact

Intervention

Other factor –context or

contribution of another

intervention

Alternative intervention

Mackie, 1974, Mark, 2001

Causal analysis of problemsCausal analysis of problemsPredisposition Triggering events or conditions

Impacts

Causal analysis of interventionsCausal analysis of interventionsIntervention Favourable context

Impacts

Eg of developing testable hypotheses Eg of developing testable hypotheses not involving a comparison groupnot involving a comparison group

Change in rate of all road fatalities

Change in rate of road fatalities on Fri and Sat nights

SimpleSimple ComplicatedComplicated ComplexComplex

Deciding Deciding impactsimpacts

Likely to be agreed

Likely to differ, reflecting different agendas

May be emergent

Describing Describing impactsimpacts

More likely to have standardised measures developed

Evidence needed about multiple components

Harder to plan for given emergence

Analysing Analysing causecause

Clear counter-factual likely

Causal packages and non-linearity

Unique, highly contingent causality

Reporting Reporting and and supporting supporting useuse

Clear messages

Complicated message

Uptake requires further ongoing adaptation