Intervention Logic A Presentation to the Pathfinder Project Karen Baehler Victoria University of Wellington 463 5711 [email protected]

Intervention Logic

A Presentation to

the Pathfinder Project

Karen Baehler

Victoria University of Wellington463 5711

[email protected]

Victoria University of Wellington 2

The problem

Citizens want to know if government is making a difference. Are we getting results in return for our taxes?

Ministers want better advice: “The most common problems which can be discerned in recent experience are:

• overstatement of what will be achieved;• under-explanation of how policy actions will achieve the

claimed outcomes.”

From Improving Policy Advice (1993) by G. R. Hawke, p. 27


The solution

• Identify goals (outcomes)• Chart a course to those goals• Measure current progress• Stop things that don’t work• Alter things that sort of work• Keep improving things that do work• Discover/invent new avenues to success


Outline

• What is intervention logic?

• What are its prerequisites?

• What are its uses?

• What are its blind spots?

• How do we minimise the blind spots?

• How do we know if an IL is working?


What is intervention logic?

• A testable theory of causation– Linked “if-then” statements– Action/reaction pairs

• A chain of conditions to be achieved• Ultimate/end outcome

= policy goal

• Intermediate outcomes and immediate impacts– Lead to the end outcome– But are not ends themselves

• A basis for confirming performance


Start with a backbone

• The vertical dimension of IL• Outcomes logic, not processes or activities

– Outcome grammar– Connectedness

• The necessary but not sufficient rule

• Advisor’s mindset– Optimistic– Skeptical


The “If you build it, they will come” backbone

• Political benefits of simplicity

• Analytical pitfalls

• What’s wrong with this backbone?

• Can the matrix fill in the gaps?

Reducetraffic

congestion

Buildbypass

Peopledriveon it

Output

Immediateimpact

End outcome


The more complex backbone

• Add intermediate outcomes

• More assumptions about indirect causation

Victoria University of Wellington 9Output: School vouchers

Ultimate outcome: Increased educational achievement/smarter kids

Parents aware of & understand program

Parents possess “appropriate” information about schools

Parents choose “best” schools for their children

“Good” schoolsgain pupils

“Bad” schoolslose pupils

“Good” schoolsget even better

Some “bad”schools fold

Some “bad”schools lifttheir game

New “good”schools come

on line


The black box at the top of the backbone

Social policy: Note the large leaps in logic that often occur at the top

Client responds wellto services

Client’s behaviour changes(How? Why?)

Ultimate outcomerealised


Plot twists

• When is an intermediate outcome also an end outcome?

• Can one agency’s / department’s intermediate outcome be another agency’s end outcome?


The backbone as a manage-ment tool: Links below outputs

Inp u t 1 Inp u t 2 Inp u t 3

A c tiv ity 1 A c tiv ity 2 A c tiv ity 3

O utp u t 1 O utp u t 2 O utp u t 3

Im m ed ia te im p a c ts

Source: R Waite


The backbone as a risk ID tool: Collateral outcomes

A c tu a l risks (p = ? )

U lt im a te o u tco m e(u n in te nd e d)

In te rm ed ia te ou tco m e(u n in te nd e d)

Im m e d ia te im p a ct(u n in te nd e d)

O u tp u t

Im m e d ia te im p a ct(in te nd e d)

In te rm ed ia te ou tco m e(in te nd e d)

U lt im a te o u tco m e(in te nd e d)


What are IL’s prerequisites?

• Agreed outcomes for the top row– Sources

• Statement of intent• Agency/departmental mission

– The importance of first principles review– The role of problem definition

• Outcomes (goals) are the flipside of problems• The “problem logic” model and the black box

• Intervention option(s) for the bottom row• Common sense


Group Exercise 1

• Work in pairs

• Choose a familiar policy/output from your work or from the news

• Produce a backbone linking the output to intermediate and ultimate outcomes

• Identify strong and weak links

Move to a matrix (Funnell 1997)

1

Outcomes

Hierarchy

2 Success Criteria

3

Factors

Within

Control

4

Factors

Outside

Control

5

Activities & Resources

6

Perfor-

mance

Ultimate outcome

Inter-mediate

outcomes

Immediate

outcomes

Output


What are its uses?

Conventional uses• Testing existing

policy hypotheses• Testing performance• Improving impacts

through design & management of risk

• Making better use of existing data

Unconventional uses• Comparing policy

options• Identifying generic

intervention templates for a department

• Discovering/inventing new interventions


Testing existing policy hypotheses

• If X, then Y• Y = f (X)

– Does the raw logic hold? (ex ante)– Does the available evidence support the logic? (ex

ante and ex post)– What additional evidence is needed to test the

logic?

• IL breaks an impact evaluation into chunks.


Testing/confirming performance

Column 6 in the IL matrix allows us to

• disaggregate performance into chunks

• distinguish chunks that are working well from those working less well– based on achievements compared agains

success criteria/targets


Improving impacts

• Identify conceptual and operational gaps in existing policy

• Target issues for review (weak links)

• Monitor– Internal and external risks– Counter-intuitive causes and effects

• Revise design


Making better use of data

• Evidence need not relate to ultimate/end outcomes to be useful

• Findings to date (from NZ or international) may shed light on immediate and intermediate links in the chain

• Role of research in the “problem logic”• Examples

– School choice research and its place in the IL


Comparing policy options (via the conventional matrix)

Criteria Option A Option B Option C

Life years saved

SA1 SB1 SC1

New incidents prevented

SA2 SB2 SC2

Equity SA3 SB3 SC3

Multiple outcomes = unlinked “criteria”


Using IL to compare options

• Compare #’s of links– More links = more

chances to stuff it up/more resources required?

– More links = less uncertainty, more robust theory?

– Fewer links = political plus?

• Compare #’s and magnitude of weak links

• Compare #’s and magnitude of possible unintended outcomes

Step 1: Prepare a backbone for each option



Step 2: Prepare an IL matrix for each option



See next slide• Compare risks across

A, B, C• Compare resources

needed A, B, C• Compare performance

contract possibilities• Compare evaluability

IL sets up more accurate cost-effectiveness analysis– Remove

unnecessary steps before costing

– Identify possible sources of extra costs

Step 3: Compare across IL matrices


A cross-cutting matrix

Option A Option B …

Outcomes (1)* Weak logic Strong logic

Success criteria (2) Measurable Unmeasurable

Internal control (3) High Low

External risks (4) Low High

Costs (5) $ per X $ per X

Institutional capacity (5)

High Low

Management (2-6) ? ?

*Numbers in parentheses refer to columns in the IL matrix


Identifying IL templates

• The case management model – Generic steps (slide 28)– Early intervention example

• The information campaign model– Generic steps (slide 29)

• The deterrence model– Mandatory sentencing laws example (slide 30)

• The pollution permits model– GHGs example (slide 31)

Victoria University of Wellington 28Output: Target group enters programme

Individual’s needs & prospects assessed accurately

Realistic objectives set for (and with) the individual

Individualised programme put in place to meet objectives

Short-term objectives for individual progressively achieved

Life circumstances/chances of individual are improved; long-term objectives are achieved

Reduced long-term costs and/or increased long-term benefitsto the community

Victoria University of Wellington 29Output: Educational literature produced

Literature passes pretest for readability, etc.

Appropriate audience receives literature

Audience reads literature

Readers learn the facts

Readers change their opinions

Readers change their behaviour

Readers influence others to change opinions/behaviour

Behaviour change leads to improved outcomes

Victoria University of Wellington 30Output: Mandatory prison sentences for crime X

Judges understand and apply them

More Xoffenders

jailed longerPast & potential offenders

aware of sentencing

Past & potential offenders includenew X sentencing risk in their

personal decision making

X offenders workharder to avoidapprehension Potential offenders

avoid crime X Fewer Xoffenders on

the street

Rates of crime X

Victoria University of Wellington 31Output: Tradable emissions permits created for GHGs

Permits auctioned to bidders (or other allocation made)

Plants calculate costs & benefits of investing in cleaner technologyv buying add’l permits v paying

fines for excessive GHGs

Regulatorssanction

Some plants invest in

clean R & D andtechnology

Innovations in cleantechnology diffuse

Govt investsauction revenuein clean R&D

Some plants buy add’lpermits

Some plantsemit GHGs

above permit

“Clean”firmsprofit

GHGs& costs


Discovering/inventing new interventions

• The brainstorming approach– Pick generic policy instruments– Apply to the problem at hand, using quick,

back-of-the-envelope backbones

• The engineering approach– Start with the “problem logic”– Find the entry points in the model– Fashion interventions for the entry points


Group exercise 2

• Same pairs• Choose an end/ultimate outcome and make it

the top “vertebra” of a backbone• Work down to identify intermediate outcomes

that might lead to that end outcome (based on your knowledge of how that outcome is “naturally” produced)

• What interventions suggest themselves as you move down?


What are IL’s blind spots?

• Might the chain of outcomes look different for different population groups of interest?

• Might risk factors differ across groups?• Might different groups need different

activities and resources to reach each intermediate outcome?

Equity


What are IL’s blind spots?

• Hidden portions of the backbone– Inputs and activities (below)– Collateral outcomes (beside)– Program/theory assumptions (beside)

• Getting trapped in a paradigm– Tikanga v cognitive-behavioural paradigms for

explaining crime

• Focusing on the lower levels of the hierarchy, where managers have more control


How do we minimise the blind spots?

• Research that contributes to robust problem logics– The poverty example– The drug harms example

• Evaluation that contributes to robust intervention logics– The welfare to work example

• Outcomes that reflect actual results in the community/real consequences


How do we know when an IL is working (or not)?

• Does it help us distinguish between apparently more and less promising interventions?

• If it just rationalises everything, not robust• Does it systematically favour some types of interventions

over others? Why? Is this warranted (cross check)?

• Does it help us make better use of existing evidence?

• Does it help us generate a research agenda?


Ex ante criteria for a good IL• Proper “grammar” in the backbone

– Outcomes, not processes or activities

• Each intermediate outcome represents a necessary but not sufficient cause of the next outcome

• Success criteria are measurable and lend themselves to targets

• Activities and resources cover all of the key factors within the programme’s control

• Activities and resources supply what is needed to get from one outcome to the next


Ex post criteria for a good IL

• Are outcomes being produced more cost effectively than prior to use of IL?

• Are intended and unintended outcomes predicted more accurately?

• Are there fewer unintended outcomes?• Are there fewer unexpected

outcomes/surprises?• Is the department accumulating better

information about its own performance?


How can IL evolve?

• Through multiple applications, learning by doing

• Through cross-breeding with other soft & hard systems approaches

• Through peer review

Challenges to be met– Accounting for new

sets of surprises– Facilitating cross-

departmental thinking on partnerships for particular outcomes

– Facilitating equity analysis

– Other

Documents

Intervention Logic A Presentation to the Pathfinder Project Karen Baehler Victoria University of Wellington 463 5711 [email protected]