wifi: Nesta Guestpassword: flourish01follow us on twitter @nesta_ukofficial event hashtag #sparkEC

Testing social policy innovation

Wednesday 12th February 2014

Simon Flemington, Chief Executive Officer, LSE Enterprise

wifi: Nesta Guestpassword: flourish01follow us on twitter @nesta_ukofficial event hashtag #sparkEC

Testing social policy innovation

wifi: Nesta Guestpassword: flourish01follow us on twitter @nesta_ukofficial event hashtag #sparkEC

Testing social policy innovation

Hélène Giacobino, Executive Director, J-PAL Europe

SPARK London Feb 12, 2014

Improving Policies and Building Knowledge: the Role of Creative Experimentation

Hélène GiacobinoExecutive Director

J-PAL Europe

SupSupport services for social policy experimentation in the EU port services for social policy experimentation in the EU

SPARK London Feb 12, 2014

The Need for Rigorous Evidence

• In a context of economic downturn, it is important to optimize public expenditure

• Promoting the implementation of rigorous evaluation methods to test the effects of new policies is a good way to achieve this goal

• Evidence-based policy is possible and highly effective• As long as we learn from our success and mistakes, this is fine,

because even a failed program helps us understand what went wrong

• Without rigorous evaluation, everybody can favor their own pet project – and even lessons from successes are lost

SPARK London Feb 12, 2014

Social Policy Experimentation

• Social policy experimentation tests the validity of policies by collecting evidence about the real impact of interventions on people

• The goal is:– to bring innovative answers to social needs,– to test impact on small scale interventions,– to scale-up the ones whose results were convincing

SPARK London Feb 12, 2014

How to do SPE?

• It is possible to undertake SPE through different methods:• Some of the most common non randomized methods are:

- Pretest-posttest (Before and after)- Differences-in-Differences- Regression discontinuity- Statistical Matching

• They always rely on strong assumptions

• Randomized Controlled Trials (RCTs) give the most rigorous results (internal validity)

SPARK London Feb 12, 2014

RCTs: a long history• Experimental psychology: late 19th century,• Education: early 20th century• Experimental sociology, early 20th century:

– rural health education,– social effects of public housing,– recreation programs for delinquent boys

• Large-scale randomized clinical trials: a norm since 1962 Drug Amendments

• Went through substantial debates but today widely accepted

• A boom in the 60’ in the USA, (250 RCTs done)• 90’: J-PAL introduce RCTs in development economics (today: 450

RCTs)

SPARK London Feb 12, 2014

BUT … not every intervention can be evaluated

• A well designed SPE should include:- an explicit and relevant policy question- a valid identification strategy- a well-powered sample - high quality data.

• Evaluation is not appropriate when:– the sample size is too small– the impact to measure is a macro impact– the scaling-up of the pilot will modify the impact a lot– the beneficiaries are in a context of urgency

9

SPARK London Feb 12, 2014

AND … not every intervention warrants an impact evaluation

• Investing in some key rigorous impact evaluations is essential to guide policy decisions and to responsibly allocate public resources

• Rigorous impact evaluations should be used to test key influential, strategically relevant or novel interventions

• No added-value to evaluate policies or programs that will benefit a very limited number of people, or to test a policy question that has been already rigorously evaluated in a similar context (except within explicit validation program)

10

SPARK London Feb 12, 2014

Ethical Issues

• SPE should be designed to follow the ethical principles applicable to evaluations and research projects involving human subjects (even if there are no rules in the country of experimentation) and be approved by an IRB.

• SPE should include rigorous protection of individual data which is paramount

• SPE can have different designs to secures fairness, impartiality and transparency (and cost effectiveness!)

SPARK London Feb 12, 2014

Financial Issues

• Rigorous SPE need good quality data: this is why, if the sample is large and you don’t have existing data, it may be costly.– but financing large-scale interventions without knowing

their effects can potentially result in a waste of resources!

• The costs depend on the program evaluated: it is more expensive if you are dealing with long-term impact

SPARK London Feb 12, 2014

The « Black Box » Issue

• One common critique: “RCTs can provide whether an intervention was effective, and even measure the size of the impact. But they cannot answer the question of how or why the impact came about.”

• This is true only if final outcomes are measured. RCTs where intermediate indicators that map to the “theory of change” are collected, and where qualitative methods are also used, can help us understand the how and why.

SPARK London Feb 12, 2014

Randomization is not a substitute for theory

• Although randomization guarantees the internal validity of the estimate, in order to interpret the result, you need a theoretical framework:The extent to which findings do or don’t generalize beyond a specific context depend on theory: only theory tells you what is likely to matter in the context, and guide replications

• Value of randomization is that you can more easily be surprised:–You cannot doubt the results when you found them: if they are surprising, instead of shelving the results, you (and others) have to think about what happened.

SPARK London Feb 12, 2014

External Validity: an issue for all types of evaluation

• External validity: the extent to which we can be confident that the results found in one context will generalize to other contexts

• External validity is a function of the program being evaluated (where is it being implemented, how replicable is the program model, how much does program implementation depend on context or interaction with the community), not of the evaluation methodology!

• Nonrandomized impact evaluations are undertaken on a specific program in a specific location, with the downside that they do not control for selection bias weaker internal validity

• A randomized evaluation tests a particular question in a specific location, at a specific time and at a specific scale. If properly conducted, and it has strong internal validity

• If we cannot be confident that the evaluations measure the true impact of the program in a specific context, then we can be less confident in generalizing conclusions to another context

SPARK London Feb 12, 2014

External validity: an issue for all types of evaluation

• The scope of an evaluation’s true external validity depends on how the evaluation sample is determined:

• if the evaluation sample is representative of the target population (randomly sampled from a larger population), then the results are generalizable to that population.

SPARK London Feb 12, 2014

• Replication is a strategy that enables us to understand how an intervention functions in various settings

• SPE, as they force researchers to pay attention to context, details, and realities on the ground, allow to test broad theories in a credible way, and produce evidence that can then feed back into our theories

• Theory and process evaluation data can help us understand the mechanisms through which the impact was produced and therefore to scale-up programs more confidently

External validity: solutions

SPARK London Feb 12, 2014

Conclusions• Social Policy Experimentation is an important tool to help improve

social programs

• It is important to strategically allocate evaluation resources to key influential, strategically relevant or novel interventions:– to facilitate scale-up– to encourage the replication of the policy in different contexts– to provide valuable information for future policymaking

• Yet not every intervention can be evaluated• And not every intervention warrants an impact evaluation

• Experimentation needs to be creative…– If we are just trying, and accept the possibility of failure, we do

not need to think inside the box – This mindset could revolutionize social policy

wifi: Nesta Guestpassword: flourish01follow us on twitter @nesta_ukofficial event hashtag #sparkEC

Testing social policy innovation

Phil Sooben, The Economic & Social Research CouncilDr Simona Milio, LSE EnterpriseHélène Giacobino, J-PAL EuropeJonathan Breckon, NestaArnaud Vaganay, LSE Enterprise