Randomisation Bias and Post-Randomisation Selection Bias in RCTs:

Barbara Sianesi Institute for Fiscal Studies

September 14, 2006

The role of non-experimental methods in the ERA demonstration

Randomised Controlled Trials in the Social Sciences: Challenges and Prospects

The University of York

Talk Outline

RCTs are the gold standard in evaluation BUT not immune from limitations

1. parameter retrieved2. outcomes that can be looked at

Judicious combination with non-experimental methods can enhance (under suitable assumptions!) what can be learnt from a RCT

Excellent example to illustrate this: ERA

What is ERA

a new package of support financial incentives (job retention,

training) to assist ND25+ and NDLP customers

obtain, retain and advance in work evaluated via RA (~14,000 in 6 districts)

Issue #1

Some eligibles in the ERA Districts did not reach the decision stage or refused to take part in research

scheme

Experimental contrast unbiased estimate of ERA impact for those who have reached the RA stage & have agreed to participate.

ERA impacton ERA

participants

Staff• discretion. choice of marketing strategy

Customer• finding job unlikely• finding job likely but no desire to stay in touch with JCP• antipathy to government, systems of support, mandatory programmes• resistant to change, …

Staff• Some may not think customer would benefit / be interested.• Some non-ERA advisers may think customers close to job entry may provide quick win.

?

?

Non-participants

Why do non-participants pose a potential issue?

Would have liked experimental estimate of impact of ERA for the full eligible population in the ERA Districts.

Benchmark: pilot/control area evaluation In ideal scenario

staff would offer ERA to any eligible (no discretion)

all eligibles would participate (no need for consent)

Why do non-participants pose a potential issue?

Would have liked experimental estimate of impact of ERA for the full eligible population in the ERA Districts.

But: ERA tested only on a subset of ERA eligibles in ERA Districts – the participants.

How to view this

a) Interested in impact on eligibles but only get impact on participants.

“Randomization Bias occurs when random assignment causes the type of persons participating in a program to differ from the type that would participate in the program as it normally operates.”

(Heckman and Smith, 95, p.99)

How to view this

b) Focus on what the RCT consistently estimates (impact on participants) and interested in how well it generalizes to wider population (impact on eligibles).

Issue of External Validity.How representative of the full eligible population?

When are non-participants a problem?

E(impact | eligibles) What we want

When are non-participants a problem?

E(impact | eligibles)

=E(impact | eligible partic.)

what we want

what we get

When are non-participants a problem?

E(impact | eligibles)

=E(impact | eligible partic.) Prob(eligible partic.)

what we want

observed

E(impact | eligibles)

=E(impact | eligible partic.) Prob(eligible partic.)

what we get

When are non-participants a problem?

E(impact | eligibles)

=E(impact | eligible partic.) Prob(eligible partic.)

+E(impact | eligible non-part.)

what we want

what we get

observed

?

When are non-participants a problem?

E(impact | eligibles)

=E(impact | eligible partic.) Prob(eligible partic.)

+E(impact | eligible non-part.) Prob(eligible non-

part.)

what we want

what we get

?

observed

When are non-participants a problem?

eligibles – partic = (nonpartic – partic)Probnonpart

bias = p

It depends on: 1. relative size of eligible non-participant

group 2. whether it is very different from RA

group

I. Descriptive Analysis

I. Extent of non-participation

II. How different

a. observable characteristics

b. outcomes (non-participants vs. controls)

II. Non-Experimental Analysis

Issue #2

Does ERA enhance hourly wages (productivity) wage growth (advancement) employment duration (retention)

Cannot use experimental contrast due to post-randomisation selection bias into employment.

?

To conclude …

“Since experiments can answer only a subset of the questions of interest to the evaluator, it remains important to build up the stock of basic social science knowledge required to successfully utilize non-experimental methods, both in themselves and as tools for more extensive analyses of experimental data.”

Heckman and Smith (1995, p.95)