Criteria for Assessing The Feasibility of RCTs. RCTs in Social Science: York September 2006 Today’s Headlines: “Drugs education is not working” “ having

Criteria for Assessing The Feasibility of RCTs

RCTs in Social Science: York September 2006

Today’s Headlines: “Drugs education is not working”

“ having reviewed research from across the world, a committee of doctors and scientists on the ACMD* concluded that the success of school-based schemes was "slight or non-existent" and could even be "counter-productive".

*Advisory Council on the Misuse of Drugs (ACMD)


Why are we here?•We share the goal of improving well being for society and individuals

• Interventions which we support as a society should benefit individuals and society and should not cause harm

•We value evidence


Who am I?•Economist•Managing Director of Matrix Research & Consultancy Ltd

•Member of the Campbell Crime and Justice Group

•Member of the Cochrane Campbell Economics Methods Group

•Advocate of improving the use of evidence to inform decisions


Who are you?•Politicians?

•Policy advisers?

•Civil Servants?

•Practitioners?

•Researchers?

•Economists…..?


Important evaluation questions (adapted: Bain 1999)

•Should it work? Theory

•Can it work? Implementation

•Does it work? Impact

• Is it worth it? Value


Source of interventions•Historical practice (“we have always done it this way”)

•Taught practice (“we teach people to do it this way”)

• Innovative practice (“I try new ideas”)

•Research (“I examine the theories”)

the popular vote………………


Role of trials for new medical interventions• Idea•Basic science•Laboratory trials•Clinical trials•Licence•Approval (e.g. NICE)

Average 10-15 years?


Particular challenges for trials in social science (adapted: Farrington 1983)

• Pace of idea to practice

• Theory base of treatment

• Definition and scope of treatment

• Context complexity

• Community vs individual interventions

• Hawthorne effects

• Contamination

• Outcome measures & measurement

• Duration and decay


Arguments for Trials• Interventions can do harm • If undertaken well trials minimize risk of bias

•Basic hypothesis: “can we identify an effect with confidence?”

• If results insignificant then:–There isn’t an effect; or–There is an effect but we haven’t detected it with confidence.


Arguments against trials•Epistemology: “the world doesn’t work like this”

•Analytical: “there are too many analytical constraints”

•Ethical: “you can’t deny treatment”

•Legal: “you might be challenged if you deny treatments”

•Logistical: “there are too many practical constraints


Research on FeasibilityResearch partners

•Matrix Research & Consultancy Ltd

•The Jerry Lee Centre, University of Pennsylvania.

Research base

•Collective research experience

•Home Office funded study for OBPs


Analytical challenges:Hierarchy

1. Internal validity: can we attribute effect to intervention?

2. Statistical power: can we measure the effect with confidence?

3. External validity: can we generalise the results?


Hurdles to ensure Internal Validity•Participant selection targeting

•Completion and attrition rates

• Inconsistent treatments & treatment measurement

•Multiple outcomes and inconsistent outcome measurement

• Independent, contamination-free alternative, treatments or no treatment


Hurdles for Statistical Power•Understanding expected/required effect size?And….Given this…•How to maximize statistical power given?

–Heterogeneity of sample?–Completion rates?–Attrition rates?

And…given this…•How many participants do you need?


Hurdles for External ValidityContext / Characteristics:

• Local service provider

•Participants

•Community

Tension between correcting for these and the increased challenges of achieving internal validity and statistical power.


Ethical Hurdles

• “denial of treatment is unethical”

• “validity of informed consent”


Legal Hurdles•E.g. for offenders tension between random assignment and requirements to sentence

•Scope of legal challenge:

–from those denied the intervention

–where programmes exist and are perceived to reduce risk to public/ stakeholders


Logistical Hurdles•Quality of trial staff

•Cost of a trial relative to the cost of the intervention and the value of the expected effect

•Sample size required when completion rates are low and attrition post completion is high

•Avoidance of Hawthorne effect


Are there Solutions?• Focus on particular well defined interventions• Appoint independent trained research teams • Randomise early• Separate control groups • Pick control treatments which enable effects to be measured• Manage and monitor implementation• Ensure programme stability • Determine sample size from desired effect size – look at the theory

and related research • Measure outcomes consistently • Identify cost effective study design• Educate and inform participants and stakeholders


If these aren’t available?Only then ask yourself:

“What is the next best alternative….and how can I minimize the risk of getting it wrong”


Final reflections•Researchers should recognise:

–decisions need to be made in real time–all information can provide evidence

•Users should recognise:–the risks of making the wrong decision increases as the quality of the evidence base declines

• In social science we are addressing the needs of people and communities who are vulnerable

Documents

Criteria for Assessing The Feasibility of RCTs. RCTs in Social Science: York September 2006 Today’s Headlines: “Drugs education is not working” “ having