Stopping Trials for Futility

RSS/NIHR HTA/MRC

1 day workshop

11 Nov 2008

Programme11.00 - 11.05 Welcome

11.05 - 11.20 Introduction

11.20 - 11.40 Andy Grieve

11.40 - 12.00 Carl-Fredrik Burman

12.00 - 12.20 John Whitehead

12.20 - 12.50 Questions

12.50 - 1.30 Lunch

1.30 - 2.45 Workshop discussions

2.45 - 3.15 Feedback and discussions

3.15 - 3.30 Summary and final voting

Workshops

• There are 4 workshop groups– 2 in the main lecture theatre– 1 in the Nightingale room – 1 in the Council chamber

• All should have been allocated to a workshop group and location

• These discussions will be facilitated by– Jenny Freeman (University of Sheffield)– Jon Nicholl (University of Sheffield)– Louise Dent (HTA)– Andrew Cook (HTA)

Workshops

• Specific questions to be addressed are:– Should we plan to do futility analyses in

publicly funded trials?– In what circumstances are they appropriate?– Should funding bodies insist upon them

before granting trial extensions?

• Three case studies have been supplied to facilitate discussions

Futility analysis

Jon Nicholl

ScHARR, University of Sheffield

Publicly funded trials

• Publicly funded trials are supported by MRC/NIHR

• Early phase trials are supported by MRC, later phases by NIHR

• Phase I – IIa DPFS (MRC), Phase IIb – EME (MRC/NIHR), phase III/IV – HTA

• Trials are also supported by NIHR through RfPB, applied programme grants, etc

HTA

• HTA budget = £88m pa at maturity• Nearly all to be spent on trials• Typical trial design

– Parallel group RCT of cost-effectiveness– 2 arm head-to-head or 3 arm with placebo– Multi-centre– 3-5 years (ts=0.5yr, tr=2yr, tf=1yr, ta=0.5yr)– £1m - £2.5m– N= 400 - 4000

Futility designs

• Futility analyses are not uncommon in Pharma trials

• But no HTA funded trials have planned a futility analysis

Why is this?Should futility analyses be designed into HTA trials?If so, in which trials?And when and how should they be done?

What is a futility analysis?

• A futility analysis is a calculation made during the course of a trial of the probability that the trial will produce helpful results

• A trial is judged futile if the probability is too small• Helpful results are usually taken to mean

‘statistically significant’ results for the primary endpoint

• So the calculation usually takes the form of a power calculation conditional on the results observed.

Should HTA do futility analyses?

• YES • Opportunity cost - spending public money on futile

trials is wrong because the money could be spent on other trials

• It is ethically wrong to continue to recruit patients to trials with little hope of achieving helpful results

• NO• All well conducted trials provide valuable evidence• The costs of designing trials with a planned futility

analysis would outweigh any savings

Futility analyses in HTA trials

Assuming they are not ruled out in principle– In what types of trial– In what circumstances– When and how

should we do them?

1. Types of trial

• In placebo controlled trials, futility suggests no worthwhile benefit (ie little or no chance of finding evidence that the intervention is better than placebo)– ie the treatment is futile as well as the trial

• In head-to-head trials this isn’t true– The treatment may be far from futile

So are futility analyses inappropriate in head-to-head trials?

2. Extensions• 54% of MRC/HTA trials seek extensions

• Usually due to long set up time and/or slow recruitment

• 55% of trials with extensions still don’t achieve N

• Cost of extensions = £50k - £1m

Extensions

• HTA currently asks for a ‘futility’ calculation“If the aim of the extension is to detect a difference between

the interventions, please state the conditional power (two-sided 5% level) given the data so far, of detecting a

currently plausible treatment effect”.

• Almost never done as intended or refused

Is it appropriate to ask for a futility analysis in extension requests?Is this the right form for the futility calculation?What level of conditional power should be taken to mean its futile to continue?

3. When and how

• Futility analyses should be conducted in large expensive trials in which tr >> tf and in which early stopping could save substantial resources.

• A futility analysis should be conducted at the time that the expected resource savings are maximised

• Expected resource savings = (resources not spent – shut down costs) x (prob shut down)

• If a conditional power calculation is used, use a conservative rule for futility, eg conditional power < 20%

• Take other information into account before deciding on stopping or not extending

4. Unconditional power calculations

• Achieved sample sizes in 122 MRC/HTA trials (from Campbell et al, HTA 2007, 11 (48))

Achieved sample size Proportion

>= 100% 31.1%

80% - 23.8%

<80% 45.1%

• We could design the trial with fixed T (and £C) and target sample size N, but variable achieved size n

• So instead of choosing N so that

Prob (Z ≥ U | H₁, N) = 1 - β• N is chosen so that

∑ Prob (Z ≥ U | H₁, n).Prob(n | N) = 1 - β• Extensions for sample size would then be

redundant

Problem is this removes any incentive to maximise recruitment and increases N, T, and £C

Possible recommendations

• Build in a futility analysis when:• N and £C are large• There is a placebo arm• Tr >> Tf

• Carry it out when the expected resources saved are maximised

• Use a conservative criterion for deciding futility, eg conditional probability <20%

• Always carry out futility analysis in extension requests unless potential savings < £?

Ask the audience

21%

53 %

9%

17%

Yes – for all Yes but Never Undecided Appropriate only for Trials (built extension into design) requests

a.m. results:Should we plan to do futility analyses in publicly-funded trials?

1. Yes – for all appropriate

trials (built into design)

2. Yes – but only for

extension requests

3. Never

4. Undecided

Only in In both placebo- undecided appropriate controlled and placebo-controlled head-to-head trials trials

29%

44%

27%

a.m. results:In what circumstances are they appropriate?

1. Only in appropriate

placebo-controlled trials

2. In both placebo-

controlled and head-to-

head trials

3. Undecided

a.m.results: Should funding bodies insist upon futility analyses before granting trial extensions?

1. Yes

2. No

3. Undecided

Yes No Undecided

47%

27% 25%

1% 5% 10% 15% 20% 30% 40% 50% It depends

0% 0% 5% 5% 5% 5% 5%

2%

72%

a.m. results:What is futile?How low can the conditional power be before stopping a trial?1. 1%

2. 5%

3. 10%

4. 15%

5. 20%

6. 30%

7. 40%

8. 50%

9. It depends

p.m. resultsShould we plan to do futility analyses in publicly-funded trials?

1. Yes – for all appropriate

trials (built into design)

2. Yes – but only for

extension requests

3. Never

4. Undecided

The percentage in the

first category increased

to about 72% (from

53%), as a result of the

undecided making up

their minds

p.m. results:In what circumstances are they appropriate?

1. Only in appropriate

placebo-controlled trials

2. In both placebo-

controlled and head-to-

head trials

3. Undecided

There was a slight shift

in the undecided

category into the first

category

p.m.results:Should funding bodies insist upon futility analyses before granting trial extensions?

1. Yes

2. No

3. Undecided

There was a shift from

the undecided into the no

category so that the

numbers answering ‘yes’

and ‘no’ were both in the

low 40s

p.m.results:What is futile?How low can the conditional power be before stopping a trial?1. 1%

2. 5%

3. 10%

4. 15%

5. 20%

6. 30%

7. 40%

8. 50%

9. It depends

The ‘It depends’

category increased

from 72% to 84%

Thanks

• Jon Nicholl, for suggesting the idea• The speakers: Andy Grieve, Carl-Fredrik

Burman and John Whitehead for agreeing to speak

• Paul Gentry and the RSS for organising things here and providing the venue

• Louise Dent and Andrew Cook for facilitating the workshops

• MRC and HTA for funding the event