Stavros Petrou26th November 2010

Adaptive designs for NIHR funded trials: A health economics

perspective

Outline

• Brief introduction to trial-based economic evaluation• Economic perspectives on adaptive designs driven

primarily by clinical, statistical and regulatory considerations:- Eligibility criteria - Statistical analysis plan

- Maintain study power - Others

- Outcomes unrelated to efficacy

- Group sequential methods

• Adaptive designs driven primarily by economic considerations

• Implications for research funders

Trial-based economic evaluation

• Aim: to maximise health gain given scare resources

• Method: compare cost and consequences of interventions

• Economic evaluation: explicit criteria for making choices

• Incremental cost-effectiveness ratio (ICER):

(CT - CC) / (ET - EC)

• Mean net benefit = Rc.E - C

Selecting treatment options

Option Total

Cost

Total

QALYs

Comparison Incremental

cost

Incremental

QALY

ICER

A 0 0 - - - -

B 10,000 0.4 B v A 10,000 0.4 £25,000

C 22,000 0.55 C v B 12,000 0.15 £80,000

D 25,000 0.5 D V C 3,000 -0.05 Dominated

E 40,000 1 E v D 15,000 0.5 £30,000

After excluding strongly dominated options:

A 0 0 - - - -

B 10,000 0.4 B v A 10,000 0.4 £25,000

C 22,000 0.55 C v B 12,000 0.15 Extended

dominated

E 40,000 1 E v C 18,000 0.45 £40,000

After excluding extendedly dominated options:

A 0 0 - - - -

B 10,000 0.4 B v A 10,000 0.4 £25,000

E 40,000 1 E v B 30,000 0.6 £50,000

Source: Gray A, Clarke P, Wolstenholme J, Wordsworth S. Applied methods of cost-effectiveness analysis in health care. Oxford University Press, 2010.

Adaptation of study eligibility criteria

• Examination of baseline information blinded to treatment assignment

• Modification of eligibility criteria to increase numbers of patients with desired characteristics

• May impair relationship between treatment effect and changed patient characteristic(s)

→Estimate net benefits from observed costs and effects, construct regression model with a treatment variable and covariates for patient characteristic(s)

→Magnitude and significance of coefficients on interaction terms provide estimates of cost-effectiveness by patient characteristic

Adaptations to maintain study power (I)

• Blinded interim analyses of aggregate data• Event rate may be below initial assumption• Increase sample size/period of follow-up

→ Trialists interested in events/time to event

→ Unlikely to collect comprehensive data on final outcomes, e.g. mortality

→ Economists interested in all events

→ Need to link intermediate endpoints to the long-term outcomes of interest

→ Extrapolation into life expectancy and quality adjusted life expectancy using survival models

Adaptations to maintain study power (II)

• Stratification of patients at baseline

• Following interim analysis focus remainder of study on subgroups with greatest event rates, lowest variance

• Subgroups no longer the focus may contain important economic information:

→ most trials underpowered on economic endpoints

→ collect follow-up event, resource use and HRQL data in refined sub-groups

• Analyse cost-effectiveness data by sub-group avoiding the danger of spurious differences in treatment effects due to chance

Adaptations based on outcomes unrelated to efficacy

• Dropping dose group(s) with unacceptable rates of adverse effects• Requirement that adverse effect is not composite of or directly

related to efficacy outcome:

→ patients in dropped dose group(s) may contain important economic information → follow-up

→ clinical assessment of risk-benefit comparison may be opaque

→ utility measures could synthesise risks and benefits

→ need for sensitive and validated utility measures

Adaptations using group sequential methods

• Alpha spending approaches to control for Type 1 errors• Compelling ethical and statistical evidence needed for

terminating early: (i) Futility (ii) Demonstrated efficacy• Is there a case for applying group sequential design and

analysis methods to economic endpoints?

→ Frequentist and Bayesian approaches available for estimating sample size requirements for economic endpoints

→ Very complex: Economists tend to focus on estimation and not hypothesis testing

→ Typically large variability in economic measures might dictate very large sample sizes, which may be neither financially nor ethically acceptable

Adaptations to statistical analysis plan

• Prospective SAP generally remains unaltered

• Limited changes may be acceptable

• Analytical plan for economic evaluation usually separate

• Is there a case for introducing a decision criterion for extrapolation modelling for cost-effectiveness?

→ TOBY trial: Total body hypothermia for neonatal encephalopathy: pCE = 0.69 at 18 months, 0.99 at 18 years

→ REFLUX Trial: Minimal access surgery amongst people with gastro-oesophageal reflux disease: pCE = 0.46 at 12 months,

0.74 over lifetime

→ When does it become ‘futile’ to extrapolate cost-effectiveness?

Other unblinded adaptive designs

Adaptive design Economic considerations

Dose selection studies Dropped dose group(s) may contain important economic information. Need to take account of adherence and completion.

Randomisation based upon relative treatment group responses

Regression model required for net benefits with a treatment variable and covariates for patient characteristic(s).

Sample size based on interim effect size estimates

Statistical adjustments required to protect against bias. May require additional funding/time.

Patient population based on treatment-effect estimates

Statistical adjustments required to protect against bias. Analyse cost-effectiveness data by subgroup.

Endpoint selection based on interim estimate of treatment effect

Relationship of new endpoint to economic outcomes may need exploration if subsequent cost-effectiveness modelling is planned

Adaptive designs driven primarily by economic considerations

Value of Information

• Expected value of perfect information (EVPI)– Equals net benefit from the best decision we could make, minus

the net benefit of the decision made based on current information– Perfect information is unattainable; EVPI is therefore the absolute

maximum we should be willing to spend on research

• Expected value of sample information (EVSI)– Represents expected value of conducting a study of a specific size– Can take account of trial costs, opportunity cost of delaying

adoption, cost of reversing initial decision, etc– Can be calculated analytically or using simulation

• Relies on many assumptions

Application of VOI to adaptive trial design

• Developed by Andrew Willan• Sample sizes not dependent on type I and II errors• Aims to maximise ENG, i.e. difference between cost of trial

and EVSI• Single stage EVSI design: sample size driven by potential no

of patients and expected values • Incorporates pilot data on INBs (means, variances, between-

patient variances). Also requires no of beneficiaries; fixed, variable & analytical costs for trial.

• Two stage EVSI design: requires further complex assumption on optimal proportion of patients in first stage.

Adaptive design of early ECV trial

• Pilot study: 323 pregnant women presenting in breech position randomised to early v late ECV

• CIHR funded trial required 730 women per arm to have 80% probability to reject Ho if treatments differed by 8% using two-sided type 1 error of 0.05

• Single stage EVSI: assumes b0=69, v0=3725, σ2=217,227, N=1,000,000, Cf=$498K,000, Cv=$1,600 and Ca=$2,000; n=345

• Two stage EVSI: assumes α0=0.45; n=290

FinancialCost ($)

OpportunityCost ($)

EVSI ($) ENG ($) Yield* (%)

N=750,000

CIHR (n=730) 2,836,000 50,345 2,298,381 -587,964 -20

Single-stage (n=279)

1,392,800 19,241 1,585,729 173,688 12.3

Two-stage (E(n)=239)

1,265,335 16,483 1,982,742 700,924 55

N=1,000,000

CIHR (n=730) 2,836,000 50,345 3,066,003 179,658 6.2

Single-stage (n=345)

1,604,000 25,793 2,364,176 736,383 45

Two-stage (E(n)=290)

1,533,200 20,001 2,942,863 1,386,662 89

N=2,000,000

CIHR (n=730) 2,836,000 50,345 6,136,488 3,250,143 112

Single-stage (n=547)

2,250,400 37,724 5,665,487 3,377,363 148

Two-stage (E(n)=448)

1,934,133 30,897 6,399,914 4,434,884 226

Source: Willan A, Kowgier M, Determining optimal sample sizes for multi-stage randomized clinical trials using value of information methods. Clinical Trials 2008;5:289-300.

Implications for research funders

• Few economic perspectives on adaptive trial designs• Adaptive designs driven primarily by clinical,

statistical and regulatory considerations will have economic implications

• Only one study has generated an adaptive design driven primarily by economic considerations

• Need for prospective experiments and validation• Adaptive designs for complex interventions, public

health interventions