(C)Stephen Senn 1

Lost Opportunities for Design Theory in Drug Development

Stephen Senn

(C)Stephen Senn 2

Basic Thesis

• Design theory has great potential in drug development

• But this potential is unrealised• Those working in so-called optimal design are so

ignorant of application realities that where their influence is not zero it is harmful

• On the other hand the understanding of design theory by biostatisticians is pitifully inadequate

• We must cooperate properly to cure this parlous state of affairs

(C)Stephen Senn 3

Outline

• Quick tutorial on cross-over trials• I shall then give two introductory examples

of nonsense– By leading design theoreticians– By leading biostatisticians

• I shall then consider ‘design nonsense’ further

• Some conclusions• After lunch a case-study

(C)Stephen Senn 4

Warning

• I am a biostatistician

• We are used to thinking of data matrices with rows as subjects and columns as measurements

• That means that we write sequences for designs with rows representing subjects and columns representing periods

(C)Stephen Senn 5

Cross-over Trials

Definition: A cross-over trial is one in which subjects are given sequences with the object of studying differences between individual treatments.

(C)Stephen Senn 6

An Example of an AB/BA cross-over in asthma

Sequence Period 1 Wash-out Period 2

for/sal formoterol salbutamol

sal/for salbutamol formoterol

(C)Stephen Senn 7

An Example from Rheumatism:2 doses of diclofenac and placebo

Period 1 Period 2 Period 3

D1 D2 PP D1 D2D2 P D1D2 D1 PP D2 D1D1 P D2

(C)Stephen Senn 8

Carry-over

Definition: Carry-over is the persistence (whether physically or in terms of effect) of a treatment applied in one period in a subsequent period of treatment.

If carry-over applies in a cross-over trial we shall, at some stage, observe the simultaneous effects of two or more treatments on given patients.

We may, however, not be aware that this is what we are observing and this ignorance may lead us to make errors in interpretation.

(C)Stephen Senn 9

Simple Carry-over

• Carry-over lasts for exactly one period

• It depends only on the engendering treatment and is unmodified by the perturbed treatment

• There is a huge literature proposing ‘optimal’ designs for this model

• There is no empirical evidence that any of this has been useful

(C)Stephen Senn 10

Three Period Bioequivalence Designs

• Three formulation designs in six sequences common.

• Subjects randomised in equal numbers to six possible sequences. – For example, 18 subjects, three on each of the

sequences ABC, ACB, BAC, BCA, CAB, CBA. – A = test formulation under fasting conditions, – B = test formulation under fed conditions – C = reference formulation under fed conditions.

(C)Stephen Senn 11

Period

Sequence 1 2 3

ABC A 0 B 1/6 C -1/6

ACB A 0 C -1/6 B 1/6

BAC B 1/6 A 0 C -1/6

BCA B 1/6 C -1/6 A 0

CBA C -1/6 A 0 B 1/6

CAB C -1/6 B 1/6 A 0

Weights for the Three Period Design:

not Adjusting for Carry-over

(C)Stephen Senn 12

Properties of these weights

• Sum 0 in any column, – eliminates the period effect.

• Sum 0 in any row – eliminates patient effect

• Sum 0 over cells labelled A– A has no part in definition of contrast

• Sum to 1 over the cells labelled B and to -1 over the cells labelled C– Estimate contrast B-C

(C)Stephen Senn 13

Period

Sequence 1 2 3

ABC A -1/24 Ba 4/24 Cb -3/24

ACB A 1/24 Ca -4/24 Bc 3/24

BAC B 4/24 Ab 2/24 Ca -6/24

BCA B 5/24 Cb -2/24 Ac -3/24

CBA C -4/24 Ac -2/24 Ba 6/24

CAB C -5/24 Bc 2/24 Ab 3/24

Weights for the Three Period Design:

Adjusting for Carry-over

(C)Stephen Senn 14

Period

Sequence 1 2 3

ABC A -1/24 Ba 4/24 Cb -3/24

ACB A 1/24 Ca -4/24 Bc 3/24

BAC B 4/24 Ab 2/24 Ca -6/24

BCA B 5/24 Cb -2/24 Ac -3/24

CBA C -4/24 Ac -2/24 Ba 6/24

CAB C -5/24 Bc 2/24 Ab 3/24

Weights for the Three Period Design:

Adjusting for Carry-over

(C)Stephen Senn 15

Properties of These Weights

• As before – Estimates B-C contrast– Eliminates, period and patient effect– Eliminates A

• Sum to zero over cells labelled a,b, and c– Eliminate simple carry-over

(C)Stephen Senn 16

Have We Got Something for Nothing?

• Sum of squares weights of first scheme is 1/3 (or 4/12)

• Sum of squares of weights of second scheme is 5/12

• Given independent homoscedastic within- patient errors, there is thus a 25% increase in variance

• Penalty for adjusting is loss of efficiency

(C)Stephen Senn 17

First ExampleSome Design Theory Nonsense

John, J. A., Russell, K. G., and Whitaker, D. (2004), "Crossover: An Algorithm for the Construction of Efficient Cross-over Designs," Statistics in Medicine, 23, 2645 - 2658.A cross-over experiment involves the application of sequences of treatments to several subjects over a number of time periods. It is thought that the observation made on each subject at the end of a time period may depend on the direct effect of the treatment applied in the current period, and the carry-over effects of the treatments applied in one or more previous periods. Various models have been proposed to explain the nature of the carry-over effects. An experimental design that is optimal under one model may not be optimal if a different model is the appropriate one. In this paper an algorithm is described to construct efficient cross-over designs for a range of models that involve the direct effects of the treatments and various functions of their carry-over effects. The effectiveness and flexibility of the algorithm are demonstrated by assessing its performance against numerous designs and models given in the literature.

(C)Stephen Senn 18

“Sometimes in a clinical trial it may be necessary to modify or extend an ongoing trial. For example, suppose that after a few periods have been completed one of the treatments is dropped from the trial. It will then be necessary to re-allocate the other treatments to the remaining periods of the trial.” (John et al, 2004 p. 2653)

“Jones and Donev [17] also consider augmenting a design to account for the removal of a treatment. The initial trial to compare four treatments A, B, C, and D in five periods using four groups of subjects used a Williams square [3] with the fourth period repeated. After the first two periods had been completed it was decided to drop treatment D from the remainderof the trial.”

What’s wrong here?

(C)Stephen Senn 19

(C)Stephen Senn 20

The reality

..in a single-dose cross-over trial in asthma in 12 patients reported by Palmqvistet al. [5] patients were treated in the first period of a cross-over on dates ranging from 5 May to 12 November 1987 [4]. They were treated in a second period on dates ranging from 18 May to 26 November 1987. Eleven of the patients had completed period two of treatment before the 12th patient was recruited. (Senn, 2005, p3675.)

A basic fact of clinical trials

You treat patients when they fall ill

(C)Stephen Senn 21

Multi-Story

Aspect Single-Dose Multi-Dose

When? Phase I/II Phase II/III

Why? PD, Dose Therapeutic

Primitive Constraints

Number of periods

Length of treatment

Carry-over Not a problem Potential Problem

(C)Stephen Senn 22

Conclusion

• Multi-dose trials real scope for design theory.

• These will employ active wash-out

• Design problem is trade-off between exploiting correlation and eliminating carry-over.

• Short vs long active wash-out periods

(C)Stephen Senn 23

Second ExampleSome Biostatistics Nonsense

Chow, S. C., and Liu, J. P. (2000), Design and Analysis of Bioavailability and Bioequivalence Studies (2nd ed.), New York: Marcel Dekker.

Have several discussions of efficiency of designs in their book which are completely beside the point. They compare designs in terms of residual degrees of freedom!

They conclude that Balaam’s design, which uses sequencesTR/RT/TT/RR

Is similar in efficiency to the more conventional TR/RT design.

They write “The degrees of freedom for the intrasubject residuals for the 2 × 2 and 4 × 2 design are 22 and 21 respectively. Therefore there is little difference in testing power.”

This is nonsense

(C)Stephen Senn 24

Second ExampleSome Biostatistics Nonsense

.Design

Source 2×2 4×2 2×3 2×4 4×4

Between 23 23 23 23

Seq 1 3 1 1

Res 22 20 22 22

Within 24 24 48 72

Period 1 1 2 3

Form 1 1 1 1

Carry 1 1 1

Res 22 21 44 67

Total 47 47 71 95

(C)Stephen Senn 25

What is wrong 1. It’s not correct design theory

• As any design expert knows residual degrees of freedom are (nearly) irrelevant to efficiency

• It is the impact of adjustment on the degree of orthogonality of the design matrix that is important

(C)Stephen Senn 26

What is wrong 2.It’s not realistic biostatistics

• In fact as any biostatistician who has had to think about it will know from a practical point of view far from being optimal Balaam’s design is simply inadmissible

• The reasons is that only half of the resources are devoted to actually measuring the treatment

• The rest are devoted to providing an adjustment for a form of carry-over that is itself implausible

(C)Stephen Senn 27

Allocation of patients for two designs

Sequence AB/BA Balaam

AB n/2 n/4

BA n/2 n/4

AA 0 n/4

BB 0 n/4

(C)Stephen Senn 28

Investigation of the real efficiency of Balaam’s design

Mathcad 2001 Program to compare the efficiency of Balaam'sdesign correcting for simple carry-over and the AB/BA design notcorrecting for carry-over.

Take simplest comparable case of four patients: one allocated to each of the sequences ofBalaam's design or two to each of the sequences of the AB/BA cross-over.

Set up elements of design matrix

Patient and period dummies common to both

Designs: AB/AB/BA/BA (standard design) or AB/BA/AA/BB (Balaam's design)Design matrix rows in order patient 1 period 1, patient 1 period 2, patient 2, period 1 etc.

(C)Stephen Senn 29

Five Reasons why the Simple Carry-over Model is not Useful

• If it applies then the investigator can design a trial which eliminates it . (double the periods)

• Implausible given pk/pd theory. (obvious)• Leads to inefficient estimators. (see investigation

to follow)• Can lead to poor designs. (ditto)• The models which incorporate it are self-

contradicting. (example: factorial X-overs)

(C)Stephen Senn 30

0 2 4

50

100

0.5125

Concentration/EC50

Response

0

Dose response: the pharmacokineticist’s version

(C)Stephen Senn 31

Dose Response:The Statistician’s Version

Dose

Response

This is what the simple carry-over model implies

(C)Stephen Senn 32

The Models Which use Simple Carry-over are Inconsistent

Consider a factorial cross-over in four periods comparing A, B and the combination of A and B to placebo. We can represent the four treatments by: **, A*, *B and AB.

Suppose we consider a patient who has received the sequence AB ** A* *B. A standard parameterisation for treatment and carry-over would be as in the following table.

(C)Stephen Senn 33

I II III IV

TreatmentCombination

AB ** A* *B

TreatmentParameters

A, B,

AB

A

B

Carry-overParameters

A, B,

AB

A

Paramaterisation of a factorial cross-over

Period

(C)Stephen Senn 34

The Rhinoceros

The rhinoceros has a kind heart, if you doubt it here’s the proof

That thing on his nose is for taking stones out of a horse’s hoof

He seldom ever meets a horse, it is this that makes him sad

When he does, then it hasn’t a stone in its hoof

But he would, if he did and it had

Flanders and Swann

(C)Stephen Senn 35

The Phoenix Bioequivalence Trials

• Analysed by D’Angelo & Potvin

• 20 drug classes

• 1989-1999

• 12 or more subjects

• 96 three period designs

• 324 two period designs

(C)Stephen Senn 36

AUC Cmax

0 : 115567899 1 : 01458999 2 : 01225568999 3 : 011335577 4 : 24688 5 : 35667788 6 : 00336667888 7 : 14444566999 8 : 011233468888 9 : 13335667899

0 : 223557888 1 : 4677799 2 : 000124566899 3 : 011124689 4 : 01223455799 5 : 00045599 6 : 000166667778 7 : 0345566779 8 : 2345779 9 : 13444556889

Three Treatment Designs

P-Values for Carry-Over

(C)Stephen Senn 37

AUC Cmax

0 : 00111111222222234444 0 : 5666777777789999 1 : 00000112222223333 1 : 5556667777899999 2 : 0011112223344444 2 : 555666788899999 3 : 00001112233344 3 : 5556666666777778888899999 4 : 001111112222223334 4 : 55666666777777788999 5 : 00000111222333344444 5 : 566677888899 6 : 000001134 6 : 55666667777888889999 7 : 111233333344 7 : 555556777888899 8 : 0000112234444 8 : 55666778888999 9 : 00011112233334444 9 : 555567777788999

0 : 00122222344 0 : 55555556666677999999 1 : 0001122233333344444444 1 : 55566667778888899 2 : 00011111122344 2 : 566667788889999 3 : 111112222233444444 3 : 555566666777778888999 4 : 000001112222333334444 4 : 5557888889999 5 : 00001122233 5 : 5555666678999 6 : 0000111222233334 6 : 55555566677788889999 7 : 000000112223344 7 : 6666777777889 8 : 0122233444 8 : 55666677888899 9 : 1111111222333444 9 : 555555556666677778889999

Two Treatment Designs

(C)Stephen Senn 38

StudyDesign

Variable Totalnumber

of studies

KSstatistic

p-value*

2-way AUC0-t 324 0.0645 0.1354Cmax 324 0.0496 0.4040

3-way AUC0-t 96 0.1048 0.2424Cmax 96 0.0542 0.9407

*H0: true cdf U[0,1] vs. H1: true cdf NOT U[0,1]

Test of Uniformity of P-Values

(C)Stephen Senn 39

Conclusions

• Distribution of P-values uniform– no evidence of carry-over

• Carry-over a priori implausible– presence testable by assay

• No point is testing for it– leads to bias

• Or adjusting for it– increased variance

(C)Stephen Senn 40

Do Bayesians do Better?

• In principle the Bayesian approach ought to allow us to be more flexible about nuisance parameters such as carry-over

• However, the Bayesian track record is not impressive here

• Realistic models have not been employed

(C)Stephen Senn 41

Hills and Armitage Eneuresis Data

10

8

14

2

12

6 1210

6

4

2

0

40 8

Dry nights placebo

Dry n

ights d

rug

Line of equality

Sequence Drug PlaceboSequence placebo drug

Cross-over trial in Eneuresis

Two treatment periods of 14 days each

Treatment effect significant if carry-over not fitted

2.037 ( 0.768, 3.306)

Treatment effect not significant if carry-over fitted

0.451 (-2.272, 3.174)

1. Hills, M, Armitage, P. The two-period cross-over clinical trial, British Journal of Clinical Pharmacology 1979; 8: 7-20.

(C)Stephen Senn 42

Identical ‘uninformative’ prior placed on carry-over as for treatment

NB Parameterisation here means that values of need to be doubled to compare to conventional contrasts

(C)Stephen Senn 43

Identical Priors for Treatment and Carryover?

• Patients treated repeatedly during trial• Fourteen day treatment period• Average time to last treatment plausibly 4 hours• Average time to previous treatment seven days• Saying that it is just as likely that carry-over

could be greater than treatment is not coherent• In any case the two cannot be independent• Is negative carry-over as likely as positive carry-

over?

(C)Stephen Senn 44

So What are Acceptable Models for Carry-over?

• Ignoring carry-over altogether (not allowing for it because one believes one has taken adequate steps to eliminate it)– This is always a reasonable strategy

• Using an integrated pharmacokinetic pharmacodynamic model (Sheiner et al, 1991)– This may work for dose-finding trials– Very difficult to implement where more than one

molecule is involved

(C)Stephen Senn 45

The Sheiner model

max

50

1

1

i

i

i j li

ijij

ij

jk T Ttk

ij ill

E dDE

D d

d D e e

PD dose response

PK model for dose-concentration as a consequence of previous dosing history

Steady state concentration for patient i in period l

(C)Stephen Senn 46

0 5 100

0.5

1

Pharmacodynamic model

Dose

Res

pons

e0.909

0

0.5

DE d 1( )

DE d 2( )

DE d 3( )

100

1 3

d

(C)Stephen Senn 47

1 2 3 40

1

2

3

4

5

Seq 1 ActualSeq 1 TheorySeq 2 ActualSeq 2 TheorySeq 3 ActualSeq 3 TheorySeq 4 ActualSeq 4 Theory

Dosing levels achieved

Period

Dos

e

Possible set of sequences for a design. These follow aWilliams square. NB This is probably not a good idea D

1

2

3

4

2

4

1

3

3

1

4

2

4

3

2

1

(C)Stephen Senn 48

1 2 3 40.2

0.1

0

0.1

Seq 1Seq 2Seq 3Seq 4Carry from 4

Carry-over by sequence

Period

Car

ry-o

ver

0

(C)Stephen Senn 49

The difference between mathematical and applied statistics is that the former is full of lemmas whereas the latter is full of dilemmas

(C)Stephen Senn 50

Advice for Design-Theoreticians

• Resist the temptation to give advice if you are unfamiliar with the application area

• Seek collaborators

• Ground your models in pharmacology

• Remember that the goal is good medicine not elegant mathematics

• Don’t defend the indefensible

(C)Stephen Senn 51

Advice for biostatisticians

• Remember that design theoreticians have many powerful results

• It’s just conceivable that some of them may even be useful

(C)Stephen Senn 52

(C)Stephen Senn 53

References

1. Senn, S.J., Is the 'simple carry-over' model useful? [published erratum appears in Statistics in Medicine 1992 Sep 15;11(12):1619]. Statistics in Medicine, 1992. 11(6): p. 715-26.2. Senn, S.J., The AB/BA cross-over: how to perform the two-stage analysis if you can't be persuaded that you shouldn't., in Liber Amicorum Roel van Strik, B. Hansen and M. de Ridder, Editors. 1996, Erasmus University: Rotterdam. p. 93-100.3. Senn, S.J., Cross-over Trials in Clinical Research. Second ed. 2002, Chichester:

Wiley.4. Senn, S.J., Statistical Issues in Drug Development. Statistics in Practice, ed. V.

Barnett. 2007, Chichester: John Wiley.