(c) Stephen Senn 20081 Statistical considerations in small proof-of-concept trials, including crossover designs Stephen Senn

(c) Stephen Senn 2008 1

Statistical considerations in small proof-of-concept trials, including

crossover designs

Stephen Senn


• People look down on marketing men

• It’s not true that they are not scientists

• They work in sell biology

• I would like to take this opportunity to draw your attention to a book I rather like

• In fact I wrote it myself


Outline

• Decision analysis and proof of concept

• Value of information perspective

• Place of cross-over trials

• Carry-over

• The potential for cross-over trials in studying individual response


A Model

1

C

E

E

C

C E

C

C

R

f

f

V f f

Probability proof of concept (POC) study successful

Probability proof of efficacy study (POE) successful if POC successful

Probability POE study successful if POC unsuccessful

Probability POE study successful

Cost of POC including any lost sales through extra delay

Cost of POE study

Expected NPV revenue if POE initiated immediately and successful

Value of strategy of POE only

Value of strategy of POC + POE

Value of POC study


Model Continued

max 0, 1

max 0, max 0, 1 max 0,

E E

C E E C

f R C

f R C R C C


Example100, 5, 25, 0.3, 0.25,

1C ER C C

Value of two strategies plotted against , the probability POE successful if POC successful

0 0.5 1

5

direct POEinitial POC

Expected return on two strategies

Prob POE successful if POC success

0.65


0 0.2 0.4 0.6 0.8

5

5

10Value of an initial POC trial

0.650.3


Value of Biomarker Information in Terms of Posterior Variance

• Suppose that over all products for this indication the correlation of true therapeutic and biomarker outcomes is 0.9

• Let the prior means be zero in this class• Let the prior variances be 1• Let the data variance of a minimal

experiment be also 1– Implies prior information equivalent to one

minimal experiment


Here n is the number of minimal experiments we run

Of course we expect a biomarker experiment to be cheaper than a therapeutic one

Nevertheless note that fairly rapidly there is no interest in getting further biomarker information

Posterior variances based on proof of concept trial

10 30 50

0.6

0.5

0

0.4

0.3

0.2

20

0.1

0.0

40

n

Poste

rior va

rian

ce

simulated therapeuticsimulated biomarkertheory therapeutictheory biomarker


A Serious Warning to Statisticians

In the mathematical formulation of any problem it is necessary to base oneself on some appropriate idealizations and simplification. This is, however, a disadvantage; it is a distorting factor which one should always try to keep in check, and to approach circumspectly. It is unfortunate that the reverse often happens. One loses sight of the original nature of the problem, falls in love with the idealization, and then blames reality for not conforming to it.

De Finetti 1975

‘The only way that human beings could ever have survived as a species for long as we have is that we’ve developed another kind of decision-making apparatus that’s capable of making very quick judgements based on very little information.

Malcolm Gladwell, Blink, 2005


My Gloomy Take on This

• We don’t really understand this topic• There may be less value in proof of

concept studies than we propose• Therapeutic studies may be valuable even

if they have low power• There is no point in undertaking POC

studies unless you can see circumstance under which they would cause you to cancel projects


Appropriate Attitudes for Cross-over Trials

• They are not suitable for all indications and questions

• They are extremely valuable for some indications and questions

• Carry-over has to be dealt with by washout• Don’t pre-test for carry-over• Don’t rely on classical statistical approaches to

carry-over• Cross-over trials have great potential in

investigating individual response


Carry-over

Definition: Carry-over is the persistence (whether physically or in terms of effect) of a treatment applied in one period in a subsequent period of treatment.

If carry-over applies in a cross-over trial we shall, at some stage, observe the simultaneous effects of two or more treatments on given patients.

We may, however, not be aware that this is what we are observing and this ignorance may lead us to make errors in interpretation.


The simple carry-over model.

This is a very popular model amongst “applied” statisticians of a mathematical bent.

The model assumes that if a carry-over effect is present

1) it lasts for one period exactly

2) it depends on the engendering treatment only and not on the perturbed treatment.


Three Period Bioequivalence Designs

• Three formulation designs in six sequences common.

• Subjects randomised in equal numbers to six possible sequences. – For example, 18 subjects, three on each of the

sequences ABC, ACB, BAC, BCA, CAB, CBA. – A = test formulation under fasting conditions, – B = test formulation under fed conditions – C = reference formulation under fed conditions.


Period

Sequence 1 2 3

ABC A 0 B 1/6 C -1/6

ACB A 0 C -1/6 B 1/6

BAC B 1/6 A 0 C -1/6

BCA B 1/6 C -1/6 A 0

CBA C -1/6 A 0 B 1/6

CAB C -1/6 B 1/6 A 0

Weights for the Three Period Design:

not Adjusting for Carry-over


Properties of these weights

• Sum to 0 in any column, – eliminates the period effect.

• Sum to 0 in any row – eliminates patient effect

• Sum to 0 over cells labelled A– A has no part in definition of contrast

• Sum to 1 over the cells labelled B and to -1 over the cells labelled C– Estimate contrast B-C


Period

Sequence 1 2 3

ABC A -1/24 Ba 4/24 Cb -3/24

ACB A 1/24 Ca -4/24 Bc 3/24

BAC B 4/24 Ab 2/24 Ca -6/24

BCA B 5/24 Cb -2/24 Ac -3/24

CBA C -4/24 Ac -2/24 Ba 6/24

CAB C -5/24 Bc 2/24 Ab 3/24


Adjusting for Carry-over

B-C contrast: illustration of treatment effect and elimination of period and patient effects


Period

Sequence 1 2 3

ABC A -1/24 Ba 4/24 Cb -3/24

ACB A 1/24 Ca -4/24 Bc 3/24

BAC B 4/24 Ab 2/24 Ca -6/24

BCA B 5/24 Cb -2/24 Ac -3/24

CBA C -4/24 Ac -2/24 Ba 6/24

CAB C -5/24 Bc 2/24 Ab 3/24


Adjusting for Carry-over

Illustration of elimination of ‘carry-over’ effects


Have We Got Something for Nothing?

• Sum of squares weights of first scheme is 1/3 (or 4/12)

• Sum of squares of weights of second scheme is 5/12

• Given independent homoscedastic within- patient errors, there is thus a 25% increase in variance

• Penalty for adjusting is loss of efficiency


The difference between mathematical and applied statistics is that the former is full of lemmas whereas the latter is full of dilemmas


The Dangers of Pre-testing

• Situation with AB/BA design– Two-stage procedure is very badly biased– CARRY and PAR are highly correlated

• 1/2 < < 1

• Three treatment design– Same problem– Carry-over and adjusted estimates correlated

= 0.45


The Phoenix Bioequivalence Trials

• Analysed by D’Angelo, Potvin & Turgeon *

• 20 drug classes

• 1989-1999

• 12 or more subjects

• 96 three period designs

• 324 two period designs

D'Angelo, G.Potvin, D.Turgeon, J. J Biopharm Stats, 11, 27-36, 2001


AUC Cmax

0 : 115567899 1 : 01458999 2 : 01225568999 3 : 011335577 4 : 24688 5 : 35667788 6 : 00336667888 7 : 14444566999 8 : 011233468888 9 : 13335667899

0 : 223557888 1 : 4677799 2 : 000124566899 3 : 011124689 4 : 01223455799 5 : 00045599 6 : 000166667778 7 : 0345566779 8 : 2345779 9 : 13444556889

Three Treatment Designs

P-Values for Carry-Over

“Significant” results in bold

Senn, S. J., G. D'Angelo, et al. (2004). "Carry-over in cross-over trials in bioequivalence: theoretical concerns and empirical evidence." Pharmaceutical Statistics 3(2): 133-142.


AUC Cmax

0 : 00111111222222234444 0 : 5666777777789999 1 : 00000112222223333 1 : 5556667777899999 2 : 0011112223344444 2 : 555666788899999 3 : 00001112233344 3 : 5556666666777778888899999 4 : 001111112222223334 4 : 55666666777777788999 5 : 00000111222333344444 5 : 566677888899 6 : 000001134 6 : 55666667777888889999 7 : 111233333344 7 : 555556777888899 8 : 0000112234444 8 : 55666778888999 9 : 00011112233334444 9 : 555567777788999

0 : 00122222344 0 : 55555556666677999999 1 : 0001122233333344444444 1 : 55566667778888899 2 : 00011111122344 2 : 566667788889999 3 : 111112222233444444 3 : 555566666777778888999 4 : 000001112222333334444 4 : 5557888889999 5 : 00001122233 5 : 5555666678999 6 : 0000111222233334 6 : 55555566677788889999 7 : 000000112223344 7 : 6666777777889 8 : 0122233444 8 : 55666677888899 9 : 1111111222333444 9 : 555555556666677778889999

Two Treatment Designs

“Significant” results in bold


StudyDesign

Variable Totalnumber

of studies

KSstatistic

p-value*

2-way AUC0-t 324 0.0645 0.1354Cmax 324 0.0496 0.4040

3-way AUC0-t 96 0.1048 0.2424Cmax 96 0.0542 0.9407

*H0: true cdf U[0,1] vs. H1: true cdf NOT U[0,1]

Test of Uniformity of P-Values


Galling as this may appear to statisticians, the cure for carry-over is more biological and pharmacological understanding not more statistics


Conclusions

• Distribution of P-values uniform– no evidence of carry-over

• Carry-over a priori implausible– presence testable by assay

• No point is testing for it– leads to bias

• Or adjusting for it– increased variance


Possible Strategy

• Run multi-period cross-overs

• Patient by treatment interaction becomes identifiable

• This provides an upper bound for gene by treatment interaction– Because patients differ by more than their

genes


Second cross-over

Responders Non-Responders

Total

First cross-over

Responders 24 0 24

Non-Responders

0 8 8

Total 24 8 32


Second cross-over

Responders Non-Responders

Total

First cross-over

Responders 18 6 24

Non-Responders

6 2 8

Total 24 8 32


Advantages and DisadvantagesPRO CON

• Cheap• Low tech• Insight into sources

of variation gained

• Only suitable for chronic diseases

• Demanding of patient’s time

• Unglamorous• Does not produce

diagnostic patents


An Overlooked Source of Genetic Variability

• Humans may be classified into two important genetic subtypes.

• One of these suffers from a massive chromosomal deficiency.

• This is expressed in.– Important phenotypic differences.– A massive disadvantage in life expectancy.

• Many treatment strategies take no account of this.

• The names of these subtypes are...


Men and Women


A Difficult Decision

• You have $100• Should you spend it

on beer?– US 20 beers– UK 15 beers

• Or on books?• In particular 1 book• Have I mentioned this

before?

Documents

(c) Stephen Senn 20081 Statistical considerations in small proof-of-concept trials, including crossover designs Stephen Senn