Beginner’s guide to randomisation

Randomisation S1

Beginner’s guide to randomisation

Research supported by TLRI

Randomisation S2

•Experiments & the Randomisation Test:

• The difference between two medians

–Watch out for: • The ‘chance is acting alone’ explanation• How we assess the plausibility of the ‘chance

alone’ explanation – (test for ‘chance alone’)

Randomisation S3

Randomisation

iNZightVIT

What does ‘chance alone’ look like?

Randomisation S4

Did she brush her teeth?

1. She has brushed her teeth.

2. The toothbrush is dry.

3. The-toothbrush-is-dry would be highly unlikely if she had

brushed her teeth.

4. Therefore, it’s a fairly safe bet she has not brushed her teeth.

I have evidence that she has not brushed her teeth.

1. Formulate statement to test.

2. Data (information at hand).

3. Consider 1. and the data:If 1. is true, then what are the chances of getting data like that in 2.?

4. Review the statement in 1. in light of 3. together with the data in 2.

Randomisation S5

Did she brush her teeth (2)?


2. The toothbrush is wet.

3. The-toothbrush-is-wet would

NOT be surprising if she had brushed her teeth.

4. Therefore, she could have brushed her teeth.

I have no evidence that she has NOT brushed her teeth.

1. Formulate statement to test.

2. Data (information at hand).


4. Review the statement in 1. in light of 3. together with the data in 2. Or she

could have just run the brush under the tap.

Randomisation S6

The Walking Babies Experiment

Treatment Exercise Control

Does a special exercise programme lower walking age?Phillip R. Zelazo, Nancy Ann Zelazo, & Sarah Kolb, “Walking in the Newborn” Science, Vol. 176 (1972), pp314-315

10 male infants (& parents) were randomly assigned to one of two treatment groups.

Randomisation S7


Treatment Age (months) Exercise 9 9.5 9.75 10 11Control 13.25 11.5 12 13.5 11.5

Does a special exercise programme lower walking age?Phillip R. Zelazo, Nancy Ann Zelazo, & Sarah Kolb, “Walking in the Newborn” Science, Vol. 176 (1972), pp314-315

10 male infants (& parents) were randomly assigned to one of two treatment groups.

First walked without support:

Randomisation S8


Age (months)9 11 1210 13 14

Does it appear that these data provide evidence that the treatment is effective?Exercise

Control

Yes.

Randomisation S9

looking through a window with ripples in the glass

“What I see …is not quite the way it really is”

Looking at the world using data is like

Randomisation S10


Age (months)9 11 1210 13 14

Is it possible that these babies’ walking ages have nothing to do with whether they undertook the exercises or not, . . .

Exercise

Control

Randomisation S11


Age (months)9 11 1210 13 14

. . . i.e., it doesn’t matter which group they were randomly assigned to, they would still have the same walking age, . . .

Exercise

Control

Randomisation S12


Age (months)9 11 1210 13 14

. . . and so the observed difference pattern is purely and simply the result of the luck-of-the-draw as to which babies just happened by chance to be assigned to which group and nothing else?

Exercise

Control

Yes.Under this scenario we say: “Chance is acting alone”.

Randomisation S13


Possible explanation:One possible explanation for the observed difference between these two groups:

Chance is acting alone (the exercise has no effect)

Randomisation S14


Is the ‘chance alone’ explanation simply not plausible?• Would our observed difference be unlikely when chance

is acting alone? • How do we determine whether an observed

difference is unlikely when chance is acting alone?Answer: See what’s likely and what’s unlikely when chance is acting alone.

Possible explanation:One possible explanation for the observed difference between these two groups:

(the exercise has no effect) Chance is acting alone

Randomisation S15

Randomisation S16

2.25

Randomisation S17

Randomisation S18

Re-randomisation distribution of differences (under chance alone)

Is chance alone likely to generate differences as big as our difference?

Our observed difference = 2.25 months

Hardly ever, they’re almost always smaller.Tail proportion:

roughly __%

Randomisation S20

The Walking BabiesExperiment

Possible explanation:One possible explanation for the observed difference between these two groups:Chance is acting alone (the exercise has no effect)

Our tail proportion of roughly __% means: • Roughly __ times out-of-a-1000 times we get a

median difference of 2.25 months or more, when chance is acting alone.

• Under chance alone, it would be highly unlikely to get a difference equal to or bigger than our observed difference of 2.25 months.

Randomisation S21



Our tail proportion of roughly __% means: • Roughly __ times out-of-a-1000 times we get a

median difference of 2.25 months or more, when chance is acting alone.

• Under chance alone, it would be highly unlikely to get a difference equal to or bigger than our observed difference of 2.25 months.

An observed difference of 2.25 months or greater is highly unlikely when chance is acting alone . . .

therefore, it probably isn’t. It’s a fairly safe bet chance is not acting alone.

Randomisation S22


• We can rule out ‘chance is acting alone’ as a plausible explanation for the difference between the two groups.

• We have evidence against ‘chance is acting alone’

• We have evidence that chance is not acting alone


Randomisation S23


• We can rule out ‘chance is acting alone’ as a plausible explanation for the difference between the two groups.

• We have evidence against ‘chance is acting alone’

• We have evidence that chance is not acting alone


If chance is not acting alone, then what else is also acting to help produce the observed difference?

Remember: Random assignment to 2 groups & each group receives different treatment.

Randomisation S24

Conclusion:Because the male infants (& parents) were randomly assigned to the groups, we may claim that the exercise was effective in lowering the walking age.

Because these subjects in this experiment were volunteers (not randomly selected), then we would need to consider carefully as to which wider group(s) this conclusion may apply.


Randomisation S25

Did she brush her teeth?


2. The toothbrush is dry.

3. The-toothbrush-is-dry would be highly unlikely if she had brushed her teeth.

4. Therefore, it’s a fairly safe bet she has not brushed her

teeth. I have evidence that she has

not brushed her teeth.

Steps

1. Statement to test.

2. Collect data (information).



Randomisation S26

Is the exercise programme effective?Steps


2. Collect data.

3. Consider 1. and the data:If 1. is true, then what are the chances of getting data like that in 2. or more?


1. Chance is acting alone. (The exercise has no effect.)

2. Diff between medians = 2.25 mths.

3. A difference of 2.25 months or more is highly unlikely when chance is acting alone.(Tail prop = roughly __%)

4. Therefore, it’s a fairly safe bet chance is not acting alone. We have evidence against ‘chance is acting alone’.

Randomisation S27

Steps


2. Collect data.

3. Consider 1. and the data:If 1. is true, then what are the chances of getting data like that in 2. or more?


1. Chance is acting alone. (The exercise has no effect.)

2. diff medians= 2.25 mths.

3. A diff of 2.25 mths or more is highly unlikely when chance is acting alone.

4. Therefore, it’s a fairly safe bet chance is not acting alone. We have evidence against chance is acting alone.

Is the exercise programme effective?

Randomisation S28

Questions…

Randomisation S29

Two types of Inference

There are two types of inference1. Sample-to-population

eg x = 172cm so the population mean is about 172cm.

2. Experiment-to-causationeg The treatment was effective

Randomisation S30

When the tail proportion is small (less than 10%).:• the observed difference would be unlikely when

chance is acting alone . . . therefore, it’s a fairly safe bet chance is not acting alone.

• we have evidence against ‘chance-is-acting-alone’

• we have evidence that chance is not acting alone

Guidelines for assessing ‘Chance alone’

Randomisation S31

When the tail proportion is large (10% or more ) then:

• the observed difference is not unusual when chance is acting alone, therefore chance could be acting alone

• we have NO evidence against ‘chance is acting alone’

• EITHER chance could be acting alone OR something else as well as ‘chance’ COULD also be acting.

-- we do NOT have ENOUGH INFORMATION to MAKE A CALL as to which one.

Guidelines for assessing ‘Chance alone’

Documents

Beginner’s guide to randomisation