Answering Questions About Cause and

Effect

PHM analyst academy session 2: impact assessment &

evaluation

HEE Leicester | 2020-02-25

1. Context

2. The need for a causal framework

3. Counterfactual reasoning

4. Natural experiments

5. Study designs

------------------------------------------------

6. ITS analysis

7. ITS walk-through

8. Summary

Contents

2

Answering Questions About Cause and Effect

3

You will need …

What is a cause?

4

What is a cause?

5

“So, what is a cause? My

take home from this

book is that even after

more than 2000 years,

philosophers don’t really

know.

A classification of data science tasks

6

1. Description

2. Prediction

3. Counterfactual

prediction

1. Description

A descriptive study only aims to describe

the data at hand e.g. we may study how

many suffer from diabetes in Indonesia.

2. Prediction

Predictions map inputs (e.g., an image of a

human retina) to outputs (e.g., a diagnosis

of retinopathy), but they do not consider

how the world would look like under

different courses of action (e.g., would the

diagnosis change if we operated on the

retina).

A classification of data science tasks

7

3. Counterfactual prediction

Scientific questions based on counterfactual

predictions can be phrased as – What would

happen if – questions. Both prediction and

causal inference require expert knowledge

to formulate the scientific question, but only

causal inference requires causal expert

knowledge to answer the question.

8

The need for a causal framework

Programmed for causal inference

9

From the moment

we’re born, we begin

learning about

cause-and-effect.

We observe how

objects and people

interact.

Cognitive bias

10

But despite our gifts, we are also very easily fooled ...

11

A mind for determinism

12

+ =

Deterministic

A mind for determinism

13

+ =

Deterministic

Probabilistic

Probabilistic reasoning

14

=

=

Statistics & probability have reconciled this problem. What’s

unknowable for an individual can become predictable for a

group!

Probabilistic reasoning

15

+ =

Causation can be understood as probabilistic not deterministic

Randomisation

16

With randomisation, this provides a potent way to identify causal effects

that we'd have little hope with experience and intuition alone.

Randomisation

17

“Given this theory ... (of probabilistic

causation) ... plus a suitable

definition of an ideal RCT, it is

possible to prove trivially that from

positive results in an RCT a causal

conclusion can be deduced”

Nancy Cartwright (philosopher of

science esp. causality)

Randomisation

18

In health and medical research, randomised controlled trials embraced

with impassioned fervour, and canonised beyond all other forms of

evidence.

Downside of randomisation

19

This limits our gaze & places vast majority of human experience beyond our scope for study.

• Unethical (anything harmful, like number of infections)

• Impractical (anything social, like your parents wealth)

• Impossible (anything fixed, like your personality)

physical (e.g. repetitive strain)

chemical (asbestos)

biological (influenza)

cultural (childhood diet)

economic (poverty)

environmental (pollution)

political (austerity)

behavioural (smoking)

personal (self-esteem)

20

“It bothers me when people try to pit RCTs against

observational studies. Both are valuable. I simply view

them as methodological options where we should

choose the best option for the question. It's about

appropriate design.

Dr Larry Svenson, University of Alberta

21

Counterfactual reasoning & potential

outcomes

22

Counterfactual reasoning & potential outcomes provide a framework for

considering causal effects. Such a framework is useful when trying to

understand some otherwise tricky ideas and can help you respond to

questions about causes and effects.

It is not, however, a replacement for careful and rigorous thinking, which

is the most important requirement for analysing observational data for

causal inference.

Counterfactual reasoning & potential outcomes

23

24

25

26

31 Aug Cholera outbreak in Soho

5 Sep Snow maps where deaths are occurring

6 Sep 83% died drank from Broad Street pump

7 Sep Snow meets parish guardians to argue for pump closure

8 Sep Snow removes handle

Cholera outbreak ends!

Potential outcomes

27

We observed outcome (Y) of what happened when the exposure (x) was 'closed' (factual)

This is written Y(x=closed) or Y(x=0) or Y0

Potential outcomes

28

We don't know the potential outcome (Y) that would have happened if the exposure (x) had

been counter to fact – left 'open' (counterfactual)

This is written Y(x=open) or Y(x=1) or Y1

Potential outcomes

29

To identify the causal effect of removing the pump handle on subsequent number of deaths we

would need to know (and compare):

The number of deaths when the pump was closed Y0

with

The number of deaths when the pump was open Y1

Fundamental problem of causal inference

30

BUT

… we can never know the potential outcome for a counterfactual exposure! For each 'unit of

analysis' we can only observe one potential outcome. This is known as the fundamental problem

of causal inference.

Fundamental problem: Within the same individual or unit, we only ever observe one outcome

(Y1 or Y0), and usually we can only treat them or not (x=1 or x=0). The counterfactual is the

imagined opposite of what we observed.

Estimating potential outcomes

31

Instead we must estimate the potential outcome for the counterfactual exposure from

exchangeable units of analysis.

Exchangeable units

Estimating potential outcomes

32

Problem: Units (e.g. people) are very different. Even the same units can respond differently at

different times.

Not exchangeable units of analysis!

Estimating potential outcomes

33

We therefore have to work with groups of units – and aim for each group to be exchangeable

on all factors except the exposure of interest. If they're exchangeable we can estimate the

average causal effect by comparing the summary outcomes between groups.

Conditional exchangeability

The probability of being assigned to a particular exposure value should be independent of the

propensity of the outcome, i.e. the units of analysis should have similar risks of the outcome.

Estimating potential outcomes

34

The easiest way to achieve exchangeability is through randomised assignment. This is why

randomised experiments can so easily estimate causal effects – they generate unconditionally

exchangeable units of analysis. Without randomisation we have to aim for conditional

exchangeability.

35

Natural experiments

Natural experiments are observational studies where 'nature' or other exogenous forces

approximates the conditions of an experiment or quasi-experiment.

(True) natural experiment: exposure assigned 'as random' (i.e. independent of the propensity of

the outcome) creating unconditional exchangeability

Quasi natural experiment: exposure assignment not strictly random, units not unconditionally

exchangeable

Natural experiments may occur due to:

- 'Acts of God' (e.g. weather, climate, disasters); which involve 'no human agency'

- geo-political events (e.g. war, famine, recession, Brexit)

- government or policy changes (e.g. smoking ban, sugar tax, austerity)

- other 'exogenous' changes (e.g. staff moved to new open-plan office)

Natural experiments

36

37

The grand experiment

38

Recall

Snow believed cholera was

contracted by ingesting 'Cholera

poison' from water contaminated

with human sewerage. But there's

no way he could have performed an

experiment!

– impractical

– unethical

The grand experiment

39

Solution

Snow noticed an opportunity for a

natural experiment.

“London was without cholera from

the latter part of 1849 to August

1853"

… “During this interval an important

change had taken place in the water

supply of several of the south districts

of London”

40

41

The Lambeth Company removed their water works from opposite

Hungerford Market to Thames Ditton; thus obtaining a supply of water

quite free from the sewage of London.

42

The Southwark and Vauxhall Company ... derived their supply from the

Thames at Battersea Fields.

43

The two Companies were in active competition ... the pipes of the

Lambeth Water Company and those of the Southwark and Vauxhall

Company pass together down all the streets of several of the south

districts.

The grand experiment

44

Units were unconditionally exchangeable:

"The pipes of each Company go down all the

streets, and into ... all the courts and alleys ...

Each company supplies both rich and poor ...

large houses and small ... there is no difference ...

in the condition or occupation of the persons

receiving the water of ... (two) Companies"

“Three hundred thousand people of both sexes ...

every age and occupation ... every rank and

station ... were divided into two groups without

their choice and ... (or) their knowledge (into

those) supplied with water containing ... sewage

... (or) water ... free from such impurity.”

The grand experiment

45

Able to directly compare mortality ratios between 'exposed' (S&V) and 'unexposed' (Lambeth) to

estimate causal risk ratio:

315 / 37 = 8.5

Natural experiments provide an opportunity to examine and estimate

causal effects in observational data.

But there are very few 'true' natural experiments, where an interesting

exposure has been assigned 'as random' to a representative sample.

Imperfect counterfactuals

Few exposures are sufficiently random or short/reversible to provide

unconditional exchangeability; many 'natural experiments' are therefore

simply pre-post studies.

True natural experiments are rare

46

47

Study designs for generating

counterfactuals (quasi-experiments)

48

Quasi-experimental study designs

1. Regression Discontinuity (RD)

2. Interrupted Time Series (ITS)

3. Instrumental Variable (IV)

4. Difference-in-Difference (DiD)

5. Matching methods (e.g. PSM)

49

Interrupted time series analysis

Improving on the standard pre-post design: a classic

example

50

A crackdown on

speeding in Connecticut

On December 23, 1955, Governor Ribicoff

announced that in the future all persons

convicted of speeding would have their licenses

suspended for 30-days on their first offense.

51

"With the saving of forty lives in

1956, a reduction of 12.3% from the

1955 motor vehicle death toll, we can

say my programme is definitely

worthwhile."

52

But to what extent are

the results claimed for

the program by

Governor Ribicoff valid?

What factors might make you doubt the

governor's claims that it was his crackdown

on speeding that caused the fall in

fatalities?

The standard pretest-posttest design fails to control for 6 common

threats

• history (events)

• maturation (processes)

• testing

• instrumentation

• instability (random variation)

• regression to the mean

Threats to the validity of experiments

53

54

Does the publication of

more recent statistics

change your view of the

governor's claims for

the effect of the

program?

55

Does this information

change your view?

A study that uses observations at multiple time points before and after

an intervention (the "interruption"). The design attempts to detect

whether the intervention has had an effect significantly greater than any

underlying trend over time.

The interrupted time series design

56

57

58

59

60

61

62

The hypothetical scenario under which the intervention had not taken

place and the trend continues unchanged (that is: the ‘expected’ trend,

in the absence of the intervention, given the pre-existing trend) is

referred to as the ‘counterfactual’. This counterfactual scenario provides

a comparison for the evaluation of the impact of the intervention by

examining any change occurring in the postintervention period

The interrupted time series design

63

64

O'Donnell A et al. Immediate impact of minimum unit pricing on alcohol purchases in

Scotland: controlled interrupted time series analysis for 2015-18. BMJ 2019;366:l5274.

doi:10.1136/bmj.l5274

Martin J, Cunliffe J, Décary-Hétu D, Aldridge J. Effect of restricting the legal supply of

prescription opioids on buying through online illicit marketplaces: interrupted time series

analysis. BMJ 2018;361:k2270. doi: 10.1136/bmj.k2270

Derde L, Cooper B, Goossens H, Malhotra-Kumar S, Willems R, Gniadkowski M et al.

Interventions to reduce colonisation and transmission of antimicrobial-resistant bacteria in

intensive care units: an interrupted time series study and cluster randomised trial. The

Lancet Infectious Diseases 2014;14(1):31-39. doi: 10.1016/S1473-3099(13)70295-0

Hawton K, Bergen H, Simkin S, Dodd S, Pocock P, Bernal W et al. Long term effect of reduced

pack sizes of paracetamol on poisoning deaths and liver transplant activity in England and

Wales: interrupted time series analyses. BMJ 2013;346:f403. doi: 10.1136/bmj.f403

Dennis J, Ramsay T, Turgeon A, Zarychanski R. Helmet legislation and admissions to hospital

for cycling related head injuries in Canadian provinces and territories: interrupted time

series analysis. BMJ 2013;346:f2674-f2674. doi: 10.1136/bmj.f2674

Examples of ITS studies

65

66

ITS walk-through

67

Smoking ban in Italy

68

In January 2005, Italy introduced regulations to ban smoking in all indoor public places, with the

aim of limiting the adverse health effects of second-hand smoke.

All 20 regions of Italy, including Sicily, were affected.

Barone-Adesi F, Gasparrini A, Vizzini L, Merletti F, Richiardi L. Effects of Italian Smoking

Regulation on Rates of Hospital Admission for Acute Coronary Events: A Country-Wide

Study. PLoS ONE 2011;6(3):e17419. doi: 10.1371/journal.pone.0017419

Objective

Estimate the effects of Italian smoking regulation on rates of hospital admission for acute

coronary events.

Design

Time series study using data on hospital admissions for ACEs from the Italian population after the

implementation of a national smoking regulation in January 2005.

Setting

The 20 Italian regions from January 2002 to November 2006.

The smoking ban in Italy study

69

The smoking ban in Italy study

70

What do you think about the choice of study

design?

What would an appropriate impact model be?

71

A publicly available dataset from a study by Barone-Adesi et al. on the

effects of the Italian smoking ban in public places on hospital

admissions for acute coronary events (ACEs, ICD10 410-411). Data are

for the Sicily region and include information on the numbers of ACE

admissions in that region between 2002 and 2006 among those aged 0-

69 years.

The Sicily dataset

72

A publicly available dataset from a study by Barone-Adesi et al. on the

effects of the Italian smoking ban in public places on hospital

admissions for acute coronary events (ACEs, ICD10 410-411). Data are

for the Sicily region and include information on the numbers of ACE

admissions in that region between 2002 and 2006 among those aged 0-

69 years.

The Sicily dataset

73

Our task is to use these data to build a model that will enable us to

estimate the effect of the smoking ban on hospital admissions for

ACEs?

Two parameters define each segment of a time series: level and trend (or

slope). It's possible to make an assessment of changes in level and/or trend by

visual inspection of the time-series. But unable to state whether changes in

level and trend are the result of (1) chance, or (2) factors other than the

intervention.

The most common method for analysis of ITS is "segmented" regression

analysis. Linear regression models are used to estimate level and trend in the

pre-intervention segment and changes in level and trend after the intervention.

Such analyses can be used to assess chance and control for other effects—

segmented regression controls for baseline level and trend (secular changes

that may have occurred in the absence of the intervention).

Statistical analysis

74

In its simplest form, an ITS analysis uses a "segmented" regression model that requires only three

variables. The exact form of model required depends on the impact model proposed.

For a change in level and slope the following segmented regression model is used:

Yt is the outcome at time t;

T is the time elapsed since the start of the study with the unit representing the frequency with

which observations are taken (e.g. month or year);

Xt is a dummy variable indicating the pre-intervention period (coded 0) or the post-intervention

period (coded 1).

Segmented regression

75

𝑌𝑡 = 𝛽0 + 𝛽1𝑇 + 𝛽2𝑋𝑡 + 𝛽3𝑇𝑋𝑡 + 𝜀𝑡

76

β0 represents the baseline level at time T=0

β1 is the change in outcome associated with a time unit increase (the underlying pre-intervention

trend)

β2 is the level change following the intervention

β3 is the slope change following the intervention (using the interaction between time and

intervention TXt)

77

Takeaways

Study of causation as a separate topic distinct from statistics. See work of Rubin and Pearl.

Causal inference is difficult and requires careful thinking! Be honest about your causal intentions.

Theory driven not data driven.

QE approaches are methods for evaluating the impact of interventions where experimental

manipulation is not feasible.

Choice of approach is best made on specifics of intervention and circumstance rather than

general rules about which methods are strongest.

Value of an analysis protocol, setting out hypotheses and methods, developed before any data

analysis is conducted.

Follow established reporting guidelines such as STROBE (Strengthening the Reporting of

Observational Studies in Epidemiology) or TREND (Transparent Reporting of Evaluations with

Nonrandomized Designs).

Takeaways

78