© Nuffield Trust 22 June 2015

Matched Control Studies:

Methods and case studies

Cono Ariti

cono.ariti@nuffieldtrust.org.uk

© Nuffield Trust

Predictive risk

modelling

Resource

allocation

Descriptive

studies Evaluations

Integrated

care

pilots

nuffield trust

Nuffield Trust Research team – data linkage projects

Risk

sharing

for CCGs

nuffield trust Combined

predictive

model

nuffield trust

Person

based

resource

allocation

nuffield trust

Social

care at

end of life

nuffield trust

Cancer

and social

care

nuffield trust

Predicting

social

care

costs

nuffield trust

Virtual

Wards

nuffield trust

WSD

nuffield trust

British Red

Cross

nuffield trust

© Nuffield Trust

Need for evaluation

Need to know what works

• In a practical setting – “real world evaluation”

• Clarify the debate

• Likely impacts – unbiased results

• Link to qualitative work

Refine programs

• Obtain feedback and learnings – the pain of implementation

• Explore sub-groups – where did it work? Where could it work?

© Nuffield Trust

Issues with evaluations

Randomised control trials

• “Gold standard”

• May not be feasible or ethical

• Inclusion and exclusion rules can limit generalisation

• Are still subject to poor implementation – can induce bias

• Potentially expensive!

Observational studies

• Typically no “natural” experiment exists

• Often no comparable control group to provide a fair assessment

© Nuffield Trust © Nuffield Trust

Matched Control Studies -

Methods

© Nuffield Trust

Matched Control Studies

The basic idea

• Match controls to those treated based on measured characteristics in existing datasets

• The control group and treated group should look similar “on balance”

• Mimics the idea of an RCT

• Based on propensity score theory (Rubin & Rosenbaum, 1983) and earlier work on matching (Cochran, 1965)

© Nuffield Trust

Matched Control Studies

Matching

• Prognostic risk score

• Demographics – age, gender, deprivation, ethnicity

• Prior acute care service use – admissions, OP and A&E attendances

• Prior diagnoses, targeted chronic conditions

Balance

• In this case all matching variables

• Additional variables such as length of stay, additional diagnoses and longer service use history

• Assures comparability between the groups

© Nuffield Trust

Matching Algorithm

Algorithm

• Exact match not possible

• Computer intensive “genetic algorithm”

• Uses a weighted Mahalanobis “distance” to determine closest match

• Automatically assesses balance and moves to an improved solution

Assessing Balance

• On overall group similarity

• Compares means and distribution of variables in the two groups

© Nuffield Trust

Analysis of matched control studies

• Standard statistical methods to estimate the difference in the two groups

• Regression models, difference in difference analysis

• By including matching variables in the statistical adjustment remaining imbalances can be reduced – “doubly robust”

• Methods exist for sensitivity analysis – impact of unobserved variables

• Some controversy around accommodating the matching in analysis

© Nuffield Trust © Nuffield Trust

Case Study 1: Telehealth

Programme

© Nuffield Trust

Case Study 1: Telehealth program

Intervention:

• Remote monitoring for patients with long term conditions

Nuffield commissioned to evaluate impact:

• Primary: Reduction in emergency hospital admissions?

• Secondary: Reductions in Emergency attendances, outpatient attendances, mortality

Methods:

• Retrospective matched control study – use of already existing administrative data

© Nuffield Trust

Description: Telehealth program

© Nuffield Trust

Matched control studies – broad aim

>30,000 individuals – resident in local area June

2010 to March 2013, did not receive telehealth

and were eligible for matching

(local controls)

Aim to find 716 individuals who match

almost exactly on a broad range of

characteristics

Use this group as study control group

716 individuals – enrolled June 2010 to March 2013

& received Telehealth intervention & eligible for

matching

© Nuffield Trust

Datasets available

Telehealth Nuffield trust

N = 716

• person details

• dates of service

• type of service

Identifiers:

Names, DOB,

Addresses, etc

• dates & place

of death for all

people in

England,

• associated

hospital (HES)

records

Identifiers:

Nuffield Trust

specific HESID

Administrative data ONS deaths Hospital inpatient, outpatient, AE

© Nuffield Trust

1

2

3

4

5

6

7

8

9

. .

1

2

3

4

5

6

7

8

9

. .

1

2

3

4

5

6

7

8

9

. .

Telehealth Data Linkage Service Nuffield Trust

New Identifier New Identifier New Identifier

(NHS no) (NHS no)

Names Names

Address Address

DOB DOB

HESID HESID

Telehealth person identifiers (File A)

© Nuffield Trust

Final datasets available for analysis

Nuffield trust

Identifiers:

HESID on all

ONS deaths Hospital inpatient, outpatient, AE Telehealth data - desensitised

Use all this

info to carry

out matched

control

analysis

© Nuffield Trust

Control group – how well matched?

© Nuffield Trust

Control group – how well matched?

Telehealth Controls

© Nuffield Trust

Control group – how well matched?

Telehealth Controls

© Nuffield Trust

Key Result 1: Risk of admissions or death

© Nuffield Trust

Key Result 2: Changes in admissions or attendances

(six months pre and post intervention)

© Nuffield Trust

Results

• Telehealth patients tended to be admitted for an emergency admission earlier than control patients

• There was no difference in mortality between the telehealth and control groups

• There were no statistically significant reductions in hospital admissions when comparing the period six months before and after the telehealth intervention

• In summary the Telehealth program did not have a significant impact on acute care outcomes

• Sensitivity analysis showed little evidence of an important unobserved variable

© Nuffield Trust © Nuffield Trust

Matched Control Studies:

Summary

© Nuffield Trust

Matched Controls: Summary

Benefits

• Makes full use existing data, with relative ease

• Techniques applicable to many different types of services and datasets

• Decisions on what seems to work (and what may not) based on more robust analyses leading to better informed decisions

Caveats

• If important unobserved variables exist results may be biased

• The routine data sources must contain the relevant data

© Nuffield Trust

Implementing locally – key enablers

Do you have … Can you …

• Access to data that contains

the outcomes relevant to your

evaluation?

• Access to data containing

relevant matching

characteristics?

• Do you have consent to

access/link the data?

• Analysis tools to apply

statistical methods to the data?

• Skilled analysts to analyse the

data?

• Link multiple sources of data?

• Handle large amounts of data

(millions of observations)?

• Identify recipients of the

intervention?

• Transform and augment that data

with bespoke variables?

• Apply sophisticated matching

algorithms routinely to this data?

• Analyse the data with a variety of

statistical methods and interpret

the results appropriately?

© Nuffield Trust

www.nuffieldtrust.org.uk

Sign-up for our newsletter www.nuffieldtrust.org.uk/newsletter

Follow us on Twitter: Twitter.com/NuffieldTrust

© Nuffield Trust

Thank you