PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea

Population Health Management

NHS England and NHS Improvement

PHM Analyst Academy26th February

09:00 -09:30 Arrival and Registration

09:30 – 09:45 Welcome & introduction to the day (Mentimeter) M Mohammed, SU and J Dellar, PHE

09:45 – 10:30 An apple a day: Logic models & Qual Methods P Mason, SU

10:30 – 10:45 (Ex-ante) Design stage evaluations / impact assessments vs

(Ex-post) summative evaluations

A Hood, SU

10:45 – 11:00 Coffee /Tea break

11:00 – 12:30 Modelling approaches (RIGHT Framework) A Hood, SU

12:30 – 13:15 Lunch

13:15 - 14:30 Experimental study designs

- Randomised controlled trials

- Cluster randomised trials

- Stepped wedge designs

P Seamer, SU

14:30 – 14:45 Coffee/Tea Break

14:45 - 15:45 Observational study designs

- Interrupted time series analysis

(exercise)

- Matched cohort design

- Synthetic controls

P Seamer, SU

15:45 - 16:00 Mentimeter questions

Review and look ahead to Session 4

J Dellar, PHE and M A Mohammed, SU

midlandsphmacademy.nhs.uk



What’s your theory of change?

A brief introduction to logic models

Paul Mason

Every action has a causal theory

5

“If we do x, then we’ll get y”

Sometimes explicit; usually implicit

“If we deliver our training package, then we will

improve the skills of care homes staff...

If staff are more skilled, then they will be more able to

cope in the event of a crisis...

If staff are more able to cope in a crisis, then there will

be fewer unplanned admissions to hospital....

If there are fewer unplanned admissions, then more

people will die in a setting of their choice. They will have

a better death; we will make better use of resources.”

‘An apple a day keeps the doctor

away…’

Giving free fruit to primary schools in order

to improve health

What’s the causal story? How do you

(logically) get from action to outcome?

Pulling out this ‘theory of change’ is essential for evaluation – and for designing initiatives

We learn via advances in theory

So the unit of analysis for evaluation

should be ‘the theory’

Results then refine, reject, (provisionally)

support the theory

Apples

Delivered

Apples

Eaten

Vitamin

Levels

Raised

Health

Outcomes

Improved

Interpretation

❌ ❌ ❌ ❌ Implementation Failure

☑ ❌ ❌ ❌Engagement Failure

(first causal link)

☑ ☑ ❌ ❌Theory Failure

(early causal link)

☑ ☑ ☑ ☑ Consistent with theory

☑ ☑ ☑/❌ ☑Theory Failure

(later causal link)

☑ ☑ ☑/❌ ☑/❌Partial Theory Failure

Works in some contexts

☑ ☑ ❌ ☑Theory Failure

(different causal path)

“An apple a day...” (Ref: Funnell & Rogers, 2013)

Logic models are one tool for representing these theories. There are different approaches; all share common elements*

* Terminology varies but basic concepts remain constant

Inputs

Resources used

Activities

Things done (measured by outputs)

Outcomes

Effects of activities

Impacts

Broader societal ‘goods’

Simple logic model for refurbishing a house (the lazy way)…

Inputs

£

Time

Activities

Source and

manage

experts

Outcomes

Improved

domestic

environment

Impacts

Increased

house value

Improved area

Inputs

£

Time

Activities

Book place to

stay, find

things to do

together

Outcomes

Reduced

stress (?)

Increased

happiness

Impacts

Improved

family

functioning

…or going on a family holiday

“If we deliver our training package, then we

will improve the skills of care homes staff...

If staff are more skilled, then they will be more

able to cope in the event of a crisis...

If staff are more able to cope in a crisis, then

there will be fewer unplanned admissions to

hospital....

If there are fewer unplanned admissions, then

more people will die in a setting of their choice.

They will have a better death; we will make

better use of resources.”

This shows the

theory that

connects activity…

…to outcomes…

…to impacts

•Impacts are the final effects that you are working towards – e.g. increased life expectancy, reduced health inequality, more sustainable services, etc

•Normally expressed at a high level. Triple / quadruple aim a useful framework

•Changes at this level only indirectly attributable to your intervention –you ‘contribute to’, rather than ‘cause’. Contextual factors a significant influence

Work back: what change do you ultimately want to see?

•These are the changes that you are trying to make / that would (logically / evidence suggests…) result from your activities

•Can be broken down into:

• Intermediate outcomes – changes in knowledge / awareness / skills / access

• Outcomes – changes in behaviour / condition / status

•Language suggesting change is therefore important: ‘reduced, increased, improved, better, worse’

Being more detailed: what outcomes do you need to achieve your impact?

•The things you do (e.g. establish apple scheme, etc)

•Measured by outputs (e.g. # people eating apples; # apples eaten)

•You don’t need to be detailed – just the main activities - the logic model is not a plan

•What ‘mechanism’ links activities to outcome? A causal ingredient?

What will you do to achieve these outcomes?

Often at this point, you’ll find

yourself working backwards

and forwards

•Should be fairly straightforward: these are the resources you have to do the things you do

•Usually measured in £

•For most programmes, cash funding is the largest element – but maybe there are in-kind inputs too, e.g. if partners have assigned staff to your programme, if you have lots of volunteers, if you are given ‘free’ facilities, etc..

And what resources will you use?

“In an Integrated Care System, NHS organisations, in partnership with local councils and others, take collective responsibility for managing resources, delivering NHS standards, and improving the health of the population they serve.”

NHS England

Inputs

Resources used

Activities

Things done (measured by outputs)

Outcomes

Effects of activities

Impacts

Broader societal ‘goods’

Take a step back and reflect on:

1: Assumptions in the model:o Practical (e.g. shows significant reliance on recruitment of…)

o Evidential (e.g. implied connection between activity x and effect y)

o Contextual (e.g. that there is no significant change in regulation of x)

Can this be used in programme planning? Is this showing risks to be managed? Would more evidence help design? What does it mean for evaluation?

2: Your overall theory of change. Policies generally use:o Sticks (beat / regulate things into place)

o Carrots (incentivise / ease the change you want)

o Sermons (eulogise and persuade)

What is the mix in your theory? Does this seem optimal given the task? If not, what is missing and can this be managed?

Parting note 1: Logic models and economic evaluation

Cost

(Economy)

Efficiency (£ per output)

Effectiveness / Benefit (£ per outcome)

Parting note 2: your model should reflect your theory, which might not be linear

Further resources / guidance

1. www.strategyunit.co.uk – search ‘logic model’ for fuller guide to using logic models

2. www.betterevaluation.org

3. HMT ‘Magenta Book’ – good all round guidance on evaluation

http://www.betterevaluation.org/



An incredibly brief introduction to qualitative

methods for evaluation

Paul Mason

Qualitative Vs

Quantitative?

There are many different types of qualitative method / approaches for evaluation…

Individual

interviews

Group

interviews

Focus groupObservation

Document

analysis

Participatory

approaches

Diaries

Social media

analysisQualitative

surveysCase studies

Ethnography

…all underpinned by some common purposes

Understanding things as

they are experienced by

the people involved

(context)

Exploring – what matters, to

whom, why

Explaining (e.g.) why did

events unfold in this way?

Creating (e.g.) what

should we do now?

(Not always, but…) democratising

the evaluation process

Interesting to note what is

not on this list that would

be there for quantitative

method

Choice of method depends on many factors

•The question! First and most important consideration…

•Time and resource. Qualitative methods can be expensive / time consuming

•Ethics. Accessing participants, what you might find, power dynamics (etc)

•Skills. Done well, it looks easy; but so easy to do badly. Analysis of non-quantitative data is a specialist skill too

•Stage of the evaluation process. Drawing out theory? Designing approaches (defining outcomes that matter)? Tracing process / implementation? Explaining effects? Interpreting findings / working up recommendations?

•Related point: where does this fit with quantitative methods?



Evaluation and Impact Assessments

Andy Hood

•Before implement - design stage evaluation / ex-ante modelling

•After implement - quantitative (ex-post) evaluation

When should we estimate the effect of a planned change?

31

time

Ex-ante

Ex-post

known

unknown

Intervention

or RiskOutcome

or Impact Applications

• commissioning plans• service plans• opportunity assessment• business case

• roll-out • decommission

• adjust plans / contracts• remedial action



Break



Design stage evaluationAndy Hood

What it is that makes each of these a model?

What are common characteristics of models?

y ~ x

•Models help us understand things that would otherwise be obscured by the complexity of the real world.•Associations, casual relationships and core dynamics – descriptive / explanatory

•Consequences, forecasts - predictive

•Help us understand what we need to do - prescriptive

•Models make us document our assumptions.

•Models help us test things that would be too costly / risky / impractical / unethical to try in real life.

•Models can act as guides or templates for complicated actions / developments.

Why do we need models in healthcare?

Modelling and simulation techniques for supporting healthcare decision making: a selection framework

A collaboration of 6 universities

•Cambridge Engineering Design Centre, University of Cambridge

•The School of Information Systems, Computing and Mathematics, Brunel University

•Brunel Business School, Brunel University

•The School of Management, University of Southampton

•The Information Engineering Research Group, University of Ulster

•The School of Mathematics, Cardiff University

The RIGHT Frameworkhttps://www-edc.eng.cam.ac.uk/downloads/right.pdf

https://www-edc.eng.cam.ac.uk/downloads/right.pdf

“This workbook is intended to provide guidance for people who are making decisions in healthcare. It is aimed at anyone who wants to find out more about different modelling and simulation techniques –what they are, when to apply them, and what resources are required to use them. It will not only help decision makers commission more appropriate modelling work, but also assist professional modellers and business consultants to expand their modelling repertoire in order to meet the diverse needs of their clients.

The workbook is not a “how-to-do” guide to modelling and simulation, rather a “what-is-it” introductory guide. That said, the further reading section at the end of the workbook will help locate further details for each technique. The RIGHT research team would also welcome any contact regarding the applications of these techniques.”

Technique characterisation (input requirements)

RIGHT framework: technique selection table

Thinking about a specific project you are involved with and using the RIGHT resources on your tables:

• What stage of the project cycle are you currently (stuck) at?

• What type/s of output are you looking for?

• What constraints are you under - time, costs etc…

When you’ve identified a potential modelling tool, your HOMEWORK is to find about a bit more about that technique and discuss with manager/project lead how you’re going to apply it.

Have a go yourself....(5 mins)

RIGHT Modelling

Framework

Design Stage

Evaluation

•A visual aid to explore how different variables in a system are related.

•A modelling method in its own right – but can also underpin quantitative modelling methods

•Comprised of

•a set of nodes, each representing a quantity or variable

•a set of arrows indicating the influence of one quantity on another

Causal loop diagrams (directed acyclic graphs)

X Y

an increase (decrease) in X tends to cause

an increase (decrease) in Y…

… all other things being equal

Population size

Number of

people with a

mental health

problem

Smoking

prevalance

Population life

expectancy

X Y

an decrease (increase) in X tends to cause

an increase (decrease) in Y…

… all other things being equal

X Y

X influences Y but through Z (i.e. not directly)

Z

Prevalence of

cardio-

vascular

disease

Premature

deaths from

cardio-vascular

disease

Smoking

prevalence

Population life

expectancy

Prevalence of

cardio-

vascular

disease

Premature

deaths from

cardio-vascular

disease

Smoking

prevalence

Population life

expectancy

Incidence of

cancer

Premature

deaths from

cancer

hungerfood

consumptionb

a balancing loop

bank balance interestr

a reinforcing loop

•Balancing and reinforcing loops can be made up of more the 2 nodes

•A loop is

Balancing – if there are an odd number of negative arrows (- - - →)

Reinforcing – if there are no or an even number of negative arrows (- - - →)

Balancing and reinforcing loops

b

r

•Work in groups of 2 or 3

•Join the nodes to indicate influence

•Decide if/where there are balancing or reinforcing loops.

•When discussions concluded, you may want to produce final / tidy version

TIP: In total, we think there are 10 arrows and 3 (+1) loops.

Exercise – Join the dots patients who

need to be

admitted

admissionsaverage

length of stay

admission

threshold

discharge

threshold

unoccupied

beds

all beds


need to be

admitted

admissionsaverage

length of stay

admission

threshold

discharge

threshold

unoccupied

beds

all beds


need to be

admitted

admissionsaverage

length of stay

admission

threshold

discharge

threshold

unoccupied

beds

all beds

b

bb


need to be

admitted

admissionsaverage

length of stay

admission

threshold

discharge

threshold

unoccupied

beds

all beds

b

bb

r

Roemer’s Law

"in an insured population, a hospital bed built is a filled bed“

Shain, M; Roemer, MI (April 1959). "Hospital costs relate to the supply of beds". Modern Hospital. 92 (4): 71–3

Parkinson’s Law

"the number of patients always tends to equality with the number of beds available for them to lie in“

patients who

need to be

admitted

admissionsaverage

length of stay

admission

threshold

discharge

threshold

unoccupied

beds

all beds

b

bb

r

www.kumu.io

http://www.kumu.io

Last year…

•Hip replacements completed – 1,253

•Mean length of stay – 7 days

•Overnight bed occupancy – 80%

•Beds – 30

•Mean waiting time – 91 days

You speak with the service and clinical lead and they say that in 5 years time they expect…

•Demand will have increased by 12%

•Mean length of stay will fall by 14%

•It would like overnight bed occupancy to reduce to 70%

How many more/less beds will we need?

Quantitative modellingan elective inpatient hip replacement service

Current Change parameter Future

Annual admissions 1,253 +12% 1,403

Average LoS 7.0 -14% 6.0

Actual occupied bed

days8,771 8,448

Bed days with

occupancy %10,964 -10% 12,069

Beds required 30 33

Therefore, under this model, we would require an additional 3 beds for our hip replacement service

Last year…

•Hip replacements completed – 1,253

•Mean length of stay – 7 days


•Beds – 30



•Demand will have increased by 12% for all levels of need

•24% of cases have only a moderate level of need. They would like to divert all these cases to receive a non-surgical service in another unit

•Mean length of stay for the non-moderate cases will fall by 14%



Last year…

•Hip replacements completed – 1,253•Moderate need - 301•High need - 752•Severe need - 200

•Mean length of stay – 7 days•Moderate need – 3.0 days•High need – 7.0 days•Severe need – 13.0 days


•Beds – 30




•24% of cases have only a moderate level of need. They would like to divert all these cases to receive a non-surgical service in another unit



How many beds will we need?

What assumptions are needed to reach this view?



Annual admissions 1,253Shift 'moderate need'

then +12%1,066

Average LoS 7.0-14% for remaining

cases7.1

Actual occupied bed

days8,771 7,575

Bed days with

occupancy %10,964 -10% 10,821

Beds required 30 30

Therefore, under this model, we would require no change in beds for our hip replacement service

Last year…

•Hip replacements completed – 1253•Moderate need - 301•High need - 752•Severe need - 200

•Mean length of stay – 7 days•Moderate need – 3.0 days•High need – 7.0 days•Severe need – 13.0 days


•Beds – 30




•24% of cases have only a moderate level of need. They would like to divert these cases to receive a non-surgical service in another unit



How many beds will we need?•What if we only manage to divert 50% of moderate cases?•What if we live with 80% occupancy?

What assumptions are needed to reach this view?



Annual admissions 1,253

Shift 50% of

'moderate need'

then +12%

1,235

Average LoS 7.0-14% for remaining

cases6.5

Actual occupied bed

days8,771 8,009

Bed days with

occupancy %10,964 No change 10,012

Beds required 30 27

Therefore, under this model, we would require 3 fewer beds in our hip replacement service

But what if the question you really need to answer was;

•If we divert moderate need patients from the beginning of year 4, how long will it take before the average waiting time falls below 84 days?

•What proportion of patients will spend more than 15 days in hospital?

•How frequently will occupancy exceed 85%?

What if the length of stay reduction is heavily dependent on the occupancy rate?

In such circumstances, basic rules-based models are not sufficient.

Two options worth considering;

•Systems dynamics modelling

•Discrete event simulation

Type Name Link

Proprietary AnyLogic https://www.anylogic.com/

Simul8 https://www.simul8.com/

Sysdea https://sysdea.com/

Open Source Insight Maker https://insightmaker.com/

Simmer (for R) https://r-simmer.org/

https://www.anylogic.com/

https://www.simul8.com/

https://sysdea.com/

https://insightmaker.com/

https://r-simmer.org/

Time dependant

I need to know about the model variables at a number of time points –

not just the start and end state of the model and I expect the changes

to evolve non-linearly over time

⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫

Individual

I need to know about variation between patients (its not sufficient to

know the average effect for groups of patients) and I have patient level

data to populate the model.

⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫

Interactions / dynamic

What happens to one patients strongly and importantly effects what

happens to others.⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫

Stochastic

I need to know about the degree of uncertainty in the model results

(either due to randomness or uncertainty of inputs / parameters) – a

point estimate with / without sensitivity analysis is not sufficient.

⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫

Basic rule based model ⚫

Systems dynamics model ⚫ ⚫ ⚫ ⚫

Discrete event simulation model ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫

Discrete event simulation (DES)

Models of sequences of discrete events within a system where future events are a condition of the current system state.

•e.g. patients flowing through a hospital.

Probability distributions used to describe the arrival and treatment of patients.

https://www.youtube.com/watch?v=4BHBJlzv4RA

Systems dynamics modelling (SD)

Used to model nonlinear behaviour in systems over time.

Causal loop models used to describe stocks and flows within a systems.

Calculus used to define the system.

https://www.youtube.com/watch?v=nTD1SL2qp3o

https://www.youtube.com/watch?v=4BHBJlzv4RA

https://www.youtube.com/watch?v=nTD1SL2qp3o

Evidence is best – but not always available. It’s often necessary to rely on expert opinion.

Establish a reference group

•One opinion is better than none, but multiple opinions are better still (https://en.wikipedia.org/wiki/Wisdom_of_the_crowd)

•Be clear what you need to know

•Provide the reference group with useful context

Ask your reference group members to provide •a central estimate (best guess) is good, but ranges are better still•Ask for best guess, low & high estimate – and use triangular distributions or ask for 90% confidence intervals•You can calibrate you reference group member’s views (https://hubbardresearch.com/publications/how-to-measure-anything-book/)

Delphi methods can be used to aggregate multiple views•Ask reference group members for views –share these and ask or second set of estimates (https://en.wikipedia.org/wiki/Delphi_method)

Practical tips on parameritising* a model?

* yes that really is a word!

https://en.wikipedia.org/wiki/Wisdom_of_the_crowd

https://hubbardresearch.com/publications/how-to-measure-anything-book/

https://en.wikipedia.org/wiki/Delphi_method



Lunch



Experimental Study Designs

Paul Seamer



Break



Observational Study Designs

Paul Seamer



ReflectionsMohammed A Mohammed and Janine Dellar



Many thanks for your participation!

Documents

PHM Analyst Academy · 2020-03-11 · 10:30 –10:45 (Ex-ante) Design stage evaluations / impact assessments vs (Ex-post) summative evaluations A Hood, SU 10:45 –11:00 Coffee /Tea