Upload
cheweb1
View
503
Download
0
Embed Size (px)
Citation preview
A methodological framework for developing the structure of Public Health economic models Dr Hazel Squires, ScHARR, University of Sheffield
Prof. Jim Chilcott, Prof. Ron Akehurst,
Dr. Jennifer Burr, Prof. Mike Kelly
Introduction
• Why we need a conceptual modelling framework
specific to Economic Modelling in Public Health
(PHE)
• Methods for developing the PHE conceptual
modelling framework
• An outline of the PHE conceptual modelling
framework
• Conclusions
Background
• Healthcare agencies such as the National
Institute for Health and Care Excellence (NICE)
need to decide how to spend their money.
• Mathematical models are used to provide a
rational, coherent & transparent framework of
the cost-effectiveness of healthcare
interventions.
• Inappropriately simple models and lack of
justification may lead to poor validity and
credibility, resulting in suboptimal allocation of
resources.
Current methods for dealing with structural uncertainty
1) Retrospective: following model implementation
by expressing the impact of uncertainties upon
the model results.
• Scenario analysis
• Model averaging
2) Prospective: considering the process through
which decisions are made around the
conceptualisation, structuring and
implementation of the model.
• Model development practice in health economics is
extremely varied (Chilcott, 2010). Chilcott J, Tappenden P, Rawdin A, Johnson M, Kaltenthaler E, Paisley S et al. Avoiding and identifying errors in health
technology assessment models: qualitative study and methodological review. Health Technol Assess 2010; 14(25).
What is a Conceptual Modelling (CM) Framework?
• ‘A set of steps that help to guide modellers
through the development of a model structure,
from developing and describing an
understanding of the decision problem to the
abstraction and non-software specific
description of the quantitative model, using a
transparent approach which enables each stage
to be shared and questioned.’
Based upon Kaltenthaler E., Tappenden P., Paisley S. and Squires H. Identifying and reviewing evidence to inform the
conceptualisation and population of cost-effectiveness models, No. 14, 2011 and Robinson S. Conceptual
modelling for simulation Part I: definition and requirements. Journal of the Operational Research Society 2008;
59:278-290.
Key potential benefits of a CM Framework
• Aids the development of modelling objectives;
• Provides tools for communication with stakeholders;
• Guides model development and experimentation;
• Improves model validation (developing the right model);
• Improves model verification (developing the model right);
• Allows model reuse;
• Helps to characterise structural uncertainties and identify
primary research needs.
Economic evaluation of Public Health compared with clinical interventions
• A key objective of Public Health is reduction of health
inequities;
• PH interventions tend to:
- Be multi-component with complex causal chains;
- Operate within dynamically complex systems;
- Be dependent upon human behaviour;
- Be dependent upon the social determinants of health,
requiring consideration of non-health costs & outcomes;
• It is much less clear what a ‘good’ outcome is;
• The culture & politics of the system is important in
choosing & assessing interventions
Methods for developing PHE conceptual modelling framework
1) Literature review of key challenges in PHE modelling
(Squires et al., 2016)
2) Review of conceptual modelling frameworks in broader
literature.
3) Qualitative research to understand modellers’ experiences
with developing the structure of PH economic models &
their views about the benefits/ barriers of using a CM
framework.
4) Pilot of the draft CM framework within a case study of
diabetes screening & prevention.
5) Evaluation via qualitative analysis & theory-based
evaluation. Squires H, Chilcott J, Akehurst R, Burr J, Kelly M. A systematic literature review of the key challenges for developing the
structure of public health economic models. Int. J. of Public Health. Epub ahead of print.
http://www.ncbi.nlm.nih.gov/pubmed/26747470
The CM framework: Key principles
(1) A systems approach to Public Health modelling should
be taken (feedback loops & unintended consequences
are important);
(2) Developing a thorough documented understanding of
the problem is imperative prior to and alongside
developing and justifying the model structure;
(3) Strong communication with stakeholders and members
of the team throughout model development is essential;
(4) A systematic consideration of the determinants of health
is central to identifying all key impacts of the
interventions within Public Health economic modelling.
The CM framework: Key principles
(1) A systems approach to Public Health modelling should
be taken (feedback loops & unintended consequences
are important);
(2) Developing a thorough documented understanding of
the problem is imperative prior to and alongside
developing and justifying the model structure;
(3) Strong communication with stakeholders and members
of the team throughout model development is essential;
(4) A systematic consideration of the determinants of health
is central to identifying all key impacts of the
interventions within Public Health economic modelling.
The CM framework: Key principles
(1) A systems approach to Public Health modelling should
be taken (feedback loops & unintended consequences
are important);
(2) Developing a thorough documented understanding of
the problem is imperative prior to and alongside
developing and justifying the model structure;
(3) Strong communication with stakeholders and members
of the team throughout model development is essential;
(4) A systematic consideration of the determinants of health
is central to identifying all key impacts of the
interventions within Public Health economic modelling.
The CM framework: Key principles
(1) A systems approach to Public Health modelling should
be taken (feedback loops & unintended consequences
are important);
(2) Developing a thorough documented understanding of
the problem is imperative prior to and alongside
developing and justifying the model structure;
(3) Strong communication with stakeholders and members
of the team throughout model development is essential;
(4) A systematic consideration of the determinants of health
is central to identifying all key impacts of the
interventions within Public Health economic modelling.
The determinants of health
Dahlgren G., Whitehead M. Policies and strategies to promote social equity in health. 1991. Institute
for Future Studies, Stockholm.
CM framework overview Aligning the framework with
the decision making process
Identifying relevant
stakeholders
Understanding the problem
i) Developing a conceptual model of the problem
ii) Establishing resource pathways
Developing and justifying the model structure
i) Reviewing existing economic evaluations
ii) Choosing specific model interventions
iii) Determining the model boundary
iv) Determining the level of detail
v) Choosing the model type
vi) Developing a qualitative description of the quantitative model
CM framework overview Aligning the framework with
the decision making process
Identifying relevant
stakeholders
Understanding the problem
i) Developing a conceptual model of the problem
ii) Establishing resource pathways
Developing and justifying the model structure
i) Reviewing existing economic evaluations
ii) Choosing specific model interventions
iii) Determining the model boundary
iv) Determining the level of detail
v) Choosing the model type
vi) Developing a qualitative description of the quantitative model
CM framework overview Aligning the framework with
the decision making process
Identifying relevant
stakeholders
Understanding the problem
i) Developing a conceptual model of the problem
ii) Establishing resource pathways
Developing and justifying the model structure
i) Reviewing existing economic evaluations
ii) Choosing specific model interventions
iii) Determining the model boundary
iv) Determining the level of detail
v) Choosing the model type
vi) Developing a qualitative description of the quantitative model
Stakeholders
Stakeholders are identified based upon the
classification from Soft Systems Methodology
(SSM):
• Customers which might include patient
representatives and lay members;
• Actors which might include clinical experts and
epidemiologic experts for all relevant diseases
and methods experts;
• System owners which might include policy
experts (in addition to some of the people
identified as actors).
CM framework overview Aligning the framework with
the decision making process
Identifying relevant
stakeholders
Understanding the problem
i) Developing a conceptual model of the problem
ii) Establishing resource pathways
Developing and justifying the model structure
i) Reviewing existing economic evaluations
ii) Choosing specific model interventions
iii) Determining the model boundary
iv) Determining the level of detail
v) Choosing the model type
vi) Developing a qualitative description of the quantitative model
Example conceptual model of the problem Contraception project example:
Why is this a problem?
_ _
Maximise health Minimise costs to NHS & PSS Minimise costs to other sectors
_ _ _ _ _ _ _ _ _
Poor health outcomes of the mother
Low birth weight babies + Poor long term socioeconomic outcomes
+ + +
Miscarriage/ stillbirth Abortion Unwanted teenage birth
+ + +
What is the problem? +
Unintended teenage pregnancies STIs
_ + +
Poor contraceptive use by sexually active teenagers
(interaction between a male & female)
+ +
Disadvantaged background
Competing risks
Dependent on history & timing of STIs
Questions to help develop diagram: The disease & determinants of health
• Have any relevant disease natural histories been
captured?
• Are the following determinants of health important in
determining outcomes & in what way:
• Age, sex & other inherent characteristics of the population of
interest?
• Individual lifestyle factors?
• Social and community networks?
• Living & working conditions & access to essential goods &
services?
• General socioeconomic, cultural & environmental conditions?
Questions to help develop diagram: Dynamic complexity
• Are there any other (positive or negative)
consequences of each concept?
• Are there any other possible causal links
between the factors? (to establish whether there
are any feedback loops)
• Could there be any other factors which explain
both of these outcomes, for links which may not
be causal, but correlated?
• Are there interactions between different sets of
people?
• Is timing/ ordering of events important?
Questions to help develop diagram: Intervention outcomes
• What is a good outcome?
• What would happen in the absence of the
interventions versus as a result of the interventions
– would behaviour be prevented or delayed?
• What evidence exists to describe the outcomes of
the intervention/ comparator over time? Are
behavioural outcomes important?
• Are there any determinants of health reported by
the effectiveness studies which are not included
within the causal diagram? Can such a relationship
be described?
Questions to help develop diagram: Dynamic complexity of interventions
• Might a third party act to reduce or increase the
impact of interventions?
• Are there any substantial impacts of social and/or
community networks upon intervention
effectiveness? Will these impacts be captured over
the long term within the effectiveness evidence?
• Are there any substantial impacts of the
interventions upon other lifestyle factors?
• Might the interventions have other impacts not
already considered?
Sources of evidence
Starting with high yield sources
Individual stakeholder
assumptions & beliefs
Stakeholder
discussion
Modeller assumptions
& beliefs
Project scope
Literature
sources
Existing diagrams/
previous work
Conceptual model of the problem
CM framework overview Aligning the framework with
the decision making process
Identifying relevant
stakeholders
Understanding the problem
i) Developing a conceptual model of the problem
ii) Establishing resource pathways
Developing and justifying the model structure
i) Reviewing existing economic evaluations
ii) Choosing specific model interventions
iii) Determining the model boundary
iv) Determining the level of detail
v) Choosing the model type
vi) Developing a qualitative description of the quantitative model
CM framework overview Aligning the framework with
the decision making process
Identifying relevant
stakeholders
Understanding the problem
i) Developing a conceptual model of the problem
ii) Establishing resource pathways
Developing and justifying the model structure
i) Reviewing existing economic evaluations
ii) Choosing specific model interventions
iii) Determining the model boundary
iv) Determining the level of detail
v) Choosing the model type
vi) Developing a qualitative description of the quantitative model
Develop understanding of the
problem
Assess whether there is
an existing model which
could be employed
Identify strengths &
limitations of different
model structures
Identify strengths &
limitations of different
model types
Identify key variables which
generally affect model results
(incl. any not already
identified) & key variables
included within the causal
diagram which do not
Identify the sort
of data available
Identify factors with not many
causal links & assess whether
they would have a substantial
impact upon the difference
between outcomes of
interventions & comparators
Identify types of
outcomes reported
Identify long term
evidence & mechanisms
Describe effectiveness of
interventions (to help
choose which to model
& for parameterisation)
Model boundary Model detail Model type
Discuss potential model perspectives,
outcomes, interventions &
populations with stakeholders
Review existing health
economic models
Review effectiveness of
relevant interventions
Review evidence of
relationships between factors
Does the factor have many causal links?
Yes No
Is the factor likely to have a substantial
impact upon the difference between costs &
effects of the interventions? This may be
based upon (though not limited to):
(1) the review of economic evaluations;
(2) the description of resource pathways;
(3) clinical papers describing the causal links;
(4) existing models in similar areas which
describe the impact of the factor;
(5) methodological choices eg. discounting;
(6) expert advice.
Yes No
INCLUDE
Is the factor associated with the interventions,
populations & outcomes being modelled?
EXCLUDE
INCLUDE
EXCLUDE
Yes No
Yes
Is the impact of the factor predominantly
captured by other included factors?
Yes
EXCLUDE
No
Would stakeholders prefer to
include the factor for model
credibility AND is it relatively easy
to incorporate in terms of
modelling skill & data availability?
INCLUDE
No
Will the model only assess one intervention AND
is the intervention likely to be cost-effective AND
does the factor only further increase benefits
Yes
No
Determining
the model
boundary
Determining level of detail
Key types of model assumptions/ simplifications:
1. The relationship between the included factors over time;
2. The extrapolation of study outcomes;
3. The level of detail used to describe each included factor;
4. How interventions will be implemented in practice.
Questions to help the modeller for each of these are
proposed within the framework.
Choosing model type (1)
A B C D
Cohort/ aggregate level/ counts Individual level
Expected value, continuous state, deterministic
Markovian, discrete state, stochastic
Markovian, discrete state
Non-Markovian, discrete state
1 No interaction
Untimed Decision tree rollback
Simulation decision tree
Individual sampling model: Simulated patient-level decision tree
2 Timed Markov model (deterministic)
Simulation Markov model
Individual sampling model: Simulated patient-level Markov model
3 Interaction between entity and environment
Discrete time
System dynamics (finite difference equations)
Discrete time Markov chain model
Discrete-time individual event history model
Discrete individual simulation
4 Continuous time
System dynamics (ordinary differential equations)
Continuous time Markov chain model
Continuous time individual event history model
Discrete event simulation
5 Strong interactions between entities. Spatial aspects important.
X X X Agent-based simulation
Revised version of taxonomy by Brennan A, Chick SE, Davies R. A taxonomy of model structures
for economic evaluation of health technologies. Health Econ 2006; 15(12):1295-1310.
Choosing
model type
(2)
Determine the most appropriate model type for the characteristics of the problem.
Is this feasible within the time and resource constraints of the decision making process given:
(i) the data available?
AND
(ii) the accessibility of any existing relevant good quality economic evaluations for use as
a starting point?
AND
(iii) the expertise of the modeller?
Are you intending to use the
model again for other
projects?
Can you answer the decision makers’ question
with a few provisos and uncertainties with a
simpler model type?
Yes No
Explore with the
decision maker the
most useful purpose of
the modelling given the
project constraints
Develop the simpler model
type, documenting the
provisos, uncertainties &
implications of the
simplifications
No Yes
Do you think a
simpler model type
would lead to the
same conclusions?
Develop
the
model
Yes No
Develop the more
complex model
Develop the simpler model,
documenting the provisos,
uncertainties & implications
of the simplifications
Yes No
Conclusions
• Very little research has been undertaken
around conceptual modelling in health
economic evaluation.
• We have developed a conceptual modelling
framework for use within Public Health
economic models (paper under review).
• This has yet to be tested on a range of case
studies.
• If anybody wants to know more about the
framework or potentially use it, let me know!
Email: [email protected]
PhD thesis: http://etheses.whiterose.ac.uk/5316/