Little pain, big gain: Data gaps and the development of cost-benefit analysis tools Jacqueline Mallender & Rory Tierney Matrix Knowledge

Little pain, big gain: Data gaps and the development of cost-benefit analysis toolsJacqueline Mallender & Rory TierneyMatrix Knowledge

Agenda

Section Page

Introduction/motivation 3

Reviews & the issue of data gaps 4

Decision modelling & dealing with data gaps 5

The creation of user-accessible cost-benefit tools

Conclusions

Little pain, big gain: Data gaps and the development of cost-benefit analysis tools


Introduction


Matrix:

• Advisory & software services

• Work across health and social care, justice, education & life sciences

• Public & private clients

• In the US, UK and continental Europe

Rory Tierney:

• Senior Economist

• Focus on life sciences & public health and justice

• Design & build models

Jacque Mallender:

• Health and welfare economist

• Founder, Director of Matrix

• Convenor of Campbell Cochrane Economics Methods Group

• Economist expert on Campbell Crime and Justice Steering Committee

Who are we?


When will the cost savings

come through (ROI)?

What of current

programs delivers most

value?

Which segment of

population will benefit most

(inequalities)?

Where will I get better outcomes

and the most value for my investment

How do I prioritise where

to invest (ranking)?

Where in the pathway should I

invest in and how much?

What are the problems of

interest in my local

population?

The questions: implicit or explicit


Reviews & the issue of data gaps


What works?

• A policymaker wants to know:

• What are the most effective programs I can spend my money on?

• How much will it cost?

• What is the return to government and society?

• What should I stop spending on?

• What kind of impact should I expect, and for whom?

• e.g. how much of a reduction in morbidity and mortality, in which populations

• e.g. how much of a reduction in recidivism, in which offenders


Systematic reviews

• Viewed as a first time approach to generate evidence base

• Should review evaluations of intervention ‘trials’ that:

• Select, critique and synthesise reliable evidence

• Draw from published and unpublished studies

• Use explicit, systematic and replicable methods

• For relevant interventions, applicable to the policy & societal context

• Quality & quantity important


The data gap

• Not enough robust published evidence

• Large variations in methodology

• General weaknesses of systematic review (Anderson & Shemilt, 2010):

• International variation in methods is likely and justifiable

• Context matters: resources required & unit costs change across geographies, time, and availability of infrastructure and services

• Same is true of benefits & their associated values

• Cannot always rely on a review of existing evidence

Anderson, R., & Shemilt, I. (2010). Chapter 3: The role of economic perspectives and evidence in

systematic review. In: Shemilt, I., Mugford, M., Vale, L., Marsh, K., Donaldson, C. (eds.) Evidence-based

decisions and economics: health care, social welfare, education and criminal justice. Oxford: Wiley-

Blackwell


Decision modelling & dealing with data gaps


Decision modelling

The usage of decision modelling is growing:

• To compare interventions in context

• To model ex-ante policy options & assess critical assumptions and required effectiveness

• To evaluate ex-post policy options and assess what would need to change to improve effectiveness

• Incremental inputs, outputs and outcomes

• Relationship between inputs, outputs and outcomes

• Value of inputs, process and outcomes• Multiple data sources


NICE: Public health

• National Institute of Health and Care Excellence

• Issues guidance on interventions & methods to improve public health (in the UK)

• Publishes on such topics as smoking cessation, substance misuse, sexually transmitted infections, nutrition etc.

• Approximately half of NICE public health interventions were evaluated using a decision model (as of 2012)

33%

NICE public health interventions

StaticDynamicNot clearNot a modelNot available

18%18%

30%

20%

13%

20%

40 Public Health

interventions

Static modelling

Decision Tree (100%)

Not a model (e.g. review)

Methodology not clear

Dynamic modelling

Markov model (75%)

Discrete Event Simulation

(8.3%)

Regression analysis (8.3%)

Other mathematical model (8.3%)


Washington State Institute for Public Policy: Criminal justice

• Produces best known example of review + economic modelling in criminal justice

• One key output is a yearly report containing a list of evidence-based interventions with independent assessment of costs and benefits of each

• Perspective of Washington citizens and taxpayers

• Models built on rigorous review of primary studies complemented with clear and published meta analytic strategy

• Produces internally consistent estimates, enabling comparison of interventions and programmes

• Detailed methodologies published: can see continuous improvement in methods over time

• Enables assessment of costs and benefits in the long-run and associated risks and uncertainties

Four-step research approach.

1. systematically assess evidence on “what works” (and what does not) to improve outcomes.

2. calculate costs and benefits for Washington State and produce a Consumer Reports-like ranking of public policy options

3. measure the riskiness of conclusions by testing how bottom lines vary when estimates and assumptions change.

4. provide a “portfolio” analysis of how a combination of policy options could affect state-wide outcomes of interest.

Has contributed to greater improvement in crime measures in the state than national average and $1.3 billion in savings over 2

year budget cycle (Urahn, 2012)

Urahn, S.K. (2012). The Cost-Benefit Imperative. Governing.com


Overcoming the data gap

• Combining jurisdiction-specific economic decision models with systematic review can help overcome the data gap

• Data & information from systematic review can be used to populate the model

• The review can be ‘expanded’ by sourcing more data & using decision-analytic techniques to combine different sources

• Expert opinion can be used to supplement findings & assess transferability of systems

• Sensitivity analysis assesses the impact of uncertain variables & assumptions

• Decision models:

• Provide a structured way of demonstrating consequences

• Involves the construction of a mathematical representation of the relationships between inputs and results

• Provide an explicit way to synthesise disparate evidence available on the outcomes and costs of alternative (mutually exclusive) interventions

• Is a socio-technical process combining quantitative evidence and deliberative techniques


Expert opinion

• There is great value in engaging experts and stakeholders in the design & development of models and identification of data

• It both involves decision makers/influencers in the analysis being done & supplements existing data (helps ‘mind the data gap’)

• Can be done through the use of surveys, interviews, workshops etc.

• Stakeholders can provide ‘inside’ knowledge not available in published studies

• By engaging them in model design, stakeholders will trust a model to make decisions

• Transparency is vital


The creation of user-accessible cost-benefit tools


Health: Alcohol misuse & physical activity

• Two VBA (Excel) based tools, developed by Matrix will be available online through NICE

• Created to measure the social return on investment of interventions for reducing alcohol use and promoting physical activity

• Allows a user to see the impact in a local area of a package of policies – can compare two scenarios at once

• Users can customise population size,

interventions, uptake and effectiveness.

• Models developed by synthesising existing studies with stakeholder input

• Mathematical/statistical techniques used to allow results to be aggregated in terms of effect sizes and subsequent health outcomes


Justice: Reducing reoffending in young adults

• Online tool developed by Matrix to allow organisations to assess the economic impact of an intervention to reduce reoffending

• Underpinned by economic model with data from a variety of different sources such as longitudinal crime studies, UK cost of crime data and pilot interventions

• Developed with stakeholder input

• Provides a ‘business case’ – allows users to set up an intervention and access a report on its outcomes and benefit:cost


Conclusions


Little pain: Data gaps can be overcome

• Systematic reviews can provide excellent data on effectiveness, benefit-cost and comparison of interventions

• However, data is often limited and may be of low quality

• Combining systematic reviews with economic decision modelling can help overcome the data gap

• The review can be ‘expanded’ by sourcing more data & using decision-analytic techniques to combine different sources

• Expert opinion can also be sourced when published data is unavailable

• Must be clear and transparent

• Can add value to a review by being locally tailored

• Provides a mathematical, structural view of the consequences of interventions or programs of interventions


Big gain: Benefit-cost can inform policymakers

• Two examples of decision modelling being successfully used to inform policymakers and practitioners:

• NICE public health guidance (UK, health)

• Washington State Institute for Public Policy (US, criminal justice)

• Shows that decision models can help policymakers understand the choices they make when investing in one program over another

• Provides an evidence base and gets policymakers thinking about benefit-cost and return on investment with different combinations of programs and interventions and see the effects

• Taking this one step further are tools which allow the user to specify inputs and see the impact of an intervention or package of interventions:

• Alcohol and physical activity return on investment tools (UK, health)

• Business case application for interventions to reduce reoffending (UK, criminal justice)

• Tools can be created based on existing reviews & analysis, stakeholder engagement and expert opinion, and provide a way for users to ‘play around’


THANKS!

Rory Tierney

Senior Economist

[email protected]

Jacqueline Mallender

Founder, Director

[email protected]

Matrix Knowledge

Kemp House

152-160 City Road

London EC1V 2NP

+44 (0)20 7553 4800

mailto:[email protected]

mailto:[email protected]


The information, materials and opinions in this presentation (the "Information") are intended to describe the activities of Matrix knowledge only. It should not be used for any other purpose or in any other context and Matrix accepts no responsibility for its use in either regard. In keeping with our values of integrity and excellence, Matrix has taken reasonable professional care in the preparation of this document. Although Matrix has made reasonable efforts, we cannot guarantee absolute accuracy or completeness of information, data and any other sources used in producing this document. This document has been produced on the basis of the information and explanations made available to Matrix at the time this document was prepared. Accordingly, no representation or warranty, express or implied, is given and no responsibility or liability is accepted by or on behalf of Matrix or by any of its employees or agents or any other person as to the accuracy, completeness or correctness of the information contained in this document or any oral information made available and any such liability is expressly disclaimed. To the fullest extent possibly, Matrix disclaims any liability arising out of the use or non-use of this document and its contents, including any action or decision taken as a result of such use or non-use. Except where permitted under the provisions of confidentiality above, this document may not be made available, reproduced, retained or stored beyond the period of validity, or transmitted in whole or in part to any person without Matrix’s prior written permission. All copyright and other proprietary rights in this document remain the property of Matrix (unless otherwise provided in this document) and all rights are reserved. If you require any clarification or further information about this document, please contact Matrix on [email protected].

Documents

Little pain, big gain: Data gaps and the development of cost-benefit analysis tools Jacqueline Mallender & Rory Tierney Matrix Knowledge