Day 2: Session IIDay 2: Session II

Presenter: Jeffrey Geppert, BattellePresenter: Jeffrey Geppert, BattelleAHRQ QI User MeetingAHRQ QI User MeetingSeptember 26-27, 2005September 26-27, 2005

Methods for Creating Methods for Creating Aggregate Performance Aggregate Performance

Indices Indices

AHRQ QI User MeetingAHRQ QI User Meeting

September 27, 2005September 27, 2005

Jeffrey GeppertJeffrey Geppert

Battelle Health and Life SciencesBattelle Health and Life Sciences

OverviewOverview

Project objectivesProject objectives Why composite measures?Why composite measures? Who might use composite measures?Who might use composite measures? Alternative approachesAlternative approaches Desirable features of a compositeDesirable features of a composite Proposed approach for the AHRQ QIProposed approach for the AHRQ QI Questions & AnswersQuestions & Answers

Project ObjectivesProject Objectives

Composite measures for the AHRQ QI Composite measures for the AHRQ QI included in the National Healthcare included in the National Healthcare Quality Report and Disparities ReportQuality Report and Disparities Report

Separate composites for overall quality Separate composites for overall quality and/or quality within certain domains and/or quality within certain domains (e.g., cardiac care, surgery, avoidable (e.g., cardiac care, surgery, avoidable hospitalizations, diabetes, adverse hospitalizations, diabetes, adverse events)events)

A methodology that can be used at the A methodology that can be used at the national, state and provider/area levelnational, state and provider/area level

Project ObjectivesProject Objectives

FeedbackFeedback Does the proposed approach meet Does the proposed approach meet

user needs for a composite?user needs for a composite? What analytic uses should the What analytic uses should the

composite address?composite address? What are the important policy What are the important policy

issues?issues? How should the composite be How should the composite be

incorporated into the AHRQ QI incorporated into the AHRQ QI software?software?

Goals of Goals of National Healthcare ReportsNational Healthcare Reports

National LevelNational Level– Provide assessment of quality and Provide assessment of quality and

disparitiesdisparities– Provide baselines to track progressProvide baselines to track progress– Identify information gapsIdentify information gaps– Emphasize interdependence of quality and Emphasize interdependence of quality and

disparitiesdisparities– Promote awareness and changePromote awareness and change

State / Local / Provider LevelState / Local / Provider Level– Provide tools for self-assessmentProvide tools for self-assessment– Provide national benchmarksProvide national benchmarks– Promote awareness and changePromote awareness and change

Unique challenges to quality Unique challenges to quality reporting by statesreporting by states

States release comparative quality information in a political States release comparative quality information in a political environmentenvironment– Either must adopt defensible scientific methodology or make conservative Either must adopt defensible scientific methodology or make conservative

assumptionsassumptions

Examples of reporting decisions:Examples of reporting decisions:– Small numbers issuesSmall numbers issues– Interpretive issues (better/worse, higher/lower)Interpretive issues (better/worse, higher/lower)

Purchasers demanding outcomes and cost information from Purchasers demanding outcomes and cost information from statesstates

Why Composites?Why Composites?

Summarize quality across multiple Summarize quality across multiple measuresmeasures

Improve ability to detect quality Improve ability to detect quality differencesdifferences

Identify important domains and drivers of Identify important domains and drivers of qualityquality

Prioritize actionPrioritize action Make current decisions about future Make current decisions about future

(unknown) healthcare needs(unknown) healthcare needs Avoids cognitive “short-cuts”Avoids cognitive “short-cuts”

Why Not Composites?Why Not Composites?

Mask important differences and Mask important differences and relationships among components (e.g. relationships among components (e.g. mortality and re-admissions)mortality and re-admissions)

Not “actionable”Not “actionable” Difficult to identify which parts of the Difficult to identify which parts of the

healthcare system contribute most to healthcare system contribute most to qualityquality

Detract from the impact and credibility of Detract from the impact and credibility of reportsreports

Independence of componentsIndependence of components Interpretation of componentsInterpretation of components

Who Might Use Them?Who Might Use Them?

Consumers – To select a hospital either Consumers – To select a hospital either before or after a health eventbefore or after a health event

Providers – To identify the domains and Providers – To identify the domains and drivers of qualitydrivers of quality

Purchasers – To select hospitals in Purchasers – To select hospitals in order to improve the health of order to improve the health of employeesemployees

Policymakers – To set policy Policymakers – To set policy in order to in order to improveimprove the health of a populationthe health of a population

ExamplesExamples

““America’s Best Hospitals” (U.S. News & World America’s Best Hospitals” (U.S. News & World Report)Report)

Leapfrog Safe Practices Score (Leapfrog Safe Practices Score (27 procedures to reduce 27 procedures to reduce

preventable medical mistakespreventable medical mistakes)) NCQA, “America’s Best Health Plans”NCQA, “America’s Best Health Plans” QA Tools (RAND)QA Tools (RAND) Veteran Health Administration Veteran Health Administration (Chronic Disease Care (Chronic Disease Care

Index, Prevention Index, Palliative Care Index)Index, Prevention Index, Palliative Care Index) Joint Commission Joint Commission (heart attack, heart failure, pneumonia, (heart attack, heart failure, pneumonia,

pregnancy)pregnancy)

National Health Service (UK) Performance RatingsNational Health Service (UK) Performance Ratings CMS Hospital Quality Incentive Demonstration ProjectCMS Hospital Quality Incentive Demonstration Project

ExamplesExamples

Measures

Com

ponents

Com

posite

Goals Utility

Alternative ApproachesAlternative Approaches

ApproachApproach GoalGoal UtilityUtility

OpportunityOpportunity Appropriate careAppropriate care Volume of Volume of opportunitiesopportunities

BurdenBurden Minimize excess Minimize excess death/costsdeath/costs

Measures with most Measures with most excessexcess

Expected qualityExpected quality Better than referenceBetter than reference Lowest ratioLowest ratio

VariationVariation Better than referenceBetter than reference OutliersOutliers

Latent qualityLatent quality Reduce variationReduce variation Measures with Measures with greatest variationgreatest variation

Desirable FeaturesDesirable Features

Valid - Based on valid measuresValid - Based on valid measures Reliable – Improve ability to detect differencesReliable – Improve ability to detect differences Minimum Bias – Based on unbiased measuresMinimum Bias – Based on unbiased measures Actionable – Interpretable metricActionable – Interpretable metric Benchmarks or standardsBenchmarks or standards TransparentTransparent Predictive – Should guide the decision-maker Predictive – Should guide the decision-maker

on likely future quality based on current on likely future quality based on current information.information.

Representative – Should reflect expected Representative – Should reflect expected outcomes for populationoutcomes for population

Proposed ApproachProposed Approach

A modeling-based approachA modeling-based approach Latent quality – observed correlation in Latent quality – observed correlation in

individual measures is induced by variability individual measures is induced by variability in latent qualityin latent quality

Individual measures with highest degree of Individual measures with highest degree of variation have larger contribution to variation have larger contribution to compositecomposite

Theoretical interpretationTheoretical interpretation Consistent with goal of reducing overall Consistent with goal of reducing overall

variation in qualityvariation in quality

Proposed ApproachProposed Approach

Latent Quality

Com

ponent

Com

ponent

Com

ponent

Com

ponent

AdvantagesAdvantages

Avoids contradictory results with Avoids contradictory results with individual measures or the creation of individual measures or the creation of composites that may misleadcomposites that may mislead

Construction of the composite increases Construction of the composite increases the power of quality appraisalsthe power of quality appraisals

Allows for both measure-specific Allows for both measure-specific estimates and compositesestimates and composites

Allows for validation with out-of-sample Allows for validation with out-of-sample predictionprediction

Advantages (Continued)Advantages (Continued)

Hierarchical – for small numbers, the Hierarchical – for small numbers, the best estimate is the pooled average rate best estimate is the pooled average rate at similar hospitalsat similar hospitals

Allows for incorporation of provider Allows for incorporation of provider characteristics to explain between-characteristics to explain between-provider variability (e.g., volume, provider variability (e.g., volume, technology, teaching status)technology, teaching status)

Gives policymakers information on the Gives policymakers information on the important drivers of qualityimportant drivers of quality

Overview of AHRQ QIsOverview of AHRQ QIs

Prevention Prevention Quality Quality IndicatorsIndicators

Inpatient Quality Inpatient Quality Indicators Indicators

Patient Safety Patient Safety IndicatorsIndicators

Ambulatory care sensitive Ambulatory care sensitive conditionsconditions

Mortality following procedures Mortality following procedures Mortality for medical conditionsMortality for medical conditions Utilization of proceduresUtilization of procedures Volume of proceduresVolume of procedures

Post-operative complicationsPost-operative complications Iatrogenic conditionsIatrogenic conditions

ExamplesExamples

IQI Surgical Mortality

0.00

0.02

0.04

0.06

0.08

0.10

0.12

0.14

High Medium Low

Latent Quality

PANCREATIC RESECTION

AAA REPAIR

CABG

CRANIOTOMY

ExamplesExamples

IQI Medical Mortality

0.00

0.02

0.04

0.06

0.08

0.10

0.12

0.14

High Medium Low

Latent Quality

HIP REPLACEMENT

AMI

CHF

STROKE

GI HEMORRHAGE

HIP FRACTURE

PNEUMONIA

ExamplesExamples

Prevention Quality Indicators

0.000

0.002

0.004

0.006

0.008

0.010

0.012

0.014

0.016

High Medium Low

Latent Quality

DIABETES SHORT TRM COMPLICATN

DIABETES LONG TERM COMPLICATN

PEDIATRIC ASTHMA

COPD

PEDIATRIC GASTROENTERITIS

HYPERTENSION

CONGESTIVE HEART FAILURE

DEHYDRATION

BACTERIAL PNEUMONIA

URINARY INFECTION

ANGINA

DIABETES UNCONTROLLED

ADULT ASTHMA

LOWER EXTREMITY AMPUTATION

ExamplesExamples

PSI Postoperative Complications

0.000

0.002

0.004

0.006

0.008

0.010

0.012

0.014

0.016

0.018

0.020

High Medium Low

Latent Quality

POSTOP PHYSIO METABOLDERANGEMENT

POSTOP RESPIRATORY FAILURE

POSTOPERATIVE PE OR DVT

POSTOPERATIVE SEPSIS

ExamplesExamples

PSI Technical Adverse Events

0.000

0.020

0.040

0.060

0.080

0.100

0.120

0.140

0.160

High Medium Low

Latent Quality

DECUBITUS ULCER

FAILURE TO RESCUE

INFECTION DUE TO MEDICAL CARE

ExamplesExamples

PSI Technical Difficulty

0.000

0.001

0.001

0.002

0.002

0.003

0.003

0.004

0.004

0.005

High Medium Low

Latent Quality

IATROGENIC PNEUMOTHORAX

POSTOP HEMORRHAGE OR HEMATOMA

ACCIDENTAL PUNCTURE/LACERATION

Hierarchical ModelsHierarchical Models

Also referred to as smoothed rates or reliability-adjusted ratesAlso referred to as smoothed rates or reliability-adjusted rates Endorsed by NQF for outcome measuresEndorsed by NQF for outcome measures Methods to separate the within and between provider level Methods to separate the within and between provider level

variation (random vs. systematic)variation (random vs. systematic) Total variation = Within provider + Between provider Total variation = Within provider + Between provider (Between = (Between =

Total – Within)Total – Within)

Reliability (w) = Between / TotalReliability (w) = Between / Total– Signal ratio = signal / (signal+noise)Signal ratio = signal / (signal+noise)

Hierarchical ModelsHierarchical Models

Smoothed rate is the (theoretical) best predictor of Smoothed rate is the (theoretical) best predictor of future qualityfuture quality

Provides a framework for validation and forecastingProvides a framework for validation and forecasting Smoothed rate (single provider, single indicator) = Smoothed rate (single provider, single indicator) =

Hospital-type rate * (1 – w) +Hospital-type rate * (1 – w) +Hospital-specific rate * wHospital-specific rate * w

Multivariate versionsMultivariate versions– Other Years (auto-regression, forecasting)Other Years (auto-regression, forecasting)– Other Measures (composites)Other Measures (composites)– Non-persistent innovations (contemporaneous, Non-persistent innovations (contemporaneous,

nonsystematic shocks)nonsystematic shocks)

Outcomes and ProcessOutcomes and Process

Source: Landrum et. al. Analytic Methods for Constructing Cross-Sectional Profiles of Health Care Providers (2000)

Hierarchical ModelsHierarchical Models

Provider T

ype

Provider

PatientPolicy Quality

Policy and PredictionPolicy and Prediction

The best predictor of future performance is often The best predictor of future performance is often historical performance + structurehistorical performance + structure

The greater the reliability of the measure for a The greater the reliability of the measure for a particular provider, the more weight on historical particular provider, the more weight on historical performanceperformance

The less the reliability of the measure for a particular The less the reliability of the measure for a particular provider, the more weight on structureprovider, the more weight on structure

Volume often improves the ability to predict Volume often improves the ability to predict performance for low-volume providersperformance for low-volume providers

Other provider characteristics (e.g. availability of Other provider characteristics (e.g. availability of technology) do as welltechnology) do as well

Area characteristics (e.g., SES) do as wellArea characteristics (e.g., SES) do as well

Socio-Economic StatusSocio-Economic Status

The Public Health Disparities Geo-coding Project - The Public Health Disparities Geo-coding Project - Harvard School of Public Health (PI: Nancy Krieger)Harvard School of Public Health (PI: Nancy Krieger)

Evaluated alternative indices of SES (e.g. Townsend Evaluated alternative indices of SES (e.g. Townsend and Carstairs)and Carstairs)

Occupational class, income, poverty, wealth, education Occupational class, income, poverty, wealth, education level, crowdinglevel, crowding

Gradations in mortality, disease incidence, LBW, Gradations in mortality, disease incidence, LBW, injuries, TB, STDinjuries, TB, STD

Percent of persons living below the U.S. poverty linePercent of persons living below the U.S. poverty line– Most attuned to capturing economic depravationMost attuned to capturing economic depravation– Meaningful across regions and over timeMeaningful across regions and over time– Easily understood and readily interpretableEasily understood and readily interpretable

Socio-Economic StatusSocio-Economic Status

PQI #1 Diabetes Short-term Complication

0.0000

0.0002

0.0004

0.0006

0.0008

0.0010

0.0012

0.0014

0.0016

1 2 3 4 5 6 7 8 9 10

% Below U.S. Poverty Decile

LimitationsLimitations

Measures and methods difficultMeasures and methods difficult Restrictive assumptions on correlationRestrictive assumptions on correlation Correlations may vary by provider typeCorrelations may vary by provider type Requires a large, centralized data Requires a large, centralized data

sourcesource

ExpansionsExpansions

Flexibility in weighting the componentsFlexibility in weighting the components Empirical – domains driven entirely by Empirical – domains driven entirely by

empirical relationships in the dataempirical relationships in the data A priori – domains determined by clinical A priori – domains determined by clinical

or other considerationsor other considerations Combination – empirical when the Combination – empirical when the

relationships are strong and the relationships are strong and the measures precise, otherwise a priorimeasures precise, otherwise a priori

Welfare-driven CompositesWelfare-driven Composites

Measures

Com

posites

Meta-com

posites

Welfare-Driven CompositesWelfare-Driven Composites

Making current decisions about future Making current decisions about future needs – maximize expected outcomes, needs – maximize expected outcomes, minimize expected costsminimize expected costs

Policymaker focus – for a populationPolicymaker focus – for a population A provider focus A provider focus –– for their patientsfor their patients A employer focus – for their employeesA employer focus – for their employees A consumer focus – based on individual A consumer focus – based on individual

characteristicscharacteristics

AcknowledgmentsAcknowledgments

Funded by AHRQFunded by AHRQ Support for Quality Indicators II (Contract No. 290-04-0020) Mamatha Pancholi, AHRQ Project Officer Marybeth Farquhar, AHRQ QI Senior Advisor Mark Gritz and Jeffrey Geppert, Project Directors, Battelle Health

and Life Sciences

Data used for analyses:Data used for analyses:Nationwide Inpatient Sample (NIS), 1995-2000. Healthcare Cost and Nationwide Inpatient Sample (NIS), 1995-2000. Healthcare Cost and

Utilization Project (HCUP), Agency for Healthcare Research and Utilization Project (HCUP), Agency for Healthcare Research and QualityQuality

State Inpatient Databases (SID), 1997-2002 (36 states). Healthcare State Inpatient Databases (SID), 1997-2002 (36 states). Healthcare Cost and Utilization Project (HCUP), Agency for Healthcare Cost and Utilization Project (HCUP), Agency for Healthcare Research and QualityResearch and Quality

AcknowledgementsAcknowledgements

We gratefully acknowledge the data organizations in participating states that contributed data to We gratefully acknowledge the data organizations in participating states that contributed data to HCUP and that we used in this study: the Arizona Department of Health Services; California Office HCUP and that we used in this study: the Arizona Department of Health Services; California Office of Statewide Health Planning & Development; Colorado Health & Hospital Association; Connecticut of Statewide Health Planning & Development; Colorado Health & Hospital Association; Connecticut - Chime, Inc.; Florida Agency for Health Care Administration; Georgia: An Association of Hospitals & - Chime, Inc.; Florida Agency for Health Care Administration; Georgia: An Association of Hospitals & Health Systems; Hawaii Health Information Corporation; Illinois Health Care Cost Containment Health Systems; Hawaii Health Information Corporation; Illinois Health Care Cost Containment Council; Iowa Hospital Association; Kansas Hospital Association; Kentucky Department for Public Council; Iowa Hospital Association; Kansas Hospital Association; Kentucky Department for Public Health; Maine Health Data Organization; Maryland Health Services Cost Review; Massachusetts Health; Maine Health Data Organization; Maryland Health Services Cost Review; Massachusetts Division of Health Care Finance and Policy; Michigan Health & Hospital Association; Minnesota Division of Health Care Finance and Policy; Michigan Health & Hospital Association; Minnesota Hospital Association; Missouri Hospital Industry Data Institute; Nebraska Hospital Association; Hospital Association; Missouri Hospital Industry Data Institute; Nebraska Hospital Association; Nevada Department of Human Resources; New Jersey Department of Health & Senior Services; Nevada Department of Human Resources; New Jersey Department of Health & Senior Services; New York State Department of Health; North Carolina Department of Health and Human Services; New York State Department of Health; North Carolina Department of Health and Human Services; Ohio Hospital Association; Oregon Association of Hospitals & Health Systems; Pennsylvania Health Ohio Hospital Association; Oregon Association of Hospitals & Health Systems; Pennsylvania Health Care Cost Containment Council; Rhode Island Department of Health; South Carolina State Budget Care Cost Containment Council; Rhode Island Department of Health; South Carolina State Budget & Control Board; South Dakota Association of Healthcare Organizations; Tennessee Hospital & Control Board; South Dakota Association of Healthcare Organizations; Tennessee Hospital Association; Texas Health Care Information Council; Utah Department of Health; Vermont Association; Texas Health Care Information Council; Utah Department of Health; Vermont Association of Hospitals and Health Systems; Virginia Health Information; Washington State Association of Hospitals and Health Systems; Virginia Health Information; Washington State Department of Health; West Virginia Health Care Authority; Wisconsin Department of Health & Department of Health; West Virginia Health Care Authority; Wisconsin Department of Health & Family Services. Family Services.

Questions & AnswersQuestions & Answers

Questions And AnswersQuestions And Answers