BIO656--Multilevel Models 1Term 4, 2006
PART 07PART 07
Evaluating Hospital PerformanceEvaluating Hospital Performance
BIO656--Multilevel Models 2Term 4, 2006
PERFORMANCE MEASURESPERFORMANCE MEASURES
Patient outcomes • Mortality, morbidity, satisfaction with care
– 30-day mortality among heart attack patients (Normand et al JAMA 1996, JASA 1997)
Process • Medication & test administration, costs
– Laboratory costs for diabetic patients– Number of physician visits
• Hofer et al JAMA, 1999– Palmer et al. (1996)
BIO656--Multilevel Models 3Term 4, 2006
DATA STRUCTUREDATA STRUCTURE
Multi-level• Patients nested in physicians, hospitals, HMOs, ...
• Providers clustered by health care systems, market areas, geographic areas
• Covariates at different levels of aggregation: – patient, physician, hospital, ...
Variation in variability• Statistical stability varies over physicians, hospitals, ..
BIO656--Multilevel Models 4Term 4, 2006
MLMs are EffectiveMLMs are Effective
Correlation at many levels• Hospital practices may induce a strong correlation
among patient outcomes within hospitals even after accounting for patient characteristics
Structuring estimation• Stabilizing noisy estimates • Balancing SEs• Estimating ranks and other non-standard summaries
BIO656--Multilevel Models 5Term 4, 2006
The Cooperative The Cooperative Cardiovascular Project (CCP)Cardiovascular Project (CCP)
• Abstracted medical records for patients discharged from hospitals located in Alabama, Connecticut, Iowa, and Wisconsin (June 1992May 1993)
• 3,269 patients hospitalized in 122 hospitals in four US States for Acute Myocardial Infarction
BIO656--Multilevel Models 6Term 4, 2006
GOALSGOALS
• Identify “aberrant” hospitals with respect to several performance measures
• Report the statistical uncertainty associated with ranking of the “worst hospitals”
• Investigate if hospital characteristics explain variation in hospital-specific mortality rates
BIO656--Multilevel Models 7Term 4, 2006
DATADATA
Outcome• Mortality within 30-days of hospital admission
Patient characteristics• Admission severity index constructed on the basis of
34 patient attributes
Hospital characteristics• Urban/Rural• (Non academic)/(versus academic)• Number of beds
BIO656--Multilevel Models 8Term 4, 2006
Why adjust for case mix?Why adjust for case mix?(patient characteristics)
• Irrespective of quality of care, older/sicker patients with multiple diseases have increased need of health care services and poorer health outcomes
• Without adjustment, physicians/hospitals who treat relatively more of these patients will appear to provide more expensive and lower quality care than those who see relatively younger/healthier patients
If there is inadequate case mix adjustment, evaluations will be unfair• But, need to avoid over adjusting
BIO656--Multilevel Models 9Term 4, 2006
Case-mix AdjustmentCase-mix Adjustment
Compute hospital-specific, expected mortality by: 1. estimating a patient-level mortality model using all hospitals 2. averaging the model-produced probabilities for all
patients within a hospital
• Hospitals with “higher-than-expected” mortality rates can be flagged as institutions with potential quality problems, but need to account for uncertainty
• Need to be careful, if also adjusting for hospital characteristics– May adjust away the important signal
BIO656--Multilevel Models 10Term 4, 2006
(as we know, very poor approach)
• Wrong SEs• Test-based
BIO656--Multilevel Models 11Term 4, 2006
Hospital Profiling of Mortality Rates Hospital Profiling of Mortality Rates Acute Myocardial Infarction Patients Acute Myocardial Infarction Patients
(Normand et al. JAMA 1996, JASA 1997)
BIO656--Multilevel Models 12Term 4, 2006
Hierarchical logistic regressionHierarchical logistic regression
I: Patient within-provider• Patient-level logistic regression model with random
intercept & slope
II: Between-provider• Hospital-specific random effects are regressed on
hospital-specific characteristics– Explicit regression
BIO656--Multilevel Models 13Term 4, 2006
Admission severity indexAdmission severity index(Normand et al. 1997 JASA)
BIO656--Multilevel Models 14Term 4, 2006
0 + 1(sevij – sevbar)
0
1
sevbar
BIO656--Multilevel Models 15Term 4, 2006
we use bwe use b0i0i + b + b1i1i(...)(...)
BIO656--Multilevel Models 16Term 4, 2006
Interpretation of parameters is different for the two levels
bb0i0i = = **0000 + N(..), etc. + N(..), etc.
BIO656--Multilevel Models 17Term 4, 2006
RESULTSRESULTS
• Estimates of regression coefficients under three models:– Random intercept only– Random intercept and random slope– Random intercept, random slope, and hospital
covariates• Hospital performance measures
BIO656--Multilevel Models 18Term 4, 2006
Normand et al. JASA 1997Normand et al. JASA 1997
BIO656--Multilevel Models 19Term 4, 2006
30-DAY MORTALITY30-DAY MORTALITY 2.5th and 97.5th percentiles for a
patient of average admission severity
Exchangeable model • Random intercept and slope, no hospital covariates
log(odds): (-1.87,-1.56) probability,scale: (0.13, 0.17)
Covariate (non-exchangeable) model • Random intercept and slope, with hospital covariates• Patient treated in a large, urban academic hospital
log(odds): (-2.15,-1.45)probability scale: (0.10,0.19)
BIO656--Multilevel Models 20Term 4, 2006
Effect of hospital characteristics Effect of hospital characteristics on baseline log-odds of 30-day mortalityon baseline log-odds of 30-day mortality
• For an average patient, rural hospitals have a higher odds ratio than urban hospitals
– Indicates between-hospital differences in the baseline mortality rates
– Case-mix adjustment may be able to remove some of this difference
BIO656--Multilevel Models 21Term 4, 2006
Estimates of Stage-II Estimates of Stage-II regression coefficientsregression coefficients
InterceptsIntercepts
BIO656--Multilevel Models 22Term 4, 2006
Effect of hospital characteristics on Effect of hospital characteristics on association between severity and mortalityassociation between severity and mortality
(slopes)
• The association between severity and mortality is modified by hospital size
• Medium-sized hospitals have smaller severity/mortality associations than large hospitals
– Indicates that the effect of clinical burden (patient severity) on mortality differs across hospitals
BIO656--Multilevel Models 23Term 4, 2006
Estimates of Stage IIEstimates of Stage IIregression coefficientsregression coefficients
SlopesSlopes
BIO656--Multilevel Models 24Term 4, 2006
Homework is on front tableHomework is on front table
BIO656--Multilevel Models 25Term 4, 2006
Observed and risk-adjusted hospital mortality ratesObserved and risk-adjusted hospital mortality ratesUrban Hospitals
Histogram displays (observed – adjusted)
BIO656--Multilevel Models 26Term 4, 2006
Observed and risk-adjusted hospital mortality ratesObserved and risk-adjusted hospital mortality ratesRural Hospitals
Histogram displays (observed – adjusted)
Substantial adjustment for severitySubstantial adjustment for severity
BIO656--Multilevel Models 27Term 4, 2006
FINDINGSFINDINGS
• There is substantial adjustment for admission severity
• Generally, urban hospitals are adjusted less than rural
• There is less variability in observed or adjusted estimated rates for urban hospitals than for rural hospitals
Can you explain why?
BIO656--Multilevel Models 28Term 4, 2006
Normand et al. JASA 1997Normand et al. JASA 1997
BIO656--Multilevel Models 29Term 4, 2006
Average the probabilitiesAverage the probabilitiesDon’t average the covariatesDon’t average the covariates
BIO656--Multilevel Models 30Term 4, 2006
k denotes a draw from the posterior
BIO656--Multilevel Models 31Term 4, 2006
Plug in the average covariatePlug in the average covariate
Keep the hospital variationKeep the hospital variation
BIO656--Multilevel Models 32Term 4, 2006
BIO656--Multilevel Models 33Term 4, 2006
Comparing measures Comparing measures of hospital performanceof hospital performance
Three measures of hospital performance• Probability of a large difference between adjusted
and standardized mortality rates
• Probability of excess mortality for the average patient
• Z-score
BIO656--Multilevel Models 34Term 4, 2006
Hospital Rankings: Normand et al 1997 JASA
BIO656--Multilevel Models 35Term 4, 2006
Hospital RanksHospital Ranks
• There was moderate disagreement among the criteria for classifying hospitals as “aberrant”
• Nevertheless, hospital 1 is ranked worst
• It is rural, medium sized non-academic with an observed mortality rate of 35%, and adjusted rate of 28%
BIO656--Multilevel Models 36Term 4, 2006
Adjusting for hospital-level charateristics Adjusting for hospital-level charateristics
Changes the comparison group in “as compared to what?”
• All hospitals (unadjusted at hospital level)
• Hospitals of a similar size, urbanicity, ...
• Percent of physicians who are board certified
• Hospitals with a similar death rate
Variance reduction and goodness of fit should not be the primary considerations
• “As compared to what?” must dominate
BIO656--Multilevel Models 37Term 4, 2006
DiscussionDiscussion
• Profiling medical providers is multi-faced and data intensive process with substantial implications for health care practice, management, and policy
• Major issues include data quality and availability, choice of performance measures, formulation of statistical models (including adjustments), reporting results
• The ranking approaches and summaries used by Normand and colleagues are very good, but some improvement is possible
BIO656--Multilevel Models 38Term 4, 2006
Multi-level models address key technical & Multi-level models address key technical & conceptual profiling issues, includingconceptual profiling issues, including
• Adjusting for patient severity
• Accounting for within-provider correlations
• Accounting for differential sample sizes at all levels
• Stabilize estimates
• Structure ranking and other, derived comparisons