Term 4, 2006BIO656--Multilevel Models 1 Midterm Open “book” and notes; closed mouth 20-25 minutes to read carefully and answer completely 60 minutes

BIO656--Multilevel Models 1Term 4, 2006

MidtermMidterm

• Open “book” and notes; closed mouth• 20-25 minutes to read carefully and answer

completely 60 minutes to think• 4 problems, with possible topics:

– Some debriefing on the role of MLMs– Something on linear MLMs variances, etc.– MLMs and shrinkage– Some basic computations and interpretations of

logit-linear models


PART 6PART 6

PROFILING, RANKING“League Tables”



RANKING IN THE NEWSRANKING IN THE NEWS


LETTERMAN’S LETTERMAN’S TOP 10 LISTTOP 10 LIST


NEW YORK’S MOST DEADLY NEW YORK’S MOST DEADLY CARDIAC SURGEONS!!!!CARDIAC SURGEONS!!!!



THE LEADING SPH IS THE LEADING SPH IS HARVARDHARVARD


HOPKINSHOPKINS IS THE IS THELEADING SPH!!!LEADING SPH!!!


PROFILINGPROFILING(League Tables)(League Tables)

• The process of comparing “units” on an outcome measure with relative or normative standards– Quality of care, use of services, cost– Educational quality– Disease rates in small areas– Gene expression

• Developing and implementing performance indices to compare physicians, hospitals, schools, teachers, genes, ........


PROFILING OBJECTIVESPROFILING OBJECTIVES(in health services)

• Estimate and compare provider-specific performance measures:– Utilization/cost– Process measures– Clinical outcomes– Patient satisfaction/QoL

• Compare using a normative (external) or a relative (internal) standard



RANKING IS EASYRANKING IS EASY

Just compute estimates & order themJust compute estimates & order them


MLE ESTIMATED SMRsMLE ESTIMATED SMRs

Provider

ML

E S

MR

0 20 40 60 80 100 120

02

46

81

0


RANKING IS DIFFICULTRANKING IS DIFFICULT

Need to trade-off the estimates and uncertainties


MLE ESTIMATED SMRs & 95% CIsMLE ESTIMATED SMRs & 95% CIs

Provider

ML

E S

MR

0 20 40 60 80 100 120

02

46

81

0


Sampling variability &Sampling variability &Systematic variabilitySystematic variability

Systematic variability• Variability among physicians/hospitals that might be

explained by hospital-specific characteristics

Sampling variability• Statistical uncertainty of physician/hospital-specific

performance measures

Use MLMs that• Incorporate patient, physician and hospital-level

characteristics• Capture all important uncertainties• Produce appropriate statistical summaries


Statistical ChallengesStatistical Challenges

Need a valid method of adjusting for case mix and other features• Patient, physician and hospital characteristics

– But, beware of over adjustment

Need a valid model for stochastic properties• Account for variation at all levels• Account for within-hospital, within-patient correlations

Need to • Adjust for systematic variation• Estimate and account for statistical variation


PROPER USE OF PROPER USE OF STATISTICAL SUMMARIESSTATISTICAL SUMMARIES

The challenge• Differences in standard errors of hospital-specific

estimates invalidate direct comparisons• In any case, large SEs make comparisons imprecise

Consequence • Even after valid case mix adjustment, differences in

directly estimated performance are due, in part, to sampling variability

(Partial) Solution, use:• Shrinkage estimates to balance and reduce variability• Goal-specific estimates to hit the right target


Comparing performance measuresComparing performance measures

Ranks/percentiles, of:• Direct estimates (MLEs)• Shrunken estimates (BLUPs, Posterior Means)• Z-scores testing H0 that a unit is just like others• Optimal (best) ranks or percentiles

Other measures• Probability of a large difference between unit-specific

“true” and H0-generated event rates• Probability of “excess mortality”

– For the “typical patient, on average or for a specific patient type

• Z-score/P-value declarations• ....



USRDSUSRDS


USRDSUSRDS






MLE ESTIMATED SMRs & CIs

Provider

ML

E S

MR

0 20 40 60 80 100 120

02

46

81

0


Poisson-Normal ModelPoisson-Normal Model(N, Y[k] , emort[k]) are inputs

model{{prec~dgamma(0.00001,0.00001)for (k in 1:N) {logsmr[k]~dnorm(00,prec)smr[k]<-exp(logsmr[k])rate[k]<-emort[k]*smr[k]Y[k] ~ dpois(rate[k])}} Monitor the SMR[k]


0.0 0.5 1.0 1.5 2.0 2.5

SE

MLE

PM

MLE, SE & POSTERIOR MEAN SMRsMLE, SE & POSTERIOR MEAN SMRs(using a log-normal/Poisson model)


Posterior Mean: estimated SMRs & CIsPosterior Mean: estimated SMRs & CIsusing a log-normal/Poisson model

(original scale)

Provider

PM

SM

R

0 20 40 60 80 100 120

02

46

81

0


Posterior Mean: estimated SMRs & CIsPosterior Mean: estimated SMRs & CIsusing a Gamma/Poisson model

(expanded scale)


• Estimated relative, physician-specific visit rate and 95% CI• Adjusted for patient demographic and case-mix (1.0 is the “typical” rate)

Caterpillar PlotCaterpillar Plot(Hofer et al. JAMA 1999)(Hofer et al. JAMA 1999)


• Amount that physician-specific, laboratory costs for diabetic patients deviates from the mean for all physicians [$/(pt. yr.)]• Lines show the path from the direct estimate (the MLE) to the shrunken estimate (Hofer et al JAMA 1999)

DIRECTDIRECT ADJUSTEDADJUSTED


Example using BUGS forExample using BUGS forhospital performance rankinghospital performance ranking



BUGS Model specificationBUGS Model specificationmodel{for k in 1:K {b[k]~dnorm(0, prec)r[k]~dbin(p[k], n[k])logit(p[k]) <- mu + b[k]}pop.mean<-exp(mu + bb)/(1+exp(mu + bb))mu~dnorm(0, 1E-6)prec~dgamma(.0001,.0001)tausq<-1/precadd~dnorm(0, prec)bb<- mu + add}

Monitor the p[k] and ask for ranks


Summary StatisticsSummary Statistics


Posterior distributions of the ranksPosterior distributions of the ranks





X = (Posterior Mean-Based Ranks) – (Optimal Ranks)

LOSLOS

==


LOSLOS


Relations among percentiling methodsRelations among percentiling methods1998 USRDS Percentiles


BACK TO THE USRDS, SMR EXAMPLEBACK TO THE USRDS, SMR EXAMPLE




False detection and non-detectionFalse detection and non-detection



K is large and we can use K is large and we can use a completely non-parametric priora completely non-parametric prior


= (1-B) = = (1-B) = 22/(/(22 + + 22) = ICC) = ICC


Probability of being in the upper 10%Probability of being in the upper 10%as a function of true percentileas a function of true percentile

= intra-class correlation


Documents

Term 4, 2006BIO656--Multilevel Models 1 Midterm Open “book” and notes; closed mouth 20-25 minutes to read carefully and answer completely 60 minutes