Term 4, 2006 BIO656--Multilevel Models 1 PART 3. Term 4, 2006 BIO656--Multilevel Models 2 NEED TO INCORPORATE ALL UNCERTAINTIES The Z versus t distribution

BIO656--Multilevel Models 1Term 4, 2006

PART 3PART 3


NEED TO INCORPORATENEED TO INCORPORATEALL UNCERTAINTIESALL UNCERTAINTIES

• The Z versus t distribution is the basic example• Want to produce a CI for a population mean• Assume a Gaussian sampling distribution,:

• Z is t with a large df• t3 is the most different from Z for t-distributions with a finite variance



ACCOUNTING FOR ACCOUNTING FOR (explaining)(explaining)

UNEXPLAINED VARIABILITYUNEXPLAINED VARIABILITY

• Including regressors can explain (account for) some of unexplained variability

• Doing so is always a trade-off in that you need to use degrees of freedom to do the explaining

• Going too far--adding too many regressors-- inflates residual variability

• In MLMs there is variance at various levels that can potentially be taken into account


TEACHER EXPECTANCYTEACHER EXPECTANCY(data are in “Datasets” )

Data are from a Raudenbush & Bryk meta-analysis of 19 studies (see Cooper and Hedges,1994)

Effect sizek = distance between treatment and control group means measured in population standard deviation units

SEk = the standard error of the effect size

Weeksk = estimated weeks of teacher-student contact prior to expectancy induction


TEACHER EXPECTANCY TEACHER EXPECTANCY (continued)

• Each study consisted of either telling teachers that a student had great potential or not

• All students received a pre-test and a post-test

• Teachers evaluated progress

• A positive effect size indicates that the teachers rated students who were “likely to improve” as having improved more than the control group

• A negative slope on “Weeks” indicates that the more a teacher got to know a student before the experiment,the less the influence of the expectancy intervention




ESTIMATING ESTIMATING 22

(when all k2 = 2 )

• Compute residuals, rk = (Yk – modelk)• Compute the Mean Squared Error: (n – df)MSE = r2

k

•Then, compute: 2 = (MSE - 2 )

• modelk is either just the intercept or intercept + weeksk

• 2 decreases if MSE decreases


A CASE STUDYA CASE STUDYON VARIANCE ACCOUNTINGON VARIANCE ACCOUNTING


DIABETES CONTROL STUDYDIABETES CONTROL STUDYPercentage Variance at the

Patient, Physician, and Clinic Levels

• Patrick J. O’Connor MD MPH, • Gestur Davidson PhD• A. Lauren Crain PhD • Leif I. Solberg MD • Robin R. Whitebird PhD • Thomas A. Louis PhD

HealthPartners Research Foundation University of Minnesota Johns Hopkins Bloomberg SPH


Conceptual Model:Conceptual Model:What Affects Diabetes Care?What Affects Diabetes Care?

A Nested Hierarchy• Health Plan• Medical Group• Clinic• Physicians• Patients• Interactions across all levels


OUTCOME MEASURE:OUTCOME MEASURE:A1c (HbA1c)A1c (HbA1c)

Glycohemoglobin, Glycated hemoglobin• Known as: Hemoglobin A1c • Used to monitor diabetes and to aid in treatment decisions• Should be assayed at first diagnosis of diabetes and then 2 to 4 times per year• Requires a blood sample • Normal values between 4% and 6%• High is bad


STUDY SITESTUDY SITEHealth Partners Medical Group, MNHealth Partners Medical Group, MN

• 175,000 adults receiving care at 19 clinics in 1995

• Medical group centrally administered and clinics have common guidelines, formulary, and culture

• So, there may be less variance at clinic or physician level than in other contexts


STUDY DESIGNSTUDY DESIGN

• Analysis of 2,463 adults with diabetes mellitus in 1994

• Follow-up A1c data in 1995, 1996, 997

• Patients nested within providers and clinics


STUDY PARTICIPANTSSTUDY PARTICIPANTS

• 2,463 adults with DM in same clinic and with same primary care physician each year from 1995-1997

• DM identification: Sensitivity = 0.91 Pred. Pos. Val. = 0.94

• To be included in analysis must have had at least one A1c test each year • A1c test rates ranged 80-87% per year 1995-1997


Number of Number of Eligible AdultsEligible Adults with diabetes with diabetes in 1995 cohortin 1995 cohort

Year Enrolled With A1c Test

1995 5,432 4,339

1996 4,835 3,941

1997 4,451 3,767


Analytic SampleAnalytic Sample

• 19 Clinics• 41 Physicians• 2,463 Patients• 3 years of time


MODEL & ANALYSISMODEL & ANALYSIS

• Multilevel multivariate hierarchical linear models (using MLwiN) to estimate variance components at each level (time, patient, physician, clinic)

• Analyzed for A1c in each year, and change in A1c across years


STUDY POTENTIALSTUDY POTENTIAL

• There is substantial variance in A1c and in change in A1c across all levels of the hierarchy

• Some of the A1c variance is a “roll-up” from lower levels

• To develop rational improvement strategies, one must understand where the variance resides and if some can be explained


HYPOTHESESHYPOTHESES

• After control for patient and physician variance, there will be no clinically significant variance in A1c change at the clinic level

• At each level that has significant variance in A1c change, we may be able to identify key variables that are related to the variance


Characteristics of study participants Characteristics of study participants who had/(did not have) who had/(did not have) 1 A1c 1 A1c tests during the 36-month study tests during the 36-month study periodperiod A1c Measurement

Status

Comparison Variables

Measured Not Meas. P-val

% Female 53% 53% 0.73

Average Age 59.5 60.4 0.10

Average Charlson 1.73 2.07 0.01

Female Physician 28% 25% 0.23

Average Physician Age

42.2 43.0 0.01

% Fam. Med. 33% 35% 0.39


Characteristics of participants who Characteristics of participants who were/(were not) assigned to were/(were not) assigned to

a primary care physiciana primary care physician

Physician Assignment Status

Comparison Variables

Assigned Not Assigned

P-val

% Female 47% 47% 0.60

Average Age 60.4 55.3 0.01

Average Charlson

1.75 1.02 0.01

A1c Value 8.29 8.06 0.01





PERCENT OF VARIANCE ACCOUNTINGPERCENT OF VARIANCE ACCOUNTING1995 A1c Level1995 A1c Level

“Vanilla” model & with Covariates

Vanilla Model

Full Model

Clinic 1.9% 2.7%

Physician 2.8% 1.4%

Patient 95.4% 96.0%


Covariates for 1995 Covariates for 1995 A1c Level (A1c Level (RR22 = 0.14 = 0.14))

Variable Coefficient SE

Pt Age < 65 0.025 0.010

Insulin Use 0.159 0.010

Sulfonyl Use 0.106 0.010

Phy. Specialty -0.015 0.014

Phy. Gender -0.016 0.057

Pt Comorbid=2 0.025 0.012


• The slope on insulin is 0.159 which is > 0

• Does this mean that insulin is bad for diabetics?

• Or, does it represent an association between A1c level and the decision to treat?

Hint: It’s the association/selection

BEWARE OF SELECTION EFFECTSBEWARE OF SELECTION EFFECTS


PERCENT OF VARIANCE ACCOUNTINGPERCENT OF VARIANCE ACCOUNTING(1997-1995) A1c Change(1997-1995) A1c Change

“Vanilla” model & with Covariates

Vanilla Model

Full Model

Clinic < 0.1% < 0.1%

Physician 0.7% 0.8%

Patient 99.3%* 99.2%*


Covariates for 1997-1995 Covariates for 1997-1995 A1c change (RA1c change (R22 = 0.11) = 0.11)

Variable Coefficient SE

Pt Age < 65 0.093 0.044

Drug Intensity -0.418 0.056

Doc Specialty 0.057 0.104

Pts Per Doc -0.002 0.002

Doc Age -0.001 0.004

Pt Comorbidity -0.014 0.018


ANALYZING CHANGE ANALYZING CHANGE REMOVES THE SELECTION EFFECTREMOVES THE SELECTION EFFECT

• The slope on insulin is now –0.418 which is < 0

• Warning: such a simple analysis will not always sort things out “Causal Analysis” is needed


RESULTSRESULTS• Models with limited set of covariates explained about 14% of variance in A1c levels in 1995

• Models with covariates explained about 35% of variance in change in A1c from 1995-97

• Over 90% of variance was at patient level or related to physician-patient interaction

• Little variance at physician level

• Little variance at the clinic level


Factors associated with change in A1cFactors associated with change in A1c

• Time: A1c got better each year

• Older patients had more improvement

• Comorbidity was related (complex)

• Drug Intensification (by drug class) was the variable that was strongest predictor

• Unidentified Patient Factors are likely


Other Clinical DomainsOther Clinical Domains

• A1c Test Rates• LDL Test Rates• Eye Exam Rates

• Generally similar results, with the majority of variance at patient level; much less variance at physician and clinic levels


STUDY LIMITATIONSSTUDY LIMITATIONS

• Relatively homogeneous medical group, may reduce variance at clinic and doc level• Clinic systems already in place• Selection effects • Paucity of covariates at clinic and provider levels


DISCUSSIONDISCUSSION

• Interventions may be made at any level, not just levels with significant variance

• However, a great deal of recent attention is directed to clinic systems

• Patient behavior and provider behavior and doctor-patient interaction need more attention

• Factors that impact drug intensification may be key


Future DirectionsFuture Directions

• Larger set of medical groups and clinics• More covariates at each level• Model selection effects • Estimate power at various levels• Strategies to handle missing data• Assess other clinical domains

Documents

Term 4, 2006 BIO656--Multilevel Models 1 PART 3. Term 4, 2006 BIO656--Multilevel Models 2 NEED TO INCORPORATE ALL UNCERTAINTIES The Z versus t distribution