Measures of Reliabilityand

Multivariate Analysis

Objectives

• To discuss measures of reliability

• To discuss multivariate analysis

Kappa Statistic

• po = observed probability of concordance between the two surveys

• pe = expected probability of concordance between the two surveys

• The standard error of the Kappa statistic is calculated by:

• To test the hypothesis Ho:=0 vs. H1:0, use the test statistic:

Percent agreement

Figure 1. Association of average faculty performance rating (from 1, bottom 20%, to 5, top 20%) and absolute rank on the National Resident Matching Program (NRMP) list (r = 0.19; P =.11).

Table 1. Discrepancy Between the DIS and SCAN for the Lifetime Occurrence of Depressive Disorder in the

Baltimore ECA Follow-up*

Table 1-Comparison of WHO and ADA diagnostic categories for undiagnosed diabetes From:   Lee: Diabetes Care, Volume 23(2).February 2000.181-186

Kappa score calculation

PlaqueNormal Expected chance agreementPlaque 140 52 192 40.70 Normal 69 725 794 625.70

209 777 986

Pe= 40.7 626 0.676Po= 140 725 0.877

k = 0.877 - .676 = 0.62----------------1.0 - .676

Kappa and prevalence

• You have two populations of 1000 each being compared by interview A and interviewer B

• One has a prevalence of 5 and the other has a prevalence of 40

• Sensitivities and specificities for observer A and B are .90 and .90 and .80 and .95 respectively

• Create a table for each population

Prevalence and Kappa

• Kappa tends toward 0 as the prevalence approaches either 0 or 1

• High values of kappa can be obtained when the marginals of the contingency table are unbalanced. (tends to be higher when the positivity prevalence is different between A and B

Kappa scores for table 8-18