Upload
georgia-dickerson
View
217
Download
1
Tags:
Embed Size (px)
Citation preview
QSU SeminarReliability and Validity:
Rita Popat, PhDDept of Health Research & Policy
Division of Epidemiology
Practical ConsiderationsDesign and Analytic Approaches
2JAMA. 2004; 292:1188-1194
What do we want to know about the measurements? Why?Dependent variable (outcome)Independent variable (risk factor or predictor)
What are other possible explanations for not detecting an association?
4
JAMA. 2004;291:1978-1986
Outline
• Definitions: Measurement error, reliability, validity
• Why should we care about measurement error?• Effects of measurement error on Study Validity
(Categorical Exposures)• Effects of measurement error on Study Validity
(Continuous Exposures)
• Measures (or indices) for reliability and validity
Measurement error
• For an individual, measurement error is the difference between his/her observed and true measurement.
• Measurement error can occur in dependent (outcome) or independent (predictor or exposure) variables
• For categorical variables, measurement error is referred to as misclassification
• Measurement error is an important source of bias that can threaten internal validity of a study
6
Reliability (aka reproducibility, consistency)
• Reliability is the extent to which repeated measurement of a stable phenomenon- by the same person or different people and instruments, at different times and places- obtain similar results.
• A precise measurement is reproducible, that is, has the same (or nearly the same) value each time it is measured.
• The higher the reliability, the greater the statistical power for a fixed sample size
• Reliability is affected by random error7
Validity or Accuracy
•The accuracy of a variable is the degree to which it actually represents what it is intended to represent
•That is: The extent to which the measurement represents the true value of the attribute being assessed.
8
Precise (Reliable) and Accurate (Valid) measurements are key to minimizing measurement errorPrecision, no accuracy Accuracy, low precision
No precision, no accuracy Precision and accuracy
9
Measurement error in Categorical Variables• Referred to as Misclassification and could be in the
• Outcome variables, or• Exposure variables
• How do we know misclassification exists?• When method used for classifying exposure lacks
accuracy
Assessment of Accuracy
Imperfect
Classification
True classification
b+da+c
c+dd
TN
c
FN
Present
-
a+bb
FP
a
TP
Present
+
Absent
-
Present
+
Sensitivity = a / (a+c) False negative = c / (a+c)
Specificity = d / (b+d) False positive = b / (b+d)
11
Criterion validity (compare against a reference or gold standard)
12
Cases(outcome +)
Controls(outcome -)
Exposure + a b
Exposure - c d
12
Misclassification of exposure
• Non-differential
• Differential
Cases Controls
Exposure + 55 34
Exposure - 45 66
Cases Controls
Exposure + 50 20
Exposure - 50 80
True exposure
Reported exposure:90% sensitivity & 80% specificity in cases & controls
Misclassification of exposure
Attenuation of true association due to misclassification of exposure
13
0.4)50)(20(
)80)(50(OR
4.2)45)(34(
)66)(55(OR
• Non-differential misclassification occurs when the degree of misclassification of exposure is independent of outcome/disease status
• Tends to bias the association toward the null• Occurs when the sensitivity and specificity of the classification
of exposure are same for those with and without the outcome but less than 100%
14
Cases(outcome +)
Controls(outcome -)
Exposure + a b
Exposure - c d
14
Misclassification of the exposure
Underestimation of a relative risk or odds ratio for…
0 1 2
Observedvalue
True value
Bias toward the null hypothesis
B. Protective factor
Modified from Greenberg. Fig 10-4, chapter 10
0 1 2
Observed value
True value
Bias toward the null hypothesis
A. Risk factor
Cases Controls
Exposure + 48 14
Exposure - 52 86
Cases Controls
Exposure + 50 20
Exposure - 50 80
True exposure
Reported exposure: Cases - 96% sensitivity and 100% specificity Controls- 70% sensitivity and 100% specificity
Misclassification of the exposure
16
0.4)50)(20(
)80)(50(OR
7.5)14)(52(
)86)(48(OR
Cases Controls
Exposure + 48 30
Exposure - 52 70
Cases Controls
Exposure + 50 20
Exposure - 50 80
True exposure
Reported exposure: Cases - 96% sensitivity and 100% specificity Controls- 70% sensitivity and 80% specificity
Misclassification of the exposure
17
0.4)50)(20(
)80)(50(OR
1.2)30)(52(
)70)(48(OR
Differential misclassification occurs when the degree of misclassification differs between the groups being compared.
• May bias the association either toward or away from the null hypothesis
• Occurs when the sensitivity and specificity of the classification of exposure differ for those with and without the outcome
18
Cases(outcome +)
Controls(outcome -)
Exposure + a b
Exposure - c d
18
Misclassification of the exposure
Overestimation of a relative risk or odds ratio for…
0 1 2
True value
Observed value
Bias away from the null hypothesis
B. Protective factor
Modified from Greenberg. Fig 10-4, chapter 10
0 1 2
True value
Observed value
Bias away from the null hypothesis
A. Risk factor
Cases
Index
Proxy (~25%)
Hormone therapy
Never
Former
Current
Pharmacy database
accuracy?
Summary so far….
• Misclassification of exposure is an important source of bias
• Good to know something about the validity of measurement for exposure classification before the study begins
• Almost impossible to avoid misclassification, but try to avoid differential misclassification
• If the study has already been conducted, develop analytic strategies that explore exposure misclassification as a possible explanation of the observed results (especially for a “primary” exposure of interest)
Measurement error in Continuous Variables• Physiologic measures (SBP, BMI)• Biomarkers (hormone levels, lipids)• Nutrients• Environmental exposures• Outcome measures (QOL, function)
Model of measurement error
•
Measurement Theory: Example contd..
24
Measurement error
Ti Ti + b Ti + b + Ei_________ b _______ ______________ E _______________
+ systematic error in + additional "random error" for subject iinstrument (bias)
EXAMPLE: One measured diastolic blood pressure (DBP) as indicator of 2-year averageDBP.
BD cuff miscalibrated -- + randomness in BP cuff mechanics measures everyone's diastolic + subject i's 10 mmHg increase over 2-year average BP as 10 mm Hg less + subject intimidated - diastolic BP 20 mmHg higher than usual
+ misreading by interviewer+ random fluctuations in current BP
Measurement theory
25
Validity of X…
•
Measurement error
Differential Measurement error
OR
Differential Measurement error
OR
Differential Bias•
Non- Differential Measurement error
The effects of non-differential measurement error on the odds ratio. ORT is the true odds ratio for exposure versus reference level r. ORO is the observable odds ratio for exposure versus reference level r.
Effects of non-differential measurement error
73.082.0 62.0/1/1 2
XTOT RRRR
Summary so far….
• Measurement error is an important source of bias• Good to know something about the validity of
measurement for exposure before the study begins• Almost impossible to avoid misclassification, but try
to avoid differential misclassification!• Non-differential measurement error will attenuate
the results towards the null, resulting in loss of power for a fixed sample size
• This should be taken into account when estimating sample size during the planning stage and
• Interpretation of results and determining internal validity of a study
So why should we evaluate reliability and validity of measurements?• If it precedes the actual study, it tells us whether
the instrument/method we are using is reliable and valid
• This information can help us run sensitivity analysis or correct for the measurement error in the variables after the study has been completed
Outline
• Definitions: Measurement error, reliability, validity
• Why should we care about measurement error?• Effects of measurement error on Study Validity
(Categorical Exposures)• Effects of measurement error on Study Validity
(Continuous Exposures)
• Measures (or indices) for reliability and validity
Type of Variable Reliability Measure(s) Validity Measure(s)
Dichotomous Kappa sensitivity, specificity
Ordinal weighted kappaICC*
misclassification matrix
Continuous ICC *Bland Altman Plots
Pearson correlation(see note)
Bland-Altman Plots
Choice of reliability and validity measures depend on type of variable . . .
*ICC – intraclass correlation coefficientNote: in inter-method reliability studies, inferences about validity can be
made from coefficients of reproducibility (such as the Pearson’s correlation )38
Assessing Accuracy (Validity) of continuous measures
• Bias: difference between the mean value as measured and the mean of the true values
• So bias = –• Standardized bias =
• Bland-Altman plots
39
X
X
x
x
XSD
Xx
Bland and Altman plots
•Take two measurements (different methods or instrument) on the same subject
•For each subject, plot the difference b/w the two measures (y axis) vs. the mean of the two measures
•We expect the mean difference to be 0•We expect 95% of the differences to be within 2 standard deviations (SD)
Yoong et al. BMC Medical Research Methodology 2013, 13:38
Yoong et al. BMC Medical Research Methodology 2013, 13:38
Suppose there is no gold standard, then how do we evaluate validity?…..We make inferences from inter-method reliability studies!
Note: will not be able to estimate bias when the two measures are based on different scales
Inferences about validity from inter-method reliability studies
• Suppose two different methods (instruments) are used to measure the same continuous exposure. Let X1 denote the measure of interest (i.e., the one to be used to measure the exposure in the study) and X2 is the comparison measure
• We have the reliability coefficient
• However, we are actually interested in the validity coefficient:
• Example: Is self-reported physical activity valid? Compare it to the 4-week diary.
44
21xx
1Tx
Relationship of Reliability to ValidityErrors of X1 and X2 are: Relationship b/w reliability and
validityUsual application
1. Uncorrelated and both measures are equally precise
Intramethod study
2. Uncorrelated , X2 is more precise than X1
Intermethod study
3. Uncorrelated , X1 is more precise than X2
Intermethod study
4. Correlated errors and both measures are equally precise
Intramethod studyIntermethod study
2121 xxTxTx
21121 xxTxxx
211 xxTx
211 xxTx
Take home message: In most situations the square root of the reliability coefficient can provide an upper limit to the validity coefficient 45
Inferences about validity from inter-method reliability studies
• In our example, X1 is measure of interest (i.e., the one to be used to measure the exposure in the study: self-reported activity) and X2 is the comparison measure (4-wk diaries)
• We have the reliability coefficient = 0.79
• Errors in X1 and X2 are likely to be uncorrelated and X2 is more precise than X1, so
0.79 < < 0.89
• So, self-reported activity appears to be a valid measure46
21xx
1Tx
Summary of Inferences From Reliability to Validity
• Reliability studies are used to interpret validity of x.
• Reliability is necessary for validity (instrument cannot be
valid if it is not reproducible).
• Reliability is not sufficient for validity - repetition of test
may yield same result because both X1 and X2 measure
some systematic error (i.e., errors are correlated).
• Reliability can only give an upper limit on validity. If the
upper limit is low, then the instrument is not valid.
• An estimate of reliability (or validity) depends on the
sample (i.e., may vary by age, gender, etc.)47
Reliability of continuously distributed variables
• Pearson product-moment correlation?
• Spearman rank correlation?
48
But…does correlation tell you about relationship or agreement?
Pearson’s Correlation coefficient=0.99Is this measure reliable?
49
Measure 1 Measure 2150 155155 158160 165163 170170 176174 184
145 150 155 160 165 170 175 180140
145
150
155
160
165
170
175
180
185
190
Measure 1
Mea
sure
2
Reliability of continuously distributed variables
• Other methods generally preferred for intra or inter-observer reliability when same method/instrument is used- Intraclass correlation coefficients (ICC): is calculated
using variance estimates obtained through an analysis of variance (ANOVA)
- Bland-Altman plots
• Correlation coefficient useful for inter-method reliability to make inferences about validity (especially when the measurement scale differs for the two methods)
50
Intraclass Correlation Coefficient (ICC)
• If within-person variance is very high, then measurement error can "overwhelm" the measurement of between person differences.
• If between-person differences are obscured by measurement error, it becomes difficult to demonstrate a correlation between the imperfectly measured characteristic and any other variable of interest.
• ICC is computed using ANOVA
51
Total variance
Within person Between person
Between person variance
Total varianceICC =
ANalysis Of Variance (ANOVA)in a reliability study
In a reliability study, we are not studying associations b/w predictors and outcome, so we will express the overall variability in the measurement as a function of between-subjects and within-subjectsvariability
• So let’s consider a test-retest reliability study, where multiple measurements are taken for each subject
52
SST = SSB + SSW
Total Variation22
12
2
11 )(...)()( XXXXXXSST nkn
53
G rou p 1 G rou p 2 G rou p 3
Resp on se , X
X
Subject 1 Subject 2 Subject 3
Between-Subject Variation22
22
2
11 )(...)()( XXkXXkXXkSSB nn
54
G rou p 1 G rou p 2 G rou p 3
Resp on se , X
1X 2X
3XX
Subject 1 Subject 2 Subject 3
Where: k1= number of measurements taken on subject 1
Within-Subject Variation
22
212
2
111 )(...)()( nnk XXXXXXSSWn
(continued)
55
G rou p 1 G rou p 2 G rou p 3
Resp on se , X
1X2X
3X11X
12X 13X
Subject 1 Subject 2 Subject 3
One way analysis of variance for computation of ICC: test-retest study
Source of variance Sum of squares Degrees of Mean square(SS) freedom (df) (MS=SS/df)
Between subjects n-1 BMS
Within subjects n (k –1) WMS (random error)
Total nk - 1
56
Here, each subject is a group.k=# times measure is repeated
2 i
i XXk
i j
iij XX2
i j
ij XX2
WMSkBMS
WMSBMSICC x )1(
ˆ
Interpretation of ICC
• If within-person variance is very high, then measurement error can "overwhelm" the measurement of between person differences.
• If between-person differences are obscured by measurement error, it becomes difficult to demonstrate a correlation between the imperfectly measured characteristic and any other variable of interest.
57
WMSkBMS
WMSBMSICC x )1(
ˆ
Interpretation of ICC
• The ICC ranges b/w 0 and 1 and is a measure of reliability adjusted for chance agreement
• An ICC of 1 is obtained when there is perfect agreement and in general a higher ICC is obtained when the within-subject error (i.e., random error) is small.
• Hence, ICC=1 only when there is exact agreement between measures (i.e., Xi1=Xi2=...Xik for each subject).
• Generally, ICCs greater than 0.7 are considered to indicate good reliability.
58
Two-way fixed effects ANOVA for computation of ICC (inter-rater reliability)
Source of variance Sum of squares Degrees ofMean square
(SS) freedom (df) (MS=SS/df)
Between subjects n-1 SMS
Between measures k - 1 MMS
Within subjects (random error) (n-1)(k-1) EMS
Total by subtraction nk - 1
59
2 i
i XXk
2 j
j XXn
i j
ij XX2
EMSknMMSknSMS
EMSSMSnx )1)(1()1(
)(ˆ
Measuring reliability of categorical variables
•Percent agreement or concordance rate
•Kappa statistic
60
Reliability of categorical variables
•Concordance rate is the proportion of observations on which the two observers agree
•Example: Agreement matrix for radiologists reading mammography for breast cancer
61
Radiologist B
Radiologist A
b+da+c
c+ddcNo
-
a+bbaYes
+
No
-
Yes
+
Overall % agreement = (a+d) / (a+b+c+d)
Concordance rates: limitations
•Considerable agreement could be expected by chance alone.
•Misleading when the observations are not evenly distributed among the categories (i.e., when the proportion “abnormal” on a dichotomous test is substantially different from 50%)
So, what reliability measures should we use?
62
Kappa
•Kappa is another measurement of reliability •Kappa measures the extent of agreement beyond that would be expected by chance alone
•Can be used for binary or variables with >2 levels
63
Cohen’s Kappa ( ): some notation
• A reliability study in which n subjects have each been measured twice where each measure is a nominal variable with k categories.
• It is assumed that the two measures are equally accurate.
• is a measure of agreement that corrects for the agreement that would be expected by chance.
64
Cohen’s KappaTable. Layout of data for computations of Cohen’s and weighted
Measure 2 (or Rater 2)1 2 . . k Total
1 p11 p12 . . p1k r1
Measure 1 2 p21 p22 . . p2k r2
(or Rater 1) . . . . . . .. . . . . . .k pk1 pk2 . . pkk rk
Total c1 c2 . . ck 1
65
Cohen’s KappaTable. Layout of data for computations of Cohen’s and weighted
Measure 21 2 . . k Total
1 p11 p12 . . p1k r1
Measure 1 2 p21 p22 . . p2k r2
. . . . . . .
. . . . . . .
k pk1 pk2 . . pkk rk
Total c1 c2 . . ck 1
The observed proportion of agreement, Po, is the sum of the
proportions on the diagonal:
66
k
iiio pP
1
Cohen’s KappaTable. Layout of data for computations of Cohen’s and weighted
Measure 21 2 . . k Total
1 p11 p12 . . p1k r1
Measure 1 2 p21 p22 . . p2k r2
. . . . . . .
. . . . . . .
k pk1 pk2 . . pkk rk
Total c1 c2 . . ck 1
The expected proportion of agreement (on the diagonal), Pe, is:
Where ri and ci are marginal proportions for the 1st and 2nd measure respectively.
67
k
iiie crP
1
KappaThen, kappa is estimated by:
Which is: Observed agreement(%)-Expected agreement (%)
100% - Expected agreement (%) • = maximum possible nonchance agreement or 100% less the
contribution of chance • = proportion of observations that can be attributed to
reliable measurement (i.e., not due to chance)• So kappa is the ratio of the number of observed nonchance
agreements to the number of possible nonchance agreements
68
e
eo
P
PP
1
eo PP
eP1
Pictorial of kappa statistic
69
agreement expected
0 100%agreement
by chancepotential improvement
beyond chance
observed agreement
Kappa = % of maximum possible improvement over thatexpected by chance alone (kappa 0.50 here)
Kappa
•Kappa ranges from –1 (perfect disagreement) to +1 (perfect agreement)
•Kappa of 0 means that: observed agreement = expected agreement
70
e
eo
P
PP
1
Reliability of categorical variables
•Example 1: Agreement matrix for radiologists reading mammography for breast cancer
71
Radiologist B
Radiologist A
12624
8683 (d)3 (c)No
-
6443 (b)21 (a)Yes
+
No
-
Yes
+
Overall % agreement = (a+d) / (a+b+c+d)=(21+83)/150=0.69
31.055.01
55.069.0
1ˆ
e
eo
P
PP