Beyond the Basics Jaleh Gholami MD,MPH, PhD candidate Beyond the Basics Jaleh Gholami MD,MPH, PhD candidate

Beyond the Basics

Jaleh Gholami Jaleh Gholami MD,MPH, PhD candidateMD,MPH, PhD candidate

Beyond the Basics

Jaleh Gholami Jaleh Gholami MD,MPH, PhD candidateMD,MPH, PhD candidate

Accuracy of research

• The inferences made from results of epidemiologic research depends on the accuracy of its methods and procedures.

Quality Assurance & Quality Control

Quality Assurance• The activities to ensure quality of the data

before data collection are regarded as quality assurance.

Quality Control:• The efforts to monitor and maintain the quality

of data during the conduct of the study are regarded as quality control.

• Study protocol : consist of a description of the general components of the investigation (10 steps).

Quality Assurance

1. Formulation of study’s main hypothesis/ hypotheses (QA)2. A priori specification of potential confounding variables

(QA).3. Definition of the characteristics of the study population

for external validity (generalizability) purposes (QA).4. Definition of the design strategy (e.g., cohort, case-

control, case-cohort) and the groups to be compared, and specification of selection procedures of internal validity. (QA)

5. Definition of the design strategy and samples for studies of reliability and validity (QA/QC)

Key features of the design of an epidemiologic study

6. specification of the study power necessary to detect the hypothesized association(s) at a given level of significance (QA).

7. standardization of procedures (QA)8. activities during data collection, including

analysis of quality control and remedial actions (QC).

9. Data Analysis10.Reporting of data

Key features of the design of an epidemiologic study

• Study protocol :

– provides a global picture of the strategies– leads to the development of more detailed

manuals of operation– describes the general design and procedures – and assists the staff in understanding the context

in which their specific activities are carried out

Quality Assurance

• Manuals of operation: – detailed descriptions of exactly how the procedures

specific to each data collection activity are to be carried out :

• Measurements• Interviews• Categorizations• Derived variables

– maximizes the likelihood the tasks will be performed as uniformly as possible (esp., in multicenter studies)

Quality Assurance

• Data collection instrument:– Questionnaires:

Development or choice or modification the previously used one?

• Validity and reliability are sometimes known• Comparing findings of the study with those of previous

studies

Quality Assurance

• Training of staff:

– Familiarizing them with the procedures under his or her responsibility– data collection– processing procedures – setting up appointments for interviews or visits– preparing materials for the interviewers and other data collectors– calibrating instruments– assigning interviewers to study participants– certification of the staff member to perform the specific procedure– periodic retraining and recertification (QC)

Quality Assurance

• Pretesting and pilot studies: feasibility and efficiency

pretest: to try to assessing specific procedures on a grab or convenience sampling in order to detect major flaws.

pilot study: is a formal rehearsal of study procedures that attempt to reproduce the entire flow of operations in a sample as similar as possible to study participants .

Quality AssuranceQuality Assurance

• Assess participant recruitment • Assess data collection procedures• A questionnaire can assessed:

– flow of questions– presence of sensitive questions– appropriateness of categorization of variables– clarity of wording to the respondent and the interviewer– clarity of the "question- by question" instructions to the

interviewer• Evaluating alternative strategies for participant

recruitment and data collection

Pretesting and pilot studies

Quality assurance

Steps:• Specify study hypothesis. • Specify general design (study protocol)• Choose or prepare instruments, develop procedures

for data collection and processing (Develop manuals)• Train staff, Certify staff. • Using certified staff, pretest and pilot-study data

collection and processing instruments and procedures; pilot-study alternative strategies for data collection

Quality AssuranceQuality Assurance

Quality controlQuality control• Quality control strategies include observation

of procedures performed by staff members, which allows the identification of obvious protocol deviations.

Observation Monitoring and Monitoring Observation Monitoring and Monitoring of Trends of Trends

• "over-the-shoulder" observation • statistical assessment of trends over time,

after adjustment for age, sex, and other relevant characteristics, for each technician or observer

• digit preference (e.g. age, blood pressure)• external or internal standards (e.g. lab data)

statistical assessmentstatistical assessment

digit preferencedigit preference

50 60 70 80 90 100110120130140150160170180190200210220230240

systolic blood presure

0%

5%

10%

15%

Validity studiesValidity studies

• A compromise between accuracy on the one hand and costs and participants' burden on the other is often necessary.

Validity studiesValidity studies

• Validity studies in subsamples of participants who undergo both the study-wide procedure and a more accurate procedure serving as a "gold standard" allow assessing the impact of these errors on the study estimates.

laboratory Measurementslaboratory Measurements

Example: serum cholesterolExample: serum cholesterol

• Screening measurements: blood obtained by the finger stick method in a non fasting state

• Gold standard measurements: The standard measurements were done in fasting state under carefully controlled conditions and assayed in a nationally recognized standard laboratory

temporal drift: differential bias over timetemporal drift: differential bias over time

Questionnaires Questionnaires

• Comparing with the gold standard: – information given by participants and physicians about

treatment– food frequency questionnaire and a 1-week diet diary

• It is important to assess samples of both "positive" and "negative" answers (estimation of both sensitivity and specificity) and collect Information separately for the groups being compared (cases and controls or exposed and unexposed) to evaluate misclassification (nondifferential or differential)

Importance of Validity Studies Importance of Validity Studies

• the knowledge of the validity of exposure, outcome, main confounding variables, and effect modifiers is important in the causal inferential process.

• many variables, even those considered "objective," have a relatively poor validity. (sensitivity and specificity)

Limitations of Validity Studies Limitations of Validity Studies

1. the "gold standard" itself may not be entirely valid.

2. validity studies samples frequently constitute a selected group of volunteers (may not be representative).

3. usually small sample size of validity studies resulting statistical imprecision.

4. sensitivity and specificity of the procedure may vary according to certain characteristics of the study population.

5. extrapolating the results of a validation study from one population to another, particularly if the data collection instrument is a questionnaire.

conclusionconclusion

Thenthe "corrected" estimate may be even less

"correct" than the original one.

Reliability StudiesReliability Studies

• reliability studies assess the extent to which results agree when obtained by – different observers, – study instruments or procedures, or by the same

observer, – study instrument, or procedure at different points

in time.• Ideally, the only source of variability in a study

should be that between study participants.

Other sources of variabilityOther sources of variability

1. Variability due to imprecision of the observer or the method, which can be classified in two types: – Within-observer (or intra-observer) or within-

method variability, – Between-observer (or interobserver) or between-

method variability

2. Variability within study participants

Within individual variabilityWithin individual variability

Control the variabilityControl the variability

• quality assurance procedures prevent or minimize within- and between-observer or method variability

• physiologic within-individual variability is not amenable to prevention.

Solution:– standardizing the timing of data collection for measures with

known temporal fluctuations (levels of cortisol or blood pressure)

– standardizing measurement conditions for variables affected by stress or activity (blood pressure)

– collect data at several points in time and use the average of all values.

Reliability studiesReliability studies

• obtaining random repeat measurements• Masked or not?

Reliability studyReliability study

misclassificationmisclassification

• If the sample size is of sufficient size, it is important to assess whether reliability estimates obtained for a sample of study participants differ according to relevant characteristics, which may result in differential levels of misclassification.

Indices of Validity and ReliabilityIndices of Validity and Reliability

Sensitivity and SpecificitySensitivity and Specificity


• Sensitivity and specificity are fixed properties of the test (or diagnostic criteria or procedure) itself, regardless of the characteristics of the study population.

Do you agree on this?


1. dependence of sensitivity and specificity estimates on the cutoff level (arbitrariness in assessing and reporting the validity)

2. the sensitivity and specificity of a given definition of a condition (based on a cutoff in a continuous distribution) depend on the distribution of the severity of the condition.



3.The validity of a test can vary from population to population when the test is not a direct marker of the condition (OB and prevalence of Peptic ulcer, PPD and prevalence of atypical mycobacterium)


4.sensitivity and specificity can vary according to the characteristics of individuals (limitation to the generalizability). if these variables represent the exposures of interest or are correlated with them, differential misclassification may arise.


Youden 's J Statistic Youden 's J Statistic

J = Sensitivity + Specificity - 1.0

Maximum value of the index =1 When there is perfect agreement

Minimum value of the index =0 When sensitivity and specificity = 0.5

Theoretically the value of J can be negative

Percent Agreement Percent Agreement

% Agreement = (20 + 60) / 100 = 80 %

MD#1

Yes No

MD#2Yes 20 10 30

No 10 60 70

30 70 100


• Although the percent agreement is a reliability indice, it can be also used to assess validity (the agreement between the test and gold standard)

• the simplest method of summarizing agreement

for categorical variables

• it can be calculated for any number of categories, not just two


MD#1

Yes No

MD#2

Yes 1 3

No 2 94

MD#1

Yes No

MD#2

Yes 43 3

No 2 52


Percent agreement= 95%

limitation of the percent agreement limitation of the percent agreement

• its values tend to be high whenever the proportion of negative-negative results is high (resulting from a low prevalence of positivity in the study population), particularly when the specificity is high.

Percent Positive Agreement Percent Positive Agreement

• Percent positive agreement:

• Chamberlain’s percent positive agreement:

1002

2100

2

)()(

cba

abaca

aPPA

100'

cba

asPPAnChamberlai

Calculate PA, PPA, CPPACalculate PA, PPA, CPPA

KappaKappa statisticstatistic

• kappa statistic is defined as the fraction of the observed agreement not due to chance in relation to the maximum nonchance agreement.

Agreement Expected - 1

Agreement Expected -agreement Observed

1

e

eo

P

PPk

ExampleExample

MD#1

Yes No Total

MD#2

Yes 43 3 46

No 2 52 54

Total 45 55 100

Calculate the expected values (chance)Calculate the expected values (chance)

ACTUAL MD #1

Yes No

MD#2 Yes 46

No 54

45 55 100

Calculate the chance agreement and KappaCalculate the chance agreement and Kappa

ACTUAL MD #1

Yes No

MD#2 Yes 20.7 25.3 46

No 24.3 29.7 54

45 55 100

%4.50100

7.297.20

ementChanceAgre

%90%4.501

%4.50%95

Kappa

%95100

5343

reementObservedAg

KAPPAKAPPA

+ 1 means that the two observers are perfectly reliable. They classify everyone exactly the same way.

0 means there is no relationship at all between the two observer’s classifications, above the agreement that would be expected by chance.

- 1 means the two observers classify exactly the opposite of each other. If one observer says yes, the other always says no.

Agreement Expected - 1

Agreement Expected -agreement Observed

1

e

eo

P

PPk

Interpretation of KappaInterpretation of Kappa

Weighted Kappa Weighted Kappa

ew

ewoww P

PPK

1

weaknesses of Weighted Kappa weaknesses of Weighted Kappa

• The weights assigned to cells, although somewhat arbitrary, on the basis of the investigators' perception of how serious the disagreement is.

Dependence of Kappa on Prevalence Dependence of Kappa on Prevalence

Dependence of Kappa on Prevalence Dependence of Kappa on Prevalence

• for the same sensitivity and specificity of the observers, the kappa value is greater in the population in which the prevalence of positivity is higher.

• kappa tends toward 0 as the prevalence approaches either 0 or 1.

conclusionconclusion

• comparisons of kappa values among populations may be unwarranted.

• kappa should be used and interpreted with caution. Use it in conjunction with other measures of agreement, such as the percent agreement .

• take into consideration its variability as a function of the prevalence of the condition and of the degree of similarity between observers with regard to the prevalence of positivity.

the 45° lines, intercept = 0, regression coefficient = 1.0 unitIt is not as sensitive as alternative graphic techniques

Indices of Validity/Reliability for Continuous Data:Indices of Validity/Reliability for Continuous Data:Correlation Graph (Scatter Diagram)Correlation Graph (Scatter Diagram)

• Pearson's r is probably one of the most frequently used measures of agreement for continuous variables .

• it is also one of the least appropriate!1. It is insensitive to systematic differences (bias) between two

observers or readings and it is misleading assessment of agreement.

Correlation CoefficientCorrelation Coefficient

2. the value of r is very sensitive to the range of values. (broader distribution)

3. Pearson's correlation coefficient is unduly sensitive to extreme values (outliers)

Correlation CoefficientCorrelation Coefficient

lntraclass Correlation Coefficient (ICC) lntraclass Correlation Coefficient (ICC) or reliability coefficient (R) or reliability coefficient (R)

• Vb = variance between individuals

• Vt = total variance

• Ve = unwanted variance (error)

eb

b

T

b

VV

V

V

VICC

• The ICC is equivalent to the kappa statistic for continuous variables.

• The range of ICC is from -1.0 (more realistically from 0) to + 1.0

• ICC is affected by the range of values in the study population (like Pearson's correlation coefficient)when Vb is small, ICC also will be small. This is particularly important in studies within populations in which the exposure values are either very high or very low.

lntraclass Correlation Coefficient (ICC) lntraclass Correlation Coefficient (ICC)

K is the number or repeat readings n is the number of individual studies or specimens being studied

Calculation of ICC from ANOVA tableCalculation of ICC from ANOVA table

Coefficient of Variability Coefficient of Variability

100i

ii

X

SDCV

2)( iiji XXV

• This calculation would have to be repeated for all pairs of measurements, and the overall coefficient of variability would be the average of all pair-wise coefficients of variability.

Bland-Altman Plot Bland-Altman Plot

Bland-Altman Plot Bland-Altman Plot

• the difference between the paired measurements (A - B in the ordinate) and their mean value [(A + B)/2, in the abscissa).

• it is much easier to assess:– The magnitude of disagreement (including

systematic differences)– Outliers– Any trend

Documents

Beyond the Basics Jaleh Gholami MD,MPH, PhD candidate Beyond the Basics Jaleh Gholami MD,MPH, PhD candidate