AACB Harmonisation Workshop September 12, 2016
Using commutable materials toassess bias by a clinical laboratory
Greg Miller, PhDVirginia Commonwealth University
Richmond, Virginia, [email protected]
Financial disclosures
Siemens Healthcare Diagnostics, consultant
Roche Molecular Diagnostics, speakinghonorarium and travel expenses, focus group
Abbott Diagnostics, research grant
Outline
What is bias
What is commutability
How to assess bias usingcommutable materials
-4 -3 -2 -1 0 1 2 3 4 5
Num
ber o
f obs
erva
tions
ExpectedValue
Measured valuePrecision
ObservedValue
BIAS
Types of Bias
Bias vs a reference measurement procedure
Bias vs other measurement procedures in EQA
Bias vs other measurement procedures in ahealthcare system
Bias within a measuring system when changingreagent or calibrator lots
Commutability (Commutable)
Property of a reference material demonstratedby the closeness of agreement betweenresults for a reference material and results forclinical samples
.
(Derived from VIM 3: 2008)
0
2
4
6
8
10
0 2 4 6 8 10
Measurement procedure 1
Mea
sure
men
t pro
cedu
re2
Patient samplesReference orEQA/PT materials
Commutable: RM has same relationshipas observed for patient samples
0
2
4
6
8
10
0 2 4 6 8 10
Measurement procedure 1
Mea
sure
men
t pro
cedu
re2
Patient samplesReference orEQA/PT materials
Non-commutable: RM has different relationshipthan observed for patient samples
0
2
4
6
8
10
0 2 4 6 8 10
Measurement procedure 1
Mea
sure
men
t pro
cedu
re2
Patient samplesReference orEQA/PT materials
Apparent agreement for non-commutable RMmeans patient samples DO NOT agree
Terminology
Correct terminology:
A reference material is commutable withpatient samples among a group ofmeasurement procedures
.
Incorrect terminology: patient sample resultsare commutable among a group ofmeasurement procedures
Correct terminology: patient sample resultsare equivalent (i.e. harmonized) among agroup of measurement procedures
o Equivalent means within limits suitable for use inmedical care decisions
.
Commutability can be different with differentreagent lots or batches
Patient sample results and QC sample results werecompared for 18 QC materials before and after 1496reagent lot changes for 83 analytes using 7 differentinstrument platforms
41% of the reagent lot changes had a significantdifference between the results for QC samplescompared to those for patient samples
Miller et al. Clin Chem 2011;57:76-83.
Internal QC assesses relative bias
Assesses bias relative to a calibration condition
Does not assess bias vs a reference system
Must adjust the expected value of a QC materialwhen influenced by a reagent lot change
Measure patient and QC samples withboth current and new reagent lots
New reagent lot is acceptablefor patient testing
Agreementfor patientsamples isacceptable
Investigate new reagent lot; donot report patient results using
new reagent lot
Yes
No
Protocol for assessing QC for a reagent lot change
Step 1
Miller, QC chapter in Henry’s Clinical Diagnosis and Management by Laboratory Methods, 23rd ed. 2016
Measure patient and QC samples withboth current and new reagent lots
Agreementfor QC is
acceptable
New reagent lot is acceptablefor patient testing
Agreementfor patientsamples isacceptable
Investigate new reagent lot; donot report patient results using
new reagent lot
Yes
No
Yes
No
Continue with sametarget value
• Update QC target value• Do not change SD
Protocol for assessing QC for a reagent lot change
Step 1
Step 2
Miller, QC chapter in Henry’s Clinical Diagnosis and Management by Laboratory Methods, 23rd ed. 2016
25
30
35
40
45
50
0 5 10 15 20 25 30 35
QC sequence
QC
val
ue
Target
Rule limit
Rule limit
New Reagent LotMeaninglessQC failures
Should beQC failures
Reagent lot change influence on QC
Miller, QC chapter in Henry’s Clinical Diagnosis and Management by Laboratory Methods, 23rd ed. 2016
25
30
35
40
45
50
0 5 10 15 20 25 30 35
QC sequence
QC
val
ue
Target
Rule limit
Rule limit
AcceptableQC values
Real QCfailures
Adjust QC target value for meaningful assessment
Miller, QC chapter in Henry’s Clinical Diagnosis and Management by Laboratory Methods, 23rd ed. 2016
Calibrator lot change
Commutability is a property influenced byreagents
QC results will correctly assess bias caused by anew lot of calibrator when the reagents do notchange at the same time
Outline
What is bias
What is commutability
How to assess bias usingcommutable materials
How can a laboratory estimate bias
1. Use commutable certified reference materials
2. Use commutable EQA materials
3. Use a set of individual patient samples
4. Use pooled patient samples (validate orassume commutability)
Bias estimation
Observed bias = biasP + biasM
Where:
biasP = bias that affects patient results
biasM = matrix-bias not observed in patientresults (non-commutability bias)
Bias estimation
Observed bias = biasP + biasM
Where:
biasP = bias that affects patient results
biasM = matrix-bias not observed in patientresults (non-commutability bias)
CommutableRMs have nomatrix bias
CLSI C37 Commutable Frozen Human Serum Pools
<15 min.
Clot 3-4 hours room temp
Pool Filter Aliquot Freeze -70C
CLSI C37 includes assessment of commutabilityof the pooled serum
1. Prepare aliquots of the individual donors
2. Measure concentration in pool and in individualdonors
3. Commutable if value for pool = mean value ofindividual donors proportional to their volumefraction
Pooled patient samples may not be commutable
Pooling can cause artifacts
Validating commutability is recommended
CDC pooled serum EQA for HDL-C Pools were prepared according to CLSI C37 protocol
Commutability was lot and criteria dependent
Korzun et al. Clin Chem 2015;61:1107-13.
Supplemented pools may be commutablePercent of SY.X <3 for EQA vs panels of patient samples
Cobbaert et al. Clin Chim A 2012;414:234-40.
What bias question does a laboratory want to know
Bias vs a reference measurement procedure
Relative bias vs other measurement procedures in EQA
Relative bias vs other measurement procedures in ahealthcare system
Bias vs a reference measurement procedure
Practical:
1. Use a commutable certified reference material2. Use EQA with commutable samples and values
assigned by reference measurement procedures
Not practical:
3. Use individual patient samples with values assigned bya reference measurement procedure
4. Use pooled patient samples with values assigned by areference measurement procedure
Commutable certified reference materials
Not all CRMs have had commutability validated
Many CRMs are intended for use with referencemeasurement procedures, not with clinical laboratorymeasurement procedures
JCTLM did not require commutability data prior to 2014for CRMs intended for use with clinical laboratorymeasurement procedures
WHO has not required commutability validation
Many WHO RMs for immunoassay are not commutable
Examples, commutable CRMs
NIST SRM 967 Creatinine in frozen human serum
NIST SRM 965 Glucose in frozen human serum
NIST SRM 956 Electrolytes in frozen human serum
JRC ERM DA451/IFCC Cortisol reference panel offresh frozen human sera
JRC ERM DA470k/IFCC Human serum proteins
JRC ERM DA471/IFCC Cystatin C
JRC ERM C-reactive protein
EQA with commutable samples and valuesassigned by reference measurement
procedures
Perfect when available
EQA may assume commutable samples basedon how they were prepared
o May be a reasonable assumption
o May be the only practical approach available
EQA for HbA1c correlates with NGSP certification status
NGSP.org.
CAP Accuracy Based Creatinine Survey
Used with permission from the College of American Pathologists
CAP Accuracy Based Creatinine Survey
Used with permission from the College of American Pathologists
Value assign a pool to be traceable toa commutable CRM (or EQA)
Residual specimens
POOL
Commutable RM
Assay pool and RM Same run Replication Multiple instruments
Pool value = Value RM xMeasured poolMeasured RM
Possibleaggregates; filter
or centrifuge
Pool values should be similarto those of the RM
Value assign a pool to be traceable toa commutable CRM (or EQA)
Assumes the pool is commutable
Reasonable assumption when valueassigned using the same measuringsystem(s) for which the pool will beused to assess bias in a laboratory
MP 1 MP 2
CRM traceablepool value 95 110
What if the pool value is different for two differentmeasurement procedures?
The pool is not commutable
The pool is still useful because the assigned valuescorrect for the non-commutability bias for thespecific measurement procedures used
What bias question does a laboratory want to know
Bias vs a reference measurement procedure
Relative bias vs other measurement procedures in EQA
Relative bias vs other measurement procedures
Bias vs other measurement procedures in EQA
Only when commutable EQA materials
Clin Chem 2011;57:1670-80.
Influence of reagent lot on peer group mean
Clin Chem 2016;62:708-15.
Peer group mean
EQA samples were commutable when reagent lot H was used
Clin Chem 2016;62:708-15.
Clin Chem 2016;62:708-15.
Commutability with the next reagent lot is unknown
What bias question does a laboratory want to know
Absolute bias vs a reference measurement procedure
Relative bias vs other measurement procedures in EQA
Relative bias vs other measurement procedures
Bias vs other measurement procedureswithin a health delivery system
Several instruments in different locations
o Some could be different methods
Responsibility of laboratory is that results reflectphysiology not differences in methods orinstruments
Different model instruments from the same manufacturerSame reagent and calibrator lots
-50
-40
-30
-20
-10
0
10
0 5 10 15 20 25 30
Trop
onin
I Di
ffere
nce
(ED-
Mai
n) n
g/m
L
Troponin I Concentration, ng/mL
Mfr Calib
-50
-40
-30
-20
-10
0
10
0 5 10 15 20 25 30
Trop
onin
I Di
ffere
nce
(ED-
Mai
n) n
g/m
L
Troponin I Concentration, ng/mL
Mfr Calib
CorrectionFactor
Result x 1.65
Different model instruments from the same manufacturerCorrection factor used to achieve agreement in results
CLSI EP 31
Verification of Comparability of PatientResults Within One Health Care System
Uses residual patient samples or commutablematerials
Criteria for agreement is based on a criticaldifference derived from medically allowable errorand precision performance of measurementprocedures
Lab can value assign a pool for usewithin the laboratory to assess bias
Residual specimens
POOL
Assay pool Same run Replication All methods
Pool value =
Possibleaggregates; filter
or centrifuge
Average of all methods
Selected comparativemethod
Lab can validate the commutability of a pool
Pool
Individual Specimens
Pool commutable
Measure withmethod 1 and
method 2 Pool not commutable
Do for allcombinations of
different methods
See CLSI EP-14 for details
Internal Round RobinPatient sample measured on several instruments
Criteria +/- Inst 1 Inst 2 Inst 3 Inst 4 Min Max
NA 4 mmol/L 138 139 137 136 136 139 Pass
K 0.3 mmol/L 4.5 4.6 4.5 4.2 4.2 4.6 Fail
CL 4 mmol/L 102 103 103 100 100 103 Pass
CO2 4 mmol/L 27 25 27 26 25 27 Pass
GLU 6 mg/dL or 5%* 118 119 119 116 116 119 Pass
UREA 3 mg/dL or 10%* 23 24 23 26 23 26 Pass
CREAT 0.2 mg/dL or 10%* 0.98 0.97 0.94 0.98 0.94 0.98 Pass
* which ever is higher
Within instrument vs. between instrument variability
SDa
4 instruments; same IVD• 3 Main lab• 1 ED lab
Within instrument vs. between instrument variability
SDa
For Change in Troponin
Between instrument variability dominates
Summary: bias assessment
It’s all about the matrix - use commutablesamples
Commutable CRMs or EQA samples
Pooled patient specimens Be aware of possible commutability limitations
Value assign a pool to a commutable CRM orEQA; or within lab to verify consistency
Individual patient specimens are always suitable
Bibliography
CLSI EP 30, Characterization and qualification of commutablereference materials for laboratory medicine
CLSI EP 14, Evaluation of commutability of processed samples
CLSI EP 31, Verification of comparability of patient results withinone health care system
CLSI EP 26, User evaluation of between reagent lot variation
CLSI C 37, Preparation and validation of frozen human serumpools as secondary reference materials for cholesterolmeasurement procedures