Infectious Disease Testing: QC and Risk Management Johns, Inf Dis, AG... · 4/12/2010 · Lab CV / Peer CV = 5.0% / 5.0% = 1.0 CVR BEFORE Lab CV / Peer CV = 10.0% / 5.0% = 2.0 CVR

Infectious Disease Testing: QC and Risk ManagementSt. John’s, Newfoundland, CanadaMike Toyoshima, BSMT(ASCP)SC, CLS; [email protected]

2 |

http://www.youtube.com/watch?v=FX30K9alQiU

“…sail in a little bituncharted waters of QC in semi-quantitative assays in immulogy and serology”

What They Do, And Their Goals

4 |

Even though 1/2 are S/CO quantitative and1/2 are qualitative, can use QC principles because qualitative tests have “a number behind it”

Why Their Lab Uses Software

Why They Use Software

• Results for their QC has been ‘online’ for chemistry for a long time but for immunology ‘has been a big improvement’

• Can capture, in a single place, different QC & Levels• Can set reject and warning rules for those combos• Remarks recorded for maintenance, et cetera• Same units between multiple, same instruments• Comparison of their values with peers, using same units• Able to set analytical goals, different for each test• Documentation and reports available (review and inspections)

What Rules Do They Use?

• Use 1-3s, 7Trend for most tests• 1-2s or 2-2s for some others• ‘No benefit to use sophisticated rules’• Capture all data for retrospective review/Inspections

Special Aspects of QC in Serology

• Existence of negative values; two measurements, QC below cutoff

• Definition of cutoff values, leading to false pos/neg values, which are different from just a number reported to clinician.

• Lack of primary standards, leading to variation in values• Use of arbitrary units by manufacturers = no comparison• No specifications for biological variation

• Example: If cutoff for CMV IgG is 7 IU/L, patient’s first visit value is 10 and subsequent visit is 15, is this value reflective of re-infection, reactivation or just biological variation?

Lack of Standardization

• Technical and cost issues (cross-reactivity, donors)• Regional variation of antigens• Some kits use old standards• Standards should use relevant isotypes – IgM, IgA• No standardization across platforms• Though molecular testing was proposed as a ‘standard’ for

serological testing, test values are not interchangeable

Different Control Materials Necessary

• In-Kit Controls• 3rd Party Commercial• If values of above are far from medical decision level, may

need to use Pooled patient sera, with drawbacks• Specific ‘important’ sera or seroconversion panels

So, no single standard, no reference analyzer….should one judge by most prevalent instrument/kit mean or compare to all methods mean?

Quantitative Values, Yet Qualitative Report

Another Example, Values & Interpretation

EQAS sample results forPertussis IgM: 2+, 2_!

Serological Testing and Diagnosis

ID from culture or +PCR for diagnosis

Reagent variation, day to day or by lot, can affect patient outcome

Patient positives, two reagent lots, over time

Using patient pop. values:

1650=20%

1914=10%

She indicated that variation could be due to the season

QC Materials Matrix

Mira’s Conclusion:

“….quality control in immuno-nology-serology remains a bit of art and not so science.”

http://www.youtube.com/watch?v=A7tWVUzmOic

“Risk Management”

Not QC, but Risk Management in ID Testing

QC is based on risk in your lab, your context…If averse to risk, may miss opportunities

Steps in Risk Management

Risk Management (Potential of wrong/delayed/no result)• Identification (What)• Analysis (Probability/likelihood, Impact, Proximity/Time)• Risk Response (Actions – corrective or preventative)• Monitoring Actions (Complaints, repeat tests, et cetera)

Risk should be documented in the form of:• Condition: “There is a risk that….”• Cause: “It is caused by….”• Consequence: “That may result in….”• Context: “….”(for patient population affected)

Fishbone Chart, Testing

What parts of the testing conditions may result in anincorrect test result?

Analytical Phase

Flow of testing, Screen to Confirmation

Close S/CO Ranges for Interpretation

A quantitative value, S/CO, is interpretedas NEG, IND, or POS

Since NEG/POS values are so close,if avoidance of false + important, more QC at lower level

Sample Outcomes After Confirmation

Green = confirmed POSRed = confirmed NEGYellow = confirmation unknown

Savings by Eliminating Confirmation?

To save cost, may simply report initial > 11 as positive,but risk is that a few may actually be negative

Why False Positive Might be ‘Better’

To reduce risk, may run QC in the 11 range, to ensure thatis correct; may compare with other labs to verify that >11 = POS

False pos less damaging, as confirmation follows

Antibody levels may vary over time

Case Study #1: Same Patient, Variable Results Over Time

Same kit, same patient, over timeread as POS, NEG, IND

Tough, because hematology wanted CMV IgG status for transplantation

What Happened?

What Happened?

IgG after infection

IgG level waned

after IgGtransfusion

IgG level waned

‘A man with two watches doesn’t knowhow late it is’ = single point measurement unreliable

Case Study #2, Variable Patient Results

Identification of condition: Patients appeared to changeCMV Ab status, lab conclusion = serostatus indeterminate

Note: Hematology worked closely with clinicians -phone calls daily, weekly meetings

What They Discovered in the Process

CAUSE: •Donor samples were submitted to the lab under the ID#of the patient that was to receive unit•CMV IgG results went directly into patient registration database•Latest results overrode previous ones•Though note on reports indicated ‘Indeterminate Results’, clinicians did not read it

What Actions Were Taken

Consequence/Actions:• Went to a patient ID system that differentiated from donors• Disconnected PID from registration system• For ‘indeterminate’ results, lab talks directly to patient’s clinician;in an academic setting, this can be difficult; note of this in chart• In a general clinical lab, single result seen, no serial values=risk

Risk Assessment May Be Subjective

Risk analysis affected by emotions,

Dependent on test situation

“We never make mistakes in the lab” (analytical phase, shown to have low error component rate in studies)

A More Objective Way to Assess Risk

Talk With All Personnel to Determine Impactand Probability

Objectively balances‘quality, price, time’

Impact doubles, solow probability canstill be a ‘red’situation

Tools For Troubleshooting St. John’s, Newfoundland, CanadaMike Toyoshima, BSMT(ASCP)SC, CLS; [email protected]

Tools For Troubleshooting Out of Control Conditions

• How Do You Detect These?• Immediately - QC Out of Control Rule Violations• Upon review of QC Reports

68.0%95.5%99.7%

Arriving at Decision Limits

s = SD = variability

§99.7%

Gaussian Distribution – the Key

68%95.5%99.7%If +/- 1S ~ 2/3 in; 1/3 out

If +/- 2S ~ 19/20 in; 1/20 outIf +/- 3S ~ 99.7/100 in; 0.3/100 = 3/1000 ~ 1/333 out - rare

Troubleshooting and Westgard Rules, Recap

• 12S: Warning, initiate further testing• 13S: Rejection, large RE, or SE start• 22S: Rejection, SE• R4S: Rejection, RE• 41S: Warning, SE• 10X: Warning, SE

Immediate QC Failure, Troubleshooting Information

• Identify source of error• Consider if other analytes affected• Localized to a particular incubator, wavelength,

reagent/sample dispense • Identified and documented from a previous service or

problem log?

Immediate QC Failure, Random Error Investigation

• Look at charts to see if trending or one time• Consider if values out have been deleted• Related to a change in test conditions?

• Do a quick precision check• Compare with within run imprecision from IFU• Single occasion outlier?

Immediate QC Failure, Systematic Error Investigation

• Look at charts to see if trending or one time• Consider if values out have been deleted• Related to a change in test conditions?

• Verify the bias• Alternate Lot of QC material with values? • Proficiency material, thawed• Alternate Calibrator lot, not what you used previously• If related to new cal or rgt, repeat patient samples• CALL your QC program!

Immediate QC Failure, Systematic Error Investigation

• If mechanical/instrument related• Consider if other analytes affected (sample size, UV)• Electrical (too low or varied? Line voltage conditioner)

• Verify the bias• Alternate Lot of QC material with values? • Proficiency material, thawed• Alternate Calibrator lot, not what you used previously• If related to new cal or rgt lot, repeat patient samples

QC Report Interpretation,CVR and SDI to Troubleshoot

• CVR - Indicator of Random Error / Imprecision• Lab CV / Peer CV• Ideal CVR is </= 1.0

• SDI - Indicator of Systematic Error / Bias• [Lab Mean – Peer Mean] / Peer 1 SD• 3 variables = 3 sources of error• Ideal SDI is 0.0

Investigating• Look at your CV – compare month vs cumulative.

Are they the same or increased? Before / After:Lab CV / Peer CV = 5.0% / 5.0% = 1.0 CVR BEFORELab CV / Peer CV = 10.0% / 5.0% = 2.0 CVR AFTERLab CV / Peer CV = 5.0% / 2.5% = 2.0 CVR AFTER

• Did your CV increase or your peer decrease?

Increased CVR on Your Report

Possible Reasons for INCREASED CVR:• Instrument / Test System Example, Coag:

• Mechanical• Electrical

• New Reagent Formulation / New Calibrator Lot• Matrix Effects – mix of old and new QC

during a single month

Elevated CVR, Sources

Investigating an INCREASED CVR:• Look at your QC – any other related analytes high?• Mechanical• Weak pump tubing on MLA• Leaking sample syringe on small volume tests

• Electrical • Weakening UV bulb; light output affects analyte value

Increased CV, Mechanical or Electrical

Why does this INCREASE CVR?• If the change is small = no rejections, but monthly CV

is a mix of old & new values, may increase CV for crossover month

• If change is larger, and some/many values are rejected in your lab, again CVR will be/may be increased, again for crossover month

• What happens if other labs are switching too?

Increased CV, New Reagent or Calibrator Lot

Investigating • Consider your monthly versus cumulative CV. Same

or greater?• If the same, did your peer group decrease?• What happens to CVR if other labs are switching

too? What happens if over different times depending on distribution/release of new formulation or calibrator?

Increased CVR, New Reagent or Calibrator Lot

Investigating• Look at your CV – compare month vs cumulative. Is

it the same or decreased? Before / After:Lab CV / Peer CV = 5.0% / 5.0% = 1.0 CVR BEFORELab CV / Peer CV = 2.5% / 5.0% = 0.5 CVR AFTERLab CV / Peer CV = 5.0% / 10.0% = 0.5 CVR AFTER

• Did your CV decrease or your peer increase?• If your peer increased, how is this for other methods?

So, if your lab uses EIA, how about Immunoturbidimeric?

Decreased CVR on Your Report

Investigating • Consider your monthly versus cumulative CV. Did

you improve?• If the same, peer group got larger• Why would this happen?

Decreased CVR, Causes

Possible Causes & Call to QC Program• Peer labs may be submitting wrong level values or

have incorrect unit coded (Calcium, 8.6 mg/dL = 2.5 mmol/L)

• Mix of old & new reagent/calibrator values• For elevated peer CV, determine if limited to

particular lab or instrument family• Check another QC lot to see if group CV elevated

Increased Peer CV

Internal QC Data (ID=994) — One-Way ANOVA Plot

M T W T F M T W T F M T W T F M T W T F M T W T FWeek 1 Week 2 Week 3 Week 4 Week 5

1820

2224

2628

3032

X L

X122

3 Fe

rritin

, ng/

mL

-3-2

-10

+1+2

+3

Your lab’s CV

Peer CV

One Lab’s CV vs Peer CV

Possible Causes• Instrument or test system• Reagent Lot Change, Reformulation• Calibrator Lot Change with New Values

Increased SDI

Instrument / test system, Mechanical or Electrical• Look for changes in:• QC, Repeat Proficiencies, Alternate lots of Calibrator• Repeated patient samples

• Improve by:• Maintenance or repair• Decreasing electrical or environmental effect

Increased SDI

Calibrator Value Reassignment• Look for changes in:• QC, Repeat Proficiencies, Alternate lots of Calibrator• Repeated patient samples

• Reconcile by:• Change in reference interval (normal range)• Change in therapeutic range

Increased SDI

Reagent Reformulation/Matrix Effects• Look for changes in:• QC, Repeat Proficiencies • Repeat of stored proficiencies• Alternate lots of calibrator, run as patients• Repeated patient samples not affected

• Reconcile by:• Contacting QC Program to see if new code available, use it to segregate ‘different’ QC values

Increased SDI

• Lab called, L1 QC data rejected, ALT, SieDim Vista• Using new, calibrated kit, ALTI, as of 6/12 data• Their lab getting ALTI values of 28 / 183 U/L• Using Unassayed Multiqual, no values

• Steps to investigate• Look at insert, but of ASSAYED Product• Look at peer group data • Call QCProgram to view data & contact labs

Rejected Data Investigation (RDI)

• Look at control insert, Assayed control for hints:•

RDI – Control Insert Info from Different Lot

ALT, UV with P5P, L1/L3 = 28.4 / 197 U/LALTI, UV with P5P, L1/L3 = 26.7 / 191 U/L, IFCC Ref Procedure, Calibr.ALT, UV with P5P, L1/L3 = 21.5 / 180 U/L, Corr Factors to ~ IFCC Ref Proc

• Manufacturers Report, Dade (old name)•

RDI – Peer Group Data

If labs run 2 levels per day, and labs run 1 & 3, why so few L1?

• Manufacturers Report, Dade (old name)•

RDI – So, What Happened?

Preliminary 8/12 data, now L1/L3 = 27/184 U/L!

• Manufacturers Notification:•

Help From Manufacturer

• Manufacturer’s Notification, excerpts:•

Help From Manufacturer 2

• Looking at values:• UV with P5P group, Dedicated, L1 = 28.4 U/L• Applying correlation factors: (28.4 x 0.949) – 5.033 = 21.9

• QC Program actions:• Contacted labs with low values, OUS• Verified they were using correlation factors• Advised to change reagent code to Factored

RDI – Resolution

• Unity Bulletin Item for QC submissions:

••

RDI – Resolution 2

Coding Example & A Question

1. Instrument model breakout – Siemens Dimension RxL and Xpand, e.g. Urine Cl, UN, Creatinine

2. Can an SDI be too small?

Internal QC Data (ID=994) — One-Way ANOVA Plot

M T W T F M T W T F M T W T F M T W T F M T W T FWeek 1 Week 2 Week 3 Week 4 Week 5

1820

2224

2628

3032

X L

X122

3 Fe

rritin

, ng/

mL

-3-2

-10

+1+2

+3

Your lab’s CV

IncreasedPeer CV, larger peer SD

Same Slide, New InterpretationSince SDI =[Lab Mean – Peer Mean] / Peer 1 SD, IF peer SD larger, lab SDI gets SMALLER!

• Linear Regression for Method Comparison

• Hints from EP9-A2•

EP9A – Linear Regression

Accuracy Testing - CLIA

• Interpretive Guidelines: 493.1253(b)(1)(i)(A):

• “Lab is responsible for verifying that the method produces correct results. Verification of accuracy may be accomplished by:”

• Testing reference materials;• Comparing lab test values versus a reference

method;• Comparing split sample results obtained from a method

with clinically valid results

• CHM.13800 Phase II

• If the laboratory uses more than one instrument to test for a given analyte, the instruments are checked against each other at least twice a year for correlation of patient/client results.

•NOTE: This requirement applies to tests performed on the same or different instrument makes/models or by different methods. This comparison must include all nonwaived instruments/methods.

•CAP, Revised 6/15/09

Linear Regression - Uses

• Study of Two Methods to Describe their Relationship

• Compare new method to old• Compare two instruments• Compare new and old reagent formulations

• Two, New Errors – Constant & Proportional

Linear Regression - Uses

Linear Regression• Constant Error

Caused by Interferent. Affects accuracy (bias) throughout range, seen as change in Y-Intercept

0

250

500

750

1000

0 250 500 750 1000

EXPECTED VALUES

ACTU

AL

VA

LUE

S

}Constant Error

Actual results

Line of identitySlope, m = 1.00

Look for: Parallel lines, but one higher

Linear Regression• Proportional Error

Varies the Y value as the concentration of X increases. The results are seen as a change in the slope from 1.00

0

250

500

750

1000

0 250 500 750 1000

EXPECTED VALUES

ACTU

AL

VAL

UES

Line of identitySlope, m = 1.00

Actual results

}Proportional

Error

Look for: Change in slope of the line

Slope = 1.10,

10% higher

Positive Proportional Error

Slope = 0.90,

10% lower

Negative Proportional Error

Linear Regression –Comparison Studies

• Perform the ComparisonsOld versus New, Manual versus Automated• Population sample size should be at least 40• Split specimens, in duplicate• Distribution of sample values – at least 50% should

be outside the normal (reference) range• Use fresh specimens, when possible• Test in a narrow time window, daily• Extended testing period, minimum 5 days

Linear Regression

• Assumptions:

The original method is assumed to be precise and accurate, free from interference. So, variation from the line of identity due to proportional or constant error is assumed to be due to the new method

Linear Regression• X Axis

Old, Reference, “A”• Y Axis

New, Comparative, “B”

Old, Reference

New

, Com

para

tive

• Test Yields: Line equation Y = mX + bCorrelation N, number ofcoefficient, r tested pairsSy/x

Linear Regression

Linear Regression – 2 Tips• >50% points outside Reference Range

• All points same weight

Linear Regression –What You’ll See

Y = mX + b ErrorY = 1.00(X) + 0. Y = X, identical (None)Y = 1.05(X) + 0. Y is 5% > X (Proportional only)Y = 1.00(X) + 5. Y = X + 5 (Constant only)

• r = correlation between methods; 1.00 is best• S y/x = random error between methods; smaller is

better• Least squares regression


Best

Fit

Residual

Linear Regression –“Least Squares”, Here’s Why


Best

Fit

Residual

Deming Regression –Both Systems might have error

Consider Deming Regression

Residual is perpendicular to the line!

Method Comparison – Bias Plot


X axis the same; Y axis is the difference between Y and X

Y - X

X


Fluke, or more points needed at this concentration?

Linear Regression

Courtesy of Paul Durham,

Siemens (formerly DPC)

Popular Kit

% Change?

Higher or lower?

Linear Regression

‘Reference Method’

% Change?

Higher or lower?

Linear Regression

New vs Now

Look for this

% Change?

Higher or lower?

Linear Regression

Linear Regression Tips

• Suggestions:

• Adequate Number of Samples• Appropriate Range of Values• Watch for Outliers• Tested Range Must Fit Your Patients• CALL with any questions!

Linear Regression – Case StudyEstradiol II

• Initial notification – reformulation with:Improved Low End Precision Improved Low-End Sensitivity

• Crossover instructions:1. Run samples using original reagent2. Remove existing E2 assay from disk3. Install new reagent, recalibrate4. Analyze same previous samples

SLOPE = 1.00; What % proportional error?

Original Regression Statistics

• Estradiol II, Linear Regression• E2 II versus E2 1st Generation (from Manufacturer): • Y = 1.001(E2) – 10.25; R=0.981; N=120• Range (pg/mL): 0 – 600

• If E2 = 100, then Y = 1.001(100) - 10.25= 89.9• If E2 = 600, then Y = 1.001(600) - 10.25 = 590

• My lab called…..so I called Manufacturer

Manufacturer’s Supportive Data –Initial Linear Regression

1.1836X

Positive or Negative?

What % change?

Revised Linear Regression Data

All points have the same weight – why was it reformulated?

Original Regression, Revisited

See points >300

Remember the

CLSI suggestion:

At least 50% points outside ‘normal’?

Recall all points have the same ‘weight’

Original Regression Statistics

Remember My Tech’s Complaint?

What may really happen

Our Level 3

Consider Constant andProportional Errors

Linear Regression Tips

• Conclusions:• Systems must be comparable• Look for exact new vs. old regression• Compare reported change with yours• Compare well? Experiment successful• Don’t compare well? CALL Manufacturer. Ask for

revised regression data

If y = 0.94x, what percent change is this? Higher or lower than X?

Linear Regression Quiz

Linear Regression Update

New Dimension ALTI, DF143




Important:Linear regressionsdone using humansamples; QC may differ


All lots now FC2083+

•A graphic representation of the probability of rejection versus increasing error.

•Two types, one for RE, one for SE

• Program Variables• Number of control levels• Magnitude of error, SE or RE• Number of runs

Power Function Graphs andQuality Control Planning

Power Function Graph (SE)

0.00.10.20.30.40.50.60.70.80.91.0

0.00 1.00 2.00 3.00 4.00

Systematic Error (ΔSE, multiples of s)

Prob

abili

ty fo

r Rej

ectio

n (P

)

Probability of rejecting runs having systematic errors when using multi-rule procedures with N of 4 .

13S/22S/R4S/41S/10X 4 3

13S/22S/R4S /41S 4 1

N R

SDI

Power Function Graphs, In General


0.00.10.20.30.40.50.60.70.80.91.0

0.00 1.00 2.00 3.00 4.00


Prob

abili

ty fo

r Rej

ectio

n (P

)

Probability of rejecting runs having systematic errors when using multi-rule procedures with N of 4 .

13S/22S/R4S/41S/10X 4 3

13S/22S/R4S /41S 4 1

N R

SDI

Increasing Error

Increasing Rejection

Power Function Graphs, SE

Power Function Graph (RE)

0.00.10.20.30.40.50.60.70.80.91.0

1.00 2.00 3.00 4.00

Random Error (ΔRE, multiples of s)

Prob

abili

ty fo

r Rej

ectio

n (P

)

Probability of rejecting runs having random errors when using multi-rule

13S/22S/R4S 4 1

13S/22S/R4S 2 1

N R

CVR

Increasing Rejection

Increasing Error

Power Function Graphs, RE

• 12S, SEWhen SE = 0, RE = 1.0 and N = 1, there is a 5%chance of rejection when the system is in a steady state Power Function Graph (SE)

0.00.10.20.30.40.50.60.70.80.91.0

0.00 1.00 2.00 3.00 4.00


Prob

abili

ty fo

r R

ejec

tion

(P)

12S 4 112S 3 112S 2 112S 1 1

N R

When N=2 or 4, the probability of rejection rises to 9% and 19%, respectively.

What can be said about the 1-2s?Power Function Graphs, SE

• 12S, SEWhen SE = 0, RE = 1.0 and N = 1, there is a 5%chance of rejection when the system is in a steady state Power Function Graph (SE)

0.00.10.20.30.40.50.60.70.80.91.0

0.00 1.00 2.00 3.00 4.00


Prob

abili

ty fo

r R

ejec

tion

(P)

12S 4 112S 3 112S 2 112S 1 1

N R

When N=2 or 4, the probability of rejection rises to 9% and 19%, respectively.

It is sensitive, up to 20% false rejections!Power Function Graphs, SE

• 12S, REWhen SE = 0, RE = 1.0 and N = 1, there is a 5% chance of rejection when the system is in a steady state


0.00.10.20.30.40.50.60.70.80.91.0

1.00 2.00 3.00 4.00


Prob

abilit

y fo

r Rej

ectio

n (P

)

12S 4 112S 3 112S 2 112S 1 1

N R

When N=2 or 4, the the probability of rejection rises to 9% and 19%,

What can be said about the 1-2s RE?


• 12S, REWhen SE = 0, RE = 1.0 and N = 1, there is a 5% chance of rejection when the system is in a steady state


0.00.10.20.30.40.50.60.70.80.91.0

1.00 2.00 3.00 4.00


Prob

abilit

y fo

r Rej

ectio

n (P

)

12S 4 112S 3 112S 2 112S 1 1

N R

When N=2 or 4, the the probability of rejection rises to 9% and 19%,

Same Pfr so cannot distinguish RE from SE with this rule

QUESTION: If there is a 5% chance of rejection when the system is in a steady state for a single level, ~ 9% with a bilevel control, what does the Tech see when running a daily QC panel of >30 analytes on a chemistry instrument’s menu? What happens if the QC > 60 analytes?

Another look at false rejections with 1-2s….

Power Function Graphs

13S, SEWith SE = 0, RE = 1.0 and N = 1, the probability of rejection is low. Not sensitive to false error detection


0.00.10.20.30.40.50.60.70.80.91.0

0.00 1.00 2.00 3.00 4.00


Prob

abili

ty fo

r Rej

ectio

n (P

)

13S 8 113S 6 113S 4 113S 3 113S 2 113S 1 1

N R


13S, REWith SE = 0, RE = 1.0 and N = 1, the probability of

rejection is low. Not sensitive to false error detection


0.00.10.20.30.40.50.60.70.80.91.0

1.00 2.00 3.00 4.00


Prob

abili

ty fo

r Rej

ectio

n (P

)

13S 8 113S 6 113S 4 113S 3 113S 2 113S 1 1

N R


Comparison 12s vs 13s

Is 13s always better than 12s? Not if you must detect error where the (red) arrow is

12s

13s

Multirule, SEIf N is “fixed,” then the rejection rate increases with the number of runs on SE WHY?


0.00.10.20.30.40.50.60.70.80.91.0

0.00 1.00 2.00 3.00 4.00


Prob

abili

ty fo

r Rej

ectio

n (P

)

13S/22S/R4S/41S/10X 4 313S/22S/R4S/41S/10X 2 513S/22S/R4S/41S 4 113S/22S/R4S/41S 2 213S/22S/R4S 2 1

N R


Multirule, REWith RE graphs the rate increases with N

(levels of control) – WHY?Power Function Graph (RE)

0.00.10.20.30.40.50.60.70.80.91.0

1.00 2.00 3.00 4.00


Prob

abili

ty fo

r Rej

ectio

n (P

)

13S/22S/R4S/41S 4 1

13S/22S/R4S 2 1

N R


• Repeat the Control• If out 1 out of 20, then in 19 out of 20, right? • If rules chosen properly, less need to repeat• See next section

• Open a New Bottle of Control• Proper preparation & storage – training• Expensive

• Recalibrate• Introduces bias, may mask other problems

From “QC - The Out of Control Problem”, Elsa Quam, Westgard.com

Bad Habits of Quality Control

• View your charts for patterns• Relate the error pattern to possible causes

• Consider common causes on multichannel analyzers (same filter, small sample, etc)

• Relate the problem to recent changes• Verify the solution then document the remedy• Regularly review your quality system

From “Unity Real Time Reference Guide for Expert QC Data Management”

Good Habits of Quality Control

Tools For QC Planning St. John’s, Newfoundland, CanadaMike Toyoshima, BSMT(ASCP)SC, CLS; [email protected]

Tools For QC Planning –Analytical Goals

• Imprecision, using Biological Variation• Used to set the range if you provide the mean value• Uses published BV information• You select Minimum, Desirable, Optimal • You provide mean based on history or enter manually

Performance Goals-Definitions

• Optimum• Tests for which desirable performance limits may

be considered too liberal or easily obtained• Desirable

• Most widely used and generally accepted quality specification

• Minimum• Tests for which technology is not yet able to

achieve desirable limits

Controlling laboratory imprecision using biological variation: formulas for possible targets (choices) based on a performance goal selection

Optimum: CVA < 0.25 CVw

Desirable: CVA < 0.50 CVw

Minimum: CVA < 0.75 CVw

Better performance expected

Imprecision and BV, Use CVw (or CVi)


• Imprecision, using Biological Variation

Controlling laboratory bias using biological variation: formulas for possible targets (choices) for analytical bias

Optimum: BAA< 0.125 (CVw2 + CVb

2)1/2

Desirable: BAA<0 .250 (CVw2 + CVb

2)1/2

Minimum: BAA< 0.375 (CVw2 + CVb

2)1/2

Better performance expected

Bias and BV, Uses CVw (or CVi) & CVb (or CVg)

Definition of Total Error based on biological variation

• TE p<.01= (imprecision BV target)(2.33)+|bias BV target|• the imprecision BV target for the test• some fraction of within-subject biological variation

• the fraction is determined by choice of minimum, optimum or desirable performance

• multiplied by 2.33• plus the bias BV target for the test• some fraction of within-subject and between-subject

biological variation

KEY TOOL – Table of Area Under a Normal Curve

*2.33 = 0.01 = 1% probability of values outside of limit = 99% probability ‘in’

Z-value Area

0.00 0.5000000.50 0.3085381.00 0.1586551.50 0.0668072.00 0.0227502.50 0.0062103.00 0.0013503.50 0.000233

1.65 = 0.05 = 5%


• Total Error, Using Biological Variation• Based on published ‘within-subject’ and ‘between-

subject’ data• Used to set the range, using your bias & imprecision • Uses published BV information• You provide Min, Des, Optimal • You provide the TEa specification


• Total Error Using Biological Variation


• State of the Art• Based comparator group imprecision data

• Peer• Measuring Method• All labs • Or, you may manually enter the %CV

• Used to set the range if you select the mean • 30 day,60 day or cumulative• You may manually enter the mean


• State of the Art• Targets lab performance to the imprecision of a

comparator group – Instrument, Method, or All Labs• Review numbers of labs in each comparator group –

larger numbers at the peer level best


• State of the Art• Using your mean, apply CV of Peers, Method, or All Labs to

obtain range:• Lab mean = 100 mg/dL, Peer CV = 3%• Then, 1 SD = 3 and 2SD range is 94 – 106• Lab mean = 100 mg/dL, Method CV = 5%• Then, 1 SD = 5 and 2SD range is 90 – 110• Lab mean = 100 mg/dL, All Labs CV = 7%• Then, 1 SD = 7 and 2SD range is 86 - 114


• Imprecision Using State of the Art

Can select Peer, Method, or All

Can select period of comparator data


• Medical Relevance• Amount of error that would cause a clinician to change

a patient’s diagnosis, prognosis or therapy• Use to differentiate a statistical from a medically

important change • Must be determined by laboratory director• Expressed as percent or absolute values

Medical Relevance

• How Does It Work?• Sets a limit based on medical importance,

determined clinically• What Does it Mean?• A tool to help assess overall (total) error• Alert when TE exceeds limits; useful for tests where

statistical limits are more sensitive than medical requirements

Medical Relevance - Application

1. Take target value from lab’s 30 or 60 day rolling, OR cumulative mean

2. Set a Decision Limit, concentration or %3. Develop a range, example:

Lab mean = 100 mg/dL; Decision Level = 10%Range = 90-110 mg/dL

Medical Relevance – Sample Calculation

• Example: Potassium• Set a Decision Limit, clinically:

e.g. Absolute value = +/- 0.5 mmol/L• Develop a range

Lab mean = 5.0 mmol/L, thenRange = 4.5 – 5.5 mmol/L by MR

Medical Relevance – Sample Calculation & Comparison

Potassium example, continued:• Normally derived QC: 1 SD = 0.06 mmol/L. Then, 3 SD = 0.06 x 3 = 0.18 and range ~ 5.0 +/- 0.2 = 4.8 – 5.2 mmol/L• Using MR, from previous slide range = 4.5 – 5.5 mmol/L• Traditional QC narrower than that obtained by Medical Relevance, even though +/- 3 SD!

Example TEa (= TAE) for Glucose

Compare TEa for Glucose

• From BV, using 5% probability, TEa = 7.9%• At a mean value of 200 mg/dL, Limit +/- 16

• From CLIA TEa, limit is 6 mg/dL or +/- 10%, whichever is greater. Using 10% and a mean value of 200 mg/dL, Limit +/- 20

Article: Managing Quality in Networked Laboratories…Westgard QC 1/2012

• Conditions Within the Network:• 9 Siemens ADVIA Chemistry models• 7 Siemens Centaur XP• Initial Rule, 1-2s, N = 2• Chemistry run hourly• Immunoassay, 3 times / day

Testing and Evaluation Conditions

• Quality Requirements for 71 Tests• Blood and Urine Chemistries• Defined at clinical decision concentrations• In terms of Total Allowable Error (TEa), CVa (allowable

imprecision), Ba (allowable Bias)• Specs determined by Biological Variation (46), Pharmaco-

kinetics (9), and Expert/EQA (16)

Here’s What They Found

• QC Combinations:• 35 of 71 tests controlled by a single rule• 30 of those tests used BV for their TEa• 32 analytes now at 1-2.5s or 1-3s, N = 2• 8 of 71 had 1-3s/2-2s/R-4s/4-1s, N = 4• For 11 of 71, 8 mean rule added • 14 of 71 needed site-specific rules

• Establish comparability of laboratory QC results with other laboratories using the same method = evaluation of bias and imprecision

• Evaluate long term trends (if present) within a given method, instrument, reagent or control

• Detect performance changes in instruments/reagents

• Become Aware of “between - method variations”

• Educational aspects• Trace the effects of corrective actions• Access peer group data, educational material, discussion

forums, and product inserts via the Internet

Benefits of Daily QC and Interlaboratory Comparison

Documents

Infectious Disease Testing: QC and Risk Management Johns, Inf Dis, AG... · 4/12/2010 · Lab CV / Peer CV = 5.0% / 5.0% = 1.0 CVR BEFORE Lab CV / Peer CV = 10.0% / 5.0% = 2.0 CVR