1

ModuleModule

PANEL TESTINGPANEL TESTING

High level

2

Content OverviewContent Overview

• What is panel testing?• What is panel testing used for? • PT advantages and disadvantages• Preparation of test smears• Validation of panel batches• Panels’ composition• Organization of a panel testing round• Analysis of results; scoring system• Interpretation and feedback• Forms

3

What is Panel Testing?

• One of EQA methodsOne of EQA methods

• System of sending stained and/or System of sending stained and/or unstained smears from NRL to unstained smears from NRL to peripheral laboratories to check peripheral laboratories to check proficiency in performing AFB proficiency in performing AFB smear microscopy and reporting smear microscopy and reporting AFB resultsAFB results

• Tests individual performance of a Tests individual performance of a laboratory worker, not the laboratory worker, not the laboratory overalllaboratory overall

4

• Rapid assessment of performance of a laboratory Rapid assessment of performance of a laboratory staff to prioritize training and supervisory activitiesstaff to prioritize training and supervisory activities

• Quick detection of problems associated with very Quick detection of problems associated with very poor performancepoor performance

• Evaluation of competency of laboratory technicians Evaluation of competency of laboratory technicians prior to and following trainingprior to and following training

What is Panel Testing Used For?

5

• A minimal step for EQA with limited resources A minimal step for EQA with limited resources

• Monitoring performance of individuals in Monitoring performance of individuals in absence of a rechecking programabsence of a rechecking program

• Supplements rechecking programsSupplements rechecking programs• Investigation of excessive errors found in recheckingInvestigation of excessive errors found in rechecking

What is Panel Testing Used For? (2)

6

• Low workload for a peripheral centerLow workload for a peripheral center

• Improves laboratory credibilityImproves laboratory credibility

• Rapid response countrywide possibleRapid response countrywide possible

• Possible identification of gross deficienciesPossible identification of gross deficiencies

• Use of stained and unstained smears can Use of stained and unstained smears can help identify source of problemhelp identify source of problem

Advantages of Panel Testing

7

• Technicians know they are being evaluatedTechnicians know they are being evaluated

• Does not measure routine performanceDoes not measure routine performance

• High workload for NRL High workload for NRL

• need forneed for additional resources -additional resources - appropriate equipment, appropriate equipment, highly highly qualified staff to produce panelsqualified staff to produce panels

• a system for panel sets distribution, data collection, a system for panel sets distribution, data collection, analysis analysis and feedbackand feedback

• May not be motivating to improve daily May not be motivating to improve daily performanceperformance

Disadvantages of Panel Testing

8

Preparation of Test Smears

• Routine patient smears:• Problems with consistency• Only stained smears available• Smears of the required AFB quantification may not

be easily available in the needed quantity

• Specially manufactured smears:• Can be stained or unstained • Provide uniformity of the test for technicians• Provide known quantification of AFB required

9

Manufacturing Smears for PT

• Should be done at NRLShould be done at NRL• requires a safety hood, centrifuge, requires a safety hood, centrifuge,

vortex, water bath, lab supplies vortex, water bath, lab supplies (pipettes, tubes, slides, boxes etc.)(pipettes, tubes, slides, boxes etc.)

• BIOSAFETY MEASURES!BIOSAFETY MEASURES!

• Smears are prepared from known Smears are prepared from known positive and negative sputapositive and negative sputa

• Reference for the manufacturing Reference for the manufacturing procedures: procedures: AFB Smear Microscopy EQA Guidelines

• Requires time practice and expertise

10

Validation of Panel Batches

• Mandatory requirement!

• Pre-validation:• Validation of consistency of panel batches prior to

sending test panels out to periphery

• Post-validation:• Validation of panel slides / batches after receiving

aggregate results from all laboratories

• Keep accurate records of batches prepared Keep accurate records of batches prepared and detailed results of the validation processand detailed results of the validation process

11

Pre-validation

• Stain at least 6 slides from each batch to be Stain at least 6 slides from each batch to be examined independently by 3 or more examined independently by 3 or more technicianstechnicians

• Calculate the average results and standard Calculate the average results and standard deviation (SD)deviation (SD)

• The average minus 2 SD should be > 0 to The average minus 2 SD should be > 0 to accept the batchaccept the batch

• Use the Validation Log to record resultsUse the Validation Log to record results

12

Normal Distribution

• All values are All values are symmetrically symmetrically distributed around distributed around the meanthe mean

• Characteristic “bell-Characteristic “bell-shaped” curveshaped” curve

• Assumed for all Assumed for all quality control quality control statisticsstatistics

Fre

qu

ency

x Variable

13

What is Standard Deviation?

The principle calculation used in the laboratory to measure dispersion of a group

of values around a mean

Standard Deviation – Statistical FormulaStandard Deviation – Statistical Formula

1

)( 2

n

XXiSD

14

Standard Deviation and Probability

• For a set of data with a normal For a set of data with a normal distribution, a value will fall distribution, a value will fall within a range of:within a range of:• +/- 1 SD 68.2% of the time+/- 1 SD 68.2% of the time• +/- 2 SD 95.5% of the time+/- 2 SD 95.5% of the time• +/- 3 SD 99.7% of the time+/- 3 SD 99.7% of the time

• Laboratories use the +/- 2 SD Laboratories use the +/- 2 SD criteria for the limits of the criteria for the limits of the acceptable range for a control acceptable range for a control valuevalue

• When the QC measurement When the QC measurement falls within that range, there is falls within that range, there is 95.5% confidence that the 95.5% confidence that the measurement is correctmeasurement is correct

68.2%

95.5%

99.7%

Fre

qu

ency

-3s- 2s -1s Mean +1s +2s+3s

X

15

Sample Form: Validation Log for AFB Panel Testing Slide Batches

(pre-validation)

1 2 3 4 5 6 Average

1 7/6/2006 17 10 10 50 15 10 11 17.7 15.96 -14.2 reject 1+

2 9/6/2006 34 0 0 0 0 0 0 0.0 N/A N/A accept negative

3 9/6/2006 40 7 2 4 3 9 2 4.5 2.88 -1.3 reject scanty

4 10/6/2006 23 5 6 2 9 12 9 7.2 3.54 0.1 accept scanty

5 15/6/2006 17 30 27 28 36 43 50 35.7 9.22 17.2 accept 1+

6 16/6/2006 30 3500 3700 1500 1700 2600 2900 2650.0 907.19 835.6 accept 3+

7

8

9

10

VALIDATION LOG FOR AFB PANEL TESTING SLIDE BATCHES

Ba

tch

№ Slide Preparation Slide evaluation

Date slides made

Number of slides

made

Slide test results (AFB per 100 fields) Standard deviation (SD)

Consistency (average minus

2 standard deviations)

ACCEPT or REJECT?

Report result

• Intended positives should never be negative• Intended negative smear should never be positive• Quantification differences should not reach 2 steps on scale

16

• The same smear error reported by a majority of technicians may represent a problem with the panel slide / batch:

• Technical difficulties in preparing panel slides• Error in the pre-validation• Incorrect recording of the expected result• Fading of smears during transportation to

peripheral sites

Post-validation

17

Logbook of Panel Slides Sets / Post-validation

Central Laboratory administering panel test:

District where panel testing is conducted:

Date slide set(s) sent to peripheral laboratories:

Slide set(s)' number(s):

1 2 3 4 5 6 7 8 9 10

1

2

3

4

5

6

7

8

9

10

Supervisor:

Slide Sets Numbers

CommentsPeripheral Laboratories' Results

Form PT2: PANEL SETS' LOGBOOK / AGGREGATE RESULTS

(Record of a set of 10 slides selected from Form PT1)

Slide No

Batch NoStained or unstained

Expected result (from Form PT1)

18

Coding of Panel Smears

• Ensure that result can not be guessed by an Ensure that result can not be guessed by an examinee – to avoid reading biasexaminee – to avoid reading bias

• Make identification of a panel smear clear to a Make identification of a panel smear clear to a supervisor in charge of a panel testing exercisesupervisor in charge of a panel testing exercise

Example of a smear’s code:

62-45-1a panel set number

a smear serial number in a panel

a batch number

19

Panels’ Composition

• The composition of a panel set is determined by NRLThe composition of a panel set is determined by NRL

• Number and types of slides to reassure that correct Number and types of slides to reassure that correct or incorrect results are not accidentalor incorrect results are not accidental

• At least 10 slides provides a valid and fair test At least 10 slides provides a valid and fair test • Batch of stained and unstained smearsBatch of stained and unstained smears

• Unstained smears:Unstained smears:• Evaluate staining technique; provide information about Evaluate staining technique; provide information about

stain preparation and qualitystain preparation and quality

20

Examples of Panel Sets Compositions

1 slide graded 3+1 slide graded 3+

1 Slide graded 2+1 Slide graded 2+

1 slide graded 1+1 slide graded 1+

2 slides graded 2 slides graded

1-9 / 100 fields1-9 / 100 fields

5 negative slides5 negative slides

1 slide graded 3+1 slide graded 3+

1 slide graded 2+1 slide graded 2+

2 slides graded 1+2 slides graded 1+

3 slides graded 3 slides graded

1-9 / 100 fields1-9 / 100 fields

3 negative slides3 negative slides

1 slide graded 2-3 +1 slide graded 2-3 +

2 slides graded 1+2 slides graded 1+

3 slides graded 3 slides graded

1-9 / 100 fields1-9 / 100 fields

4 negative slides4 negative slides

A panel test should represent a challenge in A panel test should represent a challenge in terms of difficulty:terms of difficulty:

-- some scanty and low-positive smearssome scanty and low-positive smears

21

Getting Started: Issues to Consider•System for sending slides

•Frequency of testing

•Forms to record and report results

•Time allowed for technicians to complete PT

•Availability of microscopes

•Performance criteria

•Feedback and corrective action if needed

•Mechanism to resolve discrepant results

22

Implementation of Panel Testing

• Responsibility of the NRL- from preparation of slides to Responsibility of the NRL- from preparation of slides to analysis of results and feedbackanalysis of results and feedback

• Determine the number of AFB technicians who will Determine the number of AFB technicians who will participate in PT (ensure preparation of the needed participate in PT (ensure preparation of the needed number of panels) number of panels)

• Communicate with Public Health Directors regarding Communicate with Public Health Directors regarding EQA activitiesEQA activities

• Prepare the schedule for panel testing in each locationPrepare the schedule for panel testing in each location• Collaborate with intermediate laboratoriesCollaborate with intermediate laboratories

23

Sending Slides

• Delivery system based on services, regulations, resources available:• mail/post• courier• supervisory visit

• Turnaround time

• Safe package to prevent breakage of slides:• strong plastic slide holders

24

National reference laboratory

Intermediate laboratory

Peripheral laboratories

A POSSIBLE SCHEME OF A A POSSIBLE SCHEME OF A PANEL TESTING ROUNDPANEL TESTING ROUND

Peripheral laboratories

25

Performing a Panel Test Round• Frequency: at least one to two times a yearFrequency: at least one to two times a year

• A standardized PT reporting form / an accompanying letter to A standardized PT reporting form / an accompanying letter to provide instructionsprovide instructions

• Individual, not group workIndividual, not group work

• No incentives or punitive actions as a result of the PT exerciseNo incentives or punitive actions as a result of the PT exercise

• Time allowed to complete the PT exercise, Time allowed to complete the PT exercise, maximum:maximum:• 2 hours for a stained slide set2 hours for a stained slide set• 3 hours for an unstained slide set3 hours for an unstained slide set

26

Individual Results of Panel Testing / Feedback

Form

Central Laboratory Use Only:

Test slide set No: Passsing score:

Date of sending set: Date results received:

Peripheral Laboratory:

District:

Date PT conducted:

Name of technician reading test smears:

Slide Number Result Expected result Error Type Points

Total Points: Pass / Fail:

HFP: HFN: LFP: LFN: QE:

Recommended Action:

Supervisor:

Central Laboratory Only

Feedback:

Form PT3: PANEL TESTING INDIVIDUAL RESULTS AND FEEDBACK FORM

Note: If more than one technician performs AFB microscopy in the laboratory, each technician should read all 10 smears and record their results on a separate form. Technicians should not discuss results or share forms until all results have been sent back to the central laboratory. Forms for all technicians should be sent to the central laboratory for evaluation.

27

Management of PT During a Supervisory Visit

• Administration of PT during on-site visits:Administration of PT during on-site visits:• can be effective in some circumstances can be effective in some circumstances • provides direct observation of work under PT exerciseprovides direct observation of work under PT exercise• corrective action may be easily facilitatedcorrective action may be easily facilitated• BUT: may be impractical in routine conditions BUT: may be impractical in routine conditions • can be done in a special surveycan be done in a special survey

• Important:Important: PT must not disrupt routine patients’ PT must not disrupt routine patients’ examinationsexaminations, therefore consider:, therefore consider:

• Careful planning of a supervisory visit Careful planning of a supervisory visit • Allocating sufficient time for a visitAllocating sufficient time for a visit

28

Analysis of PT Results

• A scoring system is to be developed prior to testA scoring system is to be developed prior to test

• Distinguish major and minor errorsDistinguish major and minor errors• false positive/negative related to 1+, 2+ or 3+ errors are false positive/negative related to 1+, 2+ or 3+ errors are

major errorsmajor errors• quantification errors (at least a 2 grade difference) and quantification errors (at least a 2 grade difference) and

false positive / negative errors in the scanty group (1-9 false positive / negative errors in the scanty group (1-9 AFB) are considered minorAFB) are considered minor

• Determine successful scoreDetermine successful score

• Determine plan of action for poorDetermine plan of action for poor performancesperformances

29

Result of Controller Result of Technician Negative 1-9 AFB/100 f 1+ 2+ 3+ Negative correct LFN HFN HFN HFN 1-9 AFB/100 f LFP correct correct QE QE 1+ HFP correct correct correct QE 2+ HFP QE correct correct correct 3+ HFP QE QE correct correct

Types and Classification of Errors

Correct: No errorsQE Quantification error Minor errorLFN Low False Negative Minor errorLFP Low False Positive Minor errorHFN High False NegativeMajor errorHFP High False Positive Major error

30

Example of PT Scoring

• Set of 10 slides, each slide is worth 10 points, total Set of 10 slides, each slide is worth 10 points, total possible score = 100possible score = 100• HFP and HFN scores 0HFP and HFN scores 0• LFP, LFN and QE scores 5LFP, LFN and QE scores 5

(QE = 2 grades difference) (QE = 2 grades difference) • Passing score = 80 – 90Passing score = 80 – 90

31

Analysis of PT Results

• Study the aggregate results from all laboratories Study the aggregate results from all laboratories

• Post-validate panel slides/batchesPost-validate panel slides/batches• Assure that poor performance is not due to panel slide Assure that poor performance is not due to panel slide

problemsproblems

• If a majority of technicians fail to report correct results for the same slide/batch it may represent a problem with panel slide preparation:

→ exclude this slide from scoring exclude this slide from scoring

→ check returned discrepant slidescheck returned discrepant slides

→ detect problems in preparation of panel smearsdetect problems in preparation of panel smears

→ undertake measures to improve the quality of panel undertake measures to improve the quality of panel smears preparationsmears preparation

32

PT– Interpretation of Results

• False positive and negative errors should be False positive and negative errors should be considered separatelyconsidered separately

• False positives - lack of proficiency / faulty False positives - lack of proficiency / faulty microscopemicroscope

• False negatives - poor stain / inadequate False negatives - poor stain / inadequate examination time / poor microscopeexamination time / poor microscope

33

Feedback to Laboratories on PT results

• Timely and confidentialTimely and confidential

• Individual and aggregate test results Individual and aggregate test results

• Criteria for acceptable performanceCriteria for acceptable performance

• Reports to TB program coordinator should provide Reports to TB program coordinator should provide appropriate background information and appropriate background information and recommendations and not simply scores recommendations and not simply scores

• Poor performance often requires a visit to laboratoryPoor performance often requires a visit to laboratory

34

PT Aggregate Results of Multiple Laboratories

District: District Supervisor:

Supervising Laboratory: Period PT conducted:

Panel test slide set(s): Passing score:

Peripheral Lab Annual volume SPR, %Technician(s)

participated in PTPT score HFP HFN LFP LFN QE Total errors

District Averages

Report submitted by: Date:

SPR: slide positivity rate; PT - panel testing; Annual volume - annual volume of smear examinations; HFP - high false positives; HFN - high false negatives; LFP - low false positives; LFN - low false negatives; QE - quantification errors

Form PT4: PANEL TESTING REPORT OF MULTIPLE LABORATORIES FOR DISTRICT SUPERVISOR AND NTP

35

PT Aggregate Results Report: Example

36

Conclusion:

Laboratories submitting unacceptable PT results with documented consistency and quality of PT slides experience serious problems with AFB microscopy.

Additional resources should be obtained for supervisory visits, correction of problems identified, including replacement of faulty microscopes (and/or stains), retraining if needed, and follow-up panel testing.

37

Key Messages (I):

• PT is an effective method when it is necessary to quickly obtain information about capabilities of individual laboratory technicians to read smears and report results according to standards approved by NTP.

• PT is considered to be less effective than rechecking because it does not monitor routine performance; BUT

• PT can be more effective than rechecking in the areas where prevalence of positives is low.

38

Key Messages (II):

• The main prerequisite to start a panel testing program The main prerequisite to start a panel testing program in a country is availability of a laboratory with a highly in a country is availability of a laboratory with a highly qualified staff capable to safely produce panel sets of qualified staff capable to safely produce panel sets of appropriate quality and required composition.appropriate quality and required composition.

• Validation of panel batches (pre- and post-validation) is Validation of panel batches (pre- and post-validation) is

the mandatory requirement for PT. the mandatory requirement for PT.

• A well functioning system should be established to A well functioning system should be established to distribute panels, collect and analyze data; provide distribute panels, collect and analyze data; provide timely feedback to peripheral laboratories.timely feedback to peripheral laboratories.