Cap.org v. 1 Gynecologic Cytopathology Quality Consensus Conference Working Group 4: Cytologic-Histologic Correlations June 4, 2011

cap.org v. 1

Gynecologic Cytopathology Quality Consensus ConferenceWorking Group 4: Cytologic-Histologic CorrelationsJune 4, 2011

• Barbara A Crothers, DO FCAP, Chair

• Bruce A Jones, MD FCAP, Senior Author

• Leigh Ann Cahill, CT (ASCP)CMIAC

• Ann T Moriarty, MD FCAP

• Dina R Mody, MD FCAP

• William D Tench, MD FCAP

• Rhona J Souers, MS, CAP Biostatistician

Working Group 4 Members

© 2011 College of American Pathologists. All rights reserved. 2

• Cytologic-Histologic Correlation (CHC)o CHC is first as valuable QA measure

− 4.2 out of 5 points

o Most find CHC statistical assessment worthwhile

o 94% actively monitor the correlation between Pap test and biopsy results

o 64% use CHC as part of QA reportso Corrective action for CHC is infrequent

− 15% of 133 participants

Summary of Laboratory Trends


• §493.1274(c)(2)- Laboratory comparison of clinical information, when available, with cytologic reports and comparison of all gynecologic reports with a diagnosis of HSIL+ with the histopathology report, if available in the laboratory, and determination of the causes of any discrepancies

• CAP LAP CYP.01900, .07530, .07543, .07556, and .07569

CLIA ‘88/ CAP


“This monitor should not be viewed as just an evaluation of the performance of the cytopathology laboratory. This is a true ‘system’ monitor.

The statistics represent the performance of all personnel and processes in the system of obtaining, processing, and evaluating both cytology and biopsy specimens.

Performance must be evaluated at each step of the process to identify improvement opportunities.”


Pap – Biopsy Correlation

Jones BA, Davey DD. Quality Management in Gynecologic Cytology Using Interlaboratory Comparison Arch Pathol Lab Med. 2000;124:672–681

• Traditionally, a measure of Pap test performance

• Actually represents many quality variables

• Biopsy not a gold standard

• Poor interobserver reproducibilityo ASCUS, CIN1, LSIL, CIN2

• Major cause of discordant pairs is sampling* o Both Pap test and cervical biopsyo Of FN cytology (n=1444), 85% sampling

erroro Of FP cytology (n=1527), 95% sampling error

Focus of CHC- Pap test


*Jones and Novis. Cervical biopsy-cytology correlation: a CAP Q-Probes study of 22,439 correlations in 348 laboratories. Arch Pathol Lab Med 1996;120:532-531.

• “Real Time” correlation- review of slides prior to issue of biopsy reporto Provides critical information for patient

follow-upo Resolves/confirms discrepancieso Timely report to healthcare providers

• “Retrospective” correlation- review of slides after issuance of both reportso Monitor performance and processes of

cytology and biopsy for laboratory quality improvement

Dual Role of CHC


• 67% of laboratories resolve discrepancies at biopsy sign-out and have a written policy (70%)

• 61% address discrepancies in the biopsy report

• 85% have written policy requiring retrospective CHC for statistical purposes

• 58% review cases retrospectively* (Sec 4, Table 33*)

• Bidirectional correlation was also the most commonly described CH activity in the post-survey Web-based public comment (n=13/27); only 33% (9/27) initiated correlation solely based on the biopsy result

• 59% ( n=16/27) felt optimum time for CHC was during sign out of biopsy

“Bidirectional” CHC


*Multiple answers were permitted

CHC Consensus Statement #1

Cytologic-histologic correlation may be performed “real time,” retrospectively or

both.

“Real time” CHCo Preferentially impacts immediate patient care

o Strongly preferred for HSIL Pap test/ negative biopsy regardless of the result of the review (TP or FP)

Retrospective correlationo Ease of statistical data collection

o Ease of identifying laboratory trends

Must notify provider for HSIL Pap and negative biopsy

Monitor results weekly, monthly, quarterly or annually

Justification: Literature, survey, professional opinion

Consensus Statement #1: CHC may be performed “real time,” retrospectively or both.


Laboratories should be allowed to define the process for CHC (“real time,” retrospective, or both).

A. Agree 87.5%

B. Disagree 12.5%

CS 4.1 QUESTION #37:



At a minimum, review all available slides for HSIL Pap tests with a negative biopsy.

• Recommended time interval between Pap and biopsy-within 6 months of biopsy; with 3-4 months preferred when possible

• Method for notification of care giver and documentation o May be in biopsy report

o May be in QA report

o Does not require disclosure of QA information- may choose to issue generic comment(s)

• For “real time” review, assumes that pathologist takes necessary steps to ensure adequate biopsy orientation and leveling

• Justification: Survey, professional opinion, literature

Consensus Statement #2: At a minimum, review all available slides for HSIL Pap-negative biopsy discrepancies.


• Post-survey Web-based public comment:o 77% review slides for both biopsy and

cytology in discordant caseso 59% correlate the Pap and biopsy but only

review the Pap if biopsy not availableo 25% attempt to obtain the biopsy from

another laboratoryo 66% obtain levels to expose a deeper lesion

− 48% because s/o dysplasia−24% do not obtain add’l levels

o Reviews are sometimes at the discretion of the pathologist or requested by clinicians

Survey Results


• For “incident” (first Pap with…) HSIL/AIS/cancer Pap test target- correlate with most abnormal tissue within 6-month intervalo Exclude Pap test in conjunction with

biopsy unless no access to incident Pap test

o Exclude ECC without cervical biopsies unless containing SIL/AIS/cancer

• Include LEEP/Cone/ Hysterectomy

Specimens to correlate for standard statistics: Pap Target


• For HSIL/AIS/cancer biopsy target, correlate with the most abnormal prior Pap test within 6-month interval

• Exclude Pap tests taken at the time of biopsy unless no earlier Pap test is available*

• As additional quality monitor, laboratories may choose to review less abnormal Pap tests when multiple Pap tests are available

Specimens to correlate for standard statistics: Biopsy target


* Panos et al. Usefulness of concurrent Papanicoloau smear at time of cervical biopsy. Diag Cytopathol. 2001;25(4):270-273.

The correlation interval between the Pap test and the biopsy should preferably be within 3-4 months, but no greater than 6 months.

A. Agree as stated 89%

B. Disagree - too long 3.03%

C. Disagree - too short 7.58%



CHCConsensus Statement #3

Standardization of metrics and CHC process is desirable.

• Allows for inter-laboratory comparison

• Monitor:o Total number of CHC pairs

o Number of positive correlations (“true positive,” as defined prior to review)

o Number of negative correlations (“false positive,” as defined prior to review)

• Calculate Positive Predictive Value (PPV) of a positive Pap test

• Tabulate statistics at least annuallyo More frequently for high-volume laboratories

• Justification: Survey, professional opinion

Consensus Statement #3: Standardization of metrics and CHC process is desirable.


• 50% want CHC standardized among laboratories

• 35% unsure

• Prefer to see:o 67%- Acceptable actions for discrepancies

and statisticso 72%- Type of statistics to maintaino 50%- Time frame limit from abnormal Pap

test to cervical biopsy

• 62% support development of standardized comments for follow up recommendations for positive Pap, negative biopsy

Post-Survey Web Comments


• 78% monitor the percent of positive Pap tests that correlate with biopsies, but only 13% monitor the “PPV” of a positive Pap test (the same computation) (Sec 4, Table 35)

• Larger laboratories are more likely to measure screening/interpretive sensitivity or Pap sensitivity/ specificity (Survey volume analysis data)

Survey Results


Standardization of metrics and CHC process is desirable.

A. Agree 93.65%

B. Disagree 6.35%




PPV of a positive Pap test

is the preferred standard CHC metric.

• For Pap tests, PPV = TP / TP + FP, whereo True Positive= “positive” pair

o False Positive= “positive” Pap test but “negative” biopsy

• “Screening” role of Pap test

• Standardized correlation methods required

• Assumes biopsy gold standard

• A minimum of 20 total events necessary for meaningful data

• Justification: Literature, survey, professional opinion

Consensus Statement #4: PPV of a positive Pap test is the preferred standard CHC metric.


Screening result

Diagnosis as determined by biopsy Metric

Positive Negative

Positive Pap test

True positive(TP)

False positive(FP)

TP/ TP+FP=Positive

predictive value

Negative Pap test

False negative(FN)

True negative(TN)

TN/ TN+FN=Negative

predictive value

Metric TP/ TP+FN=Sensitivity

TN/TN+FP=Specificity

Cytologic-Histologic Calculation


• There are few or no correlations for negative Pap results

• False positive Paps over-represented

• PPVo A measurement close to the % of positive Pap

tests correlating with biopsies , the most measured CHC metric in survey

• Sensitivity is preferredo Biased due to low number of negative Pap

correlations

• PPV median = 83-88%o Range 71-94% (CAP Q-Track data, 2005-2010)

Evidence for PPV


• Any biopsy interpretation and Pap interpretation, in any combination, from the “positive” list:o LSIL/ CIN1/ HPV changeso HSIL/ CIN 2, 3/ CISo SIL, indeterminateo AIS*o Carcinoma (squamous, adenocarcinoma, NOS

or othero Other malignant diagnosis

• ASCUS, ASC-H, AGC are excluded

“Positive” Correlation


• Any normal/ negative/ reactive biopsy with a Pap result from the previous list

• Any NILM/ reactive/ infectious Pap test with a biopsy result from the previous list

“Negative” Correlation


• A positive Pap test (from the list) with a negative biopsyo Review of both specimens may reveal

that the Pap test interpretation is correct- for purposes of PPV, this remains a ‘false positive’ value

o Laboratories may tabulate Pap/biopsy review results separately as a correct interpretation

“False Positive” Pap test


The PPV of a positive Pap test is the preferred standard CHC metric.

A. Agree 69.84 %

B. Disagree 26.98%

C. Other 3.17%



htuniso

3 answers in Excel document?

• Investigate cytologic diagnostic accuracy and intradepartmental variabilityo Group consensus slide reviewo Outlier individuals associated with

discordance−Correlate with individual/group statistics of

other monitors:−ASC:SIL ratio−Rescreen variances−Diagnostic category rates

o Investigate biopsy qualityo Investigate colposcopic quality

Suggested approaches to Low PPV performance


• High PPV may indicate identification of only the most obvious lesions, indicating under-recognition of subtle lesionso Compare with overall laboratory/ individual

abnormal rates (lower than benchmarks?)o Rescreen error rates (higher than

benchmarks?) o PT performance evaluation

• Lack of biopsies of subtle colposcopic lesions or presence of transformation zone- discussion with care givers

Suggested approaches to High PPV performance


1. Where possible, only one final result per patient (one pair)

2. Original interpretations used for “positive” or “negative” correlation statistics rather than the review interpretation

3. Exclude ASC-US, ASC-H, AGC, “equivocal” biopsy results

4. Maintain laboratory statistics-a minimum of 20 events required for statistical significance

• Difficult to collect sufficient number of events to monitor for individuals unless in high-volume laboratories

CHC guidelines for statistical standardization


Laboratories should use the positive predictive value (for the whole laboratory) to formulate QA monitors.

A. Agree 65.22%

B. Disagree 28.99%

C. Other 5.80%



htuniso

3 Answers in Excel document?


When there is a negative biopsy with confirmed HSIL+ Pap test, the caregiver is notified of the discrepancy resolution in a

timely manner.

• Optimizes patient careo Minimizes over-emphasis of biopsy findings

o Prevent unnecessary additional procedures

• Documentation of notification may be in the biopsy report, cytology report or QA document

• Justification: Professional opinion, survey, literatureo Of 53 cases of negative biopsy/SIL Pap test, 24

(45%) were found to have subsequent SIL- indicating necessity to follow up women with discordant pairs*

Consensus Statement #5: When there is a negative biopsy with confirmed HSIL+ Pap test, the caregiver is notified of the discrepancy resolution in a timely manner.


* Anderson and Jones. False positive CV cytology-: follow-up study, Acta Cytol 1997;41(6):1697-1700.

A negative biopsy with a confirmed HSIL+ Pap test requires caregiver notification.

A. Agree 92.19%

B. Disagree 6.25%

C. Other 1.56%



htuniso

3 answers in Excel doc


Laboratories should attempt to obtain correlation biopsy information for all patients

with a HSIL Pap test.

• Laboratories that perform primarily Pap test interpretation may not have access to biopsy follow-up, but should ask for follow-up information in the Pap report or by other means.

• Laboratories should notify the care provider if a Pap test is HSIL and no follow-up information is obtained within a laboratory-defined period of time (suggest within 6 months).

• Laboratories should be able to show documentation that an attempt to gain follow-up information has been performed.

• The follow up method should be determined by the laboratory, be described in a written procedure and results documented in their QA program.

• Justification: Survey, professional opinion, literature

Consensus Statement #6: Laboratories should attempt to obtain “HSIL follow-up.”


• CYP.07556: When a follow-up histologic report or material is not available within the laboratory, there is a documented effort to obtain follow-up histologic information for correlative review when gynecologic cases with significantly abnormal (HSIL) or malignant cytologic findings are reported.

• §493.1274 (c)(5)(iv): [Laboratories must keep] An annual statistical laboratory evaluation of the number of---Gynecologic cases with a diagnosis of HSIL, adenocarcinoma or other malignant neoplasm for which histology results were available for comparison.

Regulatory Requirements


• 83% have written policy on how to proceed for HSIL+ Pap with no documented follow-up received (Sec 4, Table 34)

• 91% notify the physician’s officeo Physician’s office contact time ranged from 0-

45 months, average 5 months (Sec 4, Table 34)

• Post-survey Web-based public comments: o 75% (n=24/32) find it useful to track % HSIL+

for which follow-up is received

Survey Results


When you do not have biopsy results for an HSIL+ Pap test, the best way to obtain follow up is to send a formal request for this information to the caregiver that collected the Pap test?

A. Agree 87.3%

B. Disagree 12.7%




Microscopic review of all slides from discordant Pap/biopsy pairs is desirable.

• If slides are unavailable, the original interpretation stands as the review interpretation

• Laboratories may define other “non-correlation” metrics (beyond HSIL mismatch) for QA purposes

• Results of microscopic review should be documented by the laboratoryo May be within the biopsy or cytology report

o May be in a separate QA document

• Justification: Professional opinion, survey

Consensus Statement #7: Microscopic review of all slides from discordant Pap/biopsy pairs is desirable.


• 61% laboratories address discrepancies in the biopsy report (Sec 4, Table 33)

• Most (71%) address NILM Pap with CIN 2,3 biopsy in report, and HSIL Pap with negative biopsy (89%) (Sec 4, Table 38)

• Laboratories are divided on whether to report concurrences (Sec 4, Table 34)

• Post-survey public commentso 77% (n=20/31) review all available glass slides for both

Pap/biopsy in discordant cases

o 19% (n=5/31) review all available glass slides for both biopsy and cytology in ALL cases, concordant and discordant (suggesting this is the sign-out practice)

o 15% (n=4/31) review only Pap tests but annotate the concordance statistics

Survey Results


It is desirable to microscopically review all discordant pairs (as laboratory-defined) for CHC.

A. Agree as stated 86.79%

B. Disagree – should be required 3.77%

C. Disagree – not necessary 5.66%

D. Other 1.89%

E. Other 1.89%



htuniso

5 answers on Excel document?!?


CHC is optimum with a “multilayered” approach.

• Multilayered, laboratory-directed approach “drills down” in potential problem areas and can be tailored to laboratory size, issues and practice

• Additional QA monitors may be continuous or interval effortso Interval efforts may target specific pairs for a pre-defined

period (i.e., quarterly) to acquire a “snapshot” of laboratory performance for that indicator

o Continuous efforts may be desirable for laboratories with high personnel turn-over, disruptive environments, or mitigating variables outside of the laboratory’s control

• Justification: Professional opinion, survey, literature

Consensus Statement #8: CHC is optimum with a “multilayered” approach.

© 2011 College of American Pathologists. All rights reserved.

48

• A favored activity among laboratories- learn from mistakes

• Leverages group experience

• Encourages uniformity of interpretation through consensus

• Survey Results (Section 6, Table 62):o 60.4% of laboratories conduct in-house review of

Pap tests

o 89.1% share interesting cases

o 46.6% compare criteria for increased diagnostic precision

Suggested Additional Quality Practices: Group Educational Review of Cases


• CHC is ideal time to investigate biopsy qualityo Reorient tissue in blocko Obtain additional levelso Perform ancillary studieso Record transformation zone presence/ absence

• Laboratories should develop trend based policies:o Number of routine serial sections

o Number of levels on cervical biopsies and ECCs

• Openly discuss methods to improve biopsy samples


Suggested Additional Quality Practices: Optimize the Biopsy

• Record # events where subsequent actions are necessary for discrepant biopsies

• Record negative cervical biopsies that:o Lack transformation zoneo Are less than 2 mm size or consist of only 1-2

fragmentso Are not associated with additional biopsies

or ECCo Are poorly orientedo Require additional levels

Suggested Additional Quality Practices: Monitor biopsy characteristics


• Investigate variables adversely influencing interpretation

• Tabulate “both correct”- Sampling error

• Record false positive/ false negative Pap tests that:o Demonstrate staining or processing irregularitieso Have obscuring factorso Show atrophic changeso Did not have subsequently-detected cells marked

(screening variance)o Demonstrate difficult patterns of detection

(“Litigation cells,” “hyperchromatic crowded groups”) (interpretive variance)

Suggested Additional Quality Practices: Monitor Pap test characteristics


• LSIL Pap and HSIL biopsy

• ASC-H Pap and HSIL biopsy

• AIS / LSIL discordance

• LSIL/ CIN1 and negative discordances

• ASC-US, HPV+ and LSIL/HSIL biopsies

• ASC-US, HPV- and LSIL/HSIL biopsies

• AGC Pap and negative cervical biopsies, ECC/ EMBx

• HSIL Pap in pregnant patient and post-partum biopsy


Suggested Additional Quality Practices: Periodically monitor other discrepancy pairs

• Obtain additional opinion for disagreement with original interpretation

• Triage specific cases for automatic peer review (HSIL Pap, HSIL biopsy, AGC, disagreement between CT/Path)

• For retrospective review- blinded review of all Paps and biopsies, then correlate results with original and review interpretations

• Different individuals performing slide reviews

• Different CT review of Pap for discrepancies

• Review with original observer or in consensus conference


Suggested Additional Quality Practices: Minimize review bias

A multi-layered approach to CHC, suited to laboratory size and staffing, optimizes opportunities for improvement.

A. Agree 97%

B. Disagree 3%



Working Group 4

Additional Voting Questions


86. For standardization of the PPV of a positive Pap test calculation, should ASCUS interpretations be included?

A. Yes 23.08%

B. No 61.54%

C. Uncertain 15.38%

VotingWG4

© 2011 College of American Pathologists. All rights reserved. 57© 2011 College of American Pathologists. All rights reserved.

87. For standardization of the PPV of a positive Pap test calculation, should ASC-H interpretations be included?

A. Yes 55.10%

B. No 34.69

C. Uncertain 10.20%

VotingWG4


88. For standardization of the PPV of a positive Pap test calculation, should AGC interpretations be included?

A. Yes 53.85%

B. No 38.46%

C. Uncertain 7.69%

VotingWG4


89. Laboratories should be allowed to define the process for CHC (“real time”, retrospective, or both).


B. Disagree – not robust enough 15.38%

C. Disagree – too restrictive 3.85%

Formerly voting question #37.

VotingWG4


90. The correlation interval between the Pap test and the biopsy should preferably be within 3-4 months, but no greater than 6 months.





VotingWG4


91. Laboratories should use the positive predictive value (for the whole laboratory) to formulate QA monitors.





VotingWG4


92. Caregivers should be personally notified when review of a discordant pair shows a negative biopsy and an HSIL+ Pap reinterpreted as NILM.

A. Agree 92.59%

B. Disagree 7.41%

VotingWG4


COMMENTS?

Group 4: CHC


Documents

Cap.org v. 1 Gynecologic Cytopathology Quality Consensus Conference Working Group 4: Cytologic-Histologic Correlations June 4, 2011