Journal of the American Academy of DermatologyVolume 28, Number 6 Briefcommunications HI03

Interobserver agreement in a community skin cancer screening setting

David J. Leffell, MD,a Ya-Ting Chen, MPH,a Marianne Berwick, PhD,band Jean L. Bolognia, MDa New Haven, Connecticut, and New York, New York

To evaluate the appropriateness of screening as apreventive health tool, it is necessary to estimate itsvalidity (or accuracy) and reliability (or reproduc­ibility). Accuracy is determined by measures of sen­sitivity, specificity, and positive predictive value,whereas reliability reflects the consistency of resultswhen the test is performed more than once on thesame person under the same conditions. l The accu­racy and reliability of skin cancer screening have notbeen examined extensively. A few studies haveassessed positive predictive value by correlating his­topathologic findings with screening diagnoses2-6;

although one study addressed the reproducibility ofthe screening examination itself, it reported interob­server agreement between dermatologists and der­matology nurses.6 Our study examines interobserveragreement between dermatologists in a communityskin cancer screening setting.

MATERIAL AND METHODS

A public oon cancer screening was conducted on May19, 1990, in New Haven, Connecticut. A total of 20 der­matologists and dermatology residents participated. Thesame protocol was followed as previously described.6

Thirteen percent ofthe screenees (n = 53) were examinedindependently by two dermatologists, and the assignmentof patients was random. Eight dermatologists partici­pated in this part of the screening, and they had an aver­age of 11 years (range 1 to 26 years) in practice since theirresidency training.

Of the group screened by two dermatologists, thosewith positive findings (n = 37) were contacted within thesubsequent 18 months by mail (n = 32) and then if nec­essary by telephone (n = 8) to determine medical follow­up. Five of the 37 patients had been seen by attendingphysicians in the Yale Department of Dermatology, andtherefore information was available from their medical

From the Department of Dermatology, Yale University, School ofMedicine. New Haven"; and Cancer Prevention Research Institute,New Yorkb

Reprint requests: Jean Bolognia, MD, Yale University School of Med­icine, Department of Dermatology, 333 Cedar St. New I-laven, CT06510.

.J AM ACAD DERMATOL 1993;28:1003-5.

Copyright @ 1993 by the American Academy of Dermatology, Inc.

0190-9622/93 $1.00 + .10 16/54/43969

charts. The identified physicians were contacted by mail(n = 25) and then by telephone (n = 3) to determine theclinical and/or histologic confirmation of screening diag­noses. Included in the physician's mailing was a site-spe­cific list of the clinical diagnoses of both screeningdermatologists as well as a body chart with a more exactlocalization of these sites. In the case of actinic keratoses(AKs), treatment such as liquid nitrogen was oftenadministered without a biopsy of the lesion; this wasscored as clinical confirmation.

Data from the questionnaires were analyzed by x2, andinterobserver agreement was assessed by the K index'?' 8

RESULTS

Four hundred twenty-five persons were screenedand 99% (423 of 425) had a total body skin exam­ination performed. Thirteen percent of the screenees(n = 53) were examined independently by two der­matologists. The demographics were similar whenthose who had two examinations were comparedwith those who had one examination. In 16 (30%) ofthe 53 "double-examination" participants, both ex­aminers thought there were no suspect lesions; onepatient had a negative examination versus a singleAK. The remaining 36 persons (68%) had at leastone suspect lesion noted by both examiners.

A site-specific comparison of clinical diagnoseswas done for the 36 screenees who had two positiveexaminations. The interobserver agreementbetweenthe dermatologists is outlined in Table L A total of143 lesion sites were identified as either AK, multi­ple AKs, basal cell carcinoma (BCC), squamous cellcarcinoma (SCC), atypical nevus (AN), or multipleatypical nevi (ANs). There was positive agreementin 90 (63%) of the lesion sites. The greatest interob­server agreement was found with multiple AKs(K = 0.78) and the least with SCC (K =0.38). Clin­ical disagreement with regard to screening diagno­sis was seen in 53 (37%) of the lesion sites. Themajority of disagreements (96%; 51 of 53) involvedthe discrimination of a suspect lesion versus nolesion.

Of the 37 persons with a positive screen, 3 (8%)were lost to follow-up because of an inaccurate ad­dress or telephone number; 1 (2%) died; and II

1004 BriefcommunicationsJournal of the American Academy of Dermatology

June 1993

Table I. Interobserver agreement between dermatologists, by types of lesion

Type oflesion

BecsecANANsAKAKs

Positive agreement*(n = 90)§

171

175

2921

Disagreementt(II= 53)§

113~

10420~

7

Negative agreementt(n = 53~1

384937472940

K

0.630.380.660.680.520.78

AK, Actinic keratosis; AN, atypical nevus; BCC, basai cell carcinoma; SCc, squamous cell carcinoma.*Both examiners saw the lesion and agreed on the clinical diagnosis.tBoth examiners saw a lesion, but disagreed on the clinical diagnosis, or only one screening physician saw a lesion.tBoth examiners agreed that the patiellt had no such lesion.§n, No. of lesions.~n, No. of patients.!Two lesions counted twice, that is, included in both see and AK.

Table II. Clinical follow-up of "positive" screenees

Follow-up Other diagnosesType of Total agreementt

Jlesion Total No. A~ment* followed up (clin/hist) Clinical Histologic

Bec 28 +17 15 0/3 AK (4); Nev (1); NS (4) sec (1); AK (1); SK (1)-11 9 0/0 AK (3); NS (4) SK (1); seb hy (1)

sec 4 +1 1 0/0 AK (1)-3 3 0/0 AK (1); SK (1) f01 (1)

AN 27 +17 7 0/4 der N (1); jun N (1) Bee (1)-10 6 0/0 jun N (1); NS (3) jun N (1); com N (1)

ANs 9 +5 2 0/0 DF (1); NS (1)-4 3 0/0 NS (3)

AK 49 +29 25 14/1 NS (9) Bow (1)-18 10 6/0 NS (4)

AKs 28 +21 21 18/0 NS (2) sec (1)-7 4 2/0 NS (2)

Calculation based on number of lesions."'Agreement: +, Both screening dermatologists saw a lesion and agreed on the clinical diagnosis; -, only one screening physician saw a lesion, or bothsaw a lesion but disagreed on the clinical diagnosis.tBya third observer.Bow, Bowen's disease; clin/hisf, clinical/histologic; com N, compound nevus; der N, dermal nevus; DF, dermatofibroma;jol, folliculitis;jull N, junc­tional nevus; Nev, nevus; NS, no lesion seen; seb hy, sebaceous hyperplasia; SK, seborrheic keratosis.

(30%) failed to seek medical follow-up. We wereable to obtain complete follow-up information on allof the 22 "positive" screenees (60%) who soughtmedical care (Table II). Of the 106 lesions detectedin these 22 "positive" screenees, the follow~up diag­nosis was "no lesion seen" for 32(30%) ofthelesions,so a comparison was done between those lesions inwhich the screening diagnoses were in agreementversus those in which there was disagreement. Onlyin the case ofAN/ ANs was a follow-up diagnosis of"no lesion seen" more likely when there was dis­agreement (agreed vs disagreed: no lesion seen in 1of 9 vs 6 of 9; P =0.01), However, when there wasagreement in the screening diagnoses, follow-up

confirmation was more likely if the screening diag­nosis was BCC/SCC or AN/ ANs (p = 0.04, 0.02,respectively).

DISCUSSION

To calculate sensitivity and specificity, the num­ber of patients who are "true negative" and "truepositive" must be determined. In a community skincancer screening, it is difficult to assess the exactnumber of false-negative results because screen­negative persons are not followed up. In addition,clinical follow-up of 100% ofthe "positive" screeneesis necessary to determine the number of "true"-pos­itive results; however, the extent of clinical follow-up

Journal of the American Academy of DermatologyVolume 28, Number 6 Briefcommunications 1005

in previous studies has varied from 52%4 to 93%6 ofpositive screenees.

Itis possible, however, to measure the reliabilityor reproducibility of a skin cancer screening on sitewith fewer patients. In this study, we chose toexamine interobse~agreement between derma­tologists. Among the 53 "double-examination"screenees, 52 (98%) were categorized by both der­matologists as having either a positive or a negativeexamination (16 negative and 36 positive). Less im­pressive were the K indices for interobserver agree­ment, which varied from 0.78 (AKs) to 0.38 (SCC).However, in 51 of 531esion sites in which there wasdisagreement, the disagreement centered on the veryexistence of a lesion; this observation must be keptin mind when evaluating these K indices. In addition,the low K value for SCC may not be an accuratevalue given the few lesions observed.

In the case of ANs, a possible explanation for thedisagreement regarding a suspect lesion versus nolesion could be that both physicians saw apigmentedlesion, but only one believed that it had atypical fea­tures. In the case of AKjAKs, the dermatologistmay have thought that identifying one site as AKjAKs was sufficient in the screening setting. For ex­ample, of the 15 patients with at least one site inwhich the diagnosis was AKjAKs versus no lesion,11 had additional sites in which both examinersnoted an AK or AKs. Clinical or histologic confir­mation was more likely in the case of precursor le­sions (AKjAKs) compared with cutaneous carci­nomas (BCCjSCC; Table II). Five previous studiesof skin cancer screening have assessed positive pre-

dictive values for cutaneous carcinomas and theywere as follows: 35% to 50% for melanoma, 39% to69% for BCC, and 35% to 100% for SCc.2-6

Other studies comparinginterobserver agreementon pigmented lesions were based on specific lesionspreselected for observation.9In this study, we did notmark lesions, but presented the entire patient for as­sessment, as is the usual case in a clinical examina­tion.

REFERENCES1. Mausner JS, Kramer S. Screening in the detection of dis­

ease. In: Epidemiology-an introductory text. Philadel­phia: WB Saunders, 1985:214-38.

2. Snow SN, Cripps D. Skin cancer prevention and detection:results ofa free screening program in Madison, Wisconsin.Wis Med J 1986;85:23-5.

3. Green A, Leslie D, Weedon D. Diagnosis of skin cancer inthe general population: clinical accuracy in the Namboursurvey. Med J Aust 1988;148:447-50.

4. Rigel DS, Friedman RJ, Kopf AW, et al. Importance ofcomplete cutaneous examination for the detection of ma­lignant melanoma. J AM ACAD DERMATOL 1987;14:857­60.

5. Koh HK, Caruso A, Gage I, et al. Evaluation of melano­ma/skin cancer screening in Massachusetts: preliminaryresults. Cancer 1990;65:375-9.

6. Bolognia JL, Berwick M, Fine JA. Completefollow-up andevaluation of a skin cancer screening in Connecticut. JAMACAD DERMATOL 1990;23:1098-106.

7. SAS Institute Inc. SAS Procedures Guide, release 6.03Edition. Cary, NC: SAS Institute Inc, 1988.

8. Fleiss JL. The measurement of interrater agreement. In:Statistical methods for rates and proportions. 2nd ed. NewYork: John Wiley & Sons, 1980:212-36.

9. Barnhill RL, Roush Ge, Ernstoff MS, et at. Interclinicianagreement on the recognition ofselected gross morphologicfeatures of pigmented lesions. J AM ACAD DERMATOL1992;26:185-90.

Pyoderma gangrenosum treated successfully with potassium iodide

CASE REPORT

A 70-year-old woman had a large, painful ulcer of theright inguinal crease. Previous treatment with oral pred-

From the Department of Medicine, Division of Dermatology, Univer­sity of Louisville.

Reprint requests: Jeffrey P. Callen, MD, 310 E. Broadway, Suite 200,Louisville, KY 40202.

JAMACAD DERMATOL1993;28:1005-7.Copyright © 1993 by the American Academy of Dermatology, Inc.0190-9622/93 $1.00 + .10 16/54/43737

Jeffrey B. Richardson, MD, and Jeffrey P. Callen, MD Louisville, Kentucky

Pyoderma gangrenosum (PG) is a chronic, ulcer- be effective in treatment include topical, intrale-ative dermatosis that is frequently associated with an sional, and systemic corticosteroids4•6; dapsone7;

underlying systemic disorder. 1-3 Agents reported to sulfasalazine8; sulfapyridine3;c1ofazimine9;minocy-c1ine10; rifampin3; azathioprine3; cyclophospha­midell; chlorambuci112; and cyc1osporine.13Wr: re­port the successful treatment of PG with potassiumiodide.