THE USE OF STANDARD FILMS IN RADIOLOGICAL …Standard Films.-The standard films used were the "Victor Medium" films selected in the way that has already beendescribed. Apreliminary

Brit. J. industr. Med., 1951, 8, 138.

THE USE OF STANDARD FILMS IN THE RADIOLOGICALDIAGNOSIS OF COALWORKERS' PNEUMOCONIOSIS

BY

C. M. FLETCHER and P. D. OLDHAMFrom the M.R.C. Pneumoconiosis Research Unit, Llandough Hospital, near Cardiff

(RECEIVED FOR PUBLICATION APRIL 4, 1951)

In an earliet paper (Fletcher and Oldham, 1949),we reported the results of an experiment in whichten doctors, experienced in the radiological diagnosisof coalworkers' pneumoconiosis, were asked toclassify 102 films of coal miners, by degree ofabnormality, into five categories. Serious disagree-ment was found between the readings of differentobservers (inter-observer error), and between thefirst and second readings of the same observers(intra-observer error), and we suggested that suchinaccuracies of diagnosis might be lessened by theuse of standard films, exemplifying the limits of thecategories, which could be used for direct side-by-side comparison with the films to be classified.We now report the results of an experiment designedto test the value of standard films for thispurpose.

In analysing the results of our previous experiment,we found that although the various readers differedin the quantitative classification of simple pneumo-coniosis, it could be shown that on the averagetheir use of the four categories which we had sug-gested did in fact subdivide the range of abnormalityof simple pneumoconiosis, from the earliest detect-able change to the most advanced abnormality, intofour roughly equal divisions. Moreover, sequentialradiographs of the same cases have suggested thatprogression of simple pneumoconiosis is much moreclosely related to quantitative increase of radio-graphic abnormality than to qualitative changes.(Davies, Fletcher, Mann, and Stewart, 1949). Atthe Pneumoconiosis Research Unit it was, therefore,decided to use a quantitative classification of simplepneumoconiosis and abandon the use of qualitativeterms such as " reticulation ", "stippling", "nodula-tion ", and so on. The precise basis for this quanti-tative classification was worked out by Davies andMann (1948), and has been more fully discussed byFletcher, Mann, Davies, Cochrane, Gilson, andHugh-Jones (1949). Qualitative differences between

various types of films are still recognized, but themain basis for classification is a subdivision ofsimple pneumoconiosis of coalworkers into fourcategories, increasing in degree of abnormality,numbered 1 to 4, the figure 0 being reserved forfilms which are within normal limits. Films showingevidence of progressive massive fibrosis (P.M.F.)are separately classified in four categories A, B, C,and D. This classification has been used by thePneumoconiosis Research Unit for three years andhas also been adopted by the pneumoconiosismedical panels of the Ministry of National Insurance.A closely similar system has recently been recom-mended by the International Conference of Expertson Pneumocoiiosis (Fletcher, 1950; I.L.O., 1950)., From the " Victor Medium " group of films usedin our first experiment, standard films were selectedaccording to the classification given during thatexperiment. A film on which opinion was equallydivided between two categories was taken as definingthe lower limit of the higher of these two categories.In this way, four films were selected exemplifyingthe lower limit of categories 1, 2, 3, and 4, andthese were used in the routine work of radiographicassessment in the Pneumoconiosis Research Unit.

It was clearly important to assess experimentallythe value of such standard films in increasing theaccuracy of diagnosis. To this end a group offilms was selected and four observers on the medicalstaff of the Pneumoconiosis Research Unit readthem with and without standard films, classifyingthem according to a number of characteristics.This test is Part I of our experiment. It was incertain respects unsatisfactory and too complicated,so a selection was made from the same group of films,and six observers of the pneumoconiosis medicalpanels of the Ministry of National Insurance readthis shorter series with and without standards usingonly the simple quantitative classification. This setof readings is Part II of the experiment.

138

on July 22, 2020 by guest. Protected by copyright.

http://oem.bm

j.com/

Br J Ind M

ed: first published as 10.1136/oem.8.3.138 on 1 July 1951. D

ownloaded from

http://oem.bmj.com/

STANDARD FILMS IN RADIOLOGICAL DIAGNOSIS OF PNEUMOCONIOSIS 139

A further short trial of the use of standard filmsby observers wholly inexperienced in the classificationused by the Pneumoconiosis Research Unit wascarried out later.The results of the experiments were less conclusive

than had been hoped, but they do suggest thatstandard films, when properly used, may result inincreased accuracy of radiological diagnosis andmay assist the inexperienced to use a classificationwith reasonable accuracy.

Methods: Part IObservers.-Observers W, X, Y, and Z were all

members of the medical staff of the PneumoconiosisResearch Unit (P.R.U. Observers). W, X, and Y hadhad considerable experience in the use of the Pneumo-coniosis Research Unit's system of classification of films.Z was well acquainted with the system but had had lessexperience in its use.

Selection of Films.-We decided not to use the sameset of films as in our previous experiment, becausethese included many films of unsatisfactory technicalstandard. Moreover, the standard films which we wereusing had been extracted from the original set, whichwas thus incomplete. We hoped originally to use theexperiment to investigate not only the diagnosis ofsimple pneumoconiosis but also to test the accuracy ofdiagnosis of abnormalities of lung markings, emphysema,and the earlier stages of progressive massive fibrosis.For these reasons, we decided to select 120 films fallinginto the following groups:

(1) Sixteen normal films from men who had neverbeen exposed to a dust risk and had no respiratorysymptoms; (2) 16 films of men attending an asthmaclinic, who had had no coal dust exposure; (3) 16 filmsof coal miners who had been exposed to dust inhalationfor at least 10 years but whose radiographs still appearedto be within normal limits; (4) 16 films of coal minerswith category 1 simple pneumoconiosis; (5) 16 filmsof coal miners with category 2 simple pneumoconiosis;(6) 16 films of coal miners with category 3 simplepneumoconiosis; (7) eight films of coal miners withcategory 4 simple pneumoconiosis; (8) 16 films ofcoal miners with category 3 or 4 simple pneumoconiosiswith, in addition, early progressive massive fibrosis ofcategory A.

The selection was made by one of us (C. M. F.) withMr. W. G. Clarke (chief radiographer of the Pneumo-coniosis Research Unit), from films already in the Unitfiles. Care was taken to ensure that the age distributionof the cases and the radiographic technique was similarin each group. Only those films were accepted in whichthe vertebral column was visible through the heartshadow but in which the intravertebral spaces could notbe distinguished. Even within these limits there may bea considerable difference in penetration, but care wastaken to ensure that within each group there was thesame proportion of slightly under-penetrated or slightlyover-penetrated films. All the films were taken with

the technique described in our earlier paper for the"Victor Medium " films.

Reading of Films.-The films were arranged in anorder derived from a table of random sampling numbersand each was given a serial number from 1 to 120.The set was read four times by each observer, (1) usingstandard films, (2) without standard films, (3) withoutstandards, and finally (4) with standards again. Thisorder was chosen because we intended at this stage toinclude inexperienced readers in the trial who, weassumed, would have to use the standards for the firstreading in order to appreciate the degree of abnormalityrepresented by each category.During all the readings, the observers had the following

written definitions of the categories before them.

Categories of Simple PneumoconiosisThere are four categories of increasing abnormality,

in addition to films that fall within normal limits(Category 0). These categories are:

Category 1 (Minimal Simple Pneumoconiosis).-Inthese films a small number of typical opacities can beseen, usually in the second, third, or fourth anteriorrib spaces in the middle longitudinal zone of the lung,more commonly on the right than on the left. Thelung markings are clearly visible.

Category 2 (Moderate Simple Pneumoconiosis).-Theopacities are more numerous and are distributed in allzones with the exception of the outer third of the lungfields where they are sparse or absent. The lung mark-ings are still visible although not so clearly as in theprevious category.

Category 3 (Marked Simple Pneumoconiosis).-Withthe exception of an occasional large vessel in the upperand lower zones, the lung markings are now obscuredby minute, small, or medium opacities, singly or incombination, profusely scattered throughout both lungfields, including the outer third of the lung.

Category 4 (Maximal Simple Pneumoconiosis).-Inthese films, the lung fields are, as it were, filled " tocapacity" with typical opacities.

It should be noted that the distinction of the categoriesis made chiefly on the following grounds: the profusionof the opacities, the presence or absence of the opacitiesat the periphery of the lung fields, and the degree ofobscuration of lung markings.

In cases in which the severity of simple pneumoconiosisvaries from one area of the lung field to another, thecategory is determined by the most advanced diseasewhich is present over at least half of one lung field.

During the readings with standards, five standardfilms were used, exemplifying the lower limits ofcategories 1, 2, 3, and 4 of simple pneumoconiosis andof the least definite localized shadow acceptable fora diagnosis of progressive massive fibrosis. As eacbfilm was placed on the viewing screen, the observerasked for the appropriate standard film which was thenplaced on an adjacent screen. Thus if he were debatingin his mind whether to diagnose category 1 or 2, he


http://oem.bm

j.com/

Br J Ind M


ownloaded from

http://oem.bmj.com/

140 BRITISH JOURNAL OF

would ask for standard 2. If he considered the abnor-mality to be as great as or greater than that in thestandard film, he would diagnose category 2. Thestandard film category 3 would then be placed onanother adjacent screen, so that he could be sure thatthe abnormality was not as great as the lower limit ofcategory 3. Thus, in the reading of nearly every film,two standards were used for comparison. The secondstandard was unnecessary when the unknown film wasclose in degree of abnormality to the first.

In connexion with each film, the observer was askedto state the category of simple pneumoconiosis if thiswere present. If it was not, he was asked whether hethought that there was any evidence in the film that theman had worked as a miner. (The purpose of thisquestion was to see whether there was any abnormalityeven earlier than category 1 simple pneumoconiosis bywhich men with dust exposure could be distinguishedfrom those without exposure.) Finally, he was askedwhether there was any evidence of tuberculosis or massivefibrosis, whether the lung markings were abnormal andwhether there was any evidence of emphysema. (Thefindings from these questions will not be reported inthe present paper.) The observers' readings wererecorded by one of the authors or by an assistant. Thefilms were read alternately in the orders 1-120, and 120-1.The intervals lapsing between the various readingswere two to six days.

After the individual readings had been carried out,observers W, X, Y, and Z met together and read thefilms in consultation using standard films, each filmbeing discussed until agreement was reached. Whereone observer insisted on disagreeing with the others, hisopinion was disregarded, the majority opinion beingaccepted. Where there was an equal division of opinion,two observers differing from the other two, both opinionswere recorded. This consultative reading was repeatedafter two days and, finally, W, X, Y, and Z met oncemore to reconsider those films on which their jointopinion on the two previous occasions had differed.A final agreed opinion was thus reached on every film.

Standard Films.-The standard films used were the"Victor Medium " films selected in the way that hasalready been described.A preliminary analysis. of the results of Part I

showed that the observers had not all increasedtheir agreement or consistency by using standardfilms. Nevertheless, their performance was muchbetter than that found in the trial reported in ourprevious paper. It was thought that this resultmight be due to the fact that observers were so wellacquaintec with the classification and with thestandard films that they had a clear picture of themin their minds and thus little further advantagewas to be gained from using the actual films them-selves.We therefore decided to extend the trial to other

observers without experience of the use of standards.At the same time, we decided to simplify the experi-ment, for observers W, X, Y, and Z had found it

INDUSTRIAL MEDICINE

distracting to have to consider localized shadows,lung markings, and emphysema in addition to thecategory of simple pneumoconiosis, and they hadalso found 120 films rather more than could beread without fatigue in a single session. Thisextension of the experiment in a simpler formconstitutes Part II.

Methods: Part IIObservers.-These were all on the medical staff of

the South Wales pneumoconiosis medical panels of theMinistry of National Insurance. They had all beenusing the P.R.U. method ofclassification for some months.Since. they were accustomed to reading films in pairs,it was agreed that each " observer " should consist ofa pair of doctors working together. These observersare referred to as A, B, C, D, E, and F, or the P.P.observers.

Selection of Films.-On the basis of the final consulta-tive reading of observers W, X, Y, and Z, the followinggroups of films were selected from the 120 films inPart I: 16 normals (without dust exposure); eight" normal miners ", who were cases that W, X, Y, and Zhad agreed to have less abnormality than category 1,but which they considered showed evidence suggestive ofdust exposure; 16 cases of simple pneumoconiosiscategory 1; 16 cases of simple pneumoconiosis category2; 16 cases of simple pneumoconiosis category 3;eight cases of simple pneumoconiosis category 4; thusmaking a total of 80 films.

Reading of Films.-The films were placed in the orderin which they had appeared in Part I of the experimentbut they were re-numbered 1-80. Before the reading,the observers were given the same written definitions ofthe categories of simple pneumoconiosis that observersW, X, Y, and Z had used. Each observer then readthe film in the light of these written definitions butwithout standard films. Two observers worked simul-taneously, the first reading films 1-40 while the secondread films 41-80. The first then read 41-80 while thesecond read 1-40. They were asked to state only thecategory of simple pneumoconiosis of each film, andthe results were recorded by one of the authors or anassistant. An interval of half -an hour followed, andthe standard films were demonstrated and discussed.Finally, the reading of the films was repeated in identicalorder using standard films. On- the few occasions whenthe doctors comprising each observer failed to agree, theopinion of the senior doctor was accepted.

It might be thought that the short interval betweenthe two readings would have enabled the, observers toremember their previous opinions, but apparently thisdid not happen. An observer sometimes commented:"I remember this film ", but invariably added, " I wonderwhat I called it last time

Later the readings without and with standards wererepeated after an interval of four weeks in cases ofC, D, and F and eight weeks later for A, B, and E.(On the second occasion, the junior doctor of " observerA" was absent on leave so the senior doctor readalone.)


http://oem.bm

j.com/

Br J Ind M


ownloaded from

http://oem.bmj.com/


Standard Films.-Since the films were read by twoobservers simultaneously, it was necessary to have twoidentical sets of standard films and it was, therefore,impossible to use the same set which had been used byobservers W, X, Y, and Z, for, of these, a duplicateexisted only of the standard lower limit category 2.The standards used were considered by observers W,X, Y, and Z to correspond very closely to the originalset except that the new films of lower limit 1 showed alittle less abnormality, and the new films of lower limit 3had the lung markings a little less obscured. ObserversA, B, C, D, E, and F all commented on the extremelysmall evidence of abnormality in the standard lowerlimit category 1 which they used, but they accepted theother films without criticism.

In summary then, the differences between Part I andPart II of the experiment were the following.

We shall make the same assumption as in ourprevious paper, that the best estimate of the " true "category of a-film may be taken to be the categorynearest to the mean of the category numbersallocated to the film by all the observers. This willbe called the standard category. Since it may beargued that the categories defined in words, andthose defined by the standards may differ, a separatecalculation of the standard categories was madefrom the readings with and without standard films.In fact, the standard categories with and withoutstandard films were the same for all but four films.We may now consider the performance of indi-

vidual observers and groups of observers by relatingthe opinions they gave on a film to its standard

Part I Part II

Observers.. .. W, X, Y, Z members of Pneumoconiosis A, B, C, D, E, F members of. pneumo-Research Unit. Read as individuals. coniosis panels. Read in pairs.

(P.R.U. observers.) (P.P. observers.)

Films .. .. 120 films including 80 used in Part II 80 films

Order of Reading.. 1: standards 1 : no standards2: no standards 2: standards3: no standards 3: no standards4: standards 4: standards

Opinions Required Category of simple pneumoconiosis. Normal or simple pneumoconiosis cate-Presence and nature of localized shadows. gories 1, 2, 3, or 4Abnormalities of lung markings.

Standard Films .. Original " P.R.U. set" Temporary set

Results

Agreement.-One of the most remarkable findingsin our previous experiment was the wide range ofopinions given on the same film. In Parts I and II

of the present experiment there was less disagreement,even without the use of standard films. Unfor-tunately, the two experiments are not comparablesince both the observers partaking and the filmsused were different, so we cannot say from thesefindings whether the more precise written definitionsof the categories used in our present experimentwere responsible for the smaller range of extremeopinion. The present results are given in Table 1.Even without standards only two films showed a

range of three categories, and more than half thefilms had a range of one category or less.When standard films were used, the number of

films showing a wide range of classifications wasfurther reduced, though one film remained with a

range ofthree categories, being classified as follows:Category 1

,, 2,, 3,, 4

1 opinion8 opinions

. . 10 opinionsopinion

category (using for the four films the standardcategory appropriate to the method of reading thatwas used). Since only five of the 3,200 opinionsgiven in the first two parts of the experiment differedby more than one category from the standardcategory, we introduce only a small error by regardingall such differences as of equal weight and consideringfirst of all simply the proportion of opinions whichdisagreed with the standard categories. These aregiven in Table 2. It is seen that the P.R.U. observersappear mostly to be assisted by standards in thenormal films, whereas the pneumoconiosis panel ob-servers are not. In the abnormal films the situation isreversed, the pneumoconiosis panel observersapparently finding the standards a help, the P.R.U.observers being for the most part unaffected. Weshall discuss this finding later. However, we findthat very few of the differences that appear can beassumed to indicate real change of performance,and not mere random fluctuations. Only thosedifferences given in italics in Table 2 are statisticallysignificant at the 5%1' level. That is, observer Ximproved significantly with standards in the " normalminer" films, observer D in category 1 films, and


http://oem.bm

j.com/

Br J Ind M


ownloaded from

http://oem.bmj.com/

BRITISH JOURNAL OF INDUSTRIAL MEDICINE

TABLE 1

RANGE OF OPINIONS BY ALL OBSERVERS ON 80 FILMS COMMON TO BOTH PARTS OF THE EXPERIMENT

a Without standards

With standards ..

Range of Extreme Opinions of

Nil One Two Three No. of FilmsCategory Categories CategoriesN

No. % No. °/% No. % No. % No. %

5 6-3

5 6-3

41 51'3 32 40 0

54 67 5 20 25-0

2 25

1 1-3

80 100

80 100

TABLE 2PERCENTAGE. OF OPINIONS DIFFERING FROM STANDARD CATEGORY (ANALYSIS BY OBSERVERS AND GROUPS OF FILMS WITH

SAME STANDARD CATEGORY)

Groups of Films by Standard Categories

Normal Simple Pneumoconiosis_____ All All AllObservers Non- M Cate- Cate- Cate- Cate- Normals Abnor- Films

miners iners gory I gory 2 gory 3 gory 4 mals

NS S NS S NS S NS S NS S NS S NS S NS S NS S

W.. .. 9 3 25 17 31 24 22 35 18 34 37 31 14 7 25 31 22 24X .. .. 3 0 75 17 42 35 25 24 21 25 14 31 23 5 28 28 26 22Y .. .. 19 6 83 50 56 35 16 18 26 16 7 0 37 18 30 20 32 19Z .. .. 13 6 33 42 33 47 37 41 18 28 29 19 20 14 29 36 27 30

All P.R.U... 11 4 56 29 40 35 25 29 21 26 21 20 23 11 28 29 27 24

A.. .. 6 6 8 8 28 26 31 18 47 19 43 6 7 7 36 19 28 16B .. .. 9 31 8 33 53 41 56 59 68 41 7 0 9 32 53 40 41 38C .. .. 0 9 0 50 53 24 28 21 32 31 21 13 0 20 36 23 26 23D .. .. 6 6 17 8 39 12 25 21 38 13 7 0 9 7 31 13 25 11E .. .. 38 50 17 42 25 50 28 15 21 6 37 13 32 48 26 22 28 29F _. 6 3 8 33 58 41 53 44 21 19 0 0 7 11 39 30 30 25

All P.P. 11 18 10 29 -43 32 36 29 38 21 19 5 11 21 38 25 30 24

All.. .. 11 12 28 29 42 34 32 29 31 23 40 23 16 17 33 26 28 24

S = with standards. NS = without standards. Figures in italics show a significant change with use of standards.

TABLE 3OPINION OF OBSERVERS W, X, Y, AND Z ON 32 FILMS OF MEN WITHOUT DUST EXPOSURE

Number of Opinions ofTotal NumberReadings Normal "Evidence of Pneumoconiosis of Opinions

Dust Exposure" Category 1

Individual without standards .. .. 178 56 22* 256

Individual with standards .. .. 189 57 10 256

First two consultative with standards 60 4 0 64

Final consultative with standards .. 32 0 0 32

* Including one opinion of Category 2

142


http://oem.bm

j.com/

Br J Ind M


ownloaded from

http://oem.bmj.com/


observer A in category 3 films, while observer Cwas worse with standards in the " normal miner"films.*The results of the final consultative readings by

observers W, X, Y, and Z using standard films,were interesting. The classification they reachedonly differed from the appropriate standard categoryin three films (40%). This provides supportingevidence for the view that the standard category,as we have defined it, adequately represents theclassification that a film would get if errors ofdiagnosis were eliminated, that is, the " true"classification.

In Part I of the experiment there were 32 filmsof men (16 of them clinically quite normal, 16 ofthem asthmatic) who had never been exposed to anoccupational dust-hazard, and can therefore bepresumed to be free from pneumoconiosis. Theaccuracy of consultative reading is suggested againby the fact that none of these films was ever calledcategory 1 pneumoconiosis in a consultative reading,though " evidence of dust exposure " was reportedin four of them. Moreover, the final consultativereading even rejected these classifications and classedall the men as absolutely normal. These resultsand the readings given by observers W, X, Y,and Z on the films of non-miners with and withoutstandards are given in Table 3. It will be seen thatthe use of a standard category 1 film reduced thenumber of times a wrong diagnosis ofpneumoconiosiswas made by the individual observer, but did notaffect the frequency with which a mistaken diagnosisof " evidence of dust exposure " was made. It mayalso be noted that the " standard" category of the16 of these absolutely normal films which wereincluded in Part II of the experiment was alsoalways 0, which gives another check on the validityof using this averaging method for determiningthe " true" category of a film.According to Fletcher and others (1949) the

diagnosis of the earliest stage of coalworkers'pneumoconiosis depends on the detection of charac-teristic minute opacities. Similar minute opacitiesare occasionally seen in normal films especiallywhen fine vascular markings run at right angles tothe plane of the film. There is a tendency forindividual observers to mistake such opacities forevidence of coalworkers' pneumoconiosis. Duringthe consultative readings, however, where oneobserver suggested such an abnormality in a normalfilm, one of the others would object, and in thecourse of critical discussion there was finally a

* Each percentage was transformed to a variable x wherex = sin-',,/(percentage) and the differences in x were comparedwith the theoretical standard errors calculated from a2 = 820 7/nwhere n is the number of films, or from an analysis of variance ifthis gave a significantly larger standard error.

rejection of all such spurious diagnoses. Thatexperience is important in the use of standardfilms in the avoidance of these mistakes is shownby the later experiment with inexperienced readers.

Indices of Disagreement.-The effect on agreementbetween observers of using standard films may beillustrated graphically by using the index ID.*

B

y

zFAExCD

W

WithoutStandards

B

WithStandards

INDICES OF DISAGREEMENT

FIG. 1.-Improvement is shown by a lower index.

In Fig. 1 the indices of disagreement for eachobserver for all films are plotted on vertical scales.The effect of using standard films is indicated bythe lines joining the corresponding indices with andwithout standard films. It will be seen that sevenobservers (A, B, C, D, F, X, and Y) appear to be con-siderably improved, one (Z) not influenced and two

* The index of disagreement, ID, and the index of inconsistency, Ic,are defined by:-

n2ID = 2 (Xi + Yi-2zi)2/4ni=1

IC = 2 (xi-yi) 2/2n

where xi and yi are the first and second opinions of an observer onthe itO film of a group of n and Zi is the standard category of the film.


http://oem.bm

j.com/

Br J Ind M


ownloaded from

http://oem.bmj.com/


(E and W) made worse with standards, but a test ofsignificance shows that only the changes in A, D,and Y can be taken as definitely established.

Consistency.-The extent to which observerschanged their opinion on a film between two readingsis shown in Table 4. Only six observers changed

TABLE 4

CONSISTENCY OF CLASSIFICATION WITH AND WITHOUTSTANDARD FILMS

Percentage of Films in which First andSecond Opinions Differed by

Observer Nil 'One TwoN Category CategoriesNS S NS S NS S

A 56 81 41 19 3B 67 60 32 39 1 1C 81 73 18 27 1 -

D 66 83 33 17 1E 55 74 45 26 -

F 71 70 28 30 1 -

W 71 76 29 24 -

X 75 84 25 16 - -Y 69 78 31 22 -

Z 65 68 31 30 4 2

P.P. 66-2 73-3 32-5 26-4 1-3 0-3P.R.U. 70-0 76-3 29-1 23-1 0-9 0-6All 69-3 74 5 28-7 25-1 2-0 0 4

Consultative - 85 - 15 -

NS = without standards. S = with standards

their opinion on a film by more than one categorywithout standards and only two changed theiropinions by this amount with standards. Thepercentage of films on which the two opinions wereidentical was increased by the use of standards inthe case of all but three observers (B, C, and F).The P.R.U. observers were more consistent thanthe P.P. observers with and without standards.The percentage of consistent opinions of the wholegroup of observers changed from 69% to 75% bythe use of standards, but this apparent improvementis not technically significant. The highest level ofconsistency was reached in the two consultativereadings, but these were closely approached byobservers X and D, using standards.The effect of standards on the index of incon-

sistency Ic which was defined in our previous paper,*is shown in Fig. 2. Observers A, D, and E can

* See footnote on previous page.

be considered as significantlyX and Y are possibly improved.

0*250 t

zE

DB

Fy

w

x

C

WithoutStandards

improved, while

0-250

B

0-200 Z

0*150

0*100

F

CE

wy

ADx

WithStandards

INDICES OF CONSISTENCY


Skill.-The average of the indices of disagreementand inconsistency for each observer with andwithout standards is shown graphically in Fig. 3.There is a tendency towards improvement with theuse of standard films in the case of all observersexcept B and C but this improvement is onlysignificantly established in the case of observersA, D, and Y.

Readings by Observers with no Previous Know-ledge of the Classification.-In view of the ratherinconclusive results which we have described, wethink it appropriate to refer here to a further smallextension of our experiment from which usefulconclusions may be drawn.The International Labour Office conference of

experts on pneumoconiosis at Sydney adopted inFebruary, 1950, a classification of radiographicappearances in pneumoconiosis very similar tothat put forward by Fletcher and others (1949)." Simple pneumoconiosis" was there described as" pneumoconiosis with discrete opacities", and it

144


http://oem.bm

j.com/

Br J Ind M


ownloaded from

http://oem.bmj.com/


was divided into three categories of increasingabnormality.*One of us (C. M. F.) had the opportunity of

arranging for two members of this conference totry out this classification on the same 80 films thatwere used in Part II of our experiment. Later,

B0 200AzE

YDF

xwC

WithoutStandards

WithStandards

INDICES OF SKILLMean of indices of disagreement and inconsistency


two other observers in South Wales, one a chestphysician and the other a consultant radiologist,with experience of pneumoconiosis but with noexperience in classifying films into categories, wereasked to do the same. These four " inexperienced "observers (called P, Q, R, and S) read the filmsonce according to the above definitions alone, andonce with the additional help of standard films.Both readings took place on the same afternoon.The standard films were those used as a routineby the Pneumoconiosis Research Unit and now(February, 1951) also used by the pneumoconiosismedical panels of the Ministry of NationalInsurance.

The results of this short test showed first, thatreading with the written definitions but withoutstandards these inexperienced observers all mademore mistakes than any of the other observersexcept B (Table 5 compare with Table 2), and thisdespite the fact that they had fewer opportunitiesfor making mistakes since they were only usingthree categories of abnormality, whereas observersA-F and W-Z were using four categories. Onthe other hand, with the help of standard filmsthey reached a level of accuracy similar to that ofthe more experienced observers. Secondly, standardfilms affected the accuracy of these observersdifferently in the group of normal films and thegroup of abnormal films. Standard films reducedtheir average percentage of mistakes from 52% to23% in the abnormal films but increased theirmistakes from 26% to 43% in the qormal films.In this respect they behaved in a similar fashionto observers A to F but the contrast between theeffect of standards in normal and abnormal filnswas even more striking. This trend was mostmarked in the films of " normal miners ". Thirdly,indices of disagreement both with and withoutthe use of standards may be calculated for theseobservers and for observers A, W, X, and Z, whotook part in the experiment described in our previouspaper, making the latter comparable by amalgama-ting their diagnoses of categories 3 and 4. A veryrough comparison may be made with a similarlymodified index of disagreement calculated from the" Victor Medium " group of films in our previouspaper,* excluding for this purpose the normal non-miner films from our present experiment.

This comparison is made in Fig. 4, in whichdotted lines connect the indices of those observerswho partook in both experiments. There is noevidence that the more accurate verbal definitionsused in the present experiment resulted in increasedagreement on classification, the lower indices ofobservers A, E, Z, W, andX (E, F, C, A, and B respec-tively in the previous experiment) being possiblydue to the fact that they were all to some extentconversant with the method of classification beforethe experiment took place. The chief benefit ofstandard films is clearly shown in the reduction ofthe gross disagreement of the inexperienced observersto levels similar to those of the more experienced.

* Category 1.-In these radiographs a small number of opacitiesmay be seen in at least two anterior rib spaces extending over notmore than half of the medial two-thirds of the lung field.

Category 2.-In these radiographs opacities extend over more thanhalf of the medial two-thirds of the lung field, but are sparse or absentin the lateral third.

Category 3.-In these radiographs profuse opacities extend overthe whole of both lung fields including the lateral third althoughthey may be sparse or absent above the clavicles.

c

I Discussion and ConclusionsAs we pointed out in the discussion in our

previous paper, it is dangerous to argue too closelyfrom a particular test on a small number of selected

* The " Victor Medium " group of films was of similar radiographictechnique and covered a similar range of abnormality.


http://oem.bm

j.com/

Br J Ind M


ownloaded from

http://oem.bmj.com/


TABLE 5PERCENTAGE OF OPINIONS DIFFERING FROM STANDARD CATEGORIES (ANALYSIS BY OBSERVER AND GROUP OF FILMS WITH

SAME STANDARD CATEGORIES)

Groups of FilmS by Standard Categories

Normal Simple PneumoconiosisObserver ~~~~~~~~~~AllAll All

Non|mi Miners Category I Category 2 Categories Normals Abnormals FilmsNS S NS S NS S NS S NS S NS S NS S NS S

P 38 38 33 67 94 29 81 12 4 8 36 45 52 15 49 24Q 0 6 0 17 61 59 56 12 42 21 0 9 51 30 38 24R 19 38 17 83 67 41 94 0 67 33 18 50 72 26 59 33S .. 56 56 33 100 61 47 31 12 17 8 50 68 34 21 39 34All 28 34 21 67 71 44 66 9 32 18 26 43 52 23 46 29

NS = without standards

erM

films and observers to the general problem of accu-rate radiological diagnosis in pneumoconiosis. Webelieve nevertheless that certain conclusions mayreasonably be drawn from our results and furtherlines of investigation suggested.We find no evidence that increased o.8 -

care in the verbal definition of theradiographic appearances in thecategories assists observers to agree 0*700in their classification of films. Theuse of standard films enabled mostof the observers to improve their 0-600accuracy of diagnosis, but this in-creased accuracy was not uniform.The most experienced observers (W to o.5oZ) were only assisted in the classifica-tion of normal films. These observersachieved reasonable accuracy in the 0 400abnormal films without standards and Dgained no additional accuracy by the Guse of standards. With the less ex- Kperienced observers and especially with F 0-30(the least experienced observers of all, Jthe position was reversed. Considerable Himprovement in accuracy was achieved CE 0-200by the use of standards in the case of B -abnormal films, but these less experi-enced readers appeared to be misled by A 0-100the standard film of category 1 intodiagnosing abnormality where noneexisted. 0

It might be suggested that this result Previous ExpVictor

was found because the standard film Filmscategory 1 pneumoconiosis was reallywithin normal limits. If this were so,normal films might often show a similarappearance when compared with thestandard, and would be diagnosed FIG. 4.--as having category 1 simple pneumo- Obser

S - with standards

coniosis on this comparative basis. This possibilityrequires careful consideration, for several radio-logists and chest physicians of experience have saidthat they consider our films ofcategory I to be within

p 0.800

R 0-700

- 0-600

s0-500

-0-400

Q

0(300 P

QS

0(200 Rz~~

--A-X W

EW A

0ment Present Experiment

Without WithStandards Standards

Excluding normal non-minersand grouping Cats. 3 and 4

INDICES OF DISAGREEMENT

Comparison of previous experiment and present results.rvers common to both experiments joined by dotted lines.

146


http://oem.bm

j.com/

Br J Ind M


ownloaded from

http://oem.bmj.com/


normal limits. Our reason for rejecting thisexplanation is chiefly based on the performance ofobservers W to Z, who, both in their individualreading and especially in consultation, were appar-ently able to see some particular appearance inthe standard film which was not present in thefilms of non-miners. Very careful scrutiny wasnecessary to make this distinction which was onlymade with complete accuracy when they discussedthe films in consultation.

It appears that considerable experience is necessaryfor the identification of the earliest characteristicradiological abnormalities of pneumoconiosis andthat only observers W, X (observers A and B ofour first paper), A, D, and F showed themselvescapable of detecting this appearance with reasonableaccuracy. The other observers, when using thestandard appear to have been misled by someincidental appearance not characteristic of pneumo-coniosis. We are engaged upon a further experimentto test this point, using a larger number of normalfilms. Preliminary results suggest that experiencedreaders can consistently distinguish films of minerswith no more abnormality than the lower limit ofour category 1 from films of men who have neverbeen exposed to dust. Methods of assisting theinexperienced to use the standard film of category 1effectively are being explored.

TIhere is probably considerable individual varia-tion in ability to use standard films. Certainreaders may never use them easily. For instance,observer W, whose accuracy was the best of thewhole group of observers when working withoutstandards, was a quick competent reader andappeared unwilling to change his immediate impres-sion of a film when the standard was presentedto him. He appeared to prefer his own mentalstandard to the objective standard films.We have observed the high consistency of a

consultative reading in which four observers readthe films using the standards and discussed eachfilm until they reached agreement. When thisconsultative reading was repeated on two occasions,and the small number of films on which the twoseparate readings were different were further dis-cussed until agreement was reached, the final resultagreed with the average opinion of all the observerswhen they were using standards in all but three ofthe 80 films. Moreover, both the consultativeopinion and the standard category gave the correctdiagnosis when films which were known to benormal were in question. If it is accepted that theaverage opinion of a set of observers gives the truecategory of a film, consultative readings will alsotend to give this true category.We have at present, therefore, adopted the

consultative technique in the routine reading offilms taken on radiological surveys by the Pneumo-coniosis Research Unit. The films are read ontwo occasions by two or three observers usingstandard films, and those films on which the tworeadings are different are subjected to a thirdscrutiny to reach a final opinion. This techniqueis similar to that suggested by Yerushalmy, Harkness,Cope, and Kennedy (1950) for the most accurateradiographic diagnosis of tuberculosis.

Certain drawbacks and difficulties in the use ofstandards became apparent in the course of ourexperiment. First, there is considerable variety inthe type of appearance in radiographs of coal-workers' pneumoconiosis (Fletcher and others, 1949).The standards used in Parts I and II of our experi-ment were all of the " mixed" type of pneumo-coniosis. A number of films in the series were ofa much finer type (pinhead) than the standards and,in such cases, the observers complained of inabilityto use the standard films for comparison. Theywere being asked, they said, to compare dissimilarappearances. Some confirmation of this dissatis-faction was found by considering those films whichwere predominantly of the pinhead type with thosefilms which were predominantly of the mixed ornodular type. One of us (C. M. F.) and observer Ysorted all the films in category 2, 3, and 4 into thosepredominantly of the pinhead type and thosepredominantly mixed or nodular. It was foundthat in Parts I and II ofthe experiment, the proportionof correct opinions in the pinhead group was 65%/oworking without standards, and 66% working withstandards, whereas in the nodular and mixed group,72% of the opinions were correct without standardsand 79% correct with standards. The lack ofimprovement with standards in the pinhead groupconfirms the observers' expressed difficulty inasses,sing the pinhead films with standards. Forthis reason we have decided that, at least in thecase of category 3, it is necessary to have twostandard films, one of the mixed nodular type andthe other of the pinhead type, so that like can bematched with like. (The set of standards used bythe "inexperienced" observers P, Q, R, and Sincluded a " mixed " and a " pinhead " category 3film.)

Secondly, there is the labour involved in readingwith standards. More than one viewing screen isrequired and for each film read at least one standardhas to be selected and placed on the screen forcomparison. This means that reading with standardsslows the speed at which films can be scrutinized.It is, however, possible that by a specially designedviewing screen, the use of standards could bemade less laborious.


http://oem.bm

j.com/

Br J Ind M


ownloaded from

http://oem.bmj.com/


Even when standard films are used, our resultssuggest that the level of accuracy achieved by theobservers who took part in our experiments is stillfar from satisfactory.

Radiographs are generally used as the chief meansof diagnosis of pneumoconiosis in life for purposesof diagnosis and compensation. It has been sug-gested that men with less than category 2 simplepneumoconiosis are unlikely to be disabled by thisdisease and may be rejected on scrutiny on theirfilms if they apply for disablement benefit on accountof pneumoconiosis. The cases in our experimentwith films of agreed category 2 might then beconsidered for benefit. Of this group of films,20% of the opinions were of category 1 when readwith standards, so that if observers A to F andW to Z were responsible for scrutiny of those cases,we could expect 20% of them to be incorrectlydeprived of assessment for disability benefit, evenif standards were used.

Again, it has been suggested that progressivemassive fibrosis, which is responsible for, the mostserious disablement in coal miners' pneumoconiosis,seldom arises in cases with less than Category 3simple pneumoconiosis. If this were confirmed, andmen were removed from dust exposure before theyreached this level, they would then be protectedfrom developing the serious disability associatedwith massive fibrosis (Fletcher, 1950). To do thisit would be necessary to diagnose Category 2 withaccuracy. We find that, working with standards,10% of the opinions given on films of Category 2simple pneumoconiosis were of Category 3 and19% of the Category 3 films were called Category2. Thus our observers would make some 15%mistakes in diagnosing cases at this " critical stage ".On the other hand, observers W, X, Y, and Zworking together in consultation only mis-diagnosedone of the 17 cases of Category 2 and noneof 16 cases of Category 3. We suggest that forimportant decisions based on radiographical diag-nosis ofpneumoconiosis the technique ofconsultativereadings on at least two separate occasions shouldbe used.

Lastly, we must consider briefly the influence ofradiographic technique on diagnosis which, in ourprevious paper, we showed was considerable. We-did not explore this question in our present experi-ment but we have considerable further experiencein our routine work of the serious influence ofchanges in radiographic technique. Two films ofthe same man on the same day, taken with differenttechniques, can lead to opinions differing by onecategory or occasionally by two categories evenwhen consultative reading with standards is used.We can only emphasize that for accurate radio-

graphic diagnosis it is essential that films shouldonly be used which are of first-class standardizedtechnique, such as that recently recommended bythe Medical Research Council (1950).

SummaryAn experiment was designed to test the value of

"standard films" in the quantitative classificationof radiographs of simple pneumoconiosis of coal-workers, the standard films being examples of thelimits of the categories and used in side-by-sidecomparison with the films to be classified.The experiment fell into two main parts. In the

first, experienced readers from the PneumoconiosisResearch Unit took part; in the second, membersof the pneumoconiosis medical panels, and in asubsequent extension, observers who had no previousexperience of the method of classification.We have no evidence that increased precision in

the verbal descriptions of the categories enabledthe observers to classify the films with any greateragreement than had been found in an earlier experi-ment with less precise definitions.The use of standards enabled some, but not all

of the observers to achieve a greater accuracy andconsistency in classification. The least experiencedobservers benefited most, but they tended to makefalse diagnoses of abnormality in normal filmswhen,working with standards.. On the other hand,the more experienced readers received greater helpin the discrimination of normal films from thosewith slight abnormality, than in the classificationof the more abnormal films. This finding isdiscussed.The greatest consistency was achieved by four

experienced observers reading in consultation withstandard films and their classification agreed veryclosely with the standard categories derived fromaveraging the opinions of all the observers.The chief practical use of standard films appears

to be to enable wholly inexperienced observers touse a radiographic classification with an accuracyapproaching that of the more experienced observers.

It is suggested that in routine work films shouldbe read by at least two observers on two independentoccasions, the films on which the two opinionsdiffer being given a third and deciding reading.Only films of first-class radiographic technique canbe used for accurate diagnosis.

We should like to thank all the observers who helpedus in this experiment -and also Dr. C. L. Sutherland,Chief Medical Officer of the Pneumoconiosis MedicalBoard of the Ministry of National Insurance, foragreeing to the collaboration of observers from thepneumoconiosis medical panels.

148


http://oem.bm

j.com/

Br J Ind M


ownloaded from

http://oem.bmj.com/


REFERENCESDavies, I., Fletcher, C. M., Mann, K. J., and Stewart, A. (1949)

Proc. 9th Int. Congr. Industr. Med. London, 1948, p. 774.,Bristol.

, and Mann, K. J. (1948). Ibid., p. 769.Fletcher, C. M. (1950). Proc. 3rd Int. Conf. on Pneumoconiosis,

Sydney. International Labour Office, Geneva. To bepublished.

Fletcher, C. M., Mann, K. J., Davies, I., Cochrane, A. L., Gilson,J. C., and Hugh-Jones, P. (1949). J. Fac. Radiol., 1, 40.and Oldham, P. D. (1949). British Journal of IndustrialMedicine, 6, 168.

International Labour Office (1950). Proc. 3rd Int. Conf. on Pneumo-coniosis, Sydney. To be published.

Medical Research Council: Industrial Pulmonary Diseases Com-mittee (1950). Brit. J. Radiol., 23, 500.

Yerushalmy, J., Harkness, J. T., Cope, J. H., and Kennedy, B. r.(1950). Amer. Rev. Tuberc., 61, 443.


http://oem.bm

j.com/

Br J Ind M


ownloaded from

http://oem.bmj.com/

Documents

THE USE OF STANDARD FILMS IN RADIOLOGICAL …Standard Films.-The standard films used were the "Victor Medium" films selected in the way that has already beendescribed. Apreliminary