British Journal of Industrial Medicine 1986;43:667-676

Inter-rater agreement in assessing occupationalexposure in a case-control studyM S GOLDBERG,'2 J SIEMIATYCKI,2 M GERIN3From the Department ofEpidemiology and Biostatistics,1 McGill University, Montreal, Department ofEpidemiology and Preventive Medicine,2 Institut Armand-Frappier, Laval, and Departement de m'decine dutravail et d'hygie'ne du milieu,3 Universite de Montreal, Montreal, Quebec, Canada

ABSTRACT The identification of occupational carcinogens in the workplace is a major concern ofepidemiologists. A novel case-control approach has been developed which includes as a keycomponent the assessment of a subject's occupational exposure history by a two stage process.Firstly, the subject is interviewed to obtain a detailed lifetime job history. Then a team of chemistsand hygienists, hired and trained to Po this work on a full time basis, translates each job into a listof potential occupational exposures. The present study investigated the inter-rater agreement in thistype of retrospective exposure assessment. Six trials were carried out over a four year period withdifferent raters and different sets ofjob files. Some trials involved only internal raters from the "inhouse" group whereas others involved comparisons between the internal raters and other externalraters who had expertise in certain industries. In assessing exposure as simply present or absent, twosummary indices of agreement were used: per cent with perfect agreement and Cohen's kappa. Inmost of the trials the per cent with perfect agreement among raters ranged from 95% to 98%, withkappa ranging from 05 to 07. The kappas were slightly higher for internal-internal comparisonsthan for internal-external ones. These results indicate a relatively high degree of inter-rateragreement and lend credibility to the validity of this type of retrospective exposure assessment.

The discovery of occupational exposures which maybe harmful to health is one of the foremost problemsin occupational health. Epidemiological approachesto the problem depend on the juxtaposition of themortality or morbidity experience of workers andtheir occupational exposure histories. Most epi-demiological case-control or surveillance studies ofoccupational disease have used job or industry titlesas indicators for exposure. In some instances high riskoccupations have been identified using this approach.The use ofjob titles, however, may obscure an associ-ation if only a subset of workers with a given job titlewere exposed to the active agent.' 2 Statistical powerwill also be reduced when occupations having a com-mon exposure are not combined.2 Furthermore, evenif an excess risk was observed it would be difficult topinpoint the harmful agents. Ideally, then, studiesshould be based on chemical and physical substancesto which subjects may have been exposed and notsimply job or industry titles.

Since 1979, a cancer case-control study has been

Accepted 3 March 1986

underway in Montreal in which detailed job exposurehistories are ascertained for cases and controls.23 Thekey element is the use of "in house" experts in chem-istry and industrial hygiene to evaluate each subject'sjob history in order to infer a list of past exposures.Other investigators have also turned to the "expertpanel" approach in determining historic exposures ofstudy subjects in population based case-controlstudies4 6 and in cohort studies.7'The credibility of results from such studies depends

in large part on the accuracy of the assessment ofexposure. Little has been published on the validity ofretrospective assessment of occupational exposure byexpert raters, mainly because this approach is com-paratively recent and because of inherent difficultiesof validation. The purpose of the present study was toinvestigate the accuracy of attributing occupationalexposure by a group of chemical coders who useddetailed, lifetime job descriptions obtained from indepth interviews of subjects enrolled in the cancercase-control study mentioned above (referred to asthe "cancer study"). Results will be presented from aseries of six trials that investigated the extent of inter-rater agreement in attributing occupational exposure

667

copyright. on S

eptember 28, 2020 by guest. P

rotected byhttp://oem

.bmj.com

/B

r J Ind Med: first published as 10.1136/oem

.43.10.667 on 1 October 1986. D

ownloaded from

668

among cancer study raters, and between these ratersand other chemists and hygienists from industry.Materials and methods

CANCER STUDYThe objective of the cancer study was to discover pre-

viously unsuspected occupational carcinogens. Abrief summary is given below, but the rationale anddetailed methods are described elsewhere.2 3 10-12

Men, aged 35-70, who were diagnosed in any of themajor hospitals in the greater Montreal area withcancer of any of 19 sites were eligible for entry intothe study. An interviewer visited the subject and con-

ducted a two part interview, composed of a structuredand a semistructured part. The structured sectionrequested information on important potential con-

founders. The semistructured section elicited detaileddescriptions of the occupations that each subject hadheld during his working life. For each reportedoccupation, the interviewers were trained to obtain asmuch information as the subjects were able to supplyon the company's activities, including the raw materi-als; final product; machines used; any responsibilityfor machine maintenance; the type of room or build-ing in which the work was carred out; the activities ofworkmates; the presence of gases, fumes, or dusts;and any other information that could furnish a clueregarding possible chemical or physical exposures.Each completed questionnaire was examined by ateam of coders, composed of chemists, industrialhygienists, and engineers, who then translated eachreported occupation into a list of potential exposures.The team of coders relied on the following sources as

a basis for estimating exposure: their own industrialexperience and chemical knowledge, old and newtechnical and bibliographical material describingindustrial processes, and consultation with expertsfamiliar with particular industries. The coding of

Goldberg, Siemiatycki, Ge'rinexposure was done on a checklist that containedchemical and physical exposures (to be referred to as"substances"). Any other substances believed to havebeen present could also be added to the list. The com-position of the checklist has evolved over the periodof the cancer study. Initially, there were 172 sub-stances; then for a period there were 259; a year laterthere were 270; and now there are 275. In the day today coding (referred to as the "routine coding")exposure was attributed through a type of consensusprocess whereby one coder intially attributedexposure and one or more other chemical codersreviewed the original codings.For each job description in each study subject's

work history, the project chemists indicated whetherany of the substances on the checklist may have beenpresent in the worker's environment. For each sub-stance which was checked off as possibly present, thecoders further indicated: (a) their confidence that theexposure had actually occurred ("possible," "proba-ble," or "definite"); (b) the average level of concen-tration ("low", "medium," or "high"); (c) the fre-quency of exposure during a normal workweek ("lessthan 5%," "5-30%," or "more than 30%"), and(d) the type of contact ("respiratory," "cutaneous,"or "both").

INTER-RATER AGREEMENT TRIALSTrue validations were impossible because of theabsence of valid historical information on theexposures of the study population. The trials of inter-rater agreement were intended, nevertheless, to shedlight on the accuracy of the chemical coder's exposureassessments. Some of the trials were designed todetermine the degree of agreement in coding amongthe chemical coders of the cancer study (hereafterreferred to as "internal raters"), whereas others weremeant to compare the assessments of the internalraters with those of external raters. The latter were

Table I General characteristics ofthe agreement trials

Job descriptions examined No ofraters

Selected or obtained No ofsubstances onTrial Date Industry/occupation from No exposure checklist at that time Internal External

A 1981 Paint manufacturing Cancer study 5 172 2 1B 1981 Manufacturing of rubber Cancer study 15 172 3 0

productsC 1982 Welding and soldering Cancer study 18 259 2 1*D 1984 Chemical manufacturing Company records 5 275 It IE 1984 Manufacturing of metal Interviews 7 275 It

productsF 1984 Miscellaneous assortment Cancer study 23 270J 2% 0

*Three external raters coded exposure as a group.tA panel of two internal raters attributed exposure as part of the routine coding.$Number of substances on the checklist at the time of the first coding.%Two routine codings by two teams of internal raters were performed at two different times.

copyright. on S

eptember 28, 2020 by guest. P

rotected byhttp://oem

.bmj.com

/B

r J Ind Med: first published as 10.1136/oem

.43.10.667 on 1 October 1986. D

ownloaded from

Inter-rater agreement in assessing occupational exposure in a case-control studyexperts in various industrial sectors, engaged solelyfor the purpose of participating in one or another ofthe trials. In total, six trials of inter-rater agreementwere carried out between 1980 and 1984; the mainfeatures of each of the trials are presented in table 1,in chronological order of the completion of the trials.Owing to personnel turnover, different internal

raters participated in the various trials. These ratershad different professional backgrounds and varyingdegrees of coding experience. As indicated above, thecoding checklist evolved over time and the varioustrials involved versions containing 172, 259, 270, and275 substances.

Since the cancer study is a population based case-control study, the whole range of occupations andindustries in the Montreal area appears among studysubjects. For most of the trials the job descriptionsthat were evaluated were taken from among cancerstudy subjects. For two of the trials (D and E), jobfiles were generated from outside sources-one fromhygiene records of a Canadian chemical manu-

facturer and one from interviews done with metalworkers whose working environment had recentlybeen measured by hygienists.For each trial, a set ofjob descriptions was selected

according to the criteria outlined below, and the vari-ous raters participating in the trial independentlycoded exposure for each job using the exposurechecklist in use at the time of the trial. In someinstances one of the "raters" was a team operating byconsensus. Whenever external raters participated theywere trained in the methodology and criteria of thecoding process, though care was taken not toinfluence their substantive judgements. More detailsabout each trial are presented below.

STATISTICAL METHODSThe primary goal of the analyses was to determine theextent of agreement between raters in identifying thepresence or absence of exposure. Thus for each trialagreement was assessed on a binary scale (present,absent). The methods used in these analyses, how-ever, were extended, as described below, when, forexample, agreement was assessed for concentration ofexposure on a four-point measurement scale.

Suppose that n job descriptions were examined by r

raters, coding exposures on a checklist of s sub-stances. For each of the s substances it was the-oretically possible to assess the agreement among r

raters on the n job descriptions. Determining agree-ment on a substance by substance basis posed twoproblems: (a) since the number ofjob descriptions, n,

was small (from 5 to 23, depending on the trial), thevariability of any agreement statistic concerning a

specific substance was quite large and (b) since thenumber of substances, s, was large (from 172 to 275,

depending on the trial), it would be difficult to assimi-late the multitude of results. Thus some method ofsummarising agreement over the s substances had tobe found. To this end, the n x s decisions of eachrater were treated as independent observations andthe agreement in rating among r raters was assessedover these n x s observations.For each trial a 2' table-that is, 2 x 2 for a two

rater trial, 2 x 2 x 2 for a three rater trial-wasderived. Each of the n x s entries were classifiedaccording to the combination of raters coding thempresent or absent. For instance, the data layout fortrial A in which there were three independent raters ispresented in table 2. There were 860 (5 x 172) obser-vations distributed among the eight combinations inthis table.The data in the 2' table was first synthesised by

counting the number of distinct patterns of agree-ment, ignoring the identities of the raters. Based onthe marginal proportions for each rater, an expectednumber for each pattern of agreement was estimatedassuming the null hypothesis that the raters' codingswere statistically independent of each other. Table 3shows the results of such an analysis for trial A, asderived from table 2. Such a table provides a fairlycomplete, yet concise, description of the results.

Several parameters were then estimated from thissummary table and, for this example, from the three2 x 2 tables derived from pairwise comparisons. Foreach rater, the number of substances attributed wassummed over all n job descriptions and then dividedby the total possible number of such assignments(n x s). (This proportion will be referred to as the"proportion of exposures attributed.") Equality ofthe three proportions was tested using Cochran'sQ-index.'3 For each pair of raters, the equality ofthe two proportions was tested using McNemar'stest.14 Both of these statistics are asymptotically dis-

Table 2 Detailedpattern ofagreement in trial A, in whichthree raters examinedfivejob descriptions using a checklistcontaining 172 substances

Substance rated as:

Rater I Rater 2 Rater 3 No of(Internal) (Internal) (External) occurrences

p p p 23p p a 19p a p 9a p p 0p a a 28a p a 8a a p 6a a a 767

Total 860*

p = Exposure was "present."a = Exposure was "absent."*860 = 5 x 172 = number of observations.

669

copyright. on S

eptember 28, 2020 by guest. P

rotected byhttp://oem

.bmj.com

/B

r J Ind Med: first published as 10.1136/oem

.43.10.667 on 1 October 1986. D

ownloaded from

670tributed as chi-squared, with the number of degrees offreedom equal to the number of raters less one.Among several measures of agreement available for

the simple two dimensional tables, 13,15 two werechosen for presentation: the index of crude agree-ment,16 po, which simply equals the proportion ofobservations for which all raters agreed that exposurewas present or absent, and Cohen's kappa index.1 5,17The latter explicitly accounts for the extent of agree-ment expected on the basis of chance, pe, assumingstatistical independence of the raters' codings. Kappais calculated by subtracting expected from crudeagreement (po- pe) and then dividing by the maxi-mum expected excess agreement (I - p). Thus avalue of kappa of zero indicates a degree of agreementthat would be expected by chance (po = pe). Themaximum value of kappa is dependent on themarginal distributions. When these are equal, kappacan attain a maximum value of unity; otherwise, it isless than one."7 Values of kappa greater than 0 5 areconsidered to represent "good to excellent" agree-ment.15,18 For trials in which the coding ofmore thantwo raters was compared simultaneously, a general-ised version of the index of crude agreement, knownas the mean majority agreement index (MMAI),'9was used. This index is equal to an average, over then x s observations, of the proportion of ratings inwhich there was unanimous agreement for eachexposure assignment. It has a maximum value of 100%.Furthermore, a generalised version of kappa,'5 basedon the intra class correlation coefficient16 was alsoused. The pairwise agreement statistics and related95% confidence intervals (95% CI) were computed bymeans of a FORTRAN program (M S Goldberg,unpublished observations).

Goldberg, Siemiatycki, Gerin

Detailed results will be presented for each trialbased on classifying exposure as either "present" oras "absent." Thus a substance labelled as present inthese analyses could have been coded with any level ofdose, as defined by the various combinations offrequency and concentration. Other analyses were infact carried out but will be presented in summary

form only. One set was similar to the present/absentanalysis but used a different threshold for dis-tinguishing present from absent-namely, only thoseexposures which were rated as "definitely present"were counted as present. Another set of analysesconcerned the extent of agreement in ratingconcentration, frequency, and type of contact.

Description and results of each trial

TRIAL ATrial A was a comparison between two cancer studyraters and an outside expert. The external rater was

an industrial chemist who had more than 10 yearsexperience in the paint manufacturing industry.Unlike the routine coding procedure, the internalraters coded these files independently without a con-sensus rating. Thus the statistical comparisons were

between the two internal raters and between each ofthem and the external rater. Five job descriptions ofpaint industry workers were selected from the bank ofcancer study files.The results from this trial are shown in tables 2

and 3. Internal rater 1 attributed a significantly largernumber of exposures than either of the other tworaters (p < 0-05). Nevertheless, the observed patternof agreement was much better than that expected bychance alone. A kappa of 0-66 and an MMAI of95%,

Table 3 Pattern ofagreement and selected summary statisticsfor trial A, in which three raters evaluatedfivejobdescriptions* using a checklist of172 substances

No ofraters attributing exposure as No ofsuch occurrences

Observed Expectedt

Present Absent No % No %

3 0 23 2-7 0.1 0.02 1 28 33 56 071 2 42 4-9 125-6 14-60 3 767 89-1 728-7 84-7

Total 860$ 100 8601 100

% Present according to§: internal rater I = 9 2%internal rater 2 = 5 8%external rater = 4-4%

Summary statisticsll: MMAI = 95%Kappa = 066 (95% CI: 0.62-0.69)

*Job descriptions were selected from those subjects in the cancer study who had been employed in the paint manufacturing industry.tThis was computed conditional on the percentage of items checked by each rater and represents the expected distribution of agreement if theexposure assessments were statistically independent.$860 = 5 x 172.§Proportion of exposures coded present by internal rater 1 was significantly greater (p < 0-05) than that coded by the other raters.IjThese statistics represent crude agreement (MMAI) and chance corrected agreement (Kappa) among all three raters simultaneously.

copyright. on S

eptember 28, 2020 by guest. P

rotected byhttp://oem

.bmj.com

/B

r J Ind Med: first published as 10.1136/oem

.43.10.667 on 1 October 1986. D

ownloaded from

Inter-rater agreement in assessing occupational exposure in a case-control study

Table 4 Pattern ofagreement and selected summary statisticsfor trial B, in which three raters evaluated 15job descriptions*using a checklist of 172 substances

No ofraters attributing exposure as No ofsuch occurrences

Observed Expectedt

Present Absent No % No %

3 0 61 1-5 0-2 0 02 1 70 1-7 13-8 031 2 94 2-3 3930 970 3 3825 94 4 3643-0 90 0

Total 4500$ 100 4500t 100

% Present according to§: internal rater I = 4 0%internal rater 2 = 3-6%internal rater 3 = 2-6%

Summary statistics 11: MMAI = 97%Kappa = 059 (95% CI: 058-061)

*Job descriptions were selected from those subjects in the cancer study who had been employed in rubber products manufacturing.tSee table 3.t4500 = 15 x 172.§Proportion of exposures coded present by internal rater 3 was significantly less (p < 0-05) than that coded by the other raters.IlSee table 3.

both indicative of excellent agreement, were observed.

TRIAL BTrial B concerned concurrent independent coding bythree of the cancer study raters. Fifteen job descrip-tions were selected from the cancer study, all ofrubber industry workers.The results of this trial are presented in table 4.

Two of the raters attributed about the same numberof exposures; the third, however, coded fewer. As inthe previous trial, the number of observed exposures

in which there was perfect agreement was greater thanthat expected by chance. This was reflected in thesummary indices which indicated excellent agreement(MMAI = 97%; kappa = 0.59).

TRIAL CIn trial C there were three external raters who attrib-uted exposure as a team. This group of raters consis-ted of an engineer engaged in welding research at a

Montreal university, a welder employed in a tradeassociation, and a welding instructor with over 30years experience in industry. Two internal ratersattributed exposure independently of each other. Forthis trial 18 job descriptions of welding and solderingworkers were selected from the files of the cancer

study.The results for this trial are shown in table 5. There

were significant differences among raters in the pro-

portion of exposures attributed. Nevertheless, theextent of crude agreement (MMAI = 93%) and

Table 5 Pattern ofagreement and selected summary statisticsfor trial C, in which three raters evaluated 18job descriptions*using a checklist of259 substances

No ofraters attributing exposure as No ofsuch occurrences

Observed Expectedt

Present Absent No % No %

3 0 196 4-2 25 0-12 1 162 3-5 85-4 1-81 2 237 51 968-6 20-80 3 4067 87-2 3605-5 77-3

Total 4662t 100 4662t 100

% Present according to§: internal rater I = 9-7%internal rater 2 = 8-4%external panel = 6-6%

Summary statisticsll: MMAI = 93%Kappa = 0-62 (95% CI: 0-60-0-64)

*Job descriptions were selected from those welders who had been interviewed during the course of the cancer study.tSee table 3.$4662 = 18 x 259.§Proportion of exposures coded present differed significantly (p < 0-05) among all three raters.IISee table 3.

671

copyright. on S

eptember 28, 2020 by guest. P

rotected byhttp://oem

.bmj.com

/B

r J Ind Med: first published as 10.1136/oem

.43.10.667 on 1 October 1986. D

ownloaded from

Table 6 Pattern ofagreement and selected summary statisticsfor trial D, in which two raters evaluatedfive job descriptions*using a checklist of275 substances

No ofraters attributing exposure as No ofsuch occurrences

Observed Expectedt

Present Absent No % No %

2 0 20 1 5 08 00I 1 35 25 67 1 490 2 1320 960 1307 1 951

Total 1375$ 100 1375+ 100

% Present according to§: internal panel = 20%external rater = 30%

Summary statistics: po = 97 5%Kappa = 0 52 (95% CI: 0 38-0-66)

*Job descriptions were generated from job function sheets supplied by an industrial hygiene department of a large chemical manufacturer.tSee table 3.$1375 = 5 x 275.§Proportion of exposures coded present by the two raters differed significantly (p < 005) from each other.

chance corrected agreement (kappa = 062) were

high.

TRIAL DTrial D was carried out with the collaboration of theindustrial hygiene department of a large Canadianchemical manufacturer. They selected for this trial a

plant located in another Canadian city, in whichdetailed job function and hygiene descriptions hadbeen recorded for their own internal use. Interviewingworkers was impossible because of the distance.Instead, job descriptions were generated, using theinformation in the job function sheet and a company

hygienist's personal knowledge of the workplace. Oneof the cancer study interviewers, the hygienist and one

of the authors (MSG) created job descriptions thatworkers might have been expected to provide through

interviews. Five such job descriptions were developed,based on jobs in different parts of the plant. Thesewere put into the routine coding work of the team ofinternal raters and they carried out their usual con-

sensus rating procedure, unaware that these jobs were

not really part of the cancer study. The company

hygienist was the external rater.There were significant differences in the two esti-

mates of the proportion of exposures attributedpresent (p < 0 05), with the internal panel coding one

third as many exposures as the external rater (table 6).Crude agreement was high (po = 97-5%), however,and was significantly greater than that expected bychance (kappa = 0 52).

TRIAL ETrial E exploited the fact that hygienists working in a

Table 7 Pattern ofagreement and selected summary statisticsfor trial E, in which two panels ofraters evaluated sevenjobdescriptions* using a checklist of275 substances

No ofraters attributing exposure as No ofsuch occurrences

Observed Expectedt

Present Absent No % No %

2 0 56 2-9 53 031 1 84 4-4 191-6 9 90 2 1785 92-7 17281 89-8

Total 1925$ 100 1925t 100

% Present according to§: internal panel = 4-7%external panel = 5 8%

Summary statistics: po = 95-6%Kappa = 059 (95% CI: 0-51-0-67)

*Job descriptions were obtained from interviews of employees currently employed at two metal fabrication plants.tSee table 3.t1925 = 7 x 275.§Proportion of exposures coded present by the two raters were significantly different (p < 005) from each other.

Goldberg, Siemiatycki, Ge'rin672

copyright. on S

eptember 28, 2020 by guest. P

rotected byhttp://oem

.bmj.com

/B

r J Ind Med: first published as 10.1136/oem

.43.10.667 on 1 October 1986. D

ownloaded from

Inter-rater agreement in assessing occupational exposure in a case-control study

local community health department had recently con-ducted hygiene surveys in selected local industries.Among these industries were two metal fabricationplants. As a result of this programme, detailedmeasurements of concentrations of some airbornematerials in the vicinity of each worker wereobtained. The survey also provided information onmaterials and equipment used by each employee. Achemical breakdown of some of the materials usedwas also obtained from some manufacturers.Two of the hygienists who had conducted these

surveys agreed to participate in the trial. Sevencurrent employees at the two plants were interviewedby one of the cancer study interviewers to obtain acurrent job description-similar to those routinelyobtained in the course of the cancer study. One of thesubjects was a drill press operator, one a metalpolisher, one a painter, two were welders, and twometal forming workers. The resulting job descriptionswere coded by the two hygienists who had carried outthe detailed surveys and their evaluations providedone consensus rating. As in the previous trial, the jobdescriptions were placed in the routine coding of theteam of internal raters.

Table 7 shows the detailed and synthesised results.Although there were significant differences inestimating the total prevalence of exposure(p < 005), there was considerable overlap in theexposures attributed by the two raters (po = 95-6%;kappa = 0-59).

TRIAL FTrial F was an outgrowth of trials D and E.Normally, in the cancer study, the interview provideda job description for each job that the subject had hadin his working life. For trials D and E, the external

raters had expert knowledge only of the worker'scurrent job, and only these were obtained in conjunc-tion with the external raters. Had only the current jobdescription for a given worker been provided, theteam of internal raters would have realised that aspecial investigation was under way. To mask this, acomplete lifetime job history was generated for the 12job descriptions that had formed the basis of trials Dand E. This was done by adding to the current jobdescription one or two others which were selectedfrom among the thousands of job descriptions onhand in the cancer study-that is, one or two jobdescriptions already coded by our internal raters atsome time in the preceding five years were copied outby an interviewer as if they were part of a recent inter-view. The internal raters did not recall that these jobdescriptions had already appeared in the study. Atotal of 23 job descriptions that had been rated pre-viously by the internal raters were recoded in trials Dand E. It was not necessarily the same individualswho rated the 23 jobs on the two separate occasions.This was because of changes in personnel through thecourse of the cancer study and because different sub-sets of raters would assess different files in the routinecoding. Nevertheless, the trial represents a com-parison of the routine coding at two points in time,thus providing what might be termed a test ofreliability of the coding system. In addition, becauseof changes in the number of substances under consid-eration, the first coding of these files used an exposurechecklist containing 270 substances whereas thesecond was based on a list of 275 substances. Thus thetrial was analysed using 270 substances.As shown in table 8, the two panels of internal

raters coded nearly identical numbers of exposures.High levels of crude (po = 98-5%) and chancecorrected agreement (kappa = 067) were observed.

Table 8 Pattern ofagreement and selectedswnmary statisticsfor trial F, in which two panels ofinternal raters evaluated 23job descriptions* using a checklist of270 substances

No ofraters attributing exposure as No ofsuch occurrences

Observed Expectedt

Present Absent No % No %

2 0 99 1-6 3-4 0-0I 1 94 1-5 285-2 4-6O 2 6017 96-9 5921-4 95-4

Total 62101 100 6210$ 100

% Present according to: internal panel I = 2-3%internal panel 2 = 2-4%

Summary statistics: po = 98-5%Kappa = 0-67 (95% CI: 0-61-0-73)

*Job descriptions were selected from the entire pool of files on hand in the Cancer Study that had been previously attributed in theroutine coding.tSee table 3.T6210 = 23 x 270.

673

copyright. on S

eptember 28, 2020 by guest. P

rotected byhttp://oem

.bmj.com

/B

r J Ind Med: first published as 10.1136/oem

.43.10.667 on 1 October 1986. D

ownloaded from

Table 9 Selected summary statistics ofagreementfor all pairwise comparisons in all trials ofinter-rater agreement

Statistics ofagreementfor

Internal-internal pairs Internal-external pairs

No of No ofpairs of Average Po No ofpairs of Average pOTrial observations* ofraters % Average Kappa ofraters % Average KappaA 860 1 97-8 0-62 2 95 5 0 51B 4050 3 97-3 0 59 0 - -C 4662 1 97-9 0-69 2 95 5 0-58D 1375 0 - - 1 97 7 0 52E 1925 0 - - 1 95 9 0.59F 6210 1 98-5 0 67 0 - -

Average 6 97-8 0-63 6 959 0 55or total:

*Equal to the number ofjob descriptions multiplied by the number of substances on the relevant coding checklist.

ADDITIONAL RESULTSSome trials compared internal raters with each other,some compared internal with external raters, andsome did both. Table 9 shows, in synthesised format,the summary agreement indices between all pairs ofraters, distinguishing the internal-internal com-parisons from the internal-external comparisons.Crude and chance corrected agreement was some-what higher for comparisons among internal ratersthan for internal-external ones.

In five of the trials there were statisticallysignificant differences between raters in the propor-tion of exposures attributed. When the same analyseswere run, however, using a more stringent thresholdfor classifying exposure as present-namely,definitely present-then differences in the proportionof exposures attributed were considerably reduced.None the less, using this more stringent threshold,values of po and kappa were virtually unchangedfrom those presented in the present/absent analyses.For each substance checked off as present, the rater

had to indicate on three point scales the concentrationand frequence of exposure and whether the exposurewas cutaneous, respiratory, or both. To assess agree-ment, absence of exposure was added as anothercategory to these scales, yielding, in effect, three four-category variables: concentration, frequency, andtype of contact. Analyses on these four point scalesfor each of these three variables were conducted,analogously to those carried out for the binarypresent/absent variable. Averaged over all pairwisecomparisons in all trials, the following results wereobtained: concentration, average kappa = 041;frequency, average kappa = 0-46; and type of con-tact, average kappa = 0-52. As in the absent/presentanalysis, agreement among internal raters was slightlyhigher than that observed between internal andexternal raters.

Discussion

Inter-rater agreement was assessed by three generalapproaches: (1) comparison of the different raters'estimates of proportion ofexposures attributed (usingMcNemar's test and Cochran's Q-index), (2) com-parison of observed and expected distributions ofrater agreement, and (3) estimation of crude andchance corrected agreement statistics (po and kappa).The interpretation of the significant differences inproportions of exposures attributed is difficultbecause the tests used are sensitive and might detectreal but small effects. Indeed, although the McNemartests usually indicated significant differences inexposure prevalence-that is, proportions of sub-stances attributed-the differences were generallysmall. In only one case (see table 3) was there a seri-ous discrepency between the raters. We believe that inevaluating inter-rater agreement, emphasis should beplaced on the crude and chance corrected indices-that is, po and kappa-and on the comparison ofobserved and expected patterns of agreement.

Despite differences in estimates of exposure preva-lence between some raters, both crude and chancecorrected statistics indicated, in all trials, that therewas "good" to "excellent" agreement in identifyingexposures. Furthermore, good agreement wasobserved in coding frequency, concentration, andtype of contact of exposure. The kappa statistics fromthese latter analyses indicated somewhat lower agree-ment than that observed in the present/absent anal-yses. This was expected since these exposure indiceswere evaluated on four point scales, thus allowing agreater chance for disagreement. Over all, the resultswere encouraging.One theoretical limitation to the use of these

methods is that expected values were calculated underthe assumption that each rater's various assessments

674 Goldberg, Siemiatycki, Ge'rin

copyright. on S

eptember 28, 2020 by guest. P

rotected byhttp://oem

.bmj.com

/B

r J Ind Med: first published as 10.1136/oem

.43.10.667 on 1 October 1986. D

ownloaded from

Inter-rater agreement in assessing occupational exposure in a case-control study

are independent of each other. Strictly speaking, thisassumption was violated, because a rater's coding ofone substance in a job description may well haveinfluenced his judgement on others. It is unlikely,however, that the results were appreciably affected.

These trials were intended to investigate the valid-ity and reliability of retrospectively assessing occupa-tional exposure by technically trained personnel. Theevidence regarding validity is indirect since none ofthe trials was based on a "gold standard" by which tojudge the coding of the cancer study raters. In sometrials the external raters' coding may be considered tobe closer to the truth than that of the internal raters,but in no instance can there be assurance that any ofthe external raters had perfect knowledge ofexposure. Indeed, it should not be assumed that theexternal raters necessarily provided more valid cod-ings than the internal raters. While the external raterswere expert in specific industrial processes, they werenot experienced in evaluating workers' exposures.Thus the lower degree of agreement between internaland external raters as compared with that amonginternal raters does not necessarily indicate that thecoding of the internal raters was less valid. The higheragreement among internal raters, however, may bepartly explained by the fact that, as a result of theirclose working arrangement, common attitudes wereengendered and a common base of knowledge wasdeveloped over time. This is best exemplified by theresults of trial F, in which a high degree of agreementwas observed between two panels of internal raterswho, using the cancer study consensus approach,attributed exposure to the same set of files on twodifferent occasions.The four trials comparing the coding of internal

and external raters showed that, for both individualand teams of coders, there was good agreement overa range of industries and occupations and timeperiods of employment. In trial E routine cancerstudy coding was compared with that of two indus-trial hygienists for a set of current occupations in themetal fabrication industry. The exposure data used bythe external raters was obtained from a fairly thor-ough occupational survey, whereas in the other trialsreliance was placed on the raters' personal knowl-edge. In trials A and C assessments of exposure weremade by individual raters for paint manufacturingand welding occupations in which employment hadoccurred in the past. In trial D job descriptions weregenerated from the hygiene records of a chemicalmanufacturer. The job descriptions were fictitiousand thus were of poorer quality than those of theother trials. Furthermore, the type of industrialoperation of the manufacturer is not found in thecancer study's catchment area, so that the internalraters had virtually no experience with the specific

chemical processes used. Nevertheless, a relativelyhigh degree of agreement between the internal andexternal raters was observed.The industries chosen as a focus for the trials-

paint, chemical, metal, welding-are among the mostcomplex and difficult for the chemical coding ofexposure. A general population case-control study,such as the cancer study, would include a crosssection of jobs and industries that on average wouldbe easier to code than those evaluated in most of thetrials of this report. Thus the levels of inter-rateragreement shown here probably represent lowerlimits of inter-rater agreement for the whole range ofdossiers in the cancer study. Combining the results ofthis study with the previous finding that there is nosubstantial underreporting of jobs by interviewees,20it appears that the translation into exposures of aworker's reported job history by a group of chemistsand hygienists provides reasonably valid data.

There have been few other reports of validations ofretrospective occupational exposure assessmentsmade by expert raters. In a study of petrochemicalworkers8 exposure to eight substances was assessedby company engineers from a dictionary of job titleswhich contained the date and area in the plant ofemployment for each employee. Much lower levels ofagreement were observed there than in the presentstudy. This may be attributed to the lack of detail inthe job title and, perhaps, to an inconsistent use of thecoding criteria. In a prospective monitoring pro-gramme of a chemical plant,7 exposure coded bycompany engineers was indirectly validated by veri-fying that those workers diagnosed with angio-sarcoma were attributed higher levels of exposure tovinyl chloride, a known liver carcinogen, than a groupof controls.The results from this study lend credibility to the

notion that past exposure to specific occupationalagents can be assessed reasonably accurately. Severalquestions remain outstanding. For example, it is notclear whether validity is affected by the nature of thejob descriptions available to the raters and by the skilland experience of the raters. Also of importance is thedegree to which agreement may vary from substanceto substance. More work is clearly required to assessthis potentially valuable epidemiological instrument.

We are grateful to Drs Graham Gibbs and GillesLebeau and Mr Jacques Guenette for permittingaccess to industrial sources of data. Chemical codingin the cancer study was performed by Denis Begin, DrJoseph Hubert, Howard Kemper, Lucien Laroche,and Christian Millet. Many thanks to Jean-PaulBoillot, J Dore, Michel Galopin, Denis Lessard, PeterNichol, Roger Tremblay, and Peter Wrzesien for theirtime and effort in performing the external codings.

675

copyright. on S

eptember 28, 2020 by guest. P

rotected byhttp://oem

.bmj.com

/B

r J Ind Med: first published as 10.1136/oem

.43.10.667 on 1 October 1986. D

ownloaded from

676

Denise Bourbonnais performed the interviews andselected job descriptions that were used in three of thetrials. Much of the data management was supervisedby Lesley Richardson who has also contributed tovarious aspects of design. We appreciate the helpfulcomments of Nancy Mayo and many thanks toClaudette Richer for preparing the manuscript.

This research was supported by grants from theInstitut de recherche en sante et en securite du travaildu Quebec, the National Health Research and Devel-opment Program, and the National Cancer Instituteof Canada. Mr Goldberg gratefully acknowledgesreceipt of personal support from the Institut derecherche en sante et en securite du travail du Quebec,the Fonds FCAC pour l'aide et le soutien a larecherche, and McGill University.

References

1 Hoar SK, Morrison AS, Cole P, Silverman DT. An occupationaland exposure linkage system for the study of occupational car-cinogenesis. J Occup Med 1980;22:722-6.

2 Siemiatycki J, Day NE, Fabry J, Cooper JA. Discovering car-cinogens in the occupational environment: a novel approach.JNCI 1981;66:217-25.

3 Siemiatycki J. An epidemiological approach to discoveringoccupational carcinogens by obtaining better information onoccupational exposures. Recent Advances in OccupationalHealth 1984;2:143-57.

4 Macaluso M, Vineis P, Continenza D, Ferrario F, Pisani P,Andisio R. Job exposure matrices: experience in Italy. In:Acheson ED, ed. Job exposure matrices. Southampton: MedicalResearch Council, 1982:22-30.

5 Hernberg S, Collan Y, Degerth R, et al. Nasal cancer andoccupational exposures: preliminary report of a joint Nordiccase-referent study. Scand J Work Environ Health1983;9:208-13.

Goldberg, Siemiatycki, Ge'rin6 Hernberg S, Korkala N, Asikainen U, et al. Primary liver cancer

and exposure to solvents. Int Arch Occup Environ Health1984;54: 147-53.

7 Greenberg RA, Tamburro CH. Exposure indices for epi-demiologic surveillance of carcinogenic agents in an industrialchemical environment. J Occup Med 1981;23:353-8.

8 Soskolne C. Upper respiratory cancer among refinery and chemicalplant workers: a case-control study in Baton Rouge, Louisiana.Philadelphia: University of Pennsylvania, 1982. (PhD dis-sertation.)

9 Bond G, Sobel W, Shellenberger R, Flores G. Validation of workhistories obtained from interviews. Am J Epidemiol1985;122:536-7.

10 Siemiatycki J, Gerin M, Hubert J. Exposure-based case-controlapproach to discovering occupational carcinogens: preliminaryfindings. In: Peto R, Schneiderman M, eds. Quantification ofoccupational cancer. (Banbury report No 9.) New York: ColdSpring Harbor Laboratory, 1981:471-83.

11 Siemiatycki J, Gerin M, Richardson L, Hubert J, Kemper H.Preliminary report of an exposure-based, case-control mon-itoring system for discovering occupational carcinogens. Terato-genesis Carcinog Mutagen 1982;2:169-77.

12 Gerin M, Siemiatycki J, Kemper H, Begin D. Obtainingoccupational exposure histories in epidemiologic case-controlstudies. J Occup Med 1985;27:420-6.

13 Landis JR, Koch GG. A review of statistical methods in the anal-ysis of data arising from observer reliability studies, parts I and2. Statistica Neerlandica 1975;29:101-23, 151-61.

14 Bennett BM. Tests of hypotheses concerning matched samples.Journal of the Royal Statistical Society B 1967;29:468-74.

15 Fleiss JL. Statistical methods for rates and proportions. 2nd ed.New York, John Wiley and Sons, 1981.

16 Rogot E, Goldberg ID. A proposed index of measuring agree-ment in test-retest studies. J Chronic Dis 1966;19:991-1006.

17 Cohen J. A coefficient of agreement for nominal scales. Edu-cational and Psychological Measurement 1960;20:37-46.

18 Landis JR, Koch GG. The measurement of observer agreementfor categorical data. Biometrics 1977;33:159-74.

19 Armitage P, Blendis LM, Smyllie HC. The measurement ofobserver disagreement in the recording of signs. Journal of theRoyal Statistical Society A 1966;129:98-109.

20 Baumgarten M, Siemiatycki J, Gibbs GW. Validity of work his-tories obtained by interview for epidemiologic purposes. Am JEpidemiol 1983;118:583-91.

Destruction of manuscripts

From 1 July 1985 articles submitted for publicationwill not be returned. Authors whose papers arerejected will be advised of the decision and themanuscripts will be kept under security for threemonths to deal with any inquiries and then des-troyed.

copyright. on S

eptember 28, 2020 by guest. P

rotected byhttp://oem

.bmj.com

/B

r J Ind Med: first published as 10.1136/oem

.43.10.667 on 1 October 1986. D

ownloaded from