Predictive Validity of the Emergency Physician and Global Job Satisfaction Instruments

234 ACADEMIC EMERGENCY MEDICINE MAR 1998 VOL SIN0 3

Predictive Validity of the Emergency Physician and Global Job Satisfaction Instruments Stephen Lloyd, MD, MSc, David Streiner; PhD, Sue Shannon, PhD

I ABSTRACT

Objective: To evaluate the predictive validity of the Emergency Physician Job Satisfaction (EPJS) and Global Job Satisfaction (GJS) instruments. Methods: Prospective mail survey of 223 Canadian emergency physicians (EPs) using a 42-item questionnaire, including 14 items evaluating their reasons for leaving emergency medicine (EM). Original (1990) EPJS and GJS scores were analyzed using l-way ANOVA and Scheffe’s test comparing the physicians who left EM with those still in their original jobs, and those who had left their original jobs but who stayed in EM. Mean scores on the 14 “reason for leaving” items were compared with scores from an earlier sample of U.S. physicians using a t-test for independent means. Criteria for statistical significance were set at a = 0.05 for all analyses. Results: The response rate for the primary study questions was 99.1%. Of the respondents, 29.4% had left their original jobs, and 10.4% had left EM altogether. The GJS scores for the physicians who left EM were significantly different from those for the physicians who stayed ( p = 0.004). The EPJS scores for the physicians who left EM were not significantly different from those for the physicians who stayed ( p = 0.56). There was no significant difference in scores between the Canadian and U.S. physicians’ reasons for leaving EM (all p- values > 0.05). Shiftwork scored the highest as a reason to leave EM. Conclusions: A low GJS score is associated with physicians’ leaving EM, but not with changing jobs. The EPJS instrument was not associated with either outcome. Canadian and U.S. EPs place similar levels of importance on potential reasons for leaving EM. Key words: emergency health services; manpower; physician workforce; job satisfaction.

Acad. Emerg. Med. 1998; 5~234-241.

.....................................................................................................................................................

I To maintain a high-quality emergency health services system, we need to keep experienced physicians with proven clinical skills on the job. While specialty recog- nition of emergency medicine (EM) has been achieved, ..................................................................................

From McMaster Universify, Hamilton, Ontario, Canada, Division of Emergency Medicine (SL, SS), Department of Clinical Epidemiology (SL., DS); and the Ontario Institute for Clinical Evaluative Sciences, Hamilton, Ontario, Canada (SL).

Received: February 18, 1997; revision received: August 15, 1997; ac- cepted: August 25. 1997; updated: September 20. 1997.

Prior presentation: SAEM annual meeting, Denver, CO, May 1996.

Address for correspondence and reprints: DI: Stephen Lloyd, Division of Emergency Medicine, McMaster University, Room 2Q61, 1200 Main Street West, Hamilton, Ontario, Canada, L8N 325. Fax: 905-521-5069; e-mail: [email protected]:ca

questions remain about the viability of EM as a career. Discussions at national, provincial, and regional meetings continue to highlight problems with emergency physician (EP) workforce shortages, and physician turnover. Earlier research suggests high turnover rates in stressful work en- vironments, particularly among above-average employees who have a choice.”’ This research also shows increased costs to the organization, high levels of job dissatisfaction, and poor work performance associated with high em- ployee turnover.’.’

The 79-item multidomain Emergency Physician Job Satisfaction (EPJS) and 12-item single-domain Global Job Satisfaction (GJS) instruments were developed in 1990 using 7-point Likert scaling devices. Responses to items within the 6 EPJS domains are summed and averaged to give domain scores, which are then summed to give the total EPJS score (potential range - 18 to + 18). Responses

Predictability of Job Satisfaction, Lloyd et al. 235

to items in the GJS are summed and then averaged to give the final GJS score (potential range -3 to +3). The EPJS and GJS instruments were found to be reliable and valid, based on available measures of construct validity.’ All correlations between EPJS and GJS scores and instruments used to measure construct validity were statistically significant ( p c 0.001), but were not so high that they ap- peared to be measuring the same thing.3 Correlations between EPJS and GJS scores and those on the CES-D Depression scale and the Emotional Exhaustion and De- personalization subscales of the Maslach Burnout Inven- tory were negative.’ Correlations between EPJS and GJS scores and those on the Index of Job Satisfaction instrument, Satisfaction with Life instrument, and Personal Ac- complishment subscale of the Maslach Burnout Inventory were p~s i t i ve .~ Finally, correlations between instruments measuring the same construct (job satisfaction) were higher than those measuring related but different con- structs (e.g., life satisfaction and personal accomplish- ment).’

The current study further evaluates these instruments by measuring another form of construct validity: predictive validity. With statistically significant correlations between EPJS and GJS scores and instruments used to measure construct validity, it is expected that higher EPJS and GJS scores will be associated with physicians remaining at their current EM jobs, or at least within EM, while lower scores will be associated with physicians changing EM jobs or leaving the practice altogether. The project is designed to answer 2 primary, and 1 secondary, questions. The primary questions are: “Do scores on either the EPJS or GJS instrument predict which physicians will leave their current job or the practice of EM altogether?” and “What EPJS or GJS scores best identify physicians who will leave their jobs or EM altogether?” The secondary question is: “Do Canadian EPs differ from their U.S. colleagues in the reasons they cite for leaving EM?” Estab- lishing the predictive validity of the instruments should provide an opportunity to identify physicians at risk, and intervene before they leave their jobs or the practice of EM altogether.

I METHODS

Study Design and Population: This study was a prospective mail survey of 223 full-time Canadian EPs. To be included subjects had to have EPJS and GJS scores as a result of completing the 1990 study that developed the instrument^.^ The study was approved by the McMaster University, Faculty of Health Sciences, Research Ethics Committee.

. . . . . . . . . . . . . . . , . . . . . . . . . . . . . .

Survey Instrument and Process: The survey booklet contained 42 pretested questions, including 4 questions on current job status, 12 demographic questions, the 12-item

GJS instrument, and for those who left EM, 14 “reason for leaving” items incorporating a 3-point scaling device. The demographic questions included items such as salary, method of payment, publication success, research activities, and whether the respondent held a position as department chief.

To enhance compliance we followed methods outlined by Dillman’ for mail surveys, including an introductory letter mailed 1 week before the survey booklet, a cover letter explaining the research with the survey booklet, a postcard reminder 1 week later, a second survey booklet 4 weeks later if the first was not yet returned, and a telephone reminder 6 weeks later if neither the original nor the second booklet was returned yet. Finally, 12 weeks after mailing the original survey, nonresponders were con- tacted by telephone to determine whether they remained in their original jobs, had left for new EM jobs, or had left EM altogether.

Data Analysis: The 1990 study3 showed responder (n = 223) scores on the EPJS ranged from -12 to +12 (mean = 2.32, SD = 3.8). and scores on the GJS ranged from -3 to +3 (mean .= 0.90, SD = 1.1). A difference in scores of 2 EPJS units, and 0.5 GJS units, was considered to be a clinically important difference; each being =0.5 the instrument score SD. With statistically significant correlations demonstrated between EPJS and GJS scores and instruments used to measure construct validity, we expected higher EPJS and GJS scores (>O) to be associated with physicians remaining at their original EM jobs, or at least still in full-time EM, and lower scores (SO) to be associated with physicians who had left the practice altogether.

The respondents’ original (1990) EPJS and GJS instrument scores were analyzed using a 1-way analysis of variance (ANOVA) and Scheffe’s test“ comparing those physicians who were still in their original jobs with those who had left their original jobs but remained in EM and those who had left EM altogether. Instruments with significantly different scores on ANOVAs had receiver op- erating characteristic (ROC) curves constructed to identify the scores that best predicted (highest sensitivity and specificity) who would leave their jobs, or the practice altogether. ROC curves were constructed by plotting points representing values for sensitivity (true-positive rate) and 1 -specificity (true-negative rate), corresponding to each cutoff for the diagnostic test result (GJS scores).’ True- positive and true-negative rates were calculated for every 1-unit cutoff in the diagnostic test result [e.g., proportion of physicians who left EM with GJS 1-3 (true positive) compared with the proportion who stayed with scores 5 -3 (true negative)].

Mean scores on the 14 “reason for leaving” items were compared with scores from Hall et al.’s U.S. physician sample? using a t-test for independent means. The 14 “reason for leaving” items were also ranked, within

236 ACADEMIC EMERGENCY MEDICINE MAR 1998 VOL 5/NO 3

the U.S. and Canadian groups, according to mean score, to allow simple comparison.

Statistical significance were set at a = 0.05 (2-sided) for all analyses, and f3 = 0.1 for the primary study questions, assuming a response rate of 90%.

I RESULTS

Of 223 survey questionnaires mailed, 209 were returned, for a booklet response rate of 93.7%. The current job status of an additional 12 physicians was determined through telepho.ne follow-up, giving a response rate for our primary study questions of 99.1% (221/223). Seventy- eight percent (1 8/23) of the physicians who left EM altogether completed the booklet “reason for leaving” items. The remaining 22% (5/23) were identified through telephone follow-up and did not complete the “reason for leaving” items. Not all respondents answered every question; however, the nonresponse rate for any one question was 4 5 % .

Surprisingly, 19% (42/221) of the respondents had left their original positions for new EM jobs and 10.4% (23/ 223) had left the practice of EM altogether. For the respondents still in EM, 89% were in full-time practice, 26% were department heads, 7 1 % worked in teaching hospitals, 82% worked in hospitals where EM had department status, 82% were involved in medical education, 38% had published in a medical journal in the preceding 2 years, and 16% were involved in research (Table 1). For those involved in medical education, 92% taught medical students, 92% taught interns, and 35% were involved in resident education (Table 1). For those involved in research, 87% were involved in funded projects and 52% in

........................................................................

nonfunded projects (note that some respondents were involved in both funded and unfunded research; therefore, the total research sums to >loo%). Most (87%) respondents were married. Most (65%) were reimbursed for clinical work through fee-for-service, compared with 35% through other mechanisms such as salary or hourly rate.

Comparison of demographic data, on marital status and annual salary, showed some significant differences (Table 1). More respondents (87%) remaining in EM were married vs those (66%) who left EM (x2 = 24.5, p < 0.001). Department chiefs earned more ($162,583 2 $42,767; Canadian dollars, mean 2 SD) in gross annual salary (from all EM sources excluding benefits) than did physicians doing straight clinical work ($149,668 2 $35,526; p c 0.05). The gross annual salaries were similar at teaching hospitals ($151,232 f $36,622) and nonteach- ing hospitals ($159,092 2 $41,923; p > 0.2).

One-way ANOVA with Scheffe’s test showed a significant difference (p = 0.004) between GJS mean scores for physicians who left the practice of EM altogether, compared with those who stayed in EM (either their original jobs or new EM positions) (Table 2). There was no difference (p = 0.56) in EPJS mean scores for physicians who left the practice of EM altogether compared with those who stayed in EM (either in their original jobs or new EM positions) (Table 2). There was no significant difference in either GJS or EPJS scores for physicians who stayed in their original jobs compared with those who moved to new EM jobs (Table 2). The multiscale EPJS scores did decrease from a high of 2.48 for physicians in the same job to 1.59 for those who left EM altogether. However, these differences were not statistically significant.

I TABLE 1 Characteristics of the Respondents Still in Emergency Medicine ....................................................................................................................................................................

Responders

Yes No Nonresoonders

Total cohort (n = 198) Married Full-time practice Department head Teaching hospital Department status Published in preceding 2 yr Fee-for-service Involved in medical education Involved in research

Education cohort (n = 156) Teach medical students Teach interns Teach residents

168 (86.6%) 172 (89.1%) 51 (26.3%)

136 (70.5%) 159 (82.4%) 73 (37.6%) 67 (65.3%)

156 (82%) 31 (16%)

143 (91.7%) 144 (92.3%) 54 (34.6%)

26 (13.4%) 21 (10.9%)

143 (73.7%) 57 (29.5%) 34 (17.6%)

121 (62.4%) 126 (34.7%) 34 (18%)

163 (84%)

13 (8.3%) 12 (7.7%)

102 (65.4%)

4 (2.0%) 5 (2.5%) 4 (2.0%) 5 (2.5%) 5 (2.5%) 4 (2.0%) 5 (2.5%) 8 (4.0%) 4 (2.0%)

0 (0%) 0 (0%) 0 (0%)

Research cohort 01 = 31) Funded research 27 (87.1%) 4 (12.9%) 0 (0%) Nonfunded research 16 (51.6%) 15 (48.4%) 0 (0%)


I TABLE 2 Mean (and Standard Deviation) on Instrument Scores for Each Physician Group

Stayed in the Same Left Job for a New Left Job and EM EM? Job EM Position Altogether Degrees of

Instrument* (Range) (n = 156) (n = 42) (n = 23) F Ratio Freedom

GJS (-3 to +3) 0.98 (1.03) 0.89 (1.05) 0.15 (1.61) 5.64 220

EPJS (-18 to +18) 2.48 (3.66) 2.42 (3.58) 1.59 (4.41) 0.57 220

*GJS = Global Job Satisfaction; EPJS = Emergency Physician Job Satisfaction. tEM = emergency medicine. No significant difference between scores connected by underlining.

When the a-error was increased to 10% and ANOVA with Scheffe’s test was repeated, there was still no difference in EPJS scores between the physicians who left EM and those who stayed (either at their original jobs or new EM positions), or for scores on either instrument for physicians who stayed in their original jobs compared with those who moved to new EM jobs. Like scores on the multiscale EPJS, scores for the individual scales within the EPJS, such as “Administrative Autonomy” and “Re- sources,” were not predictive of who would leave and who would stay (all p-values >0.1 using ANOVA with Scheffe’s test).

The ROC curve for the GJS instrument scores showed that the best values for sensitivity and specificity were 0.47 (47%) and 0.8 1 (81%), respectively, corresponding to GJS scores of 5 0 for those likely to leave EM, and >O for those more likely to stay (Fig. 1). The sensitivity, spec-

ificity, positive predictive value (PPV), and negative predictive value (NPV) for this dichotomous test result (GJS scores S O or >O) are shown in Table 3. The likelihood ratio for a positive test result [likelihood a physician has left EM in the presence of a positive test result (GJS I 0) = sensitivity/l - specificity] was 2.6. An ROC curve was not plotted for the multiscaled EPJS instrument scores because ANOVA showed it was not predictive of who would stay in EM and who would leave.

Mean scores for Canadian physicians, on the 14 “Tea- sons for leaving” EM items, showed “shiftwork” to be the most strongly held reason, followed in decreasing or- der by “emotional stress,” “workload pressures,” “family considerations,” and “physical stress.” At a score of 1.28, “conflict with peers” and “conflict with ED staff” received the lowest scores (Table 4). Simple ranking of the top 5 “reasons for leaving” scores shows only slight

c .- > .- c .- rn

C

al 0

(5-2, . _

(5-3, >-3y 0.2 0.4 0.6

1 - Specificity

0.8 1

I FIGURE 1. ROC curve for Global Job Satisfaction (GJS) scores-physicians who stayed in EM vs those who left EM. Sensitivity and 1 -specificity values are based on the proportion of physicians who left EM vs those who stayed, for the GJS scores (in brackets) plotted along the curve.

~


Score 0 11

I TABLE 3 Sensitivity, Specificity, and Predictive Values* of Global Job Satisfaction Instrument Scores S O and >O ....................................................................................................................................................................

38 PPV = a/a + b = 22.4%

Score > 0 c l d

12 160 NPV = d c + d = 90.4%

Total 23 198 22 1 Sensitivity = a/a + c = 47% Specificity = d/b + d = 81% Prevalence = a + cla + b + c + d = 10.4%

*PPV = positive predictive value; NPV = negative predictive value. tEM = emergency medicine.

differences between the U.S. and Canadian samples. Both groups scores placed “shiftwork,” “emotional stress.” and “family considerations” first, second, and third (Ta- ble 4).

The greatest difference between the Canadian and U.S. sample “reasons for leaving” scores were for “govem- ment regulations,” “workload pressures,” “low earnings,” and “other compensation issues” (Table 4). The power to detect a significant difference between the Ca- nadian and U.S. physician groups, for these 4 “reasons for leaving,” was 538%. Differences between the 2 groups for the other 10 “reasons for leaving” were still smaller, with even less power to detect a significant difference.

I DISCUSSION ..............................................................................

At 99.1 %, the response rate for our primary questions was excellent. This response was achieved through telephone follow-up, which determined current job status for an additional 12 physicians, thereby raising the 97.3% booklet response rate for these questions.

With 19.0% of our sample changing EM jobs, and

another 10.4% leaving EM within 3.5 years (fall 1990 to winterhpring 1994), there appears to be great mobility within Canadian EM. The 29.4% turnover rate (19.0% t 10.4%) from our sample is less than the 47.7% five-year turnover rate identified in an informal telephone survey of EDs across Canada.’ The informal design, and the urban teaching hospital orientation of the telephone survey, may have inflated the reported rate. Still, it is difficult to deny the instability of ED workforce inherent in the results of our study. This attrition rate far exceeds the current annual workforce production of Canadian university residency programs and will add to the current Canadian full-time physician workforce shortage.’ The 10.4% who left EM is smaller than the 12.9% reported in the Hall et al.“ study of U.S. residency-trained EPs; however, our 10.4% leaving rate is determined over only 3.5 years, while the U.S. figure was for a 10-year period.

The single-scale GJS instrument was associated with who would leave EM and who would stay. There were both clinically (difference >0.5 units) and statistically (p < 0.05) significant differences in GJS scores between those who left EM altogether and those who stayed (original job or a new EM job). However, with a difference in

I TABLE 4 Canadian and US. Scores on Reasons for Leaving Emergency Medicine ...................................................................................................................................................................

Reason Canadian Score (95% C1) U.S. Score (95% C1) t-test p-value

Shiftwork 2.67 (2.36-2.98) 2.48 (2.30-2.66) 0.97 >0.20 Emotional stress 2.17 (1.84-2.50) 2.04 (1.82-2.26) 0.59 >0.20 Family considerations 2.11 (1.72-2.50) 1.96 (1.74-2.18) 0.63 >0.20 Workload pressure 2.1 1 (1.76-2.46) 1.76 (1.56-1.96) 1.60 >0.10 Physical stress 2.06 (1.69-2.43) 1.90 (1.70-2.10) 0.72 >0.20 Dissatisfaction with patients 1.61 (1.28-1.94) 1.80 (1.60-2.00) 0.86 20.20 Conflict with consultants 1.61 (1.26-1.96) 1.51 (1.33-1.69) 0.50 >0.20 Other compensation issues 1.61 (1.26- 1.96) 1.34 (1.14-1.54) 1.29 20.10 Malpractice issues 1.50 (1.17- 1.83) 1.63 (1.41-1.85) 0.55 >0.20 Government regulutions 1.50 (1.13-1.87) 1.17 (1.03-1.31) 1.83 9 . 0 5 Low earnings 1.44 (1.09-1.79) 1.21 (1.07- 1.35) 1.35 20.10 Health reasons 1.39 (1.04- 1.74) 1.29 (1.11-1.47) 0.51 >0.20 Conflict with peers 1.28 (1.03-1.53) 1.32 (1.16-1.48) 0.24 20.20 Conflict with ED staff 1.28 (1.03- 1 S3) 1.28 (1.12-1.44) 0.00 >0.20

For all analyses: Canadian n = 18, U.S. n = 70, degrees of freedom = 86.


GJS scores of only 0.09 units, there was no clinical or statistically significant difference between those physicians who stayed in their original jobs and those who moved to new EM jobs. Because Scheffe’s test is considered very conservative, we post hoc increased the ac- cepted a-error to lo%, but still found no significant differences in GJS scores between the physicians who stayed in their original jobs and those who moved to new EM positions. Thus, the GJS score was not associated with who would leave their original EM jobs for new ones.

The GJS may help identify individuals at risk of leaving the practice of EM entirely and may provide an opportunity for intervention. Job satisfaction is made up of both intrinsic (e.g.. achievement) and extrinsic (e.g., work- ing conditions) factors. Intrinsic factors are strongly influ- enced by personality and are not readily amenable to change. Extrinsic factors are independent of personality and interventions are more likely to bring about change. Thus, intervention could be at the level of the individual (time off or schedule changes), the department (counsel- ing, a better schedule, double coverage), provincial or national EM governing bodies (lobbying for changes to re- muneration or hospital resource allocation), or residency training programs (making candidates aware of the reality of EM and selecting those most suited to it).

We wanted to treat the GJS like a test result, where under ideal conditions the uppermost left-hand comer of an ROC curve denotes a perfect test [where true positive, or sensitivity = 1 (meaning all those who will leave are identified), and false positive, or 1 - specificity = 0 (meaning none who will stay are falsely labeled as leav- ers)]. Therefore, any point on the ROC curve closest to the uppermost left-hand corner is the best cutoff in terms of making the fewest mistakes. The GJS scores providing the best values of sensitivity and specificity for predicting who will leave and who will stay are 10 and >O, respectively. The 22.4% PPV for GJS scores 50 suggest the instrument is not very predictive of who will stay and who will go; however, the 90.4% NPV for GJS scores >O suggests that the instrument is quite predictive of who will stay in EM. One problem with predictive values is that they change considerably depending on the prevalence of the outcome of interest.’ Likelihood ratios are less sensi- tive to changes in prevalence and thus provide a more useful measure of a test’s accuracy.’ The positive likelihood ratio for GJS scores 50 suggests that the probability of such a test result is 2.6 times as likely to come from physicians who have left EM as from physicians who stayed. Thus, if physicians with GJS scores S O are to remain in EM, some form of intervention is required; most likely at the level of the individual and department.

We are disappointed with the poor predictive validity of the EPJS. There were no significant differences in EPJS scores (with a-error set at either 5% or 10%) between the physicians who left EM and those who stayed, or between

those who stayed in their original EM jobs and those who moved to new EM positions. The multiscale EPJS was designed to provide more detailed information (on many aspects of the job) than do single-scale instruments such as the GJS. Our earlier work showed that the multiscale instrument total score and the individual scale (domain) scores were internally consistent (Chronbach’s as all >0.7) and stable when subjected to a test-retest reliability study. The sample on which the instrument was developed allowed factor analysis of individual domains (scales), but it was too small to allow factor analysis of the multiscale instrument as a whole. Factor analysis of the individual scales showed that 4 of the 6 scales had > I factor. Thus, while the individual scales were internally consistent, they were more complex than desired. This complexity may explain why the EPJS was not predictive on the small sample surveyed.

Lifestyle and stress issues, such as shiftwork and physical and emotional stress, received the highest scores for why our sample physicians left EM (Table 4). Previous research shows burnout is the result of high levels of un- resolved When these high stress levels are com- bined with the reality of shiftwork, with its negative personal and family costs, the option of leaving EM altogether appears to become more attractive.

Knowing the highest rated reasons physicians attribute to their decisions to leave EM helps us understand the stresses related to full-time EM practice. Discovering that the top 3 rated “reasons for leaving” EM are the same for our Canadian sample and U.S. physician sample of Hall et al.4 is of particular value. If the reasons for leaving EM are similar in the 2 countries, a workable solution in 1 country may well apply in the other. Results from both studies suggest that a key component to enhanced career longevity in EM is an effective schedule. Scheduling is an extrinsic component of job satisfaction and thus is a component amenable to change. Both studies also suggest a need to discover ways to decrease job stress, such as double coverage, effective resource management, and healthy relationships with patients and consulting staff.

The limited 3-point range of the scale of Hall et a1.: and small power available to answer questions on why physicians left EM, may have prevented us from finding real differences between the sample of U.S. physicians and our Canadian sample. Three-point scaling devices are less reliable and do not provide sufficient range to differ- entiate responses. To achieve acceptable reliability, ex- perts in scale development suggest the minimum number of response categories be in the region of 5-7.13 The only “reason” that approached statistical significance was “government regulations” (t = 1.83, p < 0.1. power = 38%), with Canadian physicians feeling more regulated than their U.S. colleagues.

The demographic data show a slight increase in the proportion of our current sample, compared with those


answering our 1990 s ~ r v e y , ’ ~ who work in an ED with hospital departmental status, work in teaching hospitals, get compensated for their clinical work through fee-for- service, are department heads, have published in the preceding 2 years, and are married. These increased propor- tions may support the notion that career longevity is associated with people having these characteristics. The results could also reflect a migration of physicians toward institutions where these characteristics are present, thus increasing the number of people with these characteristics, and therefore their proportion in the sample. The slight increase in marital status may reflect general aging, or that a higher proportion of people who stay in EM are married. The latter is suggested since 82% of the physicians remaining in EM were married vs only 66% married among those who left EM. Hall et al. found a similar result, with 87% of the physicians still in EM being married vs only 67% of the physicians who left.4 These findings are sup- ported by our earlier research, which found significantly higher Satisfaction with Life scores among manied physicians (p c 0.002), and that significantly fewer married physicians fell within the range of depressive sympto- matology on the CES-D depression scale (p c 0.017) when compared with their nonmarried colleagues.’

If either the increase in the proportion of physicians with these “positive” characteristics is associated with a migration to hospitals that support these characteristics, or physicians with them simply survive in EM longer, what can explain the decreased proportion involved in research? Our earlier study found involvement in research to be significantly associated with job satisfaction (p = 0.038).15 One explanation is that those involved in research in 1990 have completed their work and moved away from research. It may also mean that the original result was falsely inflated because some respondents lib- erally interpreted the question (including such activities as literature reviews), while others concerned about dimin- ished professional status (better to say yes than no) simply answered “yes.” Those stating they participate in research during our current sample may .represent those involved in more formal (and sustained) research activities. Of greatest concern is the possibility of fewer physicians choosing to do research. Not only do we need the new knowledge gained from research; our discipline needs mentors to develop these skills among interested EM residents and medical student^.'^"^

I LIMITATIONS AND FUTURE QUESTIONS

The results of our study must be considered in light of its limitations. Most of the limitations relate to the small, and fixed, sample of physicians who completed the EPJS and GJS in November 1990. Prior to this study, there was no information on the magnitude of difference in EPJS or GJS scores, for those who stayed in EM compared with

. , . .........................................................................

those who left, because the instruments had never been tested on physicians who had left EM. We considered that a difference of 2 EPJS units, and 0.5 GJS units, would be clinically significant. With a difference in EPJS scores of only 0.89 units, clinical significance ( 2 2 units) was not achieved, and our sample was too small to pick up a statistically significant difference of c2 units even if it did exist. Thus, the EPJS needs further evaluation. The first reevaluation step should be a factor analysis of all 79 EPJS items on a larger sample (>790 respondents). This step might improve the psychometric properties of the EPJS and ascertain whether items currently placed in spe- cific domains truly belong there.

The ROC curve is not as convex to the left upper comer of the graph as an ideal discriminator would per- form with high levels of sensitivity and specificity. The poor discrimination in part may be due to the small sample of physicians surveyed and the small proportion of those surveyed who left EM. A larger sample, with a greater number of physicians who left EM, might smooth out the curve and move it farther toward the upper left- hand comer. Likewise, a larger sample might detect significant differences in “reason for leaving” EM.

A few respondents failed to answer some questions. A number of reasons can be hypothesized for these missing values, such as concem about confidentiality (marital status and salary), professional status (hospital type, involvement in education, and research), and questionnaire design (did not see the question). Still, with >85% of respondents answering any one question, we are confident in generalizing our results to the sample as a whole. The 2 physicians we could not reach, through either address or telephone information in the Canadian Medical Direc- tory,’’ have likely moved, suggesting they have either left their original jobs or the practice of EM altogether. Add- ing these 2 physicians to either the “changed jobs,” or “left EM” groups does not dramatically affect their pro- portions: 19.1% to 20.2% and 10.5% to 11.7%, respectively. Thus, the 2 nonrespondents have little impact on the credibility of our results.

The multiscale EPJS instrument’s psychometric properties may not yet be fine-tuned enough to predict who will leave the practice of EM compared with those who will stay (either in the same job or a new one). The EPJS should be the subject of new research on a sample large enough to do factor analysis on the instrument as a whole (2790 respondents) to provide a stable factor solution. Such a study will provide an opportunity to see whether items currently attributed to a hypothetical scale (domain) truly load onto that scale. It will also provide an opportunity to reevaluate the EPJS’s predictive validity for iden- tifying respondents practicing full-time EM who will leave their jobs, or EM altogether. To achieve a sample of this size, the study will likely have to include U.S. physicians.


I CONCLUSIONS

We found GJS instrument scores to be associated with whether the physician stayed in EM. Physicians with GJS scores 50 were more likely to leave, and those with scores >O were more likely to stay. The EPJS was unable to predict physicians who would leave, or stay in EM. Neither the GJS nor the EPJS was able to identify which physicians would leave one EM position for a new one. The EPJS requires further evaluation on a larger sample to fine-tune its psychometric properties, and to carry out factor analysis on the entire instrument. Life and stress issues are the most important reasons Canadian physicians cite for leaving EM. Canadian and U.S. physicians cite similar reasons for leaving EM, with shiftwork receiving the highest “reason for leaving” scores for both groups. The predictive validity of the GJS should help identify individuals at risk of leaving the practice and provide an opportunity for intervention.

...........................................................................

The authors thank the Ontario Ministry of Health. Emergency Health Services Research Advisory Committee, for supporting this research through a peer reviewed grant (#08393S). They also thank Dr. K. Hall for allowing us to use his U.S. physician data, and the 221 physicians who participated in the study. Finally Dr. Lloyd thanks the Ontario Medical Association and Ontario Institute for Clinical Evaluative Sci- ences for providing him time to complete the research through the Adam Linton Fellowship.

I REFERENCES

1. Staw BM. The consequences of turnover. J Occup Behav. 1980; 1:

2. Price J . The impact of turnover on the organization. Work and Oc-

3. Lloyd S, Streiner D, Shannon S, Hahn E: Development of the Emer-

253-73.

CUP. 1989; 16:461-73.

gency Physician Job Satisfaction measurement instrument. Am J Emerg Med. 1994; 12:l-10. 4. Hall KN. Wakeman MA, Levy RC. Khoury J. Factors associated with career longevity in residency-trained emergency physicians. Ann Emerg Med. 1992; 21:291-7. 5. Dillman DA. Mail and Telephone Surveys: The Total Design Method. Wiley & Sons, Toronto, Canada; 1978. pp 163-99. 6. Edwards AL. Experimental Design in Psychological Research, 3rd Edition. New York: Holt, Reinhart, and Winston, 1968, pp 150-3. 7. Sackett D. Haynes RB, Tugwell P. Clinical Epidemiology: A Basic Science for Clinical Medicine. Boston: Little, Brown and Co., 1985. p 106. 8. Beveridge RC: Emergency medicine: a Canadian perspective. Ann Emerg Med. 1995; 26504-7. 9. Sackett D, Haynes RB, Tugwell P. Clinical Epidemiology: A Basic Science for Clinical Medicine. Bastofloronto: Little, Brown and Co.. 1985, p 110. 10. Phipps L Stress among doctors and nurses in the emergency department of a general hospital. Can Med Assoc J. 1988; 139:375-6. 11. Keller KL, Koenig HJ: Management of stress and prevention of burnout in emergency physicians. Ann Emerg Med. 1989; 18:42-7. 12. Koltonow SH. Who’s stressing whom? [commentary]. Ann Emerg Med. 1989; 18:103-4. 13. Streiner D. Norman G. Health Measurement Scales: A Practical Guide to Their Development and Use. New York: Oxford University Press, 1989. 14. Lloyd S, Striener D, Shannon S. Demographic characteristics of full-time emergency physicians in Canada. Am J Emerg Med. 1993: 11:

15. Lloyd S, Striener D, Shannon S. Burnout, depression, life and job satisfaction among Canadian emergency physicians. J Emerg Med.

16. Saunders AB, Fulginiti JV, Witzke DB: Factors influencing resident career choices in emergency medicine. Ann Emerg Med. 1992; 21:47- 52. 17. Saunders AB, Fulginiti JV, Witzke DB, Bangs KA. Characteristics influencing career decisions of academic and non-academic emergency physicians. Ann Emerg Med. 1994; 23231-7. 18. Canadian Medical Directory 1995, 41st Edition. Don Mills, On- tario, Canada: Southam Business Information and Communications Group Inc., 1995.

364-70.

1994; 12559-65.

Documents

Predictive Validity of the Emergency Physician and Global Job Satisfaction Instruments