Evaluation of the Nursing Research for Doctoral Programs in Nursing

1979 to 1984

Emphasis/Grants Grant Program,

WILLIAM L. H O L Z E M E R , PHD, RN, FAAN,* A N D

SUSAN R. GORTNER, PHD, MN, FAANt

This article is an evaluation of the Impact of the Nursing Research Emphasis/Grants for Doctoral Pro- grams in Nursing (NRE/DPN) grant program between 1.979 and 1984 on the quality of doctoral education in nursing. Fourteen nursing doctoral programs partici- pated in both a 1979 and 1984 national evaluation study; eight of these programs received 3 or 4 years of NRE/DPN funding and six programs received 0 or 1 year of funding. The recipient programs were able to employ younger, less experienced faculty members while maintaining their levels of productivity. The re- cipient schools' faculty reported a significant in- crease in the amount of time spent on scholarly work, whereas the nonrecipients reported spending more time on administrative and consultative activities. The NRE/DPN grants were successful in enhancing the research environments of the recipient programs. (Index words: NRE/PDN, evaluation; Nursing, doc- toral programs, grants, research environment) J Prof Nurs 4:381-386, 1988. © 1988 by W.B. Saunders Com- pany.

T H I S IS AN evaluation of the of report impact the Nursing Research Emphasis/Grants for

Doctoral Programs in Nursing (NRE/DPN) on the quality of doctoral education in nursing. Before the National Center for Nursing Research was estab- lished, the National Institutes of Health (NIH), the research branch of the Division of Nursing, the Public Health Service (PHS), and the Department of

* Professor, Physiological Nursing and Director, Office of Re- search, Evaluation, and Computer Resources, School of Nursing, University of California-San Francisco, San Francisco, California.

t Professor, Family Health Care Nursing, School of Nursing, University of California-San Francisco, San Francisco, California.

Supported in part by the grant, "Quality Indicators of Nursing Doctoral Programs," DHHS, PHS, Division of Nursing, 1 RO NU00967, P. I. Holzemer.

Dr Gormer served as the principal investigator and Dr Hol- zemer as the coprincipal investigator on UCSFs NRE/DPN grant from 1980 to 1985 (R21 NU 00788).

Address correspondence and reprint requests to Dr Holzemer: Box 0604, School of Nursing, University of California, San Fran- cisco, San Francisco, CA 94143-0604.

© 1988 by W. B. Saunders Company. 8755-7223/88/0405-00155-3.00/0

Health and Human Services (DHHS) supported aca- demic institutions with doctoral programs in nursing through NRE/DPN grants. The purpose of this spe- cial grant program was "to stimulate the develop- ment of nursing research in areas that emphasize spe- cial health needs of the nation and to enhance the research efforts and resources of faculty in schools of nursing which offer doctoral programs. 'q The special health needs included aging and chronic illness, ma- ternal and child health, terminal illness, and other problem areas that potential grantees could elect to emphasize.

The NRE/DPN grants were a unique opportunity for universities with nursing doctoral programs to strengthen and expand not only research programs but also the training environment and research sup- port facilities of the schools of nursing. Such facilities included research design and biostatistical consulta- tion, editing, grant support, and computer support. Programs were eligible for up to 5 years of funding, and awards ranged in the $100,000 per year range for direct costs.

In actuality, there was considerable variation in the schools that responded to the programs' solicitation, underwent peer review, and were funded for the ini- tial period of 3 years. In the authors' institution, fac- ulty were solicited for proposals in the empk~sis areas of aging as well as chronic and terminal illness across the life space. These proposals were reviewed by a fac- ulty peer review committee and incorporated in the final proposal. The amount of requested funding to support faculty projects was equal to that requested for the enhancement of the research training environ- ment through employment of consulting biostatistic- ians, statistical computing, proposal developers, ed- itors, and other research support. These support ser- vices became well used by faculty and by progressive numbers of doctoral students, who also served as re- search assistants on faculty projects.

Other settings placed more emphasis on individual research projects in their selected emphasis areas, also

382 HOLZEMER ETAL

involving doctoral students in these research efforts as research assistants. The extent to which institutions attempted any institutional peer review or evaluation is not known.

The first author conducted two national evaluation studies on the quality of doctoral education in nursing. 2,3 The study samples included 18 of the 22 programs in 1979 and 25 of the 29 programs in 1984. Fourteen nursing doctoral programs partici- pated in both evaluation studies. It was possible to use this data set to appraise the impact of the NRE/ DPN grants on doctoral settings because eight of the participating doctoral programs had received 3 or 4 years of NRE/DPN funding from 1979 to 1984 and six had received 0 or 1 year of funding. This appraisal targeted the faculty's perceptions of the degree of en- hancement of research efforts and resources as well as publications, not the impact of the funded research programs on the nation's health.

DESIGN

An ex post facto design was used. A list of the nursing doctoral programs (n = 14) participating in both the 1979 and 1984 national evaluation studies was crossed with a list of recipients for the NRE/ DPN grant program during the same time period. Eight programs had been the recipients of 3 or 4 years of funding and six programs had received no funding or had just begun their first year of funding in 1984. Hence, a quasiexperimental design with pretest and posttest was constructed. The pretest was adminis- tered to faculty in 1979 and the posttest in 1984.

SAMPLE

Faculty members at the 14 nursing doctoral pro- grams participating in both the 1979 and 1984 studies were the subjects. The recipient group (n = 8) received 3 or 4 years of NRE/DPN funding and the nonrecipient group (n = 6) received 0 or 1 year of funding. Overall, the faculty had a mean age of 46 years, 95 per cent were female, 92 per cent reported white as their ethnic origin, and 50 per cent held tenure. The mean number of faculty members at each program ranged from 18.5 to 28.9.

INSTRUMENTS

The Graduate Program Self-Assessment (GPSA) questionnaires-developed and commercially available from Educational Testing Service were used. The in- strument measures 16 scales, based on 60 items, in which faculty rate items on a four-point agree-to-dis- agree scale or respond yes/no. Information is assessed

about important quality-related program character- istics in seven areas: program purposes, faculty training and accomplishments, student ability and performance, resources, academic and social environ- ments of the program, program processes and proce- dures, and alumni achievements. Holzemer and Chambers reported on the psychometric properties of the scales. 4 The median internal consistency coeffi- cient (Cronbach's alpha) was 0.76. 5 Six of the 16 scales were used in this evaluation report and the scale descriptions are presented in Table 1.

ANALYSIS

Program scale scores were compared using the Wilcoxin matched pair test. The probability of ob- taining resultant Z score for each variable in the re- peated measures design is reported in each table. Be- cause of the problem of calculating multiple compar- isons, P values must be cautiously interpreted and, rather than discuss these statistical findings, Cohen's (1977) effect size or D statistic was calculated to assist in the interpretation of the data (Tables 2, 3, and 4). Effect size is "the degree to which a phenomenon is present in the population.'6 It is a measure of prac- tical significance. Effect size is Calculated by dividing the difference between the mean scores by the within-

TABLE 1. D e s c r i p t i o n of GPSA S u m m a r y Sca les

Scale 2, Scholarly Excellence--Rated excellence of the department faculty, ability of students, and intellectual stimulation in the program (5 items).

Scale 4, Faculty Concern for Students--The extent to which faculty members are perceived to be interested in the welfare and professional development of students, accessible, and aware of student needs, concerns, and suggestions (5 items).

Scale 7, Available Resources--Ratings of available facilities such as libraries and laboratories, and overall adequacy of physical and financial resources for a doctoral program (3 items).

Scale 12, Faculty Work Environment--Self-reported faculty satisfaction with departmental objectives and procedures, academic freedom, opportunities to influence decisions, and relationships with other faculty members; sense of conflicting demands and personal strain (6 items).

Scale 15, Faculty Research Activities--The extent to which faculty members report receiving awards for outstanding research or scholarly writing, editing professional journals, referring articles submitted to professional journals, and receiving grants to support research or other scholarly or creati'~ work (6 items).

Scale 16, Faculty Professional Activities--The extent to which faculty members report serving on national review or advisory councils, holding office in regional or national professional associations, and receiving awards for outstanding teaching or professional practice (5 items).

Note: The GPSA scales are often evaluated by faculty, stu- dents, and alumni, but only faculty perceptions are reported here.

EVALUATION OF NURSING RESEARCH, 1979-1984 383

TABLE 2. E v a l u a t i o n o f t he I m p a c t o f N R E / D P N o n the C o n t e x t o f R e c i p i e n t s (n = 8) a n d

N o n r e c i p i e n t s (n = 6) F r o m 1979 to 1984

t979 1984

Variables M SD M SD

P Value for

Wilcoxin Z

Effect Size*

(Cohen's d)

% Assistant professor Recipients 16.0 19.0 25.0 17.0 0.04 0.78 Noni'ecipients 17.0 8.0 23.0 9.0 0.46 0.38

% Associate professor Recipients 44.0 25.0 44.0 18.0 0.78 0.00 Nonrecipi~nts 46.0 8.0 29.0 18.0 0.04 -0.96

% Professor Recipients 39.0 27.0 27.0 18.0 0.33 - 0.53 Nonrecipients 35.0 12.0 45.0 18.0 0.05 1.11

% Full-time employment Recipients 95.0 9.0 91.0 6.0 0.45 -0.34 Nonrecipients 95.0 7.0 91.0 10.0 0.47 - 0.36

% Tenured Recipients 60.0 31.0 51.0 27.0 0.09 - 0.65 Nonrecipients 61.0 14.0 67.0 17.0 0.60 0.35

Years since receiving doctorate Recipients 8.8 2.9 7.6 1.9 0.14 - 0.57 Nonrecipients 8.8 1.5 10.3 2.8 0.35 0.43

Years in this department Recipients 7.4 2.8 6.4 1.7 0.55 - 0.32 Nonrecipients 6.2 2.4 8.8 2.5 O.07 0.93

Years total teaching experience Recipients 12.5 3.4 12.3 1.6 0.80 - 0.08 Nonrecipients 12.5 2.8 15.8 3.8 0.04 1.19

Size: no. of faculty members Recipients 18.5 8.9 28.9 12.6 0.02 0.87 Nonrecipients 19.0 7.3 23.8 13.9 0.14 0.58

Size: no. of students Recipients 28.9 22.0 52.9 24.6 0.01 1.38 Nonrecipients 17.0 19.3 37.7 22.8 0.03 1.82

Scale 7: available resources Recipients 3.0 0.38 2.99 0.34 0.89 - 0.07 Nonrecipients 2.9 0.36 2.85 0.47 0.92 - 0.04

* Effect size (Cohen, 1977): .20, small effect; .50, medium effect; .80, large effect.

group SD. Cohen suggests the following guide in in- terpreting effect sizes: .20, small effect size; .50, me- dian effect size; and .80, large effect size. Numerous authors have encouraged the use of estimated effect size for interpreting evaluation results. 7"*° Cohen's ef- fect size was chosen as an indicator of the extent of the impact of receiving NRE/DPN program grants. It was clearly recognized that there are numerous un- controlled variables in this ex post facto design and that the results ought to be interpreted within a qua- siexperimental, ex post facto framework.

Results

The results are organized around a system model for interpreting evaluations. 11 The context for se- lected characteristics of recipients and nonrecipients of NRE/DPN funding over time is presented in Table 2. The environment is presented in Table 3 and the outcomes are presented in Table 4.

CONTEXT

The change from 1979 to 1984 in the environ- ments of recipients and nonrecipients of NRE/DPN grants was strikingly similar (Table 2). Recipients ex- perienced more growth in the size of their faculty (.87 delta), and the new faculty were employed pri- marily as assistant professors (16 per cent to 25 per cent). Faculty at the nonrecipient schools were pro- moted to full professor (35 per cent to 45 per cent), with a corresponding decrease in the mean per cent of associate professors (46 per cent to 29 per cent). The recipient schuols were consistently characterized by an influx of assistant professors. For example, the per- centage tenured in these settings decreased (60 per cent to 51 per cent) as did years since receiving doc- torate (8.8 years to 7.6 years). Accordingly, so did years of teaching. The nonrecipients were character- ized by the opposite effect. In addition to increasing the size of their faculty between 1979 and 1984, the

384

TABLE 3.

HOLZEMER ET AL

Evaluation of the Impact of NRE/DPN on the Environment of Recipients (n = 8) and Nonrecipients (n = 6) From 1979 to 1984

1979 1984

Variables M SD M SD

P Value Effect for Size*

Wilcoxin Z (Cohen's d)

% of Time teaching/advising Recipients 50.0 13.0 44.0 9.0 0,02 -0.87 Nenrecipients 49.0 10.0 45.0 8.0 0.14 -0.69

fo of Time research/scholarly work Recipients 20.0 7.0 26.0 7.0 0,06 0.84 Nonrecipients 23.0 7.0 24.0 6.0 0.53 0.16

% of Time administration/consulting Recipients 30,0 13,0 31.0 12.0 0.94 0.08 Nonrecipients 29.0 9.0 32.0 7.0 0.17 0.60

No. of days away in past 12 months Recipients 16.1 7.3 15.9 4.0 0.94 - 0.04 Nonrecipients 14.5 3.4 12.2 4.0 0.68 -0.38

Scale 4. faculty concern for students Recipients 3.07 0.14 3.17 0.26 0.29 0.39 Nonrecipients 3.25 0.27 3.34 0.21 0.92 0.22

Scale 12: faculty work environment Recipients 2.98 0.28 3.12 0.14 0.33 0.44 Non recipients 2,96 0.13 3.09 0.18 0.14 0.74

Total Presentations Recipients 5.75 2.82 8.38 3.02 0.03 1.10 Nonrecipients 5.67 1.75 8,00 2.76 0.14 0.66

* Effect size (Cohen, 1977): .20, small effect; .50, medium effect; .80, large effect

recipient programs significantly increased the mean program size of their students (28.9 to 52.9), which was substantially larger in raw numbers than the nonrecipients (17.0 to 37.7). There was no change in the faculty's perceptions of available resources (scale 7) for either the recipients or nonrecipients.

ENVIRONMENT

Characteristics of the environments for the recip- ients and nonrecipients are presented in Table 3. En- vironments appear more similar than dissimilar, ex- cept that the recipients were characterized as spending significantly less time teaching and advising students (50 per cent to 44 per cent), spending more time on research and scholarly activities (20 per cent to 26 per cent), and making significantly more pre- sentations during the past 2 years (5.75 to 8.38). The environments of the recipient programs may be char- acterized as more scholarly, even with the consider- ation that these settings had also employed more ju- nior faculty. Both recipients and nonrecipients per- ceived improved faculty concern for students (scale 4) and an enhanced faculty work environment (scale 12).

OUTCOMES

Evidence to judge the impact of the NRE/DPN grant program on productivity outcomes is presented in Table 4. The only variable to have significantly

increased was scale 15, the percentage of time faculty spent on research activities for the recipient group (42 per cent to 55 per cent). Scale 15, percentage of time faculty spent on research activity, has been signifi- cantly correlated with several outcome measures, in- cluding total publications in an entire career, number of referred articles, and number of presentations in the last 2 years. 4 The nonrecipient group reported a greater increase in the number of articles and book chapters published in the last 3 years (Delta scores, 1.10 and 0.98); however, their program means were still lower than the recipient programs.

Discussion

The recipient program faculty reported a signifi- cant increase in the amount of time spent on scholarly work, whereas the nonrecipient faculty reported spending more time on administrative and consulta- tive activities. Both recipient and nonrecipient fac- ulty spent less time teaching and advising students. Both the recipient and nonrecipient settings were characterized by growth in the size of the numbers of faculty and students in their respective settings. Re- cipient programs tended to employ new junior fac- ulty, whereas nonrecipient faculty tended to employ new senior faculty. The recipient programs were able to maintain a high productivity level while em- ploying more junior faculty. The nonrecipient pro-

EVALUATION OF NURSING RESEARCH, 1979-1984 385

TABLE 4. Eva lua t ion of the Impac t of NRE/DPN on the Ou t comes fo r Rec ip ien t s (n = 8) and N o n r e c i p i e n t s (n = 6) f rom 1979 to 1984

1979 1984

Variables M SD M SD

P Value for

Wilcoxin Z

Effect Size*

(Cohen's d)

No. of articles and book chapters in last 3 years Recipients 4.50 2.67 5.75 2.05 0.29 Nonrecipients 4.00 1.89 5.33 2.34 0.07

No. of total publications in last 3 years Recipients 7.38 3.82 7.75 2.60 0.89 Nonrecipients 5.67 2.42 7.00 2.50 0.07

No. of articles and book chapters in career Recipients 13.25 10.15 t6.38 8.40 0.29 Nonrecipients 13.17 3.87 17.83 4.90 0.09

No. of total publications in entire career Recipients 20.25 11.39 20.87 9.80 0.99

.Nonrecipients 18.67 7.23 23.67 8.30 0.42 Scale 15: % of time spent on faculty research activities

Recipients 42.0 13.0 55.0 9.0 0.02 Nonrecipients 42.0 13.0 49.0 8.0 0.12

Scale 16: % of time spent on faculty professional activities Recipients 45.0 12.0 50.0 5.0 0.26 Nonrecipients 47.0 11.0 46.0 7.0 0.92

Scale 2: scholarly excellence Recipients 3.16 0.20 3.38 0.31 0.09 Nonrecipients 3.08 0.14 3,22 0.17 0.35

0.34 1.10

0.07 0.98

0.34 0.79

0,05 0.46

1.12 0.62

0.38 0.08

0.83 0.46

* Effect size (Cohen, 1977): .20, small effect; .50, medium effect; .80, large effect

grams increased in their measures of productivity, primarily by employing more senior faculty and pro- moting from within.

The NRE/DPN grant activity had a significant en- abling function for those schools that had received 3 or 4 years of funding from 1979 to 1984. The recip- ient programs were able to employ less experienced faculty while maintaining their levels of productivity. Also, these new doctorally prepared faculty members brought the advantages of recent dissertations and re- search training, thereby enhancing the recipients' en- vironments. The NRE/DPN recipient programs re- ported increased time spent on research activity and less time spent on teaching or administrative activi- ties than did nonrecipient programs. These delta scores suggest that the NRE/DPN program grants enhanced the research environments and productivity of the recipient programs.

There are several significant limitations to this data set; hence, the confidence one can have in asserting any causal relationship between the recipient of an NRE/DPN award and enhanced productivity. There was no attempt to control for any intervening vari- ables such as significant administrative changes in a program's environroent, the number of other extra- mural funded projects operating during this time pe- riod, or the general history of scholarly activity in each setting. These intervening variables could also explain the observed delta scores. However, there is

no attempt in this reporting to suggest such a causal relationship. Cohen's effect size was purposefully se- lected, given the small sample size and the potential intervening variables that operate in such an ex post facto evaluation design.

Delta scores provide a mechanism to judge the magnitude of the effect of an intervention in an evalu- ation study when the design does not allow the de- sired degree of control over extraneous variables. However, the GPSA scales have been documented to have adequate validity and reliability. Hence, one can place moderate confidence in their ability to accu- rately assess the variables of interest in this study. Although the sample size (n = 14) is small, these program means are stable because they are based upon 18 to 28 faculty members per program and are there- fore means of means. Therefore, although there are threats to internal validity in this design, stable pro- gram mean scores measured by valid and reliable scales increase confidence in the findings.

More recent work has documented the impact of working in a scholarly environment on faculty pro- ductivity. 12 The NRE/DPN grants contributed to the enhancement of the scholarly environment of the re- cipient programs. The NRE/DPN recipient programs were successful in increasing the scholarly produc- tivity of their faculty while employing more junior faculty than the nonrecipient programs. This data documents the positive effect of institutional training

386 HOLZEMER ET AL

support on the scholarly environment and faculty pro- ductivity of nursing doctoral programs. The NRE/ D P N grant program was successful in meeting its ob- jective of enhancing the scholarship and productivity among its recipients. These findings should be useful in supporting arguments for increased institutional support for nursing scholarship in the political arena.

References

1. Federal Register 1983, p 32087 2. Holzemer WL: Towards a cooperative venture in pro- gram evaluation. Proceedings of the Annual Forum on Doctoral Education in Nursing. Chicago, Rush Univer- sity, 1978, pp 65-71 3. Holzemer WL: Doctoral education in nursing: An as- sessment of quality, 1979-1984. Nurs Res, 36:111-116, 1987 4. Holzemer WL, Chambers DB: Healthy nursing doc-

toral programs: Relationship between perceptions of the academic environment and productivity of faculty and alumni. Res Nurs Health, 9:299-307, 1986 5. Clark MJ, Hartnett RT, Baird LL: Assessing Dimen- sions of Quality in Doctoral Education: A Technical Re- port of a National Study in Three Fields. Princeton, NJ, Educational Testing Service, 1976 6. Cohen J: Statistical power analysis for the behavioral sciences (rev. ed.). San Diego, Academic, 1977 7. Glass GV: Primary, secondary, and meta-analysis of re- search. Educ Res 5:3-8, 1976 8. Glass GV: Integrating findings: The meta-analysis of research. Rev Res Educ 5:351-379, 1977 9. Maracuilo LA, Serlin R: A common sense approach to theoretical statistics: A handbook for researchers and inves- tigators. (submitted for publication) 10. Pillemer DB, Light RJ: Synthesizing outcomes: How to use research evidence from many studies. Harvard Educ Rev 50:176-195, 1980 11. Holzemer WL: Quality in graduate nursing educa- tion. Nurs Health Care 3:171-189, 1982 12. Holzemer WL, Chambers DB: A contextual analysis of faculty productivity. J Nuts Educ 27:10-18, 1988