ED 177 323

AUTHORTITLE .

IiSTITUTION

SPONS AGENC!

ROB DATECONTRACTNOTE

EDES PRICEDESCRIPTORS

D00088112 11180118

Cl 022 825

Darcy, Robert L.Vocational Education Outcomes: Perspective forEvaluation. Research and Development Series No.163Ohio State Univ., Columbus. National Center forResearch in Vocational Education.Bureau of Occupational and Adult Education (DHEN/02),Nashington, D.C.79300-78-003251p.; For related dociments see CE 022 823-826 and ED170 560

MF01/PC03 Plus Postage.Evaluation; *Evaluation Criteria; *EvaluationMethods; Postsecondary Education; *ProgramEvaluation: Program Improvement; Secondary Education;*Vocational Education

ABSTRACTAs part of a project designed to identify appropriate

and feasible methods of evaluating vocational education, this essayprovides a perspective for evaluating vocational education it termsof outcomes. The lirst of four sections provides a rationale forstudying educational outcomes. Section 2 focuses on the nature andmeaning of,zputcomee and the range and diversity of vocationaleducation outcomes, and outlines an approach to outcomes evaluationwithin the general framework of vocational education evaluation. Inaddition, this section provides a list of thirty questions concerningpossible outcomes of vocational education which illustrates the broadand diverse range of possible outcomes. Among the key issuesldentitied in section 3 in the area of outcomes evaluation were (1)the definition of vocational education: (2) the specification ofvocational programs as educational treatments; and (3) sources anduses of evaluative criteria and experimental controls, and the use ofcomparison groups in evaluating vocational educaticr programs withrespect to specified outcomes. The final section suggests an agendafor improving outcome evaluation. (LRA)

***********************************************************************Reproductions supplied by EDRS ate the best that can be made

from the original document.***********************************************************************

nesearch and Development Series No. 163

-VOCATIONAL EDUCATION OUTCOMES:

PERSPECTIVE FOR EVALUATION

Robert L. Darcy

The National Center for Researchin Vocational Education

The Ohio State University1960 Kenny Road

Columbus, Ohio 43210

1979

to,

S DEPARTMENT Of WEALTH.EDUCATION iwELPANENATIONAL INSTITUTE Of

EDUCATION

1-10tuA0I Nt HAS Litt% growl.r`., r I AA( Tr, T As Stl. Cf 0.4-1) $ ROM'.1 S'I 44C04 '10.AN'/A ION t1R.C.14AAr. PtfNr%nt OP.N.01.45,A I 1)( 64 f cwhit,t v WI Plif

kit ot I A, %At :14A, .A4Stt, TI 0$Lt'.A . T. .'144 NT% .; V

THE NATIONAL CENTER MISSION STATEMENT

The National Center for Research in Vocational Education'smission is to increase the ability of diverse agencies,institutions, and organizations to solve educatioAal problemsrelating to individual career planning, preparation, andprogression. The National Center fulfills its mission by:

Generating knowledge through research

Developing educational .ograms and products

Evaluating individual program needs and outcomes

Installing educational programs and products

Operating information systems and services

Conducting leadership development and trainingprograms

3

Report of a Project ConductedUnder Contract No. OE 300780032

The material in this publication was prepared pursuant to acontract with the Bureau of Occupational and Adult Education,Office of Education, U.S. Department of Health, Education, andWelfare. Contractors undertaking such projects under government*sponsorship are encouraged to freely express their judgment inprofessional and technical. matters. Points of view or opinionsdo not, therefore, necess'arily represent official Bureau ofOccupational and Adult Education, U.S. Office of Educationposition or policy.

U.S. DEPARTMENT OF HEALTH, EDUCATION, AND WELFARE

Office of Education

TABLE OF CONTENTS S.z.

Page

LIST OF FIGURESv iv

LIST OF. TABLES.1 iv.

FOREWORD

I. INTRODUCTION 1

A. Purpose of the Essay 1B. Rationale for Studying Educational Outcomes., 2

II. EDUCATIONAL EVALUATION AND OUTCOMES ANALYSIS. 4

A. Assessment, Evaluation, and Research 4

B. The Education Enterprise andEducational Evaluation 6

C. Vocational.Education Outcomes 11D. Outcomes and Outcome Statements 20

III. SOME KEY ISSUES IN VOCATIONAL EDUCATIOgOUTCOMES EVALUATION 23

A. What is Vocational Education^ 24B. Vocational Programs as Educational

Treatments 26C. Evaluati'v C4teria and Standards, Experi-

mental Controls, and Comparison Groups 29D. Nature and Quality ofsOutcome.Data 34

6 IV. SUMMARY AND RECOMMENDATIONS 36

A. Summary 37B. Suggested Agenda fo Improving.

Outcomes Evaluation 38

REFERLNCES .41

LIST OF FIGURES

Figures Mit

1 Components of the Education Enterpriseand Types of Educational Evaluation 8

2 The Educational Production Functionas a Framework for Evaluation 10

3 An Illustratiun of Some of the DiverseTypes and Characteristics of VocationalEducation Outcomes 12

4 Educational Outcomes Evaluatioa Framework 21

5 Determinants of Vocational Education04tomes 27

LIST OF TABLES

Tables Page

Vocational Education Outcome Questions:Illustration of the Range and Diversityof Possible Outcomes 15

II Types of Data Used in VocationalEducation Outcomes Evaluation 35

iv

FOREWORD

Educational outcomes have commanded increased attentionin recent years as the focus of evaluation has shifted fromprocesses to products and impact. Are vocational educationprograms achieving their goals? What are the side effects ofvocational education? What are the tangible payoffs to indivi7duals and society from resources invested in vocational educa-tion?

This essay provides perspective for evaluating vocationaleducation in terms of outcomes. The author clarifies thenature and meaning of outcomes, expands our awareness of therange and diversity of vocational education outcomes, outlinesan approach to outcomes evaluation within the general frame-work of vocational education evaluation, identifies some issuesand problems relative to 'atcomes evalwition, and suggests anagenda for strengthening research and evaluation in the fieldof vocational education outcomes.

Audiences that may find this publication especially use-ful include the research, development, and evaluation (RD&E)community in vocational education and training; policy analystsand educational leaders; and some pracrtioners in state andlocal education agencies.

Vocational Education Outcomes: Pers?ective for Evaluation,originally appeared under the title "Examining Vocational Edu-cation Outcomes: Perspective on che State of the Art," Attach-ment "C" in Vocational Education Outcomes, Final Report on YearOne of the R&D Project "Examining Vocational Educational Outcomesand Their Correlates." The report was prepared at the NationalCenter by Robert L. Darcy, Kathleen A. Bolland, and Joanne Farleywith the assistance of Carolyn M. Taylor, and submitted to theU.S. Office of Education in January 1979. Other components ofthat report will also be published as serarate documents withthe titles Vocational Education Outcomes: A Thesaurus of OutcomeQuestions; Vocational Education Outcomes: An Evaluative BIbIi-ography of Empirical Studies; and Vocational Education Outcomes:Annotated Bibliography of Relatea Literature.

In the development of the general report and its variouscomponents, a number of people made valuable contributions.Distinguished representatives from business, labor, and educa-tion who served on the Nati.onal Workshop Panel on VocationalEduration Outcomes were: Walter G. Davis, American Federation

7

of Labor and Congress of Industrial Organizations: Mary EllenHillaire, Evergreen State College (Washington); Addison S.Hobbs, Michigan Department of Education (now with the MarylandDepartment of Education); 0. Louise Lothspeicht Oregon StateAdvisory CoUncil on Vocational Education; Fred D. Mack, CentralState University (Oklahoma); Gerald Q. Miller, Sharon SteelCorporation (Pennsylvania) and Employers National Job ServiceImprovement Committee; Philip L. Smith, The Ohio State Univer-sity; Jerry C. Olson, Pittsburgh Public Schools (Pennsylvania);Wallis E. Pereira, Industry Sducation Council of California;Clio S. Reinwald, Arizona Department of Educ.tion; Annell L.

Simcoe, Rutgers University; and J. Robert Warmbrod, The OhioState University.

We are also'grateful to Michael D. Hock of Worthington,Ohio, Douglas Sjogren of Colorado State University, and MichaelScriven of che University of San Francisco, who served as con-sultants for the study; to the five members of the NationalCpro-pr's Evaluation Technical Advisory Panel -- Carol B. Aslanian,College Entrance Examination Board; George C. Copal Universityof Minnesota; Donald W. Drewes, Conserva Inc.; Ruth P. Hughes,Iowa State University; and Daniel L. Stufflebeam, Western Michi-gan University -- and to Jo Ann M. Steiger of Steiger, Fink &Kosecoff, and David J. Pucel of the University of Minnesota, aswell as T. Lynne Peterson and William L. Hull of the NationalCenter staff, for reviewing drafts of the full report.

On behalf of the National Center, I want to express appre-ciation to the Bureau of Occupational and Adult Education, ' .S.Office of Education, for sponsoring this study and to Robert L.Darcy, Senior Research Specialist at the National Center bothfor directing the overall project and preparing the presenteslay.

Robert E. TaylorExecutive DirectorThe National Center for Re-

search in VocationalEducation

vi

VOCATIONAL EDUCATION OUTCOMES:

PERSPECTIVE FOR EVALUATION1

I. INTRODUCTION

This paper deals with the theory and practice of voca-tional education outcomes evaluation, which is part of thebroader field of educational evaluation. It is not intendedas a review and synthesis paper or comprehensive state-of-.the-art essay on vocational education evaluation but mainly.as a contribution to the conceptual basis of uutcomes analysis.

A. Purpose of the Essay

The overall purpose of the essay and the study to whichit relates is to help answer two research questions: (1) Whatparticular outcomes are appropriate for use as criteria toevaluate vocational education programs? and (2) What procedurescan be used for eiraluating vocational programs in terms ofthese outcomes?

Special attention in this paper is focused on:

- -clarifying the nature and meaning of vocational educa-tion outcomes;

- -expanding awareness of the range and diversity of out-comes;

1

,;

This essay was prepared as part of The National Center'sapnlied research and development project, "Examining Voca-t"Inal Education Outcomes and Their Correlates," under a con-tract with the Bureau of Occupational and Adult Education,Office of Education, U.S. Department of Health, Education, andWelfare. Other products of this study include an evaluativebibliography of vocational education outelome studies, a the-saurus of vocational education outcome questions, and an an-notated bibliography of literature related to vocationaleducation outcomes. The author wishes to express appreciationto John T. Grasso, Michael Scriven, Douglas Sjogren, andDaniel L. Stufflebeam for their review of an earlier draft,and his special thanks to Kathleen A. Bolland, Joanne Farley,Carolyn M. Taylor, and other colleagues at the National Centerfor their helpful comments and suggestions con.cerning the paper.

1

9

4

--developing an approach to outcomes evaluation within thegeneral framework of vocational education evaluation;

--identifying some issues, problems, and practices in theevaluation of vocational programs wlth respect to outcomes;and

--suggesting an agenda for strengthening research and eval-uation in the field of vocational education outcomes,which in turn can contribute to improving the effective-ness and efficiency of educational programs.

The present paper does not identify a blue-ribbon list ofvalid, operational outcomes nor does it lay out a procedure fordoing so. Those tasks remain to be accomplished in the future.

The audience addressed by this paper includes educationalresearchers and evaluators, vocational education policy makers,academics, and the general public. While some of the ideasand information contained in this paper may be directly usefulto practitioners, it is not intended as a how-to-do-it hand- 2book for program managers, evaluators, or classroom teachers.

B. Rationale for Studying Educational Outcumes

Outcomes analysis is important to observers of vocationaleducation for at least three very practical reasons. First,

2To help meet the needs of practitioners, the NationalCenter has developed Evaluation Handbook: Guidelines and'Practices for Follow-Up Studies of Former Vocational EducationStudents (Stephen J. Franchak and Janet Weiskott, 1978). Thebiblio4raphy alluded to in footnote 1 reports on design andprocedures that have been used in some empirical outcomestudies conducted during the past decade (see Kathleen A. Hol-land, Vocational Education Outcomes: An Evaluative Bibliographyof Empirical Studies; Columbus: National Center for Researchin Vocational Education, The Ohio State University, 1979). AnotherCenter project completed in 1978 is Interpreting Outcome Mea-sures in Vocational Education: A Final Report (Floyd L. Mc-Kinney, Kenney E. Gray, and Marie Abram). Scheduled for pub-lication by the National Center in 1979 are five state-of-the-art papers dealing with various aspects of evaluation: NeedsAssessment for Prograin Planning in Vocational Education (YRT6r-mation Series No. 154); Longitudinal Methods-as Tools for Eval-uating Vocational Education (IS No. 155); Use of EvaluativeData by Vocational Education (IS No. 156); Impact Evaluation inVocational Education: The State of the Art (IS No. 157); andData Sources for Vocational Education Evaluation (IS No. 158).In addition, the National Institute of Education is producingpapers and reports as part of its study and evaluation of voca-tional education mandated by the Education Amendments of 1976.

2

lo

the states are mandated by Congress to evaluate program ef-, fectiveness on the basis of twO specific outdome criteria:

(a) employment in training-re ated occupations, and (b) employ-er assessment of the trainin and preparation for employmentreceived by students.J Many more outcomes, such as educationalachievement and human develppment, are perceived by educatorsand others as being of importance both to individualstudents and society. Se ond, public concern over tax bur-dens, rising costs of and accountability reg*d-ing returns from educati nal spending have led to a search fortangible indicators of/program impact. Third, as educationalevaluation has developed and matured, study of the broad areaof outcomes and impact ias assumed increased importance in acomprehensive program o evaluative research alongside suchnon-outcome variables als institutional context, student char-acteristics, resources program goals, and educational pro-cesses'.

Moreover, these practical considerations are in keepingwith the fundamental purposes of scientific inquiry: to dis-cover (1) the consequences of given actions (including pro-grams, policies, and institutional arrangements) and (2) thespecific actions that will produce given consequences. Inthe policy sciences, there is a third function of research,namely to generate information which can help decision makersselect the goals that will be treated as desirable "given"consequences and provide evaluators with standards to be usedin evaluating observed consequences. In the present context,we are addressing conceptually, a question of "positive anal-vsis"--What are the various possible consequences of vocationaleducation programs? and a question of "normative analysis"--Which of these consequences are appropriate and feasiblc tobe used as criteria for evaluating vocational programs?'

3U.S. Public Law 94-482, Education Amendments of 1976,Sec. 112(b).

4Nobel prize winner Milton Friedman quotes John NevilleKeynes in distinguishing among "positive science," a system-ized body of knowledge concerning what is; "normative science,"knowledge addressing criteria of what ought to be; and "art,"a ;..ystem of rules for the attainment of a given end. Essaysin Positive Economics (Chicago: University of Chicago Press,l§53), P. I-.

II. EDUCATIONAL EVALUATION AND OUTCOMES ANALYSIS

A review of literature in the broad area of evaluation,the narrower field of educational evaluation, and the morespecialized field of vocational'education evaluation isclosesconsiderable variation in how people view the nature, pur-pose, and preferred methods of assessment and evaluation,

A. Assessment, Evaluation and Research

In the Encyclopedia of Educational Evaluation, assess-ment is defined- as "the process of gathering data and fashion-ing them into an interpretable form."5 Worthen and Sanders.in Educational Evaluation: Theory and Pra6tice identify five"schools of thought about how evaluation ougETto be defined."As practiced by their respective adherents, these approachesare characterized as: (1) "educational measurement," (2)"professional judgment," (3) "comparing performance data withclearly specified objectives," (4) "collecting information toassist decision-makers in choosin among available decisionalternatives," and (5) "systematic collection and analysisof information to determine the worth of a thing." Irrespec-tive of approach, description is acknowledged to be a basiccomponent of the evaluation process.6 Evaluation is sometimes

5Sometimes the term "assessment" is used interchangeably

with "measurement" and with "evaluation." See Scarvia B. Ander-son and others, Encyclopedia of Educational Evaluation (SanFrancisco: Jossey-Bass, 197) p. 27.

6Blaine R. Worthen and James R. Sanders, Educational

Eveuation: Theory and Practice (Worthington, Ohio: Charles

Jones, 1973), pp. 20f, 109f, 125. In their treatment of de-scription as a component of formal evaluation, Worthen andSanders focus on the Weas of Robert E. Stake. More recently,

writing in the newsletter of The Evaluation Network, Joe B.Hansen emphasizes the importance of description in programevaluation and laments the bias which evaluators display forexperimental studies to the neglect of full and accurate de-

scription. "With the advent of.modern statistical techniquesand the wide availability of computers to perform statisticalanalysis," he writes, "educational research and evaluationhave become associated with statistical methodology to thepoint of eclipsing more fundamental considerations. It could

be interpreted by practitioners of this emerging art-science

as a sign of insecurity -- (footnote continued on next page)

4

referred to as evaluative (or evaluation) research. Scrivensuggests that "the basic disanction Obetween 'straight' re-sedrch and.evaluation reaearch] seems to be that evaluationresearch-must produce as a conclusion exactly the kind ofstatement that social scientists have for years been taughtis illegitimate: a judgment of value, worth, or merit."' In .

the present paper we are adopting the Worthen and Sanders/Scriven definition of evaluation as the determination of theworth of athing. In Vlis view, evaluation involves (a) ob-servation and description of what is, and (b) a value judg-ment concerning what should be, i.i7, appraisal or judgmentof worth.

However defined, evaluation has numerous purposes andmethods. The former include accountability, .pampliance, pro-ject monitoring, program improvement, and providing usefulinformation for judging decision alternatives such as whetherto continue, expand, modify, or terminate programs.8 Needsassessments are conducted, and both formative and summativeevaluations are carried out. The diversity of evaluationmethods is suggested by some of the terms one encounters inthe literature: goal-based evaluation, goal-free evaluation,formal, informal, objective, subjective, process, on-site,product, follow-up, impact, payoff evaluation, benefit-coststudies, and cost-effectiveness analyses, among.others.

This kaleidoscope of concepts, purposes, perceptions, andprocedures adds a richness to evaluation that may, however,exact a toll. The other side of the diversity coin in educa-tiona1 evaluation is a lack of consistency, clarity, and

a

6(continued) -- that evaluators are often more concerned

with the statistical significance of their results than withthe importance of the questions to which the analyses areaddressed." See "The Formative-Summative Dichotomy and theRole of Description in Program Evaluation," Evaluation News,No. 6, August 1978, pp. 14-16.

7Michael Scriven, "Evaluationures," pp. 3-93 in W. James Pophamtion (Berkeley: McCutchan, 1974),

8.In Educational Evaluation and Decision Making (Itasca,

Ill.: Peacock, 1971T, prepared for the education societyPhi Delta Kappa by Daniel L. Stufflebeam and others, evalua-tion is defined as "the process of delineating, obtaining,and providing useful information for judging decision alter-natives."

Perspectives and Proced-(ed.) Evaluation in Educa-p. 4.

5

Credibility with respect to the knowledge base we have accum-

ulated in the field of educational outcomes.9

B. The Education Enterprise and Edncational Evaluation

It will be helpful to view educational outcomes and.out-comes evaluation in the broader framework of the overall ed-

ucation enterprise.

9A. 1976 report by the Committee on Vocational EducationResearch and Development (C)VERD) noted that "$250 millionspent by the U. S. Office of Education in vocational educationresearch and development during the last 10 years has not haddocumented, widespread impact" (Assessing Vocational EducationResearch and Development, National Academy of Sciences, p. 1).Commenting on COVERD's findings, the president of the Ameri-can Vocational Education Research Association observed that"the substance and process of vocational education researchleaves something to be desired" (J. Robert Warmbrod, "Evalua-tion Research in Vocational Education," paper presented at.the AVERA annual meeting in Houston, Texas, December 6, 1976,and reprinted in Beacon, Vol. 6, No. 1 (January 1977) pp. 1, ,

6-11.) Other evaliiii75is have noted that "most of . . . [theevaluation of investments in the human agent, such as voca-tional and technical education] is of little or no use forpurposes of making reasoned choices among competing socialprograms . . , due to the methodological errors which marmany'studies as well as the lack of effective techniques tohandle many of the more difficult issues which arise as a partof the evaluative process (Jerome Moss, Jr., and Ernst W.Stromsdorfer, "Evaluation Vocational and Technical EducationPrograms" in Gerald G. Somers and J. Kenneth Little, VocationalEducation: Today and Tomorrow, Center for Studies in-V6a=----tional and Technical Education, University of Wisconsin, 1971,

p. 221). Floyd L. McKinney found "a lack of consistency inthe use of acceptable procedures in conducting follow-up studies"(Pro ram Evaluation in Vocational Education: A Review, ER/CC ear ng ouse on Career Education, Ae centirlor VocationalEducation, Ohio State University, 1977, p. 24). Writing inan official U. S. Department of Labor magazine in 1974, aColumbia University researcher observed that "the large bodyof studies [analyzing] the outcomes of high school vocationaleducation [are] beset with conceptual problems, methodologicalpitfalls, and statistical limitations . . . . Proceedingfrom different values and assumptions, the analysts have noteven agreed on the objectives or outcomes to be tested" (BeatriceG. Ruebens, "Vocational Education: Performance and Potential,"Man owe ,,Auly 1974, pp. 23-29). For additional informationon the state of vocational education evaluation see Darcy,Robert L.; Bolland, Kathleen A.; and Farley, Joanne.Vocational Education Outcomes (Columbus: National Center forResearch in Vocational EducaAion, The Ohio State University,1979) pp. viii and passim ha, well as Bolland, Evaluative Biblio-graphy, passim).

6

14

Figure 1 lists six components of the educational system,each of which may be seen as an approOriate object for de-scription and evaluation. 'Mese interdependent lets of edu-cation variables are:

(1) Context -- the milieu of circumstahces and needs,organizational arrangements, societal values, insti-tutional settings, and the physical environment with-in which education takes place;

(2) Students -- characteristics of the potential learners,who are recipients of investment activities and whosehuman capital" 41.11 be enhanced when the investmentactivities are successfully carried out;

(3) Resources -- in money terms, the funds allocated topay for real resources employcd in educational pro-duction; in real terms, the quality and quantity ofhuman and physical factors of production so employed(e.g., teachers, physical plant, instructional sup-

.

plies and equipment);

(4) Goals -- the stated or unstated aims of educationalprograms (i.e., intended educational outcomes); and

(5) Processes -- instructional methods, program proced-ures, and educational technologies (broadly defined);

(6) Outcomes -- short-term consequences and longer-termimpact resulting from vocatidnal programs (i.e.,dependent variables associated with the interactionof the five preceding sets of independent variables).

This six-factor analytical framework is similar to the CIPPmodel (Context, Input, Process, Product)'described in evalu-ation literature, although the terms are not defined in

10Acquired capabilities that can be used in one's capac-ity as a factor of production, i.e., as a human resource.See Robert L. Darcy, "The Nature of Economic Enterprise," inEdwin L. Herr (ed.) Vocational.Guidance and Human Development(Boston: Houghton Mcrfrin 1974), p. 121. Note.that humancapital does not denote people, but capabilities embodied inpeople. For a simplified explanation of human capital invest-ment, see William E. Becker, Jr., "Investment in Human Capi-tal," pp. 59-76 in Warren G. Meyer (ed.) Vocational Educationand the Nation's Economy (Washington, D. C.: American Voca-tional Association, 1977).

COMPONENTS OF THE EDUCATIONAL ENTERPRISEAND

TYPES OF EDUCATIONAL EVALUATION

(VA1APPRAISAL

LUE JUDGMENT)

DESCRIPTION

EVALUATION ... CONSISTS OF AND REGARDING

CONTEXT

STUDENTS

RESOURCES{Money

ReeI Resources

GOALS

PROCESSES

tLilhort-TermOUTCOMES ongsr-Term Impact

precisely the same way. 11It also resembles a conventional

education production function, whicti relates input factorsto educational. products or outputs.'" The five "input" fac-tors listed i, Figure 1, which interact to constitute educa-.,tional programs or "treatments," are recast in Figure 2 (noteinput factors ni n5) in the format of a production func-tion. This analogy suggests that over a period of time, andsubject to the influence of intervening forces and error fac-tops, it is theoretically possible to observe certain outcomesas having been generated by (i.e., significantly associatedwith) given.educational treatments.

As suggested above, each component of the.educationalenterprise represents A potential obiect of systematic inquiryand evaluation. For purposes of description and appraisal, onemay single out student characteristics (e.g., age, sex, men-tal ability, socioeconomic background of vocational students);or the quality of resources employed (e.g., training and ex-perience of teacher"); or educational outcomes (e.g., skillproficiency, employment status, hourly earnings) for specialscrutiny rather than addressing, under the rubric of "programevaluation," the totality of factors relevant to educationalperformance. Indeed, this selective, "partial equilibrium"methodology, based on the principles of specialization 'and

.

division of labor, might produce the same results for vocationaleducation evaluation that are claimed in economics; sharplyincreased productivity. We need to know far more than we.pre-gently know about student.characteristics, context, resources,goals, and the processes of vocational education, as well asabout outcomes. And certainly we need to know about correla-tions and interactions among the variables. The appeal forspecial attention to particular facets td the vocational ed-ucation enterprise is not meant to imply that each componentstands in isolation from the others. In education, as in theeconomic world, everything depends on, everything else. Yet,it can still be fruitful to study parts.and facets.

11See Daniel L. Stufflebeam, pp. 128-150 in Worthen andSanders, Educational Evaluation: Theory and Practice.

12See Richard V. Kauffman, A Study of the EducationalProduction Function, Ph.D. thesis, Department of Economics,Colorado State UniVersitY, 1974, pp. 24-31 (reprinted in hardcopy by University Microfilms, Ann Arbor, 1978); and Daniel C.Rogers and Hirsch S. Ruchlin, Economics and Education: Prin-ciples and Applications (New York Free Press, 1971), pp.107f, 119ff.

9

1 7

THE EDUCATIONAL PRODUCTION FUNCTION AS AFRAMEWORK FOR EVALUATION

Oasf(S, C, R, G, P)*

Outcomes are a function of Students, Context, Resources, Goals, Processes

STUDENTS (n1)

CONTEXT (n2) VOCATIONALEDUCATIONAL

EDUCATION TimeRESOURCES (n3)OUTCOMES

GOALS (n4) PROGRAMS

PROCESSES (n3)

z

Outputs

Products

Consequences(defined as)

I I Effects

Results

Impact

*Observed outcomes are subject to the influenceof extraneous forces and error factors.

Figure 2:8

C. Vocational Education Outcomes

The focus of this essays however, is on vocational educa-tion outcomes, which arg broadly defined as the consequencesof vocational programs." Figure 3 illustrated the range anddiversity of possible outcomes associated with vocational edu-cation programs. All outputs, products, consequences, effects,results, and impact of vocational programs are recognized asoutcomes whether they are intended or unintended, positive ornegative, short-term or long-term, economic or noneconomic,direct or indirect. These outcomes may affect individual stu-dents, society as a whole, particular societal entities (suchas the local school system), or functional characteristicssuch as the human resource base of the state and local economy.The outcomes may be seen as resulting from a student's partic-

,

,.ipation in vocational education or from the existence of on-going programs in the community. Outcomes are manifest aschangbs in individual or societal capabilities (includingknowledge am skills), attitudes, attributes, status, or cir-.cumstanzes.'"

Some outcomes, such as the acquisition of occupationalskills by secondary .44chool students'who complete vocationalprograms, are objectively observable and often quantifiable.Others, such as the development of work-related attitudes orleadership capability, may be perceived more subjectivelywhile defying straight-forward empirical observation. Easeof measurement does not qualify a particular outcome as nec-essarily important or valid as a criterion for evaluatingvocational education, though it certainly may offer practical:advantages for such purposes.

Figure 3 does not purport to identify all the kinds andcharacteristics of educational outcomes. No explicit :.7efer-ence, for example, is made to physical, esthetic, or moraldevelopment. The representation is suggestive and'impression-istic, hence the guestio4 marks. Every effect that is

13Definitions of vocational education and vocational

programs are presented in Section III below.

14A distinction is sometimes 'made between educational

ou,puts and outcomes, the latter implying change while the.

former merely suggests "through-put." No attempt is made inthis paper to distinguish among such terms as educational pro-duct, output, outcome, impact, etc. As indicated in Fig-ure 3, outcomes are broadly defined to include all of theconsequences and results of the education enterprise.

11

1 9

AN ILLUSTRATION OF SOME OF THE OIVERSE TYPES

AND CHARACTEMSTICS OF VOCATIONAL EDUCATION OUTCOMES

Intended, Unintended(Attained Goals, Side Effects)

'N

Employment-ReletedEducation-Related,

Coping Skills

Effects on Individualsand on Society

Quantifiable,

OUTCOMES

Calculation, Psychomotor,Communication,

/Direct, Indirect

Short-term, Long-term

NoneconomicEconomic,? ? ? (Other) ? ? ?

Group Relations Skills

Benefits,Unfavorable

Consequences

significantly linked with a vocational program is regarded asa vocatic..al education outcome. Not all of these consequences,however, are readily observable. Professional debate, social.criticism, and aducational research have made the Americanpeople keenly aware of how much we do not know about the effectsof schooling. We therefore need to spiiE of possible outcomesof vocational programs without confining the analysis-to suchnarrow outcome sets as preannounced goals, presumed benefits,or quantitatively measured impact.

Outcomes typically are multidimensional--e.g., a mix-ture of educational and economic consequenCes, cognitive andaffective, individual and societal--and they occur as "jointproducts" in the sense of multiple outputs occurring to-gether in a single production process. Thus, just as lumberand wood particles are simultaneous outputs of the ...oggingindustry, so too are occupational skills, job-related atti-tudes, and 'a high school diploma--all "products" resultingfrom the same schooling experience. Which is the "product"and which the "byproduct" is a matter of selective per-ception. A comprehensive assessment of vocational educa-tion outcomes would acknowledge all of the observable con- .

sequences and presumably focus on the ones which are ofspecial relevande-to the evaluation task at hand. It wouldalso recognize the opportunity .costs or tradeoffs associatedwith vocational programs, i.e., the alternative educationaloutcomes foregone becausa the resources are being.utilizedin vocational programs.

Differential outcomei may be Associated with variationsin educational treatment (see Section III below), intensityof treatment, i.e., "time on topic"), educational level(high school, post-secondary, adult), student characteristics(age, sex, race/ethnicity, socioeconomic background, physi-cal or mental handicap, aptitudes, career interests, etc.)and a hose°of other variables indicated in Figures 1 and 2above. 'This follows from the recognition that variation inthe "input" factors of the educational production function(Figure 2 above) predictably will result in differentialoutcomes.

Awareness of the broad range, di.Tersity, and multi-dimensional nature of possible educational outcomes is im-portant to.practitioners, researchers, evaluators, policy

. makers, and the general public. It helps offset the narrow.perception of equating outcomes simply with goals or benefits,while ignoring "side effects." This expanded view in turntriggers a search for evaluative standards beyond preannounced

13

program objectives and can stimulate a re-examination,9fprogram implementation strategies and overall policy."

As an example, when State and local units of governmentfirst enacted genera] sales taxes, their single purpose was

to raise revenue. Attention was focused on the outcome of

fiscal productivity. Another outcome of the tax, however,

was its regressive burden: low-income consumers paid highereffective tax rates (tax payments as a percentage of income)

than high-income consumers. While the purpose of the tax

was not to redistribute income, that was nevertheless an

outcome. Awareneas of this consequence eventually led many

states to amend the general sales tax by exempting food,

thereby reducing the regressivity of the tax.

Similarly, if an important outcome of vocational educa-tion is to increase the rate of school completion eventhough the stated objective (or specified evaluation cti-terion) is job placement, then knowledge of the school-completion outcome can be useful to policy makers as theyrevise legislation and allocate resources for the support ofvarious programs. "Side effects" are consequences no lessthan attained objectives. As the environmental,and consumerprotection fields illustrate, yesterday's side effect may betomorrow's most significant outcome.

Table I lists 30 questions concerning possible outcomesof vocational education. The outcome questions (verbal ex-pressions of empirical phenomena) listed here are not intended

15In a recent monograph on noneconomic outcomes, theU.S. Department of Labor noted the "unfortunate tendency toevaluate many employment and training programs in terms ofnarrow economic', cost-benefit analyses alone." The Job Corpswas cited as "a good example of a program that should not beviewed so narrowly--a program in which the noneconomic impactsare at least as important and possibly more so, than economicbenefits such as higher earnings." The report identifies 21different noneconomic outcomes in three areas: (1) job re-lated; (2) social-attitudinal; and (3) health-related. Themtcomes are discussed in the context of "social developmentobjectives . . . rehabilitation goals, and . . . remediation."U.S. Department of Labor, The Noneconomic impacts of theJob Corps, R & D Monograph 64, Employment and Training Admin-istration (Washington, D.C.: U.S. Government Printing,-,Office,1978) pp. iii-iv. More than 200 economic and noneconomic out-comes -are identified in Joanne Farley, Vocational EducationOutcomes: A Thesaurus of Outcome Questions (Columbus:National Center for R _march in Vocational Education, The OhioL;tate University, 1979).

14

_

1

Table I : . .

VOCATIONAL EDUCATION OUTCOME QUESTIONS: ILLUSTRATIONOF THE RANGE AND DIVERSITY OF POSSIBLE OUTCOMES i

No. Outcome Questions Thematic Area

1. Do students become occupationally proficientand ready for entry-level employment as aresult pf completing a .vocational program?

.

Skill acquisition

2. What proportion of former vocational stu-dents are employed in training-related jobssix months following program completion?

4

Employment intraining-related jobs

3.,

Are dropout raLos lower for vocational stu-dents than for comparable students enrolledin general/academic curricula at the sec-ondary school level?

.

v

Schooling

.

4. Do vocational programs at all levelshighschoblo postsecondary, and adult--signifi-cantiv strengthen the human resource base ofthe local and state economy?

Economic develOpment

.

5. One year following program completion, are.former vocational students earning higherhourly wages than.comparable, general/academicstudents similar jobs?

Wage rates -

.

.

6.

.

.

Are the parents of students who are enrolledin vocational programs better satisfied withthe overall high school program than parentsof students enrolled in general/academiccurricula?

,

Parental satisfaction

wkth school program

7.

,

Do vocational students obt-ain jobs which theyconsider acceptable, whether in 'training- 3related fields or not, within a specifiedtime period following program completion?

Employment

8. Are vocatlonal programs more effective orless effective than general/academic cur-rimila in enhancing the-self-imaae ana self-confidence of qtudents who are preparing foremployment:

Student self-imagc

i

-1

continued

,.....

Table I (Continued)

.

VOCATIONAL EDUCATION OUTCOME QUESTIONS: ILLUSTRATION

OF THE RANGE AND DIVERSITY OF POSSIBLE OUTCOMES

No. Outcome Questions Thematic Area

9. Do vocational students receive higher employerratings than comparable general/academic stu-dents following a specified period of employ-ment? ..

Employer assessmentof students

.

10. Do vocational students develop leadershipability through their participation in occu-pationally-relfted youth programs such asDistributive Education Clubs of America(DECA), Future Homemakers of America (FHA).Future Farmers of America (FFA), and Voca-tional Industrial Clubs of America (VICA)?

Leadership develop-ment

,

.

11. Are students who are currently enrolled invocational programs better satisfied withtheir overall school experience than compar-,able students'enrolled in general/academiccurricula?

Student satisfactionwith schooling

.

12.

.

.

Do graduates of vocational programs expressmore, less, or the same job satisfaction ascomparable graduates of general/academicprograms holding similar jobs?

)

Job satisfaction

13. Does participation in high school vocationalprograms reduce the incidence of school

.

vandalism and othei antisocial behavior onthe part of students?

Antisocial behavior

.

14. Does vocational education (high school, post-secondary, adult) provide its graduates witha general set of skills and basic work dis-ciplines that significantly increase theirchances for successful employment?

Employabilitydevelopment

4.1

0

Table I (Continued)

VOCATIONAL EDUCATION OUTCOME QUESTIONS: TLLUSTRATIOIOF THE RANGE AND DIVERSITY OF POSSIBLE OUTCOMES

No. Outcome Ruestions Thematic Area

15. By completing a vocational program, do stu-dents acquire job-seeking skills such as howto write a resume, how to participate in an

interview, and knowing where to look for jobopportunities? .

job-search skills

16. Do students enrolled in vocational programslearn the basic cognitive skills of speaking,reading, writing, ansil numericalknalculationas effectively as comparable students inacademic/general curricula?

Educational achieve-ment

.

17. Does participation in a vocational programgive students a realistic understanding ofwhat employers will expect of them on the job?

World-of-workunderstanding

IA. Is the occupational mobility Jf vocationalstudents limited both horizontally 'andvertically by the content of the programs(e.g., trainIng in narrow skill areas or inoccupations with diminishing job opportuni-ties): , il

/

Occupational mobility,

.

11.

--------------,--Does participation in a high school vocationalprogram inhibit or enhance a student's oppor-tunities for a college education?

College education

20.

21.

Do students who complete vocational programshave a more realistic understanding of oursociety and its economic system than compar-able students who complete general/acAemicprograms?

Socioeconomicunderstanding

Are the returns from individual and socialinvestments in vocationol education lower,higher, or the same aq returns frominvestment.; made in gent.L.11/a;:ademic pro-

grams below the baccalaureate level?0

i..

Economic returnsto schooling

i

contlnuvd------O

-..........--...-..

Table I (Continued)

VOCATIONAL EDUCATION OUTCOME QUESTIONS: ILLUSTRATION

OF THE RANGE AND DIVERSITY OF POSSIBLE OUTCOMES

N . Outcom uestions Thematic Area

22. .Does participation in vocational piDgrams ,

on the high school levil contribute measur-ably to the moeal development of students?

Moral development

23. Through their participation in vocationaleducation programs (e.g., cooperative educa-tion, placement in part-time jobs), do dis-advantaged minority youth earn more incomewhile still in school than comparable studentsin general/acadetic programs?

Student earnings

.

24.

.

Do high ichool, postsecondary, and adultvocational programs increase occupationalawareness, training opportunities, and jobaccess for women in both traditional andnontraditional areas?

,

Employment opportun-ities for women

2 .

.

Does favorable community reaction to ongoingvocational programs result in general expres-sións of satisfaction and financial supportfor the entire local school system?

Community support .

for education

.

26. Do students who have participated in vocationalprograms on the high school level have moreclearly defined occuoational goals than thoseexpressed by nonvocational students who enrollfor postsecondary uoation?

Career planning .

,

27.

-Do high school home economics courses sig-

nificantly increase the consumer skills of

students who have taken them?.

.

Consumer skills

continued-----

18

Table I (Continued)t

VOCATIONAL EDUCATION OUTCOME QUESTIONS: ILLUSTRATION.

OF THE RANGE AND DIVERSITY Cf POSSIBLE OUTCOMES,

. Outcomelauestions Thtmatic Area

2 . Do certain admissions and student assignmentpractices followed in voicationaI educationtend to reduce or to reinforce occupationaldiscrimination on ihe basis of race, class,or sex? .

Occupational dis-crimination.

.

29. Is job-search time lower for students whoparticipated in vocational programs, than forcomparaple workers who did not? 4_

Unemployment.

30.,

How do the annuiii earnings of vocational grad-uates compare with those of general/academicgraduates, both in the short term and in thelong run?

,

Earnings

(continued from page 14).

to imply anything about what we actually know, believe, orsuspect concerning the consequences of vocational programs.The questions simply illustrate the broad and diverserange of possible outcomes.

It is interesting that while some of the outcomequestions relate to employment and earnings, other focus

19

On such topics as schooling, antisocial behavior, economicdevelopment, parental attitudes, and the like. Some of thequestions imply positive consequences for vocational pro-grams, while others may suggest negative or neutral effects.What is quite clear is that even a limited array of voca-tional education outcomes, as we have illustrated here, ex-tends well.beyond traditionally announced program objectives,labor market benefits to individual studenti, and alleged.harmful or beneficial effects on educational development.

Several of the outcome questions in Table I have begnaddressed in empirical research and evaluation studies.'"Undocumented assertions have been made regarding the factsalluded to in other questions. And some of the questionshave simply been posed without conjecture as to the answers.At this point in the discussion, in keeping with our intentto expand awareness of the range and diversity of outcomes,

1 we simply wish to regard these questions as illustrativeverbal expressions of possible results: waves and ripWsthat vocational programs may or may not be generating.'

D. Outcomes and Outcome Statements

Figure 4 outlines a conceptual framework for evaluatingvocational education with respect to outcomes. The outcomesthemselves (1) are empirical phenomena: real-world conse-quefices of educational programs. Outcome statements (2)are verbal expressions of these cause-and-effect phenomena.Educational outcome statements can take the form of declar-ative sentences or hypotheses: "Completers of high schoolvocational programs have lower rates of unemployment sixmonths after graduation than 'comparable completers of aca-demic/general curricula." Or they can be-phrased as ques-tions (as in Table I): "Do completers of high schoolvocational programs have lower rates of unemployment sixmonths after graduation than comparable completers ofacademic/general curricula?" An outcome statement describeswhat happens to

16For descriptive and evaluative comments on thesestudies see Solland, Evaluative Bibliography.

17See Farley, Thesaurus for a categorized listing of228 vocational educiETEZ-Eacome questions along with ananalysis of the structure and content of outcome statements.

20

EDUCATION OUTCOMES EVALUATION FRAMEWORK

(1) EDUCATIONAL OUTCOMES

(2) OUTCOME STATEMENTS/HYPOTHESES/QUESTIONS

<>.(3) OUTCOME MEASURES (DATA, EMPIRICAL INDICATORS)

(4) EVALUATIONITANDARDS (RELATIVE, ABSOLUTE)

g<>.(5) ANALYSIS, INTERPRETATION, AND EVALUATION

(6) FINDINGS CONCERNING THE CONSEQUENCES OF PARTICULAR PROGRAMS

(7) APPLICATIONS OF FINDINGS(7-A) POLICY (GOALS, RESOURCES, RESOURCE ALLOCATION)(7-B) PROGRAM DESIGN(7-C) PROGRAM MANAGEMENT(7-0) ACCOUNTABILITY

Figure .4

2 9

whom and how, and in some cases also indicates when, where,and why. As a minimum, a meaningful outcome statementclearly identifies (a) an outcome, (b) the person or thingaffected, and (c) tle educational treatment. Often thestatement (qr outcome question) will also identify (d) theagency through which the tkeatment is applied, (e) a rationalefor the posited outcome, and (f) a time framework. Expressingan educational outcome in precise verbal form affords twoimportant benefits: clarity of understanding and clarity ofcommunication. ks 126get of Thesaurus fame and others havepointed out, words are not only a medium of communicatiql,but equally impurtant, they are instruments of thought.'

In Table I, questions are used rather than statementsor hypotheses. This is to avoid giving an impression thatwe are asserting beliefs or.reporting.documented facts.

Outcome measures (point 3 in Figure 4) are empiricalindicators of the educational outcomes. These indicatorsmay be qUantitative (differences in amount, expressednumerically/statistically) or qualitative (differences inkind, described verbally)* The outcome measures are ob-servable indicators of the changed conditions or circum-stances resulting from an educational program. Thesechanges may be observed directly or indirectly. Outcomemeasures require empirical information such as employmentand wage data, and standards or norms for evaluating thedata. It is at poia74 -TE-The framework, therefore, thatpositive analysis of educational outcomes (observation anddescripiton) merges with normative analysis (appraisalof data) to produce the judgment of worth that constituteseducational evaluation (5).

The findings about program donsequences (6) that resultfrom evaluation research become 'available for a variety ofapplications (7) including policy making, proaram improvement,accountability, and decisions to terminate, expand, orotherwise modify the education enterprise. Returning to

.

18The author is indebted to Herbert A. Simon, winner ofthe 1978 Nobel Prize in Economics, for Roget's observation:"The use of language is not confined to its being the mediumthrough which we communicate our ideas to one another; itfulfills a no less important function as an instrument ofthought; not being merely its vehicle, but giving it wingsfor flight." Herbert A. Simon and Allen Newell, "Models:Their Uses and Limitations" in Leonard D. White (editor)The State of the Social Sciences (Chicago: University ofChicago Press, 1955), p. 66.

22

:3 0

4, we want to emphasize the point that 'introducingvalue criteria (standards of goodness or badness) into theresearch process is prerequisite for making evaluativejudgments. Whether a particular outcome such as the place-rent of 65% of all vocational graduates in training-relatedjobs is considered good, bad, or indifferent depends onthe evaluative standards applied. Knowledge of facts alonemay be sufficient for research, but in the absence of astandard focnbomparison, there can be no meaningfulevaluation."

Standards or norms for evaluating observed outcomes canthus be regarded as the link between positive outcomesanalysis and evaluation of vocational education in terms ofoutcomes, which by definition is a normative undertaking.Where these standards come from and how they are used aretherefore fundamental issues to be considered in this paper.20

III. SOME KEY ISSUES IN VOCATIONAL EDUCATION OUTCOMES EVALUATION

0

As noted by the authors of a Rand Corporation report onthe findings of educational research, "Education is an ex-tremely complex and subtle phenomenon." Moreover, educa-tional research is subject to serious limitations of data,measurement instruments, and experimental controls, includ-ing the influence of factors outside the schools ("perhapsstrong enough to 'swamp' the effects of variations in educa-tional practices"). The upshot is that "research has found

19Jerome Moss, Jr., The Evaluation of Occu ational

Education Programs, Technical Report, Research CoordinationUnit in Occupational Education, University of Minnesota,September 1968, p. 5.

20A8 suggested in Section I, outcomes analysis addressestwo types of questions: What are the actual consequences ofvocational education?.and What are ate desirable consequencesof vocational education? The first is iriT5T.Frcal matter..of-fact question triggering a search for information aboutwhat is. The second is a normative question leading to asearch for values indicating what ought to be. To the ex-tent that value judgments are influenced by evidence andreason, the search for values and facts will be intertwined.

nothing that consistently 40 Unambiguously makes a diffel-ence in student outcomes."'

While the Rand assessment dealt with general education(and in the context of the federal government's role inschool finance), the conclusions are not unlike those re-ported above for vocational education. .Indeed, vocationaleducation research and evaluation,faces all of,the generalproblems confronting-education plus some special difficul-ties of its own, to which attention is now turned.

A. What is Vocational Education?

The study of educational outcomes, as illuatrated inFigures 1 and 2 above, investigates the consequences ofdistinctive educational treatments or interventions. What .

is the nature of the.treatment carrying the label "voca-tional education"?

According to the Education'Amendments of 1976 (Sec. 195),the term voCational education means "organized educationalprograms which are directly related to the preparation ofindividuals for paid or unpaid employment, or for additionalpreparation for a career requiring other than a baccalaureateor-advanced degree." In further defining an organized edu-cation program, the law refers to "instruction related tothe occupation or occupations for which students are intraining or instruction necessary for students to benefitfrom such.training."

Im a generic sense, vocational education can be definedas any program of education or training that enhances one'scapability to pursue a vocation, a calling or occupa-tion or career. Defined more narrowly for programmatic

21 Harvey A. Averch and others, How Effective Is School-ing:. A Critical RdView and Synthesis of Research Findings,Rand Corporation, Santa Monica, Calif., March 1972. Seepp. ixf, 154, 160. The authors emphasize that:they are"not suggesting that nothing makes a difference, or thatnothing 'works'; the problem is that research findingsoften conflict." In a conversation with the author of thispaper, Michael Scriven pointed out, however, that cognitiveoutcomes do consistently'show up. It is when educaors"slide into grandiosity" searching for impact on life style,income distribution, and similar variables.that planned var-iations in educat.ion soem not to work.

24

i2

4

purposes, vocational education refers to occuPational pro-grams which are included in state plans developed by the re-spective State Boards for Vocational Education. A similaradministrative definition limits vocational education toprograms supported by Federal funds.`` In practice, "thevocational education program has become so broad and diversein its response to manpower demands.and the needs of allages and ethnic groups that it,As almost impossible to de-scribe in any definitive way."44

One may question whether it makes sense to speak of"the" vocational education program, not only because of themanpower demands and needs alluded to, but for at.least twoadditional reasons. It has frequently been observed thatthere is'no national vocational education program in theUnited States. 'We have 50 different state programs eachwith its own supposdd needs, resources, goals, priorities,standards, definitions, data requirements, and deliverysystems. In light of these differences, are there enoughcommonalities among state programs to discern a vocationaleducation program that is reasonably "typical" for thenation? The answer would seem to be "Yes." Despite some

22Vocational education is defined in EncyclopediaBrittanica as " instruction that is intended to fit .

persons for industrial.or commercial occupations" (Vol. 23,p. 95). The Dictionary of Education (edited by Carter V. Good)defines vocational education as program of edtwation belowcollege grade organized to prepare thejearner for entranceinto a particular chosen vocation.or to upgrade eamloyed work-ers; . . . training or retraining is given in schools orclasses under public supervision and control or under contractwith a state bbard or local education agency" (p. 645). InOhio State University's Proposal for a National Center for Re-search in Vocational Education, a distinction is noted between"Vocational Education (capitalized), the program supportedunder federal legislation; and vocational education in a gen-eric sense . . [which] embraces all activities, public andprivate, which are designed to contribute directly to occupa-tional proficiency" (Vol. I, pp. lf). Examples of such activ-ities cited in the proposal include military training, Compre-hensive Employment and Training Act (CETA) programs, public andprivate school vocational programs, and apprenticeships.

23Mary L. Ellis, A Report to the Nation on VocationalEducation (college Park, Maryland: Ellis Associates, 1975),p. 2.

25

334

interstate diffeiences in Student Characteristics (nl inFigure 2 above) and Context (n,) it seems reasonable to thiswriter that nationally shared Coals (nA) and the existenceof national markets for Resources (n,rand.Processes (n,)would combine to establish vocational education programd inthe fifty states that are in essence quite similar. If thisis in fact true, some important outcomes should be observableas a national pattern.

The other respect in which diversity might underminethe plausibility of treating vocational education as anent.ity relates to differences among particular occupationalprograms. DoeeDistributive tducation, for example, haveless in common with Vocational Agriculture than it does withthe academic/general school curriculum? Are inter-programdifferences in goals, methods, personnel, and 'philosophy sovast that the occupational curricula share little other thanthe "vocational" label? Is it true, as some have claimed,that "There are no vocational educators,- only Vo Ag, Home Ec,and T & I educators"? Operating on the premise that,themore specialized and unique a program is, the easier it isto demonstrate differential outcomes, a case can be made forevaluating highly specific occupational curricula (such asAir Conditioning, Automotive Ser'ices, Diesel Mechanics)rather than clusters (such as Trade and Industrial Occupa-tions). However promising this "component-program"approach 'to evaluation might appear, it would neverthelesspose methodological problems of its own when trying toaggregate results for.the purpose of producing generaliza-tions about "the outcomes ,3f vocational education."

B. Vocational Programs as Educational Treatments

Answering the question, What is vocational education?is more than an "academic" exercise. If evaluation entailsboth description and appraisal as suggested in Section IIabove, a necessary first step for evaluation is to identifythe entity to be described. What are the essential featuresor the distinctive educational treatment or intervention be-ing evaluated? Figure 5 depicts the relationship between"input" 'variables and educational outcomet4 in a manner thatdiffers somewhat from the production function of Figure 2.Intervention (or program treatment) is defined in Figure 5to include Resources (a1), Goals (n4), and Processes (NY;while Students (ni) and Context (n2T arc dcpicted separately.This formulation has the dual advantage of isolating the ,two"input" variables that are less amenable to variation whilealso highlighting the fact that outcomes may be observedwith respect to either (a) individual students or (b) socialcontext. The linkage on the left side of the schematic in--dicates continuous mutual interaction between Students and.

26

DETERMINANTS OF VOCATIONAL EDUCATION OUTCOMES

VOCATIONAL EDUCATION OUTCOMES"FON :AIDIVIDUAL STUDENTS tFOR SOCIETY

"InterVention or educational treatment is narrowly defined here to encompassResources (n3), Goals (n4), and Processes (n3). (See Figure 2 above.)

"Observe/ 'a effacts of distinctive educational treatments (subject to theinfluence os extraneous forces and error factors).

Figure 5

36

Context. Using this framework, we can build on the dis-cus n of program definition in the preceding section.

The focus there was on interstate differences in voca-tional education programs and possibili ies of disaggrega-tion based on occupational clusters or s ecific programcurricula. We now want to approach the u it-o,f-analysisissue from the broad perspective of the tot /educationalprogram. The question may be posed: Is it appropriate forevaluation purposes to consider vocational education, howeverthat entity may be defined, as a treatment distinctfrom the total educational program (especially the occupa-tional skills component)?.

In discussitg.the issue of criterion variables forvocational educatione.one observer, points to the "separationof vocational and general education" as a major source ofdifficulty. "As long as vocational education.claims to bethe sole input for the knowledge, skills, and attitudes re-quired for occupational competence," he argues, "it will bevocational education, not the total school program, thattakes the brunt of criticism that schools are not accoMplish-ing satisfactonily their functions of occupational . . .

preparation."`"' In other words, If outcome criteria such asoccupational proficiency, employability, actual employment,work adjustment, "satisfactoriness" to employers, and asuccessful transition from school to work are to be used forevaluating program performance, should the treatment that isbeing evaluated be (a) the total school program, or (b) thesubset of courses and experiences labeled vocational educa-4tion?

Whether vocational education does indeed claim to bethe sole source of the components of occupational competency--or has such a function ascribed to it--is a moot point. Thepractical question for vocational education evaluation iswhether (a) or (b) is regarded as the treatment. If theformer, then "evaluation of the school's occupational educa-tion function (for all of the school's students] becomes acomplicated ana methodologically sophisticated affair . .

11124Warmbrod, "Evaluation Research in Vocational

Education," pp. 9f.

28

something of the magnitude of the Coleman Report."25 It mightbe conjectured that'this strategy would also tend to removethe focused heat of educational accountability fFom vocation-al education.

Yet a legitimate question remains. So long as schoolsoffer a course or set of courses,and experiences under therubric of vocational education--and with earmarked funding--there will be a reasonable expectation for an educationalquid pro quo. What comes out of the vocational educationprocess in response to the specialized resources that gointo it? Perhaps the outcomes that can reasonably be ex-pected will be much more limited than occupational competence,employability, employment, successful work adjustment, andthe like. Some differential outcomes, however, should pre-sumably be identifiable as resulting from distinctive goalsjprocesses, and resources.

C. Evaluative Criteria and Standards, Ex erimental Controlsand Compar son Groups

According to one of the definitions listed in Section Ifevaluation is the process of comparing performance data withclearly specified objectives. This conception of evaluationis associated with the currently popular view that the soleor dominant purpose of evaluation is program 4Tprovement:"Evaluation is not to prove, but to improve." The approachhelps explain why many people equate a study of vocationaleducation outcomes with an investigation of goals and theextent to which they have or have not been achieved. Clearly,

25Warmbrod argues (on p. 9 of his Beacon article):"When vocational education is viewed as a function of thetotal school program, some important issues relevant to thedesign and conduct of evaluation studies come into focus."Fitst, the tarriet population becomes all of the school'sstudents, not just vocational students. Second, the treat-ments to be evaluated are various mixes of vocational andother courses. And third, "an evaluation of the school'soccupational education goal becomes a part of the evaluationof all the goals and functions of the schools."

26Floyd L. McKinney, Program Evaluation in Vocational

Education: A Review (Columbus: The ERIC Clearinghouse oncareer Education, center for Vocational Education, Ohiostate university, 1977) p. 2.

29

hom.lver, the observable effects of.vocational programsmay differ from the preannounced intent of such programs notonly in terms of discrepancies between stated outcome goalsand actual achievement, but also with regard to such sideeffects as (a) benefits not previously identified at goalsand (b) other consequences that might be considered neutralor negative. In short, evaluation generically defined canserve other functions besides program improvement; andevaluative criteria can be derived from sources other thanpreannounced objectives.

It was noted in the Introduction that one function of re-search is to describe the consequences of given actions, suchas the observable outcomes of particular educational programs.Positive outcomes analysis, as defined above, addresses thefull range of educational outcomes, not just traditional ob-Tgatives and presumed benefits. This breadth of scope canproduce insights which in turn may generate new goals andpriorities. In this way, knowledge of what is can contributeto informed judgments of what ought to be. yr a stated cb-jective of vocational education is preparation for employmentin a training-related occupation but an observed and perhapsunanticipated outcome is employability in a much broaderrange of occupations, with corresponding laNpr market bene-fits for individual students, .then the latter goal mightappropriately be added to or substituted for the narrower one.

As we earlier emphasized, in moving from positiveanalysis to evaluation, there is a need for evaluativestandards. Some of the standards may be goal-based; butlacking a comprehensive, omniscient statement of program ob-jectives, wliere does the full set of evaluation standardscome from? One source is professional judgment as to what

27It has been noted that an area "not well addressed

in writings on the evaluation process is the establishment.ofstandards and the relative utility, of various types ofstandards" (Quentin D. Clarkson and others, "A System forEstablishing Evaluation Standards," CEDaR Quarterly, Vol. 10,No. 1, Spring 1977, pp. 17-19). In their proposed classifica-tion system, the authors distinguish four types of relativestandards (Normative, Predicted, Control, Trend) and twotypes of absolute criteria (Arbitrary or intuitive, and Value-based). Moss and Stromsdotfer (."Evaluating Vocational andTechnical Education Programs," p. 224) have observ.ed that "the'problem of evaluating vocational education is confounded bythe fact that its.objectives and outputs are multidimensional. . . [and] successful development, weighting, and estimationof performancej.ndexes . . . has never been achieved."

30

38

S.

constitutes reasonable and ippropriate performance. Agree-ment might be reached (by a panel of evaluators or theirsponsors?) that an unemployment rate under 10% constitutesan acceptable programmatic outcome. Or'a flexible norm couldbe established for vocational graduates pegged at say 75% ofor 110% of the current unemployment rate for all labor forceparticipants in the same age group. .

Evaluative standards can be derived from observing theoutcomes of similar program activities, e.g., job-searchtime required for CETA graduates to find unsubsidized jobs.compared with job-search time for vocational school gradu-ates, or differences in outcomes resulting from the sameprogram observed over a period of years.

And, in certain instances, verbal absolutes such as"Satisfactory," "high.," "low," "good," "bad," can be positedas nonempirical standards. These seem less likely to proveuseful/and credible than criteria based on empirical ob-servat ons (whether quantitative or qualitative).

erhaps the most promising source of evaluation stand-ards ijs needs assessment, which recognizes that someone'sdesid rata or "justified goals"--though not necessarily thespons r's or producer's--aregppropriate criteria for eval-uatin educational programs.

Once again, definitional problems intrude on the issueof e aluative criteria. Who are "vocational.students," andwho re the "nonvocational students" with whom status orPerf rmance might usefully be compared? A host of related

comp risOn groups. Definitions aside, is it appropriate toissur arise, including the use of experimental controls and

compare employment status and earnings of former vocationalstudents with nonvocational (i.e., academic and general)students? Should comparisons of labor market status andperformance (e.g., finding employment in occupations relatedto training) only be made among specific occupationalcurricula within vocational education, or can such comparisonsbe made legitimately, between the totality of vocational stu-dents on one hand versus all other students? If the latter,whichqparticular entry-level jobs are "training-related" forgraduates of the high school academic and general curricula?

Consider the mandate under the Education Amendments of1136 to evaluate vocational programs "according to the extentto which program completers and leavers (a) find employment

28Scriven, "Evaluation Perspectives and Procedures," p. 51.

31

ai9

in occupations related to their training, and (b) are con-sidered by their employers to be well-trained and preparedfor employment." What are the procedural options for carry-ing out the evaluation? As an illustration, without exten-sive discussion, suppose the strategy is limited to identify-ing the former vocational students who have found training-related jobs (however defined) and then surveying their em-ployers with given instruments. Assume that 60% of "programcompleters and leavers" found jobs in occupations related totheir training and within this group, 80% were rated highlysatisfactory with respect to their training and preparationfoi employment. How are the data to be evaluated? As notedin Section II, whether outcomes are congidered good or baddepends on the comparison standards ertTloyed.29

It was suggested above that in the absence of clearlydefined performance objectives or predetermined evaluationstandards, it will be necessary to derive or develop suchstandards. One approach is to compare the status or per-formance of vocational students with another group, similarin all important respects--age, sex, mental ability, socio-economic.baCkground, etc.--except that the vocational studentsreceived a distinctive educational treatment that was notprovided to the comparison group. In the case of secondaryschool students, the comparison might be made for vocationalstudents versus nonvocational (i.e., academic plus general)who are similar in all characteristics except the treatmentvariable (participation in a vocational program). Given sucha comparison, is it reasonable to hypothesize significantdifferences in the outcome variable for the respective groups?

The general rationale for the hypothesis is that sub-stantially different educational treatments can be expectedto yield significantly different outcomes. Whether the dif-ferences between "vocational education" and the "regulartreatment" can be expected to result in employabilitydifferences is a more specific and problematic question. Ofthe two outcome variables mentioned in the Education Amend-ments of 1976, placement in a training-related job

.29It might be argued that interpretation of the facts and

evaluations can be left to others--managers, policy makers,interested parties--who can furnish their own judgmentalcriteria. The question of who does the evaluating, however,does not diminish the importance of how it is done. Proceduresand value premises will be of concern to all groups having astake in the evaluation, and whoever supplies the values willpresumably be held accountable. Dogmatic standards and vote-counting may seem expedient, but that leaves important valuequestions unanswered.

32

is a highly vestionabie.evaluation criterion forat least three reasons: (a) "relatedness" is ambiguous;(b) vocational students who were enrolled in a particularcurriculum may not, desire eMployment in a related occupation(either because their.preferred option was not offered andthey simply took something that wwavailable, or becausethe option they followed provided a-useful opportunity toexplore and reject a particular occupational fiele;;(c) there are no appropriate cdmparison groupd fG.t.whomdifferentiated- fields of training can be identified.

On the other hand, the satisfactoriness-to-employerscriterion .could.,be applied in the-following way. One couldenumerate the former nonvocational students in a given schoolclass who found jobs (irrespective .of occupational fields),draw a sample matched to the vocational Students* according tocharacteristics Such as mental ability and socioeconomicbackground, and then compare employer assessments of voca-tional and nonvocatiOnal students relative to the extent towhich they are considered "well-trained and prepared foremployment."

If the intent were to test the efficacy of theth

difr '

tinctive vocational education treatment.wi respect temployability (as measured by eniployer assessment of trainingand preparation), it would seem plausible to expect lowerrates of unemployment for students exposed to an employment-related vocational education treatment as Compared withstudents who were not.3° If no .significant differences occur

30Such expectations are plausible for the same reason

one would expect a'student who completed three years ofFrenett to enjoy- an-adv ca... .Langaayeover a comparable student who had spent three years takinghistory instead. A separate question relates to the amountof the differential in unemployment rates. Is achievIE4-aedifferential worth whateVer extra costs, if any, have beenincurred in providing the vocational education program?Efficiency issues involving calculations of marginal benefitsand marginal costs are not addressed in this paper, for tworeasons. First, until we know more about gross outcomevariables, it seems pointless if not pernicious to go throughthe exercise of nanipulating the finely tuned pecuniary valuesneeded to generate benefit-cost estimates. Second, there isso much methodological-discretion inherent in performingbenefit7cost, rate-of-return, and cost-effectiveness analysesthat the findings may reflect the tastes and style of the in-vestigator more faithfully than the nature of the empiricalevidence.

33

11

between the matched groups, or if the regular group performedbetter than the treatment group, this certainly would notdemonstrate that vocational education was an abysmal failure.It would suggest, however, that success criteria might besought elsewhere; perhaps vocational programs do not enhanceemployability differentially but have other worthwhile outcomes.Alternative tests can presumably be designed to measure these out-comes, thereby helping to disclose the real nature of the voca-tional education outcomes function, i.e., the matrix of conse-quences associated with this distinctive educational treatment.

D. Nature and Quality of Outcome Data

3 Numerous studies have commented on the availability andinterpretability of vocational education data. Evaluatorshave lamented the lack of national data, accurate data,comparable data, "hard" data, cost data, and impact data.."Our own review of empirical outcome studies and relatedinquiries concerning the credibility and persuasiveness of

,outcome data suggest the need for a profile of data typesclassified on the basis of the nature of the data and howthey are generated. Table II distinguishes five types ofdata used in describing and appraising the consequences ofvocational education.

A

31Among the data-related problems identified by McKinney,uray, ono-Abram ( In terpretinsSurcame- -Measures p. 1) , arethose of "definitions, commuffraation, and problems of handlingdata." In the same report (Ch. III), Donald W. Dreweb commentson data format and other issues hssociated with setting up

/ the proposed Vocational Education Data System (VEDS). Foradditional perspective on data problems, see Arthur M. Lee"Project Baseline: Historiographic Foundatfons for VocationalEducation Statistics, "Educational Researcher, Vol. 6, No. 7(July-August 1977), pp.-7.757-747TTOTERTETri footnote 2above, two state-of the-art papers dealing with vocationaleducation data are scheduled for publication by TheNational Center in 1979. See Boiland, Evaluative Bibliographyfor comments on data used in some recent studies ofovocationaleducation outcomes.

34

4 2

__......_ ..........._

Table II

TYPES OF DATA USED IN VOCATIONAL EDUCATICN OUTCOMES EVALUATION

Type of DataNature ofObservation Method of Observation

*

Examples

Status(Nonmonetary)

Objective Observation and reporting ofcircumstance by a second par-ty or directly by respondent.

._ .

Labor force status of former voca...tional students.

School enrollment status of stu-dents currently or iormerlyenrolled in vocational programs. ,

"oltatus.--

! (Monetary)

.

!

:

Objective

,

Observation by a second partyof circumstances reportablein money terms. ,

Hourly wages of former vocational. and nonvocational students.Annual earnings of former students.Cost and benefits of vocationalprograms.

iPersonal Judg-

1

mentSubjective Expressed directly by

respondent.

.

Student opinions or perceptionsconcerning quality and relevanceof vocational programs.Employer assessment of the extentto which students are well trainedand prepared for employment.r.

Ascriptive

- .-

Objective Characterization by a secondparty based on records, testresults, and/or observationand professional assessment.

Work-related attitudes and attri-butes of students.Vocational interests and aspire-tions of students.

Performance

.

_

Objective Written or other tests of cog-nitive and psychomotor know-ledge and skills.

Score on skill proficiency test(e.g., welding).

Score on reading/math achievementtest.

4

The literature review to which we allude disclosedheavier use of Status (Nonmonetary) and Personal Judgmentdata than the other types. In part, this reflects concernwith the outcomes specified as evaluation criteria in theEducation Amendments of 1976, namely "the extent to which'vocational program completers and leavers . . . find em-ployment in occupations related to their training, and .

are considered by their employers to be well-trained andprepared for employment" (Sec. 112). Certain kinds of dataare clearly more appropriate than other types as empiricalindicators of particular outcomes (see Figure 4 above). Infact, outcome hypotheses can be formulated so explicitlythat given types of data and particular evaluation standardsare specified for use in performing the evaluation task.

Additional factors.influencing the kinds of outcomedata used in evaluation include ease, timing, and costs ofcollection; privacy considerations; credibility; comparabilitywith other available data; amenability to statistical treatmentand analysis; potential impact; and the ingenuity and skillof evaluation designers and data gatherers. Inevitably, thereare tradeoffs between-such considerations as the ease andeconomy of collecting softer data of the Personal Judgmenttype and generating "hard" Monetary data, or Performance data,which might have higher impact on certain audiences.

IV. SUMMARY AND RECOMMENDATIONS

As an "instrumental" contribution to answeriag theresearch question "What particular outcomes are appropriatefor use as criteria to evaluate vocational education pro-grams?" this paper has attempted to identify and clarifysome key concepts and issues related to educational outcomes.Three additional products were completed during'the firstyear of the National Center's study of vocational educationoutcomes and are recommended as companion references: (1)Vocational Education Outcomes: An Evaluative Bibliography ofEmpirical Studies;' (2) Vocational E ucat on Outcomes: AThesaurus of Questions; (3) Vocational Education Outcomes:Annotated Bibliography of Related Literature. It is hopedthat these products will-help stimuTiEi-Vilaer conceptual andempirical work in the field and prove useful as points ofreference for future commentary and reporting.

36

4 4

A. Summary

Vocational education outcomes were defined in thispaper as "the consequences of vocational' programs." Figure 3depicted the range and diversity of possible outcomes asso-ciated with high school, post-secondary, and adult vocationaleducation programs. ..It was emphasized that educational out-comes are not limited simply to the goals, objectives, andbenefits of vocational programs but also include unintendedconsequences ("side effects") which may be evaluated inpositive or negative terms. Expended awareness cf the rangeand diversity of possible outcomes can.help offset narrrowperceptions that,inhibit*research, evaluation, program im-plementation, and policy making. Table I listed 30 vocation-al education outcome questions, illustrating such diversethemes as the acquisition of knowledge and occupationalskills, economid development, employment, earnings, leader-ship development, student and parental satisfaction withschooling, educational and occupational discrimination, jobsatisfaction, personal self-image, occupational mobility,work-related attitudea, career planang4 job-search skills,employment opportunities for women, and economic returns toschooling. Some of the outcomes identified in these ques- .

tions apply to individual students while others are morebroadly societal in nature.

An analytical framework for evaluation was suggested inFigure 2 based on an educational production function and sim-ilar to the familiar CIPP model (Context, Input, Process, andProduct). Figure I identified six components of the educationenterprise: Student Characteristics, Context, Resources, Goals,Processes, and Outcomes. It was suggested that vocationaleducation can be evaluated not only on the basis of what comesout of the system--educational products, outputs, effects,riFact--but also in terms of what goes into the enterprise andhow it functions. While outcomes are of great importance, itcannot be claimed that this is the only category of criteriarelevant for appraising the merit of vocational education.

, And within the outcomes category, one must resist the temp-tation to look only at one or two particular outcomes whileignoring all the other consequences of vocational programs.In order to evaluate vocational education in a balanced andmeaningful way, evaluators should avoid putting all of theireggs in one narrow basket. Figures 1 and 2 were intended toprovide perspective and focus attention on outcomes.

Figure 4 distinguished among educational outcomes(changes in empirical phenomena), outcome statements (verbalexpressions), and outcome measures (observable indicators). Thismethodological framework recognized the need for evaluation

37

15

standards in order to carry. systematic inquiry.lieyond descrip-tion and research into the sphere of evaluation proper. Figure4 also listed four potential areas in which evaluation find-ings might be applied--policy, program design, program manage-ment, and accountability--whether for program improvement orother purposes. The importance of formulating outcome state-ments explicitly and unambiguously was emphasized both forclarity of thinking and communication. An outcome statement orquestion typically identifies an educational treatment, affect-ed entity, and specific outcome as well aa the agency throughwhich the treatment is delivered. An outcome statement mayalso identify a time framework and rationale for the stated

4outcome.

Among the key issues identified in the area of outcomesevaluation were the definition of vocational edu:ation, thespecification of vocational programs as educational treat-ments (see Figure 5), sources and uses of evaluative cri-teria, experimental controls, and the use of comparisongroups in evaluating vocational education programs withrespect to specified outcomes. Tab2e II distinguished fivetypes of data used in outcomes evaluation, classifiedaccording to the nature of the observation and methods ofdata collection. A review of outcome studies disclosedheavier use of Status (Nonmonetary) and Personal Judgmentdata as opposed to Monetary, Ascriptive, and Performancedata, with corresponding implications for generalizability,credibility, and impact on potential adudiences.

o

B. Suggested Agenda for Improving OutcoMes Evaluation

Along with others, we find that empirical research hasnot documented a substantial body of knowledge concerning

- of vocational education. Additional evaluationstudies which repeat shortcomings of design and methodologythat have been noted will not add significantly to ouraccumulation of knowledge. Some new initiatives are necessaryto improve the art of outcomes evaluation.

we therefore recommend the formulation cf a research,development, and evaluation (RD&E) agenda, aimed at pro-ducing clear, accurate, relevant, credible, and communi-cable findings. Salient features of such an agenda wouldbe conceptual and empirical answers to such questions as:

(1) What is vocational education?

(2) Who are vocational students?.

38

(3) What are the outcome goals, of vocationaleducation?

(4) What outcopes should be used as criteria.to.. evaluate vocational education, and Why?52

(5) -What data are needed to describe and-assess the.effects of vocational education from the nationalai well as state and local perspectives, and hewcan such data-best be collected and analyzed?

(6) What standards should be applied to particularoutcome measures for purposes of evaluation, andhow should.they be determined?

(7) What specific "input" variables (under-the head-ings of Student Characteristics, Educational andSocial Context, Resources, Goals, and Processes)in the vocationa,/ education enterprise.are signif-icantly associated with particular Outcomes, and.how are they functionally related?

(8) To what extent do extraneous forces and errorfactors inhibit the production oCknowledgeconcerning "the correlates" of particular voca-tional education outcomes?

(9) What evaluation methods and procedures should beused to produce accurate and credible generalize-.tions at the national level regarding the outcomesof vocational education?

(10) How can evaluation findin s on vocational educa-tion 'outcomes best e communicated to relevantaudiences, including policy makers, practitioners,and other evaluators to assure that such findingswill be apRlied in the interests of improved edu-cational ettectiveness and efficiency?

32This is the research question currently being ad-dressed in the study to which this essay contributes. Wefeel that in its first year, this multi-year project pro-gressed about 35%-45% of the way toward developing somemeaningful and useful answers to the question. During YearTwo (February 1979 to February 1980), attention will befocused on approximately six to eight specific outcomes--designing operational procedures and determining the extentto which the selected outcomes and procedures are regardedby relevant audiences as appropriate for use in evaluatingvocational programs.

39

17

The conceptual and empirical answers prescribed for thesequestions must be meaningful and well documented so they

. can serve as a basis for developing authoritative, widelyshared definitions and standards. Where.value criteria arelacking and where empitical evidence is incomplete andconceptualizationsf imprecise, deficiencies should be openiyacknowledged and a cogent rationale provided to jnstifywhatever course of action is adopted as an expedient inlight of the specified ritional-empirical-normative. gaps.

Action on the recommended REME agenda requires a will-ingness to acknowledge--with neither defensiveness norresignation--existing.shortcomings as well as the strengthsof vocational education, both with respect to programperformance and the knowledge we possess about the system'scomponents and impact. And the !high tolerance.for ambig-uity" that is sometimes counseled for vocational educationresearchers and evaluators needs to be challenged in theinterests of better evaluation and vocational education'scommitment to high quality and responsiveness to the needs"of individuals .and society. .

REFERENCES

Anderson, Scarvia B.;.Ball, Samuel; and Murphy, Richard T.; andassociates. Encyclopedia of Educational Evaluation, SanFrancisco: Jossey-Bass, Inc., 1975.

Averch, Harvey A. and others. How Effective Is Schoolirtg?Critical Review and Synthesis ,of Research Findings. SantaMonica, California: Rand Corporation, iftrch 1972.

Becker, William E., Jr. "Investment in Human Capital," pp, 59-76 in Vocational Education and the Nation's Economy, editedby Warren e. Meyer. Waihington: American Vocational Associ-ation, 1977,

Bolland, Kathleen A. Vocational Education Outcomes: An Evalu-ative.Bibliography of Outcome Studies. Columbus: National,.Center for Research in Vocational Education, The Ohio StateUniversity, 1979.

Clarkson, Quentin D.; Neuburger, Wayne F.; and Koroloff Nancy."A System for Establishing Evaluation Standards," CEDaRQuarterly Vol. 10, No. 1 (Spring 1977) pp. 17-19.

Committee on Vocational Education Research and Development (COVERD).Assessing Vocational Education Research and Development,Washington: National Academy of Sciences, 1976.

Darcy, Robertl. "The Nature of Economic Enterprise," p7. 109-129 in Y22At1211.41_214.0.1111222_9111.111MSE_RM12211212t, edited byEdwin L.1-ari.;----1-3-oiatoit:lb

Darcy, Robert L.; polland, Kathleen A.; and Farley, Joanne.Vocational Education Outcomes ('ina1 ReRort on Year One ofThe R&D Protect "Examinlng Vocational Education Outcomesand Their Correlates"). ColuMbus: National Center forResearch In Vocational Education, The Ohio State University,1979.

Ellis, Mary L. A Report to the Nation on Vocational Education.College Park, Maryland: Ellis Associates, 1975.

Encyclopedia Brittanica (1973).

Farley, Joanne. Vocational Education Outcomes; A Thesaurus ofOutcome Questions. Columbus: National Center for Researchin Vocational Education, The Ohio State University, 1979.-

Franchak, Stephen J., and Weiskott, Janet. Evaluation Handbook:Guidelines and Practices for Follow-up SEMiiii-of FormerVocational aucation Students. Columbus: National Center

41

1 9

for Research in Vocational Education, The Ohio State Univer-sity* September 1978.

Friedman, Milton. Essays in Positive Economics. Chicago:, University of Chicago press, 1953.

Good, Carter V., ed. Dictionary of *Education. 3d rev. ed. NewYork: McGraw-Hill, 1973. ,

Hansen, Joe B. "The Formative-Summative Dichotomy and the Roleof Description in Program Evaluation." Evaluation News,No. 6 (August 1978) pp. 14-16.

Kauffman, Richard V. A Study of the Educational ProductionFunction. oPh.D. thesis, Department of Economics, ColoradoState University, 1974. Reprinted (in hard copy), Ann-Arbor:University Microfilms, 1978.

Lecht, Leonard A. Evaluating Vocational Education: Policies andPlans.for the 1970s. New York: 'Praeger, 1974.

Lee, Arthur M. "Project Baseline: Historiographic Foundationsfor Vocational Education Statistics." Educational ResearcherVol. 6, No. 7 (JU1y-4ugust 1977) pp. 3-9.

McKinney, Floyd L. +Program Evaluation in Vocational Education:A Review. Columbus: ERIC Clearinghouse on Career Education,5131747-Tor Vocational Education, The Ohio' State University,1977.

McKinney, Floyd L.; Gray, Kenney E.; and Abram, Marie. Inter-preting Outcome Measures in Vocational Education: A-FailReport. Columbus: National Center for Research in VocationalEducation, The Ohio State University, 1978.

Moss, Jerome, Jr. The Evaluation of Occupational EducationPrograms. Technical Report. Minneapolis: Research Coordih-ation Unit in Occupational Education, University of Minnesota,September 1968.

Moss, Jerome, Jr. and Stromsdorfer, Ernst W. "Evaluating Voca-tional'and Technical Education Programs" in VocationalEducation: Today and Tomorrow, edited by Gerald G. Somersand J. Kenneth L4t1p. Madison: Center for Studies in Voca-tional and Technical Education, University of Wisconsin*,1971.

Reubens, Beatrice G., "Vocational Education: Performance andPotential." Manpower, July 1974.

42

;50

Rogers, Daniel C. and Ruchlin, Hirsh S. Economics and Education:Erircons. New York: Free 'press, 1971.

Scriven, Michaele "Evaluation Perspectives and Procedures,". pp. 3-93 in Evaluation Education, 9dited by W.. James

N.Popham. Berkeley: McCutchan, 1974. ;

\SlitIon, Herbert A. andLimitations," pp.

\ edited by Leonard\press, 1955.

Newell, Allen. "Models: Thier Uses and66-83 in The State of the Social Sciences,D. White. Chicago: University of Chicago

Stuffiebeam, Daniel L. and otheis. Educational Evaluation andDec;sion Making. Itasca, Illinoii: Peacock, 1971.

U. S. Pubiic Law 94-482, Education Amendments of 1976.

U. S. Depaitment of.Labqr. The Noneconomic Impacts of the JobCorps, R,f D Monograph 64, Employment and Ttaining Admini-st'ration.-, Washington, D.C.: U. S. Government PrintingOffice, 1918.

\,

Warmbrod, J. Robert. "Evaluation Research in Vocational Educa-. tion." Paper presented at the American.Vocational Education

Research Association's annual meeting in Houston, Texas,December 6, 1976. Reprinted in Beacon, Vol. 6, No. 1

, (January 1977) pp. 1, 6-11.

Worthen, Blaine R. and Sanders, James R. Educational Evaluation:Theory and Practice. Worthington, ohim---inmEnrimmr,7----1973.

PUBLICATIONS RELATED TO THE.PROJECT"EXAMINING VOCATIONAL EDUCATION OUTCOMES"

* * *

Darcy, Robert L.; Bolland, Kathleen A.; and Farley, Joanne.Vocational Education Outcomes (Final Report on Year One of

e R & D-Project wExamining Vocational Education Outcomesand Their Correlates"). Coludbus: National Center forResearch.in Vocational Education, The Ohio State Univers-sity, January 1979.

* * *

Bolland, Kathleen A. Vocational Education Outcomes: An Evalu-ative Bibliography of Empirccal Studies. Columbus: NationsCenter for Research in VocatIonal Education, The Ohio StateUniversity, 1979 (forthcoMing).

Darcy, Robert L. Vocational Education Outcomes: Perspectivefor Evaluation. ColuMbus: National Center for Researchin vocational Education, The Ohio State University, 1979:

Farley, Joanne. Vocational Education Outcomes: AThesaurus ofOutcome Nestions. Columbus: National Center for Researchin Vocational Education, The Ohio State University, 1979(forthcoming).

Taylor, Carolyn M.; Darcy, Robert L.; and Bolland, Kathleen A.Vocational Education Outcomes: Annotated Bibliography ofRelated Literature. Columbus: National Center forResearch in Vocational Education, The Ohio State Univer-,sity, 1979 (forthcoming).

For Further Information

For information concerning the above publicationsand the vocational education outcomes project,please contact:

Program Information OfficeThe National Center for Research

in Vocational EducationThe Ohio State University1960 Kenny RoadColumbus, Ohio 43210(614) 486-3655

51