ED 327 738

AUTHORTITLE

INSTITUTION

SPONS AGENCY

PUB DATECONTRACTNOTEAVAILABLE FROM

PUB TYPE

EDRS PRICEDESCRIPTORS

ABSTRACT

DOCUMENT RESUME

CE 057 134

McCaslin, N. L.A Framework for Evaluating Local Vocational EducationPrograms. Information Series No. 344.ERIC Clearinghouse on Adult, Career, and VocationalEducation, Columbus, Ohio.Office of Educational Research and Improvement (ED),Washington, DC.90RI8806200537p.

Publications, Center on Education and Training forEmployment, 1900 Kenny Road, Columbus, OH 43210-1090(order no. IN344: $4.75).Information Analyses - ERIC Information AnalysisProducts (071)

MF01/PCO2 Plus Postage.Accountability; Data Collection; Educational Change;*Evaluation Criteria; Information Needs; InformationSources; *Outcomes of Education; PostsecondaryEducation; *Program Evaluation; Program Improvement;Secondary Education; *Vocational Education

To help vocational education administrators andpractitioners, this paper presents a framework for vocationaleducation program evaluation. Evaluation is defined as the act ofrend ,_,. ng judgments to determine a program's value; two major typesare formative and summative. Vocational education evaluation shouldtake a broader and more comprehensive approach than a focus onstudent achievement. This approach requires use of valid and reliableinformation of three interrelated types: needs, processes, andoutcomes. The needs for vocational education include students'interests, knowledge, attitudes, and skills; the knowledge,attitudes, and skills employers want in employees; and theeducatiohal, economic, and social needs of society. Sources ofinformation about vocational education program processes include: (1)organizational information; (2) program information; (3) supportservices and activities; (4) staff; (5) student demographiccharacteristics and educational achievement; and (6) communityinformation. Also part of the evaluation framework are economicoutcomes (labor force participation, employment and unemploymentrates, training-related placement, type of employment, earnings,employee satisfaction with work, employer satisfaction withemployees); educational outcomes (academic and higher-order thinkingskills, knowledge of the world of work, occupational skills, schoolattendance and dropout rates, continuing education rates,satisfaction with education); and psychosocial outcomes (aspirations,attitudes, values, self-esteem, citizenship, and leadership). (28references) (SK)

Information Series No. 344

A FRAMEWORK FOR EVALUATING LOCALVOCATIONAL EDUCATION PROGRAMS

N. L McCaslinDepartment of Agricultural Education

The Ohio State University

U S DEPARTMENT OF EDUCATIONOffice of Educational RInearch and Improvement

EDUCATIONAL RESOURCES INFORMATIONCENTER (ER1C)

f/This doCument has been reproduced asreceived from the person Of Organilahonofiginaffrig It

C Minor Changes have been made to improvereproduction Quality

Points of view or opinions stated in this clocu,ment do not necessarily represent officialOE RI Dosit,on or policy

ERIC Clearinghouse on Adult, Career, and Vocational EducationCenter on Education and Training for Employment

The Ohio State University1900 Kenny Road

Columbus, Ohio 43210-1090

1990

2

FUNDING INFORMATION

Project Title: ERIC Clearinghouse on Adult, Career, and VocationalEducation

Contract Number: RI88062005

Act under WhichAdministered: 41 USC 252 (15) and P.L. 92-318

Source of Contract: Office of Educational Research and ImprovementU.S. Department of EducationWashington, DC 20208

Contractor: Center on Education and Training for EmploymentThe Ohio State UniversityColumbus, Ohio 43210-1090

Executive Director: Ray D. Ryan

Disclaimer: This publication was prepared pursuant to a contract with theOffice of Educational Research and Improvement, U.S. Depart-ment of Education. Contractors undertalag such projectsunder guvernment sponsorship are encouraged to express freelytheir judgment in professional and technical matters. Points ofview or opinions do not, therefore, necessarily represent officialU.S. Department of Education position or policy.

Discrimination Title VI of the Civil Rights Act of 1964 states: "No person inProhibited: the United States shall, on the grounds of race, color, or nation-

al origin, be excluded from participation in, be denied thebenefits of, or be subjected to discrimination under any programor activity receiving federal financial assistance." Title IX ofthe Education Amendments of 1971 states: "No person in theUnitea Siates shall, on the basis of sex, be excluded from par-ticipation in, be denied the benefits of, or be subjected to dis-crimination under any education program or activity receivingfederal financial assistance." The ERIC Clearinghouse project,like every program or activity receiving financial assistance fromthe U.S. Department of Education, must be operated in compli-ance with these laws.

ii

0o

TABLE OF CONTENTS

LIST OF FIGURES

FOR EWORD vii

EXECUTIVE SUMMARY ix

INTRODUCTION

EVALUATION IN VOCATIONAL EDUCATION 3

INFORMATION NEEDED FOR VOCATIONAL EDUCATIONEVALUATION 7

Needs for Vocational Education 8Vocational Education Processes 8Vocational Education Outcomes 8

A FRAMEWORK FOR EVALUATING VOCATIONAL EDUCATION . . . 9

The Needs of Students, Employers, and Society 9Vocational Education Processes 12Vocational Education Outcomes 18

USING THE EVALUATION FRAMEWORK 23

Determine the Type, Purpose, and Extent of theEvaluation 23

Select the Components to Be Included 24Identify the Sources and Types of Information to Be

Collected 24Collect the I? equired Information 25Analyze and Interpret the Information Collected 26Prepare an Evaluation Report 26Concluding Remark 26

REFERENCES 27

LIST OF FIGURES

Figure

1. Interrelationships of evaluation information

2. A framework for evaluating vocational education

3. A sample listing of potential sources ofevaluation information

FOREWORD

The Educational Resources Information Center Clearinghouse on Adult, Career, and Voca-tional Education (ERIC/ACVE) is 1 of 16 clearinghouses in a national information systemthat is funded by the Office of Educational Research and Improvement (OERI), U.S.Department of Education. This publication was developed to fulfill one of the functionsof the clearinghouseinterpreting the literature in the ERIC database. This guide shouldbe of interest to vocational education administrators, practitioners, and evaluators engagedin assessing vocational education programs.

ERIC/ACVE would like to thank N. L. McCaslin for his work in the preparation of thispublication. Dr. McCaslin, Associate Professor in the Department of Agricultural Educa-tion at the Ohio State University, has an extensive background in vocational educationteaching, administration, evaluation, and planning. He was formerly Associate Director ofthe Evaluation and Policy Division of the National Center for Research in Vocational Edu-cation at Ohio State. He has served as an evaluation consultant in such varied settings asthe U.S. Army, communiv; colleges, universities, state departments of education, state advi-sory councils, the Unes:.:o Institute of Education, and government ministries in Mexico,Uganda, and Scotland.

The following people are also acknowledged for their critical review of the manuscript priorto publication: Willie Mott, Director of 'Vocational Education, Jackson Public Schools,Mississippi; Kay Rogers, Deputy Director, Francis Tuttle Vocational-Technical Center,Oklahoma City; Richard D. Miguel, Assistant Superintendent, Illinois Department of Voca-tional-Technical Education; and Morgan Lewis, Research Scientist, Center on Educationand Training for Employment.

Publication development was coordinated by Susan Imel. Sandra Kerka edited the manu-script, and Janet Ray served as word processor operator.

Ray D. RyanExecutive DirectoiCenter on Education and

Training for Employment

EXECUTIVE SUMMARY

One effect of educational reform movements is greater emphasis on evaluation. To helpvocational education administrators and practitioners with program assessment, this paperpresents a framework for vocational education program evaluation. Evaluation is defmedas the act of rendering judgments to determine a program's value; two major types areformative and summative. Vomtional education evaluation has been encouraged ormandated by the Vocational Education Act of 1963, the 1968 Amendments, the EducationalAmendments of 1976, and Carl D. Perkins Vocational Education Act of 1984, and therecently enacted Carl D. Perkins Vocational and Applied Technology Education Act.

Vocational education evaluation should take a broader and more comprehensive approachthan a focus on student achievement. This approach requires use of valid and reliableinformation of three interrelated types:

1. The needs for vocational education expressed by students, employers, and society

2. Vocational education program processes

3. Vocational education program outcomes

A framework for evaluating vocational education consists of the following components:

Needs

1. Students' interests, knowledge, attitudes, and skills

2. The knowledge, attitudes, and skills employers want in employees

3. The educational, economic, and social needs of society

Vocational Education Processes

1. Organizational information (organizational arrangement, administrative style, organiza-tional climate, teaching styles, type of organization)

2. Program information (goals and objectives, instructional content and delivery methods,administrative support, recruitment, facilities and equipment, cost)

3. Support services and activities information (student organizations, job placement,continuing education, planning, evaluation, personnel development)

4. Staff information (demographic characteristics, educational experiences and competence,occupational experiences and competence)

5. Student information (demographic characteristics, educational achievement)

6. Community information (linkage with business, industry, other educational agencies,other training programs; type of community)

Vocational Education Outcomes

1. Economic outcomes (labor force participation, employment and unemployment rates,training-related placement, type of employment, earnings, employee satisfaction withwork, employer satisfaction with employees)

2. Educational outcomes (academic and higher-ordei thinking skills, knowledge of theworld of work, occupational skills, school attendance and dropout rates, continuingeducation rates, satisfaction with education)

3. Psychosocial outcomes (aspirations, attitudes, values, self-esteem, citizenship, leadership)

Steps fo., using the evaluation framework are as follows:

Determining the type, purpose, and extent of the evaluation

Selecting the components

Identifying the sources and types of information to be collected

Collecting the required information

Analyzing and interpreting the information

Preparing an evaluation report

Information on vocational education program evaluation may be found in the ERIC systemusing the following descriptors: Accountability, Data Collection, Educational Change,*Evaluation Criteria, Information Needs, Information Sources, *Outcomes of Education,*Program Evaluation, Program Improvement, *Vocational Education. Asterisks indicatedescriptors that are particularly relevant.

x

oo

INTRODUCTION

Evaluation has taken on increased mean-ing and importance in the last decade withthe emphasis that has been placed oneducational reform. The educationalreform movement first became apparentwhen the Secretary of Educatio-., T. H.Bell, created the National Commission onExcellence in Education on August 26,1981, and charged it to report on thequality of education in the United States.The commission report, A Nation At Risk:The Imperative for Educational Refonn(NCEE 1983) stated:

If an unfriendly foreign powerhad attempted to impose onAmerica the mediocre educa-tional performance that existstoday, we might well have viewedit as an act of war. (p. 5)

This report stimulated interest throughoutthe country and many states initiatedsimilar study groups. These groups wereasked to examine their educational sys-tem, determine problems, and recommendchanges for improving their programs.

In the 1990s there continues to be wideconcern over the level of educationalachievement in the United States.According to "The 22nd Annual GallupPoll of the Publir's Attitudes toward thePublic Schools" (Elam 1990), more thanthree-quarters of those questioned indi-cated that they attach very high or highpriority to all six of the national goals foreducation proposed by President Bushand the 50 state governors. PresidentBush, in his concluding remarks to the

1

0

Education Summit held at the Universityof Virginia on September 28, 1990, stated:

There is a need for the first timein this nation's history to havespecific results-oriented goals (foreducation). We recognize theneed for . . . accountability.

The National Governors' Association(1990) stated that the country is facing amajor crisis in education. They reportedthat the educational system is obsoletedue to technological, economic, social,and demographic changes. According totheir report--

over the course of the next dec-ada, our nation must better edu-cate far more Americans, of allages, to new kinds and higherlevels of knowledge and skillsthan ever before. We must dothis with an increasingly diversepopulation, many of whom facesubstantial economic, social orother barriers to learning, such asthe effects of substance abuse,teen pregnancy or ina&quatehealth care. None of our citizenscan be written off. (p. 7)

Several states (for example, California,Connecticut, Iowa, Kansas, Maryland,New Jersey, and Ohio) have taken stepsto make their educational systems moreaccountable for their outcomes. Theseinclude such outcomes as basic skillachievement, attendance, and dropoutrates. Regardless of the outcomes select-ed, the message is clear: states want their

schools to demonstrge a reg. _nsible levelof accountability for.the funds that havebeen entrusted to them.

If educational programs are to reach theirobjectives, increased levels of cooperationamong the schools, parents and students,and business and industry are required.Much of this effort in the past has beenfragmented with little articulation ofeffort. Much more needs to be done tosmooth the transitions between educationand work.

Although change 1/4 necessary, educators,parents, and employers often find it dif-ficult. All too often, we are uncomfort-able with the adjustments it requires andwant to hold on to the practices of thepast. This type of behavior must be dis-carded as we approach the 21st Century.

2

The field of vocational education has notbeen unaffected by the educational reformmovement for it has also spurred interestin the evaluation of vocational educationprograms. This paper is intended to helpadministrators and vocational educationpractitioners with the evaluation of localvocational education programs It beginsby defining evaluation and then providesan overview of evaluation in vocationaleducation. Next, types of informationneeded for evaluating vocational educa-tion are described. Then, a frameworkfor evaluating vocational education, whichuses these information categories as abasis, is presented. The paper concludeswith recommendations for using the eval-uation framework.

. 1 0

EVALUATION IN VOCATIONAL EDUCATION

One of the major problems in discussingevaluation is that there is no widelyaccepted definition of evaluation. Indeed,this lack of agreement has led to thedevelopment of a variety of approaches toevaluation. Worth tn and Sanders (1987)classified the different approaches into thefollowing six categories:

1. Objectives-oriented approaches

2. Management-oriented approaches

3. Consumer-oriented approaches

4. Expertise-oriented approaches

5. Adversary-oriented approaches

6. Naturalistic and participant-orientedapproaches

For the purposes of this paper, the defi-nition of evaluation offered by Worthenand Sanders (1987) is used: "evaluation. . . (is) the act of rendering judgments todetermine (a program's) value. -worth andmerit" (p. 24).

Two major types of evaluation are gener-ally accepted in discussions of evaluation:formative and summative. MichaelScriven (1967) was the first to differenti-ate between these two types of evaluation.

Formative evaluation is conductedthroughout a program to improve its oper-ation. This type of evaluation primarily isdesigned to provide feedback for revising

3

1

or improving the procedures used in theprograms

Summative evaluation, OD the other hand,is conducted at the conclusion of a pro-gram to provide evidence of its worth ormerit. This type of information is oftenused to document the effectiveness of aprogram. In this sense, the purpose ofsummative evaluation is accountability.

Evaluation has been an area of concernfor vocational education teachers andadministrators for many years. Early val-uation efforts centered on conductingfollow-up studies to determine the degreeto which vocational education graduatesfound placements in jobs that were relatedto their training. This type of evaluationis summative. Other early forms of evalu-ation relied on an accreditation type ofvisit by external teams of individuals.This is an example of formative evalua-tion.

The 1963 Vocational Education Act wasone of the first pieces of federal legis-lation indicating the need for states toconduct evaluations of their vocationaleducation programs. Many evaluation ef-forts resulting from this mandate consistedof reviewing vocational education proces-ses and follow-up information on programgraduates.

In 1968, the Vocational EducationAmendments to the 1963 Vocational Edu-cation Act continued to emphasize stateevaluation activities. They also gave

evaluation responsibilities to state advisorycouncils.

The Education Amendments of 1976 (P.L.94-482), which revised the VocationalEducation Act of 1963, further expandedthe responsibility for evaluation and hadat least 28 references to different forms ofevaluation (Went ling 1980). These evalu-ation requirements were designed to in-crease the responsiveness of vocationaleducation to changing labor markets, andthey required states to evaluate theirprograms every 5 years and to determinethe extent to which programs completersand leavers--

found employment in occupations re-lated to their training, and

were considered by their employers tobe w-ll-trained and prepared foremployment.

Seven different groups were specificallycharged with conducting vocational educa-tion evaluations in the 1976 Amendments:

1. State boards of vocational education

2. State advisory councils for vocationaleducation

3. The National Advisory Council forVocational Education

4. The U.S. Office of Education'sBureau of Occupational and AdultEducation

5. The National Institute of Education

6. The National Center for EducationStatistics

7. The U.S. Office of Education's Officeof Evaluation and Dissemination

4

1 2

The Carl D. Perkins Vocational EducationAct (P.L. 98-524) was enacted in 1984.This legislation was designed--

to amend the Vocational Educa-tion Act of 1963 to strengthenand expand the economic base ofthe Nation, develop human re-sources, reduce structural unem-ployment, increase productivity,and strengthen the Nation's de-fense capabilities by assisting theStates to expand, improve, andupdate high-quality programs ofN;ocational-technical education,and for other purposes. (p. 2435)

The evaluation requirements of P.L. 98-524 charged the states to develop mea-sures for the effectiveness of vocationaleducation programs including the follow-ing:

The occupations to be trained for,which will reflect a realisticassessment of the labor market ofthe state;

The levels of skills to be achievedin particular occupations, wbichwill reflect the hiring needs ofemployers: and

The basic employment competen-cies to be used in performanceoutcomes, which will reflect thehiring needs of employers.(p. 2447)

Congress has recently enacted the Carl D.Perkins Vocational and Applied Technol-ogy Education Act of 1990. This legisla-tion continues to include evaluation spe-cifications. It requires states to developcore standards and measure3 of perfor-mance for secondary and postsecondaryvocational education programs. Theseperformance standards we to include thefollowing:

INFORMATION NEEDED FOR VOCATIONALEDUCATION EVALUATION

If vocational education is to provide thecitizens of the United States with theskills they need to compete in the market-place and lead meaningful lives, then theevaluation of these programs becomeseven more important. Much of the cur-rent literature on educational reformcould lead one to conclude that the onlyimportant evaluation information isrelated to results in terms of studentachievement. However, if information onstlident achievement in vocational and

ademic skills is all that is available, howcan teachers a.ld administrators knowwhat worked and did not work? Uponwhat basis should decisions to add anddrop programs be made? What should bechanged in order for vocational studentsto achit., e at higher levels?

This paper suggests that those concernedwith evaluating vocational education con-

sider a broader and more comprehensiveapproach to evaluation. This approachrequires the use of valid and reliableinformation of three major types:

1. The needs for vocational education asexpressed by clients (that is, studentsand employers) and society

2. The educational processes followedby vocational education programs

3. The outcomes achieved by vocationaleducation programs

The interrelationships of these three typesof evaluation information are presented infigure 1. A brief description of thesethree types follows.

Needs for Vocational Education

Vocational

Education

Processes

Vocational

Education

Outcomes

Figure 1. Interrelationships of evaluation information

7

1 L.4

Needs for Vocational Education

Vocational education programs in theUnited States are basically offered as elec-tives in secondary and postsecondary edu-cation agencies. Therefore, individualswill not enroll unless they feel that aprogram would meet their needs or satisfytheir desire for a type of service theywould like to receive. Thus, vocationaleducation is a "customer" driven program."Customers" here are broadly defined toinclude all groups in a community thatmight have a stake in vocational education(Rossi and Freeman 1989).

Kaufman and Stone (1983) indicated thatin assessing needs it is necessary to showand document gaps between current re-sults and desired results (in terms ofoutcomes) and then to place these gaps(needs) in priority order. Accordingly,needs are concerned with determining theextent or magnitude of a problem.

Needs can be determined by reviewing ex-isting information (for example, labormarket information and educationalachievement levels) or collecting newinformation. This information might be ofeither a quantitative or qualitative nature.

Vocational Education Processes

Historically, accreditation reviews havedevoted much of their attention to exam-ining the processes used by educationalagencies. Many states' 5-year reviews oftheir vocational education programs haverelied on this type of information. Pro-gram sponsors/funders typically placeheavy emphasis on process information indemonstrating their accountability.

8

Process information provides evaluatorswith a basis for understanding and inter-preting what has occurred. If vocationaleducation programs are to be improved asa result of an evaluation, it is essential toprovide information about how they havebeen conducted. Rossi and Freeman(1989) referred to the evaluation of pro-cesses as systematic monitoring to exam-ine program coverage and delivery.

Vocational Education Outcomes

As previously discussed, the currentemphasis in educational evaluation is ondowmenting educational effectiveness interms of student achievement. Stuff le-beam (1969, 1971, 1983) and Stufflebeamand Shinkfield (1985) referred to evalu-ating the outcomes of educational pro-grams as product evaluation. Among thequestions that product evaluation answersare the following:

What results were obtained?How well were needs reduced?What should be done with the pro-gram after it has run its course?

Rossi and Freeman (1989) referred toevaluating outcomes as "impact assess-ments (which) are undertaken to establishwhether or not interventions produce theirintended effects" (p. 225). The NationalGovernors' Association (1990) report,Educating America: State Strategies forAchieving the National Education Goals,stated that "national education goals willbe meaningless unless progress towardmeeting them is measured accurately andadequately, and reported to the Americanpeople" (p. 39).

A FRAMEWORK FOR EVALUATINGVOCATIONAL EDUCATION

Using the three types of information need-ed for evaluating vocational education, anevaluation framework is presented in fig-ure 2. This framework is not proposed asa rigid model that all evaluations shouldfollow. Rather, it is presented as a guidefor carefully thinking through what mightbe included in an evaluation of vocationaleducation. The remainder of this sectiondiscusses possible components for each ofthe three types of information.

The Needs of Students, Employers,and Society

In the Peters and Waterman (1987) book,In Search of Excellence, Lew Young,Editor-inChief of Business Week, isquoted as saying:

Probably the most importantmanagement fundamental that isbeing ignored today is stayingclose to the customer to satisfyhis (sic) needs and anticipate his(sic) wants. (p. 156)

Educators need to understand and antici-pate the needs of their customers. If theneeds of the vocational education custo-mer are to be taken seriously, educatorsmust begin collecting better informationabout students, employers, and society.

9

- ,I c

Students

Today's vocational education studentscome with a variety of needs. Increas-ingly, they come from one-parent, lowersocioeconomic status, and minority fami-lies. The Forgotten Half: Pathwuys toSuez= for America's Youth and YoungFamilies (The William T. Grant Founda-tion Commission on Work, Family andCitizenship 1988) reported that--

our two-year study of 16-24-year-olds has corvinced us that, asyoung Americans navigate . . .

from youth to adulthood, far toomany flounder and . . . fail. . . .

Although rich in material re-sources, our society seems unableto ensure that all our youth will. . . face their futures with . . .

confidence and security. This isespecially true of the 20 millionnon-college-bound young peoplewe have termed the forgottenhalf. (p. 1)

Similarly, vocational educators must be-come increasingly concerned with adultswho are struggling to survive in the work-place. Of particular concern among thepotentid students in these categories aredisadvantaged or handicapped people,people entering nontraditional occupa-tions, homemakers, immigrants, individu-als with limited English proficiency, andindividuals who are incarcerated. Theworkplace is demanding that even thosewith "good jobs" receive training and

NEEDS

Students

intc.ests

Knowledge

Attitudes

Skills

Employers

Knowledge

Attitudes

Skills

Society

Educational

Economic

Social

PROCESSES

Organizational

Information

Organizational Prrangements

AdMinistrative Style

Organizational ClimateTeaching Styles

Type of Educational

Organization

Program Information

Goals and Objectives

Instructional Content

Instructional Delivery

Methods

AdMinistrative Support

Recruitment

Facilities and Equipment

Cost

Support Services and

Activities Information

Student Organizations

Job Placement Service

Continuing Education Service

PlmningEvaluation

Personnel Develorment

Staff

Information

Demagraphic Characteristics

Educational Experiences

Educational Competence

Occupational Experiences

Occupational Competence

Student Information

Demographic Characteristics

Educational Achievement

Community Information

linkage with Business

and Industry

linkage with Other

Educational Agencies

linkage with Other

Training Programs

type of Community

rftl

4

OUTCCIES

Economic

labor Force Participation Rates

Employment and Unemployment Rates

Training-Related Placement

Type of Employment

Earnings

Employee Satisfaction with Work

Employer Satisfaction with Employees

Educational

Academic Skills

NigherOrder Thinking Skills

Knowledge of the World of Work

Occupational Skills

School Attendance and Dropout Rates

Continuing Education Rates

Satisfaction with Education

psychosocia(

Aspirations

Attitudes and Values

Self-esteem

Citizenship

leadership

Figure 2. A framework for evaluating vocational education

1 °C)

retraining to meet the rapidly changingdemands of the global market.

In attempting to detsrmine the needs ofthese vocational education customers, in-formation is needed regarding their inter-ests, knowledge, attitudes, and skillsWithout taking this type of informationinto consideration, vocational educationprograms are likely to be judged irrelevantby these individuals. As a result, they willseek education and training from otheragencies or become dropouts--further re-ducing the nation's productivity and eco-nomic growth.

Employers

Another important group of customers forvocational education are the employers.The chief executive officers (CEOs) oflarge businesses have become more vocalin expressing the needs of their organiza-tions (Iacocca and Novak 1987; Kearnsand Doyle 1988; Sculley and Byrne 1987).However, these businesses are only onetype of vocational education employer cus-tomer. Their needs may not be the sameas those of other employer customers--themedium-sized and small businesses whoalso employ vocational education studentsor the entrepreneurs who create their ownjobs after completing a vocational educa-tion program.

Cetron and Davies (1989) projected that"by the turn of the century, most of ourmid-sized institutions will have vanished,but thousands of tiny companies will beflourishing beneath the feet--and noticeof the giants" (p. 204). Whether or notthis speculation comes about, vocationaleducators need to consider who their em-ployer customers are. Regardless ofwhether these customers are large, medi-um, or small, educators need to seek theiradvice regarding the knowledge, attitudes,

and skills they would like in theiremployees.

It is also important to note that, in thelarge and medium-sized businesses, inputshould be sought from more than one in-dividual. This is particularly importantbecause businesses are increasing theamount of employee participation in man-agement and stressing the autonomy andcreativity of the worker. Information onneeds collected from such individuals aspersonnel directors, division heads, andline supervisors is essential in determiningthe knowledge, attitudes, and skills neededby these employer custome:s.

The needs of the entrepreneur as an em-ployer customer have been overlooked inmost evaluation efforts in vocational edu-cation. If Cetron and Davies (1y89) arecorrect in their prediction about the dra-matic increase in small companies, theneeds of this type of customer must bedetermined.

Society

Another important customer of vocationaleducation is society itself. These custo-mers include such groups as taxpayers,school board members, boards of trustees,legislators, policy makers, and parents.The educational, economic, and socialneeds of society are often expressed bythe media or through statistical and ana-lytical reports published by the govern-ment and private foundations.

Reports on the educational needs of soci-ety are particularly abundant in the media.Among the types of information express-ing educational needs are school cerzusdata, schiol attendance and dropout rates,basic education achievement scores, andrates of continuing postsecondary educa-tion.

A variety of ezonomic information andstatistics is published at the national, state,and local levels. These figures provideinformation that is relevant to those con-cerned about vocational education pro-grams. Among the types of economic in-formation are unemployment rates, num-ber of building permits issued, averagehours employees work per week, andlabor market projections.

A number of social indicators are also rel-evant to vocational education programs.Among the types of information thatcould be used are census information(particularly since the 1990 informationwas just collected) on numbers of school-aged students, racial composition, andmarital status. Statistics on the number ofteenaged parents, percentage on publicassistance, number receiving Aid toFamilies with Dependent Children,number of handicapped individuals, andnumber receiving free or reduced priceschool lunches represent yet anothersource of valuable information.

Vocational Education Processes

An examination of various processes ofvocational education has been a majorfocus of statewide evaluation efforts sincethe implementation of the 1968 Amend-ments to the Vocational Education Act of1963. Building upon the model used byaccreditation organizations, these effortsrelied heavily upon a process evaluation inwhich review teams went into the schools,following a self-study, to interview, ob-sk. rve, and review documents related tohow these programs were operated.These statewide evaluation systems oftenincluded specific components such as cur-riculum and instruction, facilities, equip-ment, staff, student assessment, and stu-dent placement and follow-up. Manystates continue to use these same types ofstatewide evaluations today and have

expanded the process to include othercomponents such as those dealing withequity, disadvantaged, handicapped, andnontraditional enrollments.

In preparing this paper, the author re-viewed materials from many statewideevaluation systems and other vocationaleducation evaluation studies. As a result,six major types of information emerged asimportant in conducting process evalua-tions of vocational educ.....tion: organiza-tional information, program information,support services and activities information,staff information, student information, andcommunity information.

Organizational Information

Public vocational education programs op-erate within the broader environment ofa secondary or postsecondary educationalorganization. How these organizationsbehave and operate is an importantsource of information in evaluating voca-tional education programs. Informationabout an organization can be obtained byinterviews, observations, reviews of exist-ing documents and information, and sur-veys--either individually or in combination.Potential areas of interest in an evaluationinclude the following:

Organizational arrangementsAdministrative styleOrganizational climateTeaching stylesType of educational organization

Two important considerations in evaluat-ing the organizational arrangements forvocational education are its administrativestructure and physical location. The or-ganizational chart provides an initial , ii-cation of how the school or college 'sstructured to administer vocational educa-

don programs. However, it is often idor-mative to discuss the administrative ar-rangements with faculty and administra-tors to find out how close the statedadministrative arrangements are to theactual situation. Another item to consideris the program's physical location within abuilding. The accessibility of a programto students can be an important factor indetermining who enrolls.

Administrators are often identified as akey factor in having an effective organiza-tion. These individuals usually set thetone and direction for the organization.How administrators perceive their role incarrying out their functions (administrativestyle) serves as an important signal to fac-ulty and students in determining the ex-tent to which they will take responsibilityin the teaching and learning process.

Another important characteristic to beconsidered is that of organizational cli-mate--the set of properties that are per-ceived by the administrators, faculty, andstudents as influencing the performance ofpeople in an organization (adapted fromGibson, Ivancevich, and Donnelly 1979).Therefore, organizational climate is animportant element in helping to shape theopinions of and anticipated outcomes forvocational education programs.

The teaching styles used in an educationalorganization are also imnortant considera-tions in evaluating vocational educationprograms. One of the hallmarks of voca-tional education has been that of learningby doing. This educational process re-quiro- students to become active in con-crete learning situations, whether they bedeveloping basic educational skills oroccupational skills. An evaluation ofvocational education should examine theappropriateness and relevance of theteaching styles to the outcomes expected.

13

The type of agency offering vocationaleducatici programs is often overlookedwhen evaluations of these programs areconducted. Not all secondary find post-secondary agencies are alike. Secondaryschools can be classified into organizationssuch a: comprepensive high schools con-taining their own vocational educationprograms, high schools that cooperatewith other schools in offering vocationaleducation programs at another location,and separate vocational education highschools with their own academic offerings.Similarly, postsecondary schools can beclassified as community colleges, technicalcollegeVirstitutes, or technical colleges/institutes within 4-year colleges. Thesetypes of organizations often have decided-ly different types of vocational programs.In other instances, one educational organi-zation might effer both secondary andpostsecondary programs. Therefore, thetype of educational organizations offeringvocational education programs should beidentified so that the results of the eval-uation could be compared with those ofsimilar organizations.

Program Information

Information related to how vocationaleducvItion programs are conducted is im-portant in determining what changes needto be made in these programs or in deter-mining how to implement similar pro-gram in another location. The degree towhich programs are operated as they weredesigned is essential if one is to judge theoutcomes obtained. Potential areas ofinterest include--

Goals and objectivesInstructional contentInstructional delivery methodsAdministrative supportRecruitmentFacilities and equipmentCost

The goals and objectives provide the basicstructure and road map for vocationaleducation programs Clear statements ofgoals and objectives are essential if every-one is to develop a sense of agreementabout the purposes of these programs.These statements should be included indocuments such as curriculum guides,courses of studies, or plans of instruction.

The instructional content should be con-sistent with the goals and objectives. Thiscontent should be based on current indus-try standards and should reflect the needsof students and the community. Many vo-cational education programs have deter-mined the instructional content by tech-niques such as task analyses and DACUM(Developing A Curriculun). The curricu-lum guides, courses of studies, or plans ofinstruction should also be examined todetermine the scope and sequence of theinstruction and see how safety instructionis provided.

The instructional delivery methods shouldbe well-balanced, with a variety of teach-ing and learning approaches in the class-room instruction, supervised experience,and vocational student organizations. Evi-dence related to how articulation andintegration of vocational and academicinstruction are achieved should also beobtained.

Mministrative support should not beoverlooked as an area of interest. On thesurface this area may appear to be easy todetermine. In making determinations of

administradve support for vocational edu-cation, the perceptions of several differentstakeholders need to be checked as wellas reviewing adviory committee minutesand budget allocations.

Recruitment of students is especiallyimportant for vocational education sincethese programs are elective. Teachers,administrators, and guidance counselorswill need to have recruitment and promo-tional materials available for studentsconsidering taking vocational educationprograms or courses. The content ofthese materials should be reviewed withboth accuracy and equity concerns inmind

14

.1):Th

Facilities and equipment represent yetanother area of interest related to pro-gram information. Just as students in thescience areas need access to up-to-datefacilities and equipment, vocational educa-tion students also require access to mod-em technology. The provision of facilitiesand equipment may be a shared responsi-bility of both the school and business andindustry. Some skills needed in the oper-ation of equipment might be developedduring the classroom instruction and otherrequired skills may need to be developedduring the student's supervised occupa-tional experience program.

Finally, the cost of the program should bereviewed. A major item will be personnelcosts. However, budgets should be exam-ined to see that additional funds areprovided for items such as instructionalmaterials, equipment, and supplies.

Support Services and ActivitiesInformation

To offer an effective program of voca-tional education, several support servicesand activities are recommended. Anevaluation of vocational education needs

to consider the degree to which thesezeMces and activities are provided.Potential areas of interest include--

Student organizationsJob placement serviceContinuing education servicePlanningEvaluationPersonnel development

Student organizations have been animportant aspect of vocational educationfor many years. These organizationsinclude the Distributive Education Clubsof America (DECA), FFA (formerly theFuture Farmers of America), Future Busi-ness Leaders of America (FBLA), FutureHomemakers of America (FHA), HealthOccupations Students of America(HOSA), Home Economics Related Occu-pations (HERO), Phi Beta Lambda(PBL), Vocational Industrial Clubs ofAmerica (VICA), and Technology Stu-dents Association (TSA). These organi-zations provide many opportunities forstudents to develop self-esteem, kader-ship, and citizenship. Additionally, thesestudent organizations sponsor numerouscompetitive events and contests wheretheir members' competence is evaluated.

A job placement service is often includedin vocational education programs to helpstudents find employment. Individualteachers often take the sole responsibilityfor the placement of their students. Atother times, especially in larger programs,the job placement service is centralizedwith individuals specifically responsible forthis service. Some schools have job place-ment services using some centralizedplacement functions, but leaving the ac-tual responsibility of job placement withthe teachers.

15

A continuing education service is theassistance offered to students desiring topursue additional schooling. It is asimportant to vocational education studentsas it is to academic students since educa-tion is increasingly viewed as a lifelongprocess. The increasing number of em-ployers requesting individuals with morethan a high school education also rein-forces the need for this service.

The necessity of planning from both astrategic and tactical point of view isbeing recognized as an essential compon-ent in reforming education. Ohio's Futurrat Work (Ohio Department of Education1990) is one example of the importancetha# state departments of vocational edu-cation are placing on developing strategicplans for the reform of education. Thesetypes of long-range plans develop a 3- to5-year vision for vocational education andprovide the basis for tactical (operational)plans regarding what and how it will becarried out over each of the ensuing years.

Evaluation activities are closely linkedwith planning. In planning, emphasis isplaced on stating intentions ("What do wewant to do?") whereas evaluation placesemphasis on determining the degree towhich stated expectations were met ("Didwe do what we planned?"). Evaluationshould examine the outcomes of a pro-gram and relate them to the various needsand processes contributing to or hinderingthese accomplishments so that the pro-gram can be improved.

Personnel development encompasses thoseinservice activities primarily designed todevelop or enhance the knowledge orskills of vocational education teachers andadministrators. These activities can be ofa formal or informal nature and offeredeither for credit or noncredit. The cur-rent educational reform movement willgenerate many reeds for personneldevelopment.

Staff Information

The teuhers and administrators are thekeystone in developing high quality voca-tional education programs Efforts toevaluate vocational education need toconsider information about their back-ground and experience. The types ofinformation needed include

Demographic characteristicsEducational experiencesEducational competenceOccupational experienceL.Occupational competence

"I1,

Demographic information is important indealing with issues such as the extent towhich vocational education is dealing withequity issues and nontraditional programsIn these cases it would be important toknow both the gender and race of thevocational education teachers andadministrators.

The educational experiences of vocationaleducation teachers and administrators arequite varied. Sometimes they come di-rectly from business and industry withlittle or no formal postsecondary educa-tion. Some will have completed a 2-yearprogram in a community college or tech-nical institute. Others will have com-pleted 4-year college degrees--some ineducation and some in other areas(health, business, agriculture, humanecology, engineering, and so forth).Similarly, these individuals have variedexperiences in and with vocational educa-tion. Some will have worked only asteachers, others will have worked as bothadministrators and teachers. The typeand amount of educational experiencesthat teachers and administrators possessrepresent impot tant information in evalu-ating vocational education prz)grams.

Serious questions regarding the educa-tional competence of educators have beenraised by many in the educational reformmovement. To address these concerns,information is needed regarding the basicskill levels that vocational educatorspossess in reading, writing, and computa-tion.

The amount of occupational experiencethat vocational educators possess is oftenincluded in evaluating vocational educa-tion programs. The recency of this experi-ence has been receiving increased atten-tion because of the rapid changes that areoccurring in the workplace and society.

Occupational competence represents yetanother important source of informationabout the vocational education staff. Theteachers' level of competence will affecttheir students' acquisition of the funda-mental skills necessary to become produc-tive members of the work force. Somestates require occupational competencytesting before a teacher can be certified.The educational reform concern over thedegree to which teachers are competent toteach their subject matter is as importantin vocational education as it is to the restof the educational community.

Student Information

The need for information regarding thestudents whom vocational education servesis of paramount importance in evaluation.This information can be used initiely asbaseline information in determining whatrtudents learn in vocational education.Furthermore, it can be used as a controll-ing variable when comparisons of achieve-ment with other types of students aremade. Among the types of informationneeded are--

16

1

1

Demographic characteristicsEducational achievement

Students' demographic characteristics suchas grade level, gender, race, and socio-economic status (SES) are valuable inunderstanding what type of student isparticipating in vocational education.Vocational education often has been view-ed as a second-rate education involvin5;greater proportions of minorities and stu-dents with low SES. Vocational educationhas also been cLarged with sex-role ster-eotyping. Any evaluation efforts address-ing these concerns would need to examinestudent demographic information.

The educational achievement in both aca-demic and vocational skills is especiallyimportant as more emphasis is beingplaced on the results of educational pro-grams. The present accountability move-ment has placed most of its emphasis onstudent achievement at the end of acourse or program. However, approachessuch as "value-added" evaluations, inwhich educational agencies need to docu-ment how much students learned, also willrequire educational achievement levels forstudents entering the program.

Community Information

The community has always played an im-portant role in vocational education.Since its inception, vocational educatorshave promoted the involvement of em-ployers and parents in planning and oper-ating their programs. The use of advisorygroups to review a program, particularlyto keep it up to date and relevant, hasbe,m a common practice. The Carl D.Perkins Vocational and Applied Technol-ogy Education Act of 1990 continues thistradition by calling for the involvement ofemployers, parents, teachers, students, and

other providers in the planning, operation,and evaluation of vocational educationprograms. The types of information use-

trfal in the evaluation of these programsrinclude

Linkage with business and industryLinkage with other educationalagenciesLinkage with other trainingprogramsType of community

A program's linkage with business andindustry is not a recent development.These types of arrangements have beencommon since the inception of vocationaleducation. One important use of advisorycouncils--consisting of personnel frombusiness, industry, and laboris for eval-uation. These groups are particularly wellsuited to determine needs for programs;identify and examine course content; re-view facilities, equipment and supplies;suggest procedures for coordinating andcollaborating with business and industry;and assess student occupational compe-tence.

The emphasis on linkage with other edu-cational agencies, on the other hand, is amore recent occurrence. The increasingcompetition for the limited funds availablefor supporting public education and train-ing programs has reinforced the need forthese agencies and organizations to devel-op collaborative arrangements with oneanother. Many secondary and postsecond-ary agencies have developed articulationagreements to help students move fromone program to another.

Another more recent development is theiinkage with other education and trainingprograms. Not only do these arrange-ments often result in economic benefits,they also facilitate the delivery of services

17

C., ,r-( t..)

to the participants. The 1990 Perldns Actpermits states to use funds to coordinateservices with the Job Training PartnershipAct (JTPA) and the Wagner-Peyser Actfor coordinated services.

The type of community is another cate-gory of information that needs to beconsidered when conducting evaluations ofvocational education. The various typesof communities include rural, urban, andsuburban areas. This identification willfacilitate the comparison of program re-sults with similar communities, develop-ment of appropriate conclusions, andpreparation of realistic policy recommen-dations.

Vocational Education Outcomes

Vocational education outcomes refer tothe results or accomplishments of thosewho participate in these programs. Theseoutcomes should be easily understood andagreed upon by vocational educationstakeholders. An outcome might consistof either a single statistic or a compositemeasure based on several factors. Theterms core standards, measures of perfor-mance, and performance standards aresometimes used in reference to educa-tional outcomes (as in the Carl D. PerkinsVocational and Applied Technology Edu-cation Act of 1990 and the Job TrainingPartnership Act).

Tne outcomes of vocational educationoften have been thought of solely ineconomic terms. However, more recentdiscussions and studies have included awider range of educational, psychological,and sociological outcomes. For the pur-poses of this paper, three major types ofoutcomes are suggested: economic, edu-cational, and psychosocial.

18

Economic Outcomes

As mentioned earlier, much of the re-search on the effectiveness of vocationaleducation has focussed on economic out-comes. These types of outcomes are us-ually measured in terms of the experi-ences of former vocational education stu-dents in the labor market. Examples ofeconomic outcomes include--

Labor force participation ratesEmployment and unemploymentiatesTraining-related placement1; ^ of employmentEirnitEmployee satisfaction with workEmployer satisfaction withemployees

Labor force participation rates refer tothe extent to which individuals are actu-ally working or seeking work. It does notinclude those individuals who have "givenup" and stopped seeking work eventhough they are unemployed.

The outcomes of employment and unem-ployment rates are determined by the pecentage of individuals who are eitherworking or not working. This outcome, aswell as the previous one, can be comparedwith state and national statistics for simi-lar groups of individuals if the same defi-nitions have been used in computing therates.

Historically, training-related placementhas been the major outcome for evalua-ting vocational education. This outcomerefers to the percentage of individualswho fmd employment in the occupationsfor which they were trained. This out-come often bas been criticized since itsachievement is determined by many fac-tors that are beyond the control of voca-

tional educators. However, it has remain-ed an important outcome.

The type of employment has also becomean important outcome as emphasis on vo-cational education's role in economicdevelopment has increased. Since manynew jobs are in small businesses, self-employment has become a legitimate out-come for vocational education, and manyentrepreneurship programs have been de-veloped with this specific result in mindThe percentage of males and females dis-tributed across all types of occupations isanother important factor, and Lamy pro-grams have beer specifically designed toaddress inequities in these occupationalareas.

Earnings received from working representanother type of outcome for vocationaleducation programs. In determining earn-ings, it usually helpful to think in terms ofannual earnings that reflect both the hour-ly wages and number of hours worked.Annual earnings also allow evaluators tocompare the wages of those who work asemployees with those who are self-employed. Also, it is often interesting toexamine and compare the earnings ofthose who are employed in the occupa-tions for which they were train-d withthose who are working in occupation::unrelated to their training.

The degree of employee satisfaction withwork is classified here as an economicoutcome since it is often reflected in theproductivity of an individual. Although itis also influenced by many factors beyondthe control of vocational education, it canreflect the degree to which a student'sprogram develcped a realistic picture ofthe workplace.

Similarly, employer satisfaction withemployees is classified as an economicoutcome since it can be a reflection of acompany's success in the marketplace.

19

Again, this factor is inflpsnced by manyfactors beyond how well the employer issatisfied with the educational preparationindividual employees have received.Therefore, evaluators need to interpretthe results obtained for this outcome andthe previous one with caution.

Educational Outcomes

Legislators and policy makers are movingmore toward accountability measures tiedto the instructional outcomes of educa-tional agencies. This reflects quite adeparture from the institutional or processevaluations that have been used in thepast. Thig growing concern for educa-tional outcomes is being reflected in revi-sions to statewide approaches to voca-tional education evaluation in states suchas Illinois, Minnesota, and Virginia. Pos-sible educational outcomes include--

Academic skillsHigher-order thinking skillsKnowledge of the world of workOccupational skalsSchool attendance and dropoutratesContinuing education ratesStudents' satisfaction witheducation

The development of academie skills insuch areas as mathematics and English isan outcome that more than 86 percent ofthe public perceives as important for allstudents (Elam 1990). The high level ofimportance placed on these subjects indi-cates that vocational teachers should alsoemphasize the development of these skillsin their courses by showing students howthese skills can be applied.

Higher-order thinlcing skills require stu-dents to do more than simply recall or

understand information. The develop-ment of these higher-order thinking skillsrequires students to apply, analyze, synthe-size, and/or evaluate information (Bloom1956) in solving problems. If teachers areto develop these type', of skills, the objec-tives, instructional activities, and evalua-tion processes should reflect this type ofemphasis.

Another educational outco ct is knowl-edge of the world of work. ,s Cetronand Davies (1989) predicte( the nextC:cade, new jobs will appear and obsoletejobs will disappear faster th: L cver before,individuals will need to know the types ofjobs that will be available to them as thework force changes. Among the types ofjobs that Cetron and Davis predicted asgood opportunities are 4.omputers anddata processing, telecommunications,health care, biotechnology, advancedmaterials, hazardous-waste disposal andpollution control, education and voca-tional training, and personal and businessservices.

Occupational skills have been the majoroutcome of interest to many vocationaleducators. Much of the evidence for thisoutcome has been provided through theuse of competency or performance testing.Though the public believes that vocationaleducation should be required for all stu-dents not planning to go on to college(Elam 1990), evidence on student occupa-tional achievement is still needed.

School attendance and dropout rates areparticularly important outcomes. If thecountry is to increase the educational levelof students, it is imperative that schoolattendance rates be increased and dropoutrates be decreased. These rates should beexamined for differences among racial/ethnic groups, given the wide disparity inthese rates across groups.

The continuing education rates for stu-dents represent a recent outcome forvocational education. In a national longi-tudinal study of more than 6,000 individ-uals, Laughlin (1986) reported that 62percent of vocational education initiallyenrolled in some type of postsecondaryeducation and traluing. Of those whoenrolled, 62 percent completed the pro-gram or were still enrolled at the time ofthe follow-up.

Psychosocial Outcomes

The psychosocial outcomes of vocationaleducation relate primarily to the personaldevelopment of the individual student.These outcomes are often the result ofexperiences in vocational education stu-dent organizations. Some of these out-comes are difficult to measure, and indi-cators of their effectiveness are not readilyavailable. These outcomes include

AspirationsAttitudes and valuesSelf-esteemCitizenshipLeadership

The aspirations of students refer to theirdesires and ambitions. Many students donot have role models to use as examplesin setting personal goals and selectingpotential career areas. Therefore, theexperiential learning provided by voca-tional education provides students withreal-life examples to use in planning theirfuture. Sometimes the personal contactsprovide the opportunity for students toidentify role models that they wish toemulate.

Employers often criticize the attitudes andvalues that individuals bring to the work-place. Teachers and administrators have

20

fi CI ....)

the opportunity to influence students' atti-tudes and values by the personal examplesthey give to students. This outcome canalso be influenced through the student'soccupational experience program and thevocational student organization.

Self-esteem refers to the belief studentshave in themselves. It also refers to theirself-respect. This outcome can be influ-enced by how individuals interact withstudents. Positive learning environmentsneed to be established by teachers, admin-istrators, occupational experience supervis-ors, and other students and co-workers.

The outcome of citizenship refers to howstudents demonstrate their membership in

21

society by acts such as voting and partici-pation in community organizations. Thevocational student organizations providemicrocosms in which to practice the du-ties, rights, and privileges related to thisoutcome.

Leadership skills will be needed through-out a student's life. These skills willdetermine how students direct, command,or guide their colleagues or an activity.Excellent opportunities for students todevelop these skills, as well as Citizenship,are prov;-led by the vocational studentorganization.

ii C

USING THE EVALUATION FRAMEWORK

This section describes how to use theframework in evaluating vocational educa-tion programs. As mentioned earlier, theframework is not a rigid model, but aguide for thinldng through evaluations ofvocational education programs. As such,the framework is helpful in delineatingand designing program evaluations.

To use this framework, it must be adaptedto the situation. First, it will be neces-sary to decide the type, purpose, andextent of the evaluation. Then, it will benecessary to set priorities on the mostimportant information to be collected.Once these determinations have beenmade, data can be collected and the infor-mation analyzed and irterpreted. Finally,a report can be prepared presenting theresults of the evaluation. Each of thesesteps is described here.

Determine the Type, Purpose, andExtent of the Evaluation

Earlier, a distinction was made betweentwo types of evaluation: formative andsummative. Formative evaluation occursthroughout the operation of a program todetermine how it might be revised orimproved. Summative evaluation isconducted at the end of a program toprovide evidence on the effectivenes5 ofthe program and is used primarily foraccountability purposes. The type ofevaluation to be conducted is affected bywhoever requested the evaluation.Vocational program operators most oftenare interested in formative evaluation,whereas program funders and sponsors

23

usually are interested in summativeevaluation. Therefore, it is essential todetermine the type of evaluation that isdesired. Will the primary purpose of theinformation be to improve the operationof the program (formative) or to provideaccountability information (summative)?

Several purposes for conthicting evalua-tions exist (Brinkerhoff, Brethower,Hluchyj, and Nowakowski 1983a). Amongthe major purposes that an evaluationmight address are the following:

To find out if there is a need for theprogram

To determine if the goals andobjectives of a program have beenmet

To decide if the ways in which aprogram operates are adequate

To see if a program operates in theway it was intended

To determine the outcomes of aprogram

Finally, a decision must be made regard-ing the extent of the evaluation. Is it toinvolve all the vocational programs in aneducational organization? Is it to beselective and address only vocational pro-grams that are experiencing problems inreaching their objectives or are notachieving the desired level of performancestandards? Should information be col-lected from all the students? Should

information be collected only from at-riskstudents?

The decisions regarding the type, purpose,and extent of evaluation are very impor-tant if useful and objective evaluations areto be designed and conducted. A designteam of seven to nine individuals repre-senting local advisory groups, teachers,and administrators can be used in makingthese decisions. The majority of this teamshould be composed of people outside theeducational organization if the evaluationis to be viewed as a credible assessment.

Select the Componentsto Be Included

Once the type, purpose, and extent of theevaluation have been decided, the com-ponents of the evaluation must be chosenfrom the framework. The current empha-sis on documenting the results of educa-tional programs suggest that the compon-ents of vocational education outcomesprovide an appropriate place to start.

The Carl D. Perkins Vocational andApplied Technology Education Act of1990 states that all vocational educationprograms are to evaluate their programsusing measures of learning and compe-tency gains including student progress inthe achievement of basic and moreadvanced academic skills. The componentin the framework that addresses this areais academic skills. The Perkins Act alsocalls for one or more measures of perfor-mance from outcomes such as occupation-al skills, retention in school, placementinto additional training or education, andplacement into the military service oremployment. These outcomes are includ-ed in the framework as occupational skills,school attendance rates, continuing educa-tion rates, training-related placement, andemployment rates.

24

Some states may have other types of out-comes that they wish to emphasize in theirvocational education programs. Localeducational agencies may also have thesame or other outcomes of interest tothem. All these factors need to be con-sidered when selecting the type of out-comes used to focus the evaluation ofvocational education programs

The next step is to select componentsfrom the framework that relate to voca-tional education processes and needs forvocational education. These componentsshould be those that most likely influencethe vocational education outcomes thatwere selected. This information will beneeded to help explain or interpret thefindings f the evaluation.

It should not be surprising to fmd thatthere are more components of the evalua-tion than can be included. Therefore,these components must be placed in orderof priority based on the type and purposeof evaluation that was selected in theprevious step. Once this priority has beenestablished, the selection can be based onthe resources available.

Identify the Sources and Types ofInformation to Be Collected

Once the components to be included inthe evaluation are selected, the type ofdata or information that it will b.: neces-sary to collect must be identified. A sam-ple listing of potential sources of informa-tion for each component is presented infigure 3. A wide range of potentialsources of informdtionboth qualitativeand quantitativeexists. Whenever possi-ble it is desirable to obtain more than onetype of information about a component tobe included in the evaluation of vocationaleducation programs.

If") -

,

Component

Academic Skills

Occupational Skills

Educational Competenceof Staff

Potential Sources of Information

Demographic Chart teristicsof Students

Goals and Objectives ofProgram

Instructional Content

Instructional DeliveryMethods

Teaching Styles

Educational Achievement by Grade and Age on the IoTests of Basic Skills or California Achievement Tests

a

National Occupational Competency Tests for anoccupation

College TranscriptsNational Teachers Examination

Individual Student RecordsInterviews with Students

Course of StudyInterviews with Teachers

Lesson PlansObservation of Classes

Observation of ClassesInterviews with Teachers

Observation of TeachersSurvey of Teaching Styles

Figure 3. A sample listing of potential sources of evaluation information

Collect the Required Information

Collecting information is an importantstep in conducting an evaluation of voca-tional education programs. This informa-tion will provide the evidence to be usedin judging the value of the program. Re-member that the evaluation will only beas good as the quality of information col-lected. According to Worthen andSanders (1987)

information is situational andchanges with every evaluation.Similarly, the methods evaluatorsmust use to collect information

25

',I '2

also must change. Sometimesalternative data-collection meth-ods are available, and the evalua-tor must make a choice, consider-ing cost, precision (reliability),stability, relevance, validity ofmeasurements, feasibility, politicaladvisability, and acceptability tovarious audiences. The evaluatormust be ready to use whichevermethod or methods appear mostappropriate. And "being pre-pared" usually means having priorexperience. We advise evaluatorsto gain experience with any meth-ods that may be new to them--

and that means actually using themethods, not just reading aboutthem. (pp. 298-299)

Usually it is necessary to establish agree-ments with those whose assistance will beneeded in collecting evaluation informa-tion. Large schools often have very de-tailed procedures that must be followedIn these efforts. Approval for accessingthe required information may be needed.

Analyze and Interpret theInformation Collected

The process o: analyzing and interpretingthe information collected is also animportant task. It is likely that evaluationefforts will collect a large amount ofinformation. Somehow this informationmust be assembled cohesively so thatothers can comprehend the fmdings.

Analysis of data is one way that informa-tion can be reduced to a meaningful level.Often, measures of central tendency (thatis, medians, means, modes) are used tosummarize quantitative data. Othertimes, ranges and correlations may needto be used to indicate the diversity orsimilarity of data. It may even benecessary so use statistics such as chi-square, t-tests, and analysis of variance tointerpret the information.

The analysis of qualitative informationrequires the evaluator to be involved inan ongoing process of observation andreflection, drawing conclusions and testingthese conclusions as additional informa-

26

tion becomes available. Miles and Huber-man (1984) provided an excellent discus-sion regarding the analysis of qualitativedata.

Prepare an Evaluation Report

The results of the evaluation need to bereported to the stakeholders (those direct-ly affected by the evaluation results) ofthe vocational education program. Thesereports can be used for a variety of pur-poses besides determining the worth of aprogram. Other uses include image build-ing, education, and public relations. Be-sides considering the purpose an evalua-tion report is to serve, evaluators need toconsider the audience for the report. Thepurpose and the intended audience willdictate the form of the evaluation report.Worthen and Sanders (1987) suggestedthe following types of evaluation reports:written reports, photo essays, audiotapedreports, slide-tape presentations, filmed orvideotaped reports, multimedia presenta-tions, dialogues/testimonies, hearings ormock trials, product displays, simulations,scenarios, portrayals, case studies, graphsand charts, test score summaries, andquestions/answers (pp. 344-345).

Concluding Remark

This paper is not intended to be a com-plete handbook for conducting evalua-tions. Handbooks and guidelines of thistype are available by such authors asBrinkerhoff et al. (1983a,b), Herman(1989), and Worthen and Sanders (1987).

REFERENCES

Items with ED numbers may be ordered from the ERIC Document Reproduction Service,CBIS Federal, 7420 Fullerton Road, Suite 110, Springfield, VA 22153-2852,(800) 443-3742, (703) 404-1400.

Bloom, B. S., ed. Taxcnomy of Educational Objectives: Cognitive Domain. New York:David McKay Company, Inc., 1956.

Brinkerhoff, R. O.; Brethower, D. M.; Hluchyj, T.; and Nowakowski, J. R. ProgramEvaluation: A Practitioner's Guide for Trabwrs and Educators. Design Manual.Boston: Kluwer-Nijhoff Publishing, 1983a.

Brinkerhoff, R. O.; Brethower, D. M.; Hluchyj, T.; and Nowakowski, J. R. ProgramEvaluation: A Practitioner's Guide for Trainers and Educators. Sourcebook. Boston:Kluwer-Nijhoff Publishing, 1983b.

Bush, G. H. W. "Concluding Remarks." Paper presented to the Education Summit at theUniversity of Virginia, September 28, 1989.

Carl D. Perkins Vocational Education Act. Public Law 98-524. An Act to Amend theVocational Education Act of 1963. Washington, DC: U.S. Government PrintingOf. ;ce, 1984. (ERIC Document Reproduction Service No. ED 256 926).

Cetron, M., and Davies, 0. American Renaissance: Our L4e at the Turn of the 21st Century.New York: St. Martin's Press, 1989.

Elam, S. M. 'The 22nd Annual Gallup Poll of the Public's Attitudes toward the PublicSchools." Phi Delta Kappan 72, no.1 (September 1990): 41-55. (ERIC No. EJ 413175).

Gibson, J. L.; Ivancevich, J. M.; and Donnelly, J. H., Jr. Organizations: Behavior, Structure,Processes. 3d ed. Dallas: Business Publications, Inc., 1979.

Herman, J. L., ed. Program Evaluation lat. 2d ed. Newbury Park, CA: Sage Publications,Inc., 1989.

Iacocca, L. A., and Novak, W. Iacocca: An Autobiography. Boston: G. K. Hall, 1987.

Kaufman, R. A., and Stone, B. Planning for Organizational Success: A Practical Guide.New York: Wiley, 1983.

27

:1 t

Kearns, D. T., and Doyle, D. P. Winning the Brain Race: A Bold Plan to Make Our SchoolsCompetitive. San Francisco: ICS Press, 1988.

Laughlin, S. "After High School--What Next?" In Facts & Findings, Vol. 4, No. 12.Columbus: The National Center for Research in Vocational Education, The OhioState University, Fall 1986.

Miles, M. B., and Huberman, A. M. Qualitative Data Analysis: A Sourcebook of NewMethods. Beverly Hills, CA: Sage Publications, Inc., 1984.

National Commission on Excellence in Education. A Nation al Risk- The Imperative forEducational Reform. Washington, DC: NCEE, 1983. (ERIC Document Reproduc-tion Service No. ED 226 0615).

Nat'onal Governors' Association. Educating America: State Strategies for Achieving theNational Education Goals. Washington, DC: NGA, 1990.

Ohio Department of Education. Ohio's Future at Work. Columbus: Division of Vocationaland Career Education, Ohio Department of Education, 1990. (ERIC DocumentReproduction Service No. ED 323 407).

Peters, T. J ., and Waterman, R. H., Jr. In Search of Excelknce: Lessons from America'sBest-Run Companies. New York: Harper & Row, Publishers, 1982.

Rossi, P. H., and Freeman, H. E. Evaluation: A Systematic Approach. 4th ed. NewbucyPark, CA: Sage Publication, Inc., 1989.

Scriven, M. 'The Methodology of Evaluation. In Curriculurn Evaluation. AmericanEducational Research Association Monograph Series on Evaluation, no. 1, edited byR. E. Stake. Chicago: Rand McNally, 1967.

Sculley, I, and Byrne, J. A. Ody.ssey: Pepsi to Apple. New York: Harper-Row, 1987.

Stufflebeam, D. L. "Evaluation as Enlightenment for Decision Making." In ImprovingEducational Assessment and an Inventory of Affective Behavior, edited by W. H. Beattyand A. B. Walcott. Washington, DC: Association for Supervision and CurriculumDevelopment, 1969. (ERIC Document Reproduction S.rvice No. ED 034 730).

Stufflebeam, D. L 'The Relevance of the CIPP Evaluation Model for EducationalAccountability." Journal of Research and Developmznt in Education 5, rx 1 (Febniary1971): 19-25. (ERIC No. EJ 048 749).

Stufflebeam, D. L. 'The CIPP Model for Program Evaluation." In Evaluation Models:Viewpoints on Educational and Human Services Evaluation, edited by G. F. Madaus,M. Scriven, and D. L. Stufflebeam. Boston: Kluwer-Nijhoff, 1983.

Stufflebeam, D. L., and Shinkfield, A. J. Systenuctic Evaluation. Boston: Kluwer-Nijhoff,1985.

28

0 3 E

Went ling, Tim L Evaluating Ocatpational Education and Training Programs. 2d ed.Boston: Allyn and Bacon, Inc., 1980.

The William T. Grant Foundation Commission on Work, Family and Citizenship. TheForgotten Half Pathways to Success for America's Youth and Young Families.Washington, DC: The William T. Grant Foundation, November 1988. (ERICDocument Reproduction Service No. ED 300 580).

Worthen, B. R., and Sanders, .I. R. Educational Evaluation: Alternative Approaches andPractical Guidelines. New York: Longman, 1987.

/

29

93 C

THE 1990 ERIC INFORMATION ANALYSIS PAPERS

IN 342 $8.75

Family and Intergenerational literacy Program&An Update of Me Noises of Literacy," by Ruth S.Nickse.

Family and intergenerational literacy programsrepresent one promising approach to the problem ofilliteracy, combining the agendas of adult basic skillsimprovemnt and children's literacy development.Nickse depicts the context of family/intergenera-tional literacy, examines an interdisciplinary re-search base, and describes existing programs. Atypology of four generic program models with theiradvantages and disadvantages is developed, and 12programs classified using the typology are detailed.

IN 343 $5.25

Articulation Models for Vocational Education, byMary Robertson-Smith.

The vital role of articulation in vocational educationtoday is highlighted in this review of the process ofcoordinating curricula at different levels of educa-tion. Types of interschool articulation examinedare tech-prep, cooperative education, proprietaryschools, retention of at-risk students, and college-to-college arrangements. Nonschool partnershipsieviewed include those with business and industry,government agencies, and the military.

IN 344 $4.75

A Framework for Evaluating Local Vocational Edu-cation Programs, by N. L. McCaslin.

Increased emphasis on evaluation in educational re-form movements necessitates a comprehensive, sys-tematic approach to assessment. The frameworkfor vocational eduation evaluation presented in thispaper draws information from the needs for voca-tional education of students, employers, and society;from program processes; and from program out-comes. Steps for implementing the evaluationframework in local programs are outlined

IN 345 $10.50

Competency-Based Career Development Strategiesand the National Career Development Guidelines,by Howard Splete and Amy Stewart.

Splete and Stewart give an overview of the NationalCareer Development Guidelines, which they use asa basis for analyzing over 1,500 ERIC abstracts.The abstracts are coded by National Guidelinesareas and competencies as well as educational level(elementary, middle/junior high, high school, 2-year community college, 4-year college/university,business/industry, and community agency). Samplecareer development activities relatato the NationalGaidelines are provided for each level, with recom-mendations on using the competencies for thegrowth and revitalization of career developmentprograms.

ORDERING INSTRUCTIONS

To order additional copies, please use order num-ber and title. Orders of $10.00 or less should beprepaid. Make remittance payable to the Centeron Education and Training for Employment. Mailorder to:

Center on Education and Training forEmployment

Center Publications1900 Kenny RoadColumbus, OH 43210-1090

Prices listed are in effect at the time of publica-tion of this book. All prices include postage andhandling Prices are subject to change withoutnotice.

Quantity DiscountsOrders of five (5) or more items, as listed bypublication number and title, with a total dollarvalue for the order of:

$ 50 to $100, the discount is 5%$101 to $200, the discount is 10%$201 to $300, the discount is 15%$301 to $400, the discount is 20%$401 and above, the discount is 25%

Intei national OrdersAll orders, in any amount, from outside theUnited States and its possessions are to bepaid in U.S. currency. Additional postage andhandling charges may be added for foreignshipments if necessary.

ENDU.S. Dept. of Education

Office of EducationalResearch and improvement (OERI)

ERICDate Filmed

July 17, 1991