ED 323 663

AUTHORTITLEINSTITUTION

SPONS AGENCY

PUB DATECONTRACTNOTEPUB TYPE

EDRS PRICEDESCRIPTORS

IDENTIFIERS

ABSTRACT

DOCUMENT RESUME

CE 056 237

Campbell, Paul B.; Panzano, PhyllisElements of Vocational Program Quality.Ohio State Univ., Columbus. National Center forResearch in Vocational Education.Office of Vocational and Adult Education (ED),Washington, DC.85

300-83-001647p.

Reports - Research/Technical (143)

MF01/PCO2 Plus Postage.*Educational Assessment; Educational Objectives;*Educational Quality; Evaluation Criteria; Models;*Outcomes of Educatio1; *Program Effectiveness;*Program Evaluation; *Vocational Education*Quality Indicators

This report is an attempt to conceptualize acomprehensive framework for evaluating vocational education quality.It is intended to inform policymakers of the breadth of the elementsof quality and to suggest an integrated framework for assessingquality to vocational education evaluation theoreticians. Chapter 1offers preliminary considerations. Chapter 2 introduces and definesthe elements of quality related to educational outcomes that wereidentified through the model-building process described in Chapter 1.Two general types of outcomes are included: individual and societal,with a subset of societal outcomes specific to employers. Chapter 3parallels chapter 2 in structure, but focuses on the elements ofquality related to the educational experience. The educationalexperience is divided into two domains: school context and schoolprocess. Chapter 4 presents some issues and guidelines regarding thedevelopment of quality indicators for the elements within theproposed model. General evaluation approaches and specific models arediscussed as they relate to developing indicators for the elements ofquality in vocational education. Thirty-two references are appended.(YLB)

Reproductions supplied by FDRS are the best that can be madefrom the original document.

61"

fte,

,

-rol,,1,1.0,4:,

THE NATIONAL CENTER MISSION STATEMENT

The National Center for Research in Vocational Education's mission

is to increase the ability of diverse agencies, institutions, and

organizations to solve educational problems relating to individual

career planning, preparation, and progression. The National Center

fulfills its mission by:

Generating knowledge through research

Develop;ng educational programs and products

Evaluating individual program needs and outcomes

Providing information for national planning and policy

Instaliing educational programs and products

Operating information systems and services

Conducting leadership development and training programs

For further information contact:

Program Information OfficeNational Center for Research

in Vocational EducationThe Ohio State University1960 Kenny Road

Columbus, Ohio 43210

Telephone: (614) 486-3655 or (800) 848-4815Cable: CTVOCKDOSU/Columbus, OhioTelex: 8104821894

ELEMENTS OF VOCATIONALPROGRAM QUALITY

Paul B. CampbellPhyllis Panzano

The National Center for Research in Vocational EducationTha Ohio State University

1990 Kenny RoadColumbus, Ohio 43210-1090

1995

FUNDING INFORMATION

Profect Tide: National Center for Research in Vocational Education, Informationfor Planning and Policy Development

Contract Number 300830016

Project Number: 0510C40060/0510C40061

Act under Which Education Amendments of 1976,Funds Administered: P.L. 94-482

Source of Contract: Office of Vocational and Adult EducationU.S. Department of EducationWashington, DC 20202

Contractor The National Center for Researchin Vocational Education

The Ohio State UniversityColumbus, Ohio 43210-1090

Executive Director Robert E. Taylor

Disclaimer This publication was prepared pursuant to a contract with the Officeof Vocational and Adult Education, U.S. Department of Education.Contractors undertaking such projects under Government sponsor-ship are encouraged to express freely their judgment in professionaland technical matters. Points of view or opinions do not, therefore,necessarily represent official U.S. Department of Education positionof policy.

Discrimination Title VI of the Civil Rights Act of 1964 states: "No person in theProhibited: United States shall, on the ground of race, color, or national origin,

be excluded from participation in, be denied the benefits of, or besubjected to discrimination under any program or activity receivingFederal financial assistance." Title IX of the Education Amendmentsof 1972 states: "No person in the United States shall on the basis ofsex, be excluded from participation in, be denied the benefits of, orbe subjected to discrimination under any education program oractivity receiving Federal financial assistance." Therefore, theNational Center for Research in Vocational Education Project. likeevery program or activity receiving financial assistance from theU S. Department of Education, must be operated in compliance withthese laws.

TABLE OF CONTENTS

LIST OF FIGURES

FOREWORD

EXECUTIVE SUMMARY

CHAPTER 1. INTRODUCTION 1

Some Preliminary Considerations 1

Conceptual Framework 3

Organization of This Publication 6

CHAPTER 2. EDUCATIONAL OUTCOMES OR GU-4LS 7

Individual Outcomes 7

Societal Outcomes 9The Place of Vocational Education 12

CHAPTER 3. THE EDUCATIONAL EXPERIENCE 13

General Definition of the Educational Experience . 13

Elements oi School Context 15

Elements of School Process 19

CHAPTER 4. ASSESSING QUALITY OF VOCATIONAL EDUCATION 23

Introduction 23

What Is Quality? 23Approaches to Assessing Quality 24

Developing Indicators of Quality 26

Summary 98

The Framework: Elements of Quality Vocational Education 30Assessing the Quality of Vocational Education 31

REFERENCES 33

iii

8

LIST OF FIGURES

Figure

1. Framework identifying the elements of vocational education quality 4

2. Elements of quality of school context 16

3. Elements of quality of school process 20

4. Examples of quality indicators 29

FOREWORD

Traditionally, vocational education program quality has been defined (1) in terms of the outputof the program (e.g , training-related jobs obtained by former students, satisfied employers) or(2) as the process of the program (e.g., time spent on learning tasks, skill content addressed) Bothof these are logically related to quality, but neither one by itself provides an explanation of qualitythat is sufficient to influence policy toward quality improvement.

This publication provides a discussion of the elements of quality in vocational education asviewed in the context of formal education. Reviews of these elements by knowledgeable practi-tioners and recommendations related to the development of quality indicators are also included It

is an attempt to conceptualize a comprehensive framework for evaluating vocational educationquality. The report is intended to inform policymakers of the breadth of the elements of quality andto suggest an integrated framework for assessing quality to vocational education evaluation

theoreticians.

This project was conducted in the Evaluation and Policy Division of the National Center forResearch in Vocational Education urder the direction of N L. McCaslin, Associate Director. Many

people made significant contributions in the course of its completion. We wish to thank for theirparticipation in our field discussions Dr. Nellie Dry, Dr. George Mehallis, Charlotte Hayes, andDavid Hall, associated with Broward Community College in Ft. Lauderdale, Florida; John Ryan,

Ron Smith, Joe Fabing, and Billy Keith Butt, associated with Tri-County Joint Vocational School inNelsonville, Ohio; Dr. Dwight Davis, Marvin Bausman, Dan Sczygelski, Larry Post, and CindyPogachnick, associated with North Central Technical Institute in Wausau, Wisconsin; Dr. JimFrasier, Nelson Whitemen, Harold Darling, Harold Pearson, and Beth Allen, associated with Patter-sun Cooperative High School in Dayton, Ohio; and Ray Bertleson, John Miller, Marilyn Gordon,and Matt Mediwk, associated with Westerville South High Scnool in Westerville, Ohio.

We also acknowledge the contributions of National Center staff members Roy Butler, IdaHalm, N. L. McCaslin, Allen Wiant, and Jane Reisman and the field work of Cathy Warmbrod andJim Hamilton, which was instrumental in the development of our model.

David L. Passmore, professor of vocational education, The Pennsylvania State University,George Quarles, former Director of Vocatic.nal Education for New York City, and Robert H.

McCabe, president, Miami-Dade Community College, provided helpful criticisms and suggestionsrelated to the overall context and outcomes of our conceptual framework

We wish also to thank the project staffPaul Campbell, project director, Jim Hamilton, CathyWarmbrod, and Phyllis Panzano for their work in preparing the publication. And finally, we thankJeanne Desy for her comments related to the organization of the report, Mary Beth Dauner for hereffort, cooperation, and patience in typing the manuscript with its many revisions, and Janet

Kip linger, under whose supervision the publication was edited

Robert E. TaylorExecutive DirectorNational Center for Research

in Vocational Education

VII

s

AMR"

EXECUTIVE SUMMARY

The elements of quality proposed in this publication have been deri, ed from a diverse body ofliterature and experience. We have attempted to incorporate various findings related to attributesof quality education and programs into a general yet comprehensive conceptual framework inorder to broaden our understanding of the elements of quality vocational education. It will con-tribute to an increased awareness of the major components influencing the quality of education.This publication is intended to inform policymakers of the breadth of the elements of quality andto suggest an integrated framework for assessing quality to policymakers and vocational educa-tion evaluators.

For purposes of this effort vocational education was defined as school-related activities thatoccur in secondary schools and 1- and 2-year programs in postsecondary schools; vocationaleducation was considered to be a subset of general, formal education, which in turn is a subset ofeducation coming from a variety of life experiences.

Our framework for the elements of quality grew out of a multiphased process that includedreviews of related literature, expert input, and key informant interviews. The literature providedpreliminary structure to the framework that was further refined and defined by expert opinion andkey informant commentaries.

Some vocational education has been directed by law to be evaluated in terms that include jobplacement in training-related employment, employer satisfaction, and quality and availability ofinstructional offerings. This effort, however, initially explored a variety of desirable outcomes thatmay help to define high quality vocational education programs and subsequently identified thecomponents of the educational environment that may support attainment of those outcomes. Theoutcomes were therefore considered as goals and the structures and processes within the educa-tional environment were perceived as means for achieving the goals. The resulting framework forthe elements of vocational education quality (figure 1) was conceptualized as a complex environ-ment converging on the educational experience, driven by desired individual and societal out-comes, and influenced by environmental inputs such as community setting and labor marketconditions.

The Framework: Elements of QualityVocational Education

Elements of Quality Outcome

Two general types of outcomes are included in our framework individual and societal with asubset of societal outcomes specific to employers. Four elements of quality relating to individualoutcomes were identified and defined Competency development encompasses the development

ix

of work-rehsited skills (both occupational and job specific), basic skills such as verbal and mathe-matical, and general problem-solving ability. Positive interpersonal interaction involves the abilityof people to have positive and mutually supportive relations with others in the home, the work-place, and the community. A sense of efficacy is an attitude of confidence in one's ability to utilizeskills to achieve goals. The ability to manage a large array of tasks without having to depend uponsomeone else is designated as a sense of independence These are the four general elements ofquality individual outcomes included within our conceptual framework

Social stability and emp/oyer needs are the primary elements of quality within the societaloutcome portion of our framework Social stability does not imply social stagnation, rather, itmeans gradual, orderly change Provision of essential services, quality of work products, resourcesfor self-support, and societal improvement are the subelements that support social stability

We also examined a subset of societal outcomes specific to employers To be worth theinvestment from the employer's point of view, vocational education should contribute to availableskills, reduced training time, and high productivity These are the three elements of quality relatedto employers needs Our definition of quality vocational education outcomes includes, but is notlimited to, employment-related skills. Rather, vocational education of a positive quality must alsoenhance the development of desirable or even essential skills or dispositions that are not directlyemployment related

Elements of Quality withinthe Educational Experience

Taking a systems perspective, the processes and structures that define the elements of quahtywithin the vocational education experience should relate to intended individual and societal out-comes. Our mok el divides the educational experience into two domains, school context andschoo/ process. School context deals with both physical and organizational aspects of the schoolunit Personnel, facilities, equipment, policies, and formal organizational structure all fall withinschool context. The elements of quality within school context that are formally included in ourmodel are school setting, classroom setting, teacher skills, teacher characteristic* and peer groupcharacteristics. Figuratively speaking, school context provides the setting for soh, process,which provides the "action Estimations of quality of school process would consider such thingsas course content: course-specific equipment; interaction between teacher, student, and materials.and level of formal student involvement in the vocational education program The elements of qual-ity formally included within school process are teacher, student, and material interaction; instruc-tional content and material; and student utilization of instructional process

These elements of quality within the educational experience should relate to intermediateobjectives that, in turr -`lould support attainment of desired outcomes, the linkage should be sys-tematic between quality of experience and quality of outcome in order to maintain and replicateeffective programs and make modifications as necessary.

Assessing the Ouality of Vocational Education

The elements of quality within our framework are generalizable across school unit and pro-gram However, the specific indicators of quality associated with them may vary across school andprogram and are likely to vary over time

x

Considerable effort has been expended in developing evaluative strategies to facilitate theassessment of quality. Evaluation models should combine structural, process, and outcomeapprotiches in ordur to assess the range of elements of quality within our framework

Specific indicators of quality need to be developed at the individual school unit level It is notpractically possible to tabulate exhaustively all the indicators related to the elements of our modei,nor is it financially practical to support an evaluation system capable of gathering and analyzingsuch dat. in a complete form A process for identifying and prioritizing subelements within themodel is suggested as a practical starting point for developing indicators A combination of quan-titative and qualitative methods is suited to a comprehensive evaluation of quality, and bothnormative and empirical approaches should be drawn upon for developing indicators

In summary, this discussion presents a justification and set of procedures that will permit anintegrated consideration of the elements of vocational education quality We have advocated con-sidering a broad set of elements of quality rather than a narrowly defined or segregated set, so thatthe evaluation of vocationa: education reflects its contribution to a set of living experiences inaddition to employment

XI

1 1

CHAPTER 1

INTRODUCTION

Some Preliminary Considerations

WhitA Is Thls Publication About?

When determining whether or not a pro-gram of instruction is producing the desiredoutcomes, it is first necessary to decide whatthose outcomes should be. Vocational educa-tion has been directed by law to be evaluatedin terms of job placement in training-relatedemployment and employer satisfaction.Although both of these outcomes are reason-able as well as legally required, there is alarge body of opinion that extends thedesired outcomes beyond that limited set.This publication advocates such an expan-sion for the purpose of evaluation, explores avariety of potentially desirable outcomes thatmay help to define a high-quality vocation&education program and identifies thosecomponents of the educational experiencethat may support attainment of thoseouicomes.

The elements of vocational educathnquality may be found in two placesthe out-comes of the educational experience and theeducational experience itself. In this discus-sion we present a tentative set of outcomegoals, with some description of their natureand justification for their selection. The justi-fication relies upon normative premises,empirical and theoretical evidence, and, insome cases, intuitive logic. Because the pro-cesses and structures that presumably definethe educational experience should relate tothe intended outcomes, the outcomes areconsidered first. A similar consideration ofthe processes will be the topic of a subse-quent chapter.

Wa are attempting to conceptualize ageneral yet comprehensive framework forassessing vocational education quality. It willcontribute to an increased awareness of the

1

breadth of the components that influence thequality of education.

What Is Vocational Education?

Before proceeding with a sear,:n tor ele-ments of quality in vocational education,some fences must be erected around thevaried terrain of ideas and activities to whichthe label "vocational education" is attached.From a narrow point of view, vocational edu-cation is considered by some as only thoseschool-related activities that are the recip-ients of Federal funds under the vocationaleducation acts beginning with the Smith-Hughes Act in 1917. On the other hand, itmay be considered as any school-relatedactivity, through graduate and professionalschools, that provides skills for employmentin a specific fieldfrom machining to medi-cine, or from cooking to counseling. For pur-poses of this inquiry, we have elected to limitconsideration to those school-related activi-ties that take place in secondary schools andin 1- or 2-year programs in postsecondaryschools. These school activities that we clas-sify as vocational oducation includebut aswe intend to show subsequentlyare notlimited to programs that produce salableskills specific to general classes of occupa-tion. They may or may not be supported byFederal funding.

Where Does Vocational Education Fit?

We consider vocational education to be asubset of general, formal education, which, inturn, is a subset of education coming from awiee variety of life experiences. At thesecondary level, it is part of a universal andobligatory educational experience. It is uni-versal in most Westero democracies because

12

provision is made in the laws al these Statesto provide it for nearly everyone. The mostusual exception occurs where someextremely severe mental or physical problemrenders fOrmal education impractical. Educa-tion is an obligation because most of theseStatee require some level of compulsoryattendance.

At the postsecondary level, the obliga-tory feature of education is completelyrelaxed. The univevsal quality is also muchleas apparent, although the general availabil-ity of public funds to support postsecondaryinstitutions, at least in the United States,reflects a commitment to broaden the accessof individuals to continued education.

What Should Se the Outcome of General,Formal education?

To understand where vocational educa-tion fits within this scope of general. formaleducation, Is is necessary to examine whatthe outcomes of general, formal educationare expected to be. Frankena (1974) de-scribes these outcomes as the formation ofwhat he calls "dispositions" that include abili-ties, habits, beliefs. skills, and bodies ofknowledge.

It in beyond the scope of this discussionto elaborate upon the basic normative prem-ises underlying the selection of the disposi-tions that seem to be most universally valuedin American society. Rather, those that havebroad repres-ntation in the literature, in themedia, and in practice suggest what appearsto be the society's underlying values. If weconsider both practice and commentaries,the currently existing values, as well as thosedesired, have a chance to emerge.

There seems to be general agreementthat speaking, reading, and writing the major-ity language, as well as handling basic quan-titative problem solving, are fundamentalelements of the dispositions. A Jest somelevel of comprehension of the procedures ofscience and the political organization of the

2

Nation, State, and community are alsogenerally accepted. Some National reports(e.g., by the National Commission on Excel-lence, The Twentieth Century Fund) haveadditionally emphasized such other disposi-tions as computer literacy and foreignlanguages.

That acquisition of the foregoing disposi-tions (e.g., language. math) is necessary for agood and productive life seems to be sup-ported by the observable, normative prem-ises. There is no general agreement abouthow all of these basic skills should be pro-vided, but the presence of some elements oflanguage, government, science, and mathcurriculum in virtually all high schoolsunderscores the importance attached to for-mal study of these skills by American society.

All societies are made up of individualsand of institutions that, at the time of theirfounding, were intended to provide someform of service to individuals. (This purposemay have been lost to self-perpetuation asthe institutions developed.) Americansociety, in which the form of vocational edu-cation being examined is set, has a longtradition of emphasis upon the individual.Illustrative are the writings of ThomasJefferson on the need for an educated citi-zenry, Abraham Lincoln's eloquent referenceto "government for the people, by the people,and of the people" in the Gettysburg address,and John Dewey's concern for the role of theindividual, expressed in the following Quote.

Such societies were found to be demo-cratic in quality, because of the greaterfreedom allowed the constituentmembers, and the conscious need ofsecuring in individuals a consciouslysocialized interest, instead of trustingmainly to the force of customs operatingunder the control of a superior class. Thesort of education appropriate to thedevelopment of a democratic communitywas then explicitly taken as the criterionof the further, more detailed analysis ofeducation. (Dewey 1916, p 322)

13

In this socitny, education is focused uponindividuals and seems to have two basic.goalsto help individuals maximize theirwell-being in a broad sense and to provide astability in the society through ensuring thewell-being of its members collectively.

An argument may also be made thatwork is necessary for the good and produc-tive life. As John Gardner (1961) puts it, "Thebest-kept secret in America today is thatpeople would rather work hard for somethingthey believe in than enjoy a pampered idle-ness." Although the context of Gardner'sremark is the need to invest human energy inimproving society in general, it applies tovocational education, which can and doesaddress directly a specific inclination to workthat serves ooth societal and individual ends.One can easily recall numerous expressionsof the desire to work ranging from expecta-tions to continue working expressed by lot-tery winners, through the search for part-timejobs by retired persons, to the pride withwhich workers display tangible results oftheir efforts. Vocational education providesspecific training that enables people to work,and thus addresses the individual's needrepresented by these expressions as well associety's need for the outcomes of an indi-vidual's work.

Vocational education cannot be pursued.however, at the expense of some acceptablelevel of the broad basic dispositions thatwere detailed earlier. That these are generallyvalued is evidenced by their prevalence andalso by their support by National soundingboards. To quote Dewey (1916) again.

Put in concrete terms, there is dangerthat vocational education will be inter-preted in theory and practice as tradeeducation: as a means of securing tech-nical efficiency in specialized futurerursuits.

Education weuld then become aninstrument ot perpetuating unchangedthe existing industrial order of society,instead of operating as a means of its

3

transformation. The desired transforma-tion is not difficult to define in a formalway. It aignifies a society in which everyperson shall be occupied in somethingwhich makes the lives of others betterworth living. (p. 316)

The National Commission on SecondaryVocational Education seconds Dewey'sconcern"Vocational education should notdilute the student's general education nordisenfranchise certain groups through selec-tive enrollment" (1984, p. 11). Therefore,vocational education as we have defined It ispart of the broader concept of education withits general goal of individual and societalwell-being. It shares with general, formaleducation the dispositions that are presumedto lead toward these two aspects lfwell-being.

Conceptual Framework

Before going further, it may be useful toconsider the overall setting in which voca-tional education operates. Figure 1 has beenprepared to illustrate one schematic of theconceptual framework. Briefly, figure 1 dis-plays three critical components of the overallsetting. The left-hand panel displays themajor dimensions of the context. These areconditions that exist independently of, and inmost cases, prior to, participation in voca-tional education. Although the figure doesnot make it explicit, the general context con-tinues to operate beyond formal educationand influences independently the effective-ness of the outcomes in accomplishing thegoals that underlie a conception of qualiti.The central panel displays the major dimen-sions of the vocational education setting.Three categories comprise this panel. Theyare the school-specific context, the teaching-learning process, and the formal utilization ofthat process by the student. The right-handpanel displays a set of outcomes. These aredivided into two major categories, individualand societal.

The outcomes panel is discussed first inthis report. Discussion of the school setting.

14

EDUCATIONAL VOCATIONALCONTEXT EXPERIENCE

Family Background

Personal Characteristics

Community Setting

1--Labor Market Conditions

School Context

School SettingClassroom Setting

Teacher SkillsTeacher Characteristics

Peer Group Characteristics

IStudent Utilization

of Instructional Process(Patterns of Participation)

i

Teacher, Student, andMaterial Interaction

A

Instructional Contentand Equipment

School Process

OUTCOMES

Ind ividual

Competency developmentBasic skillsInformation skillsProblem-solving skillsEmployment skillsJob-specific skillsCareer awareness

Positive interpersonal interactionsSense of efficacySense of independence

EnrichmentSelf-helping skills

Societal

Social stabilityEssential servicesQuality of work productsResources for self-supportSocietal improvement

Employer needsAvailable skillsReduced training timeHigh productivity

Figure 1. Framework identifying the elements of vocational education quality

15

teaching-learning process, and student utili-zation will follow. The overall approach canbest be understood, however, if the concep-tual framework is kept in mind.

The approach taken here assumes afunctional role for education, rather than asorting or conditioning role (e.g., Bowles andGintis 1976) or a role of securing a privilegedposition in the job queue (Thurow 1975).Human capital theory might be regarded asthe basis for this approach, but we take theposition that rational decisions in investingfor desired returns are not sufficient to cap-ture the overall value and meaning of humanexistence. Thus, although our discussion willbe in many respects consistent with humancapital conceptions, we do not limit ourselvesto expressing the values we have acceptednor the outcomes we discern as having nor-mative value, within the confines of that theo-retical perspective. In other words, we acceptan altruism that values human existence in itsown right, independent of economicinterpretations.

We also recognize that the institutionsdelivering formal education reflect the con-flicting forces and differences of opinion thatpermeate the society. They well may serve asz sorting mechanism, in part a function oftheir time-bound nature and the imperfectprofessionals who structure the experiencesfor students within them. The results ofacquiring the dispositions may be interpretedby employers as proxies for the ability tolearn new skills as yet unknown. As a conse-quence, noncredentialed workers may bedisplaced or not hired, even though theypossess the needed skills or ability to learn.These conditions reflect the imperfections ofthe enterprise that we seek to understand inorder to improve. We therefore consider theoutcomes as goals and the processes asways to move closer to the goals.

Modeling the Elements ofVocational Education Quality

The proposed framework for the ele-ments of vocational education quality was

5

the result of a multiphased process. A reviewof the literature, expert judgment, and keyinformant interviews provided input to themodeling process.

The literature on effective schoolresearch and related subjects was the initialsource considered in developing the model.Although the literature directly relating toand identifying the elements of vocationaleducation quality was quite segmented, ref-erences were discovered that addressed vir-tually all the major components of ourframework. The content of the referencesranged from general observations and com-ments about effective programs to sugges-tions for specific indicators of quality.Related references were subsequentlygrouped and the resulting significant clusterswere incorporated as model components orelements.

To illustrate briefly the method by whichreferences were utilized in model building,consider the following examples. Miller(1984) and McKinney et al. (1981) commentthat effective schools are characterized bytotal commitment of the school unit to qualityprograms. This reference could be generallyclassified as an organizational characteristicof school setting. More specifically, it couldbe cons;dered as an example of a generalpolicy statement that is common to qualityprograms. References were also found relat-ing to organizational structure (Westbrook1982) and strategy (Benson 1982). This typeof information was subsequently utilized tobuild a preliminary general framework ofschool setting that was structured as follows

Organizational AspectsstructurepOlicy

strategy

As more information was gathered, aphysical aspect of school setting was alsoidentified, structured, and added to themodel. The process was followed and even-tually resulted in a preliminary, systematicframework identifying the elements of voca-tional education quality

16

Experts in the field were involved inmodel definition through normative groupprocesses, modified Delphi approaches, andindividual review. Expert input was utilized tocross-validate components suggested by theliterature, to suggest additional componentsto our model, and to develop more detaileddefinitions of components.

The final source of information was keyinformant interviews. Nine administrators, sixinstructors/department heads, and six pro-gram graduates representing secondary andpostsecondary vocational education pro-grams participated in an unstructured inter-view process. Interviews were conducted asopen discussions on the topic of "QualityVocational Education." The role of the inter-viewers was to keep the discussion focusedon the general topic and to provide promptsor suggestions for related discussion.

In observance of Federal regulationsregarding over-collection of information, thesample size was intentionally kept small.Even though they were not technically repre-sentative of the population of interest, theinterviews were valuable to the model-formulating process. Interviews were contentanalyzed and informally used to cross-validate the basic model suggested by expert

6

judgment and the literature Interview infor-mation also aided in defining thecomponents and elements of quality and inproviding examples of potential indicators ofquality

In summary, our model was developedthrough a synthesis of the informationgathered from experts, practitioners, andrelated literature The model is proposed aszn attempt to define in an integrated fashionthe people. processes, and structures thatdefine the elements of vocational educationquality

Organization of This Publication

The remainder of this publication isdivided into three chapters. Chapter 2introduces and defines the elements of qual-ity related to educational outcomes that were1..:.,.ntified through the previously descrihedmodel-building process. Chapter 3 parallelschapter 2 in structure, but focuses on theelements of quality related to the educationalexperience. Finally, chapter 4 presents someissues and guidelines regarding the devel-opment of quality indicators for the elementswithin the proposed model

17

CHAPTER 2

EDUCATIONAL OUTCOMES OR GOALS

Two kinds of outcomes or go...Is are sug-gested in figure 1: individual and societal,with a subset of societal outcomes specific toemployers. They are not presented asexhaustive, but were selected to representthe key elements deemed necessary for aproductive and satisfying life.

Individual Outcomes

Within the parameters of individual out-comes are several components. The first ofthese is called competency development.Aspects of this outcome include the devel-opment of work-related skills, both occupa-tional and job specific. They include thedevelopmer t of the basic skills suggestedearlier in th's paper as being part of thegeneral right of individuals and obligation ofeducation. They also include the develop-ment of general problem-solving ability.

It should be understood that, as statedearlier, the development of the more recog-nizable vocational skills is not confined toformal education, either secondary or post-secondary: rather, both of these levels pro-vide avenues for their development. Theprocesses (methods) currently employed todevelop the necessary dispositions in thebasic skills, again using Frankena's terms(1974), are primarily those of formal educa-tion. This premise is supported by law, bypublished opinion, and by traditionalpractice.

Examples of the occupational and job-specific skills are such things as drafting,keyboarding, engine troubleshooting, pro-gramming, and crop planning. There are alsomore general, occupation- and job-relatedskills that include knowledge of the world of

work, job search strategies, interview pro-tocol, and work conventions (arriving ontime, following work rules). In most worksituations, some ability to communicateusing either spoken or written language isnecessary. In many, the solution of quantita-tive problems is another significant element.The general ability to make appropriatejudgments about the consequences of one'sactions has major importance. Also, a recog-nition of fundamental relationshipe that arethe stuff of science plays a part, whereasspecific understanding of certain chemical,physical, biological, and economic laws fig-ures in the world of work in many instancesFor example, failure to understand the role ofsoil nutrients leaves the development of acrop to happenstance. Or, the likelihood of abusiness failure is increased by not under-standing that when costs exceed revenues,the enterprise is bound to fail when its avail-able capital is depleted to a critical level.Competency development is judged to be akey component because it provides the indi-vidual with a currency to exchange forneeded goods and services, the necessarydispositions to accomplish self-servicesindependently of other people, and the abilityto add to the context of life in a positive waythrough contributions to society.

Another individual competency withincreasingly important potential might beidentified as "information skills." As thesociety changes from an industrial base to aninformation base, a great variety of newsources and kinds of information becomeimportantand as jobs change, these altera-tions frequently require accessing newinformation. The need for this skill is impliedwithin many of the basic skills, but it isworthy of emphasis as a separate and distinctskill because of its importance It includes

'This conception was contributed by one of our reviewers Robert McCabe, President Miami-Dade Community College

7

18

finding, evaluating, applying, and communi-cating information. It will apply not only tostability and growth within a vocation, butalso to other aspects of living as well.

The second component, positive inter-personal interaction, permeates not only thework role of individuals but also the total lifespace. Here individual fulfillment of nearly allpeople apends on having positive and mu-tually supportive relations with key peoplewhose lives mesh with theirs in thp t:ork-place, the home, and the la ger community.To do this, they need to be able to communi-cate, to be sensitive to the impact that theyhave on their colleagues, and to develop andmaintain a sense of shared responsibility.

A wide range of interpersonal interac-tions might be labeled positive. At the oneextreme are those people, some employersamong them, who want docile response torequests and directives. At the other arethose who demand individual assertivenesswithout regard for the consequences to oth-ers. We define positive, interpersonal actionsas being neither of these extremes but rathera set of behaviors involving other people.These behaviors are adjusted to the circum-stances, but the individuals do not acceptindefinitely repressive situations: they areresponsive to the needs of others, but notinsensitive to their own fulfillment. This pointof view is analogous to Sternberg's (1984)contextual subtheory of adaptation, selec-tion, and shaping. According to Sternberg'stheory, individuals first try to adapt to theirenvironments. If this is impossible or undesir-able, they try to select a new environment.Should that not work out, they try to adaptthe environment to make it a better fit. Posi-tive interpersonal relations involve activitiesof this kind. We do not assume, however, theorder implied here We do assume that thedisposition or behaviors required can bedeveloped, rather than being inherent.

A summary of research consistent witn adefinition of this sort is also found inTennyson et al. (1980). These authors con-sider both the work situation and other soc'al

8

settings They identify the functionality ofpositive interpersonal relations, which theycall "mature," in a variety of settings. In partic-ular, they imply that better problem solvingwill result from those mature relationshipsbecause less effort will be spent on defensivereactions and more on eliciting informationand developing solutions. These constructiveactions will lead to reduced conflict and lessoccurrence of defensive behavior.

A third individual outcome that we postu-late is a sense of efficacy. Stated simply, itimplies that individuals not only have someskills, but also sense their ability to use themto accomplish something. They expect to beable to get a job, to accomplish tasks deemedto be worthwhile, and to exercise a reason-able degree of control over their immediatepersonal environment.

Beginning with the postulations of KurtLewin that the likelihood of undertaking atask is related to an individual's perception ofthe likelihood of success, the role of a senseof efficacy in motivation has been detailed bynumerous researchers (For example, seeKelly 1967; Wong and Weiner 1981; Taylor1983.)

The theories developed by theseresearchers and the evidence they haveexamined strongly suggest the importance ofa sense of efficacy. In contrast, the conceptof learned helplessness (Abramson,Seligman, and Teesdale 1978) and of its con-sequences illustrates the debilitating effect ofthe opposite pole of the efficacy continuumAlso, there is empirical evidence that a senseof efficacy is associated with certain aspectsof job satisfaction. For example, a recentstudy of job satisfaction, utilizing theNational Longitudinal Survey Youth data(Campbell et al. 1982) found a consistentassociation between four different forms ofjob satisfaction and scores on a scale thatincluded items such as "I'm able to do thingsas well as most other people." These exam-ples of theoretical and empirical evidenceadd to a normative and humanistic concernfor the qualities of individual satisfaction to

13

support the inclusion of a sense of efficacyas a major individual outcome of formaleducation.

The fourth major outcome for individualsincludes to some degree the preceding three.It is designated as a sense of indeper,dence.It means that the individual can manage alarge array of ordinary tasks without havingto depend upon someone else. It means thata man can provide a balanced meal whenneeded and that a woman is not helplesswhen confronted with a flat tire. It means thatthe individual chooses among alternativesrather than being channeled along singlepathway of life episodes.

A sense of irdependence has a strongnormative background demonstrated in suchdiverse documents as the Declaration ofIndependence and the literature on therapeu-tic counseling. Its relationship to specificwork life as well as its role in the generalquality of life is suggested by a variety of evi-dence. It is reportedly related to job satisfac-tion among adults. For example, in a recentstudy, Hills and Crowley (1983) found that asense of independence, operationally definedas a sense of autonomy on the job, was sig-nificantly related to a scale score of job satis-faction. This result is a further verification ofRobert Blauner's classic study, Alienationand Freedom (1964). Katzell (1979) also pro-vides an analysis of the growing awarenessof the legitimacy of this sense among thecurrent work force. Although theserelationships do not establish a sense ofindependence as a goal, they document itspervasiveness as a human value, which, inour judgment, is the ultimate criterion forjustifying it as a goal. (See also Taylor (1974)for a discussion of the logical and philo-sophical basis for this contention.)

These four individual outcomes restupon a class of intermediate outcomes thatare not so much ends in themselves but arenecessary in some form in order to attain themore fundamental results. Examples are thedevelopment of self-helping skills (tire chang-ing, cooking, keyboarding), information

9

about alternatives (career awareness), andthe broadening of each individual's repertoireof knowledge (enrichment). Vocational edu-cation functions with respect to these out-comes in varied ways. In some cases, it maybe the sole, school-based source of oppor-tunity to achieve the outcome (cooking) Inothers, achieving the outcome may be sharedwith alternative curricula (general, academic)in the formal school setting.

Societal Outcomes

The societal outcomes are five innumber. In order of generality, social stabilityis probably highest. It is necessary for theother intermediate outcomes to be possible,but it also depends, at least in part, upontheir attainment. By social stability we do notmean social stagnation. Rather, we meangradual, orderly, and nonviolent change, withexpectations within the society being rea-sonably predictable, at least for a longenough period that changes may be planned,rather than being unpredictable shocks.Neither education in general, nor certainlyvocLiional education in particular, canaccomplish this outcome alone, but the expe-riences that occur in vocational educations''ould meet the test of a positive contribu-tion to social stability.

The remaining four societal outcomesare much more specific. They represent prod-ucts that can be addressed quite directly byvocational education.

Providing essential services to thesociety is an obvious need. Trained paople inhealth care, information processing, and thehighly skilled components of durable manu-facturing are needed for a smoothly function-ing, modern society. It is recognized that notall essential services are provided throughthe labor market or through paid work Thequality of a society also rests heavily uponthose services that are provided in the homeand by volunteers in various roles.

Although it is true that services viewed asessential by the majority of the society areprobably available through some form ofenterprise that becomes part of the labormarket, not all services are so provided.Many are available both on a paid basis andon a donated basis. Numerous examplesexist of the latter, all of which contribute tothe quality of life in a society but do notrequire an exchange of money, goods, orservices. Volunteers often provide some formof nursing care to sick patients, technicaladvice to amateurs on home repair, freeassistance on car repairs, free garden pro-duce, snow shoveling, attentive concern topeople in trouble, family management, par-enting, teaching (e.g, church school, informalmusic lessons), advice on nutrition, politics,and ways to cope with a complex society.Whether government policies and profes-sionals in these areas concur or not, thesevolunteer activities do take place and willcontinue to do so. Education, both formal,general, and vocational, becomes one of themechanisms through which volunteers canlearn appropriate skills and roles and cantherefore contribute to the well-being of thesociety.

Establishing a scale of values for out-comes of this sort is not easy. This difficulty,however, is a very inadequate reason forignoring the contribution made by the non-labor market elements of the society. Thepresence or absence of skills, and reports ofengagement in nonpaid volunteer activities,may provide a gross estimate. Some attemptshave been made to place dollar values onsuch services (e.g., the value of homemak-ing) but to a very large degree the problem isunresomed. Nevertheless, the contribution istoo great to ignore.

Closely related to the essential servicesis the outcome termed quality of work prod-ucts. It extends beyond the tangible andmaterial quality of things to the quality ofservices provided. A disillusionment withquality in this broad sense leads to an ero-sion of stability and a reduction in the overallquality of life for the society as a whole

10

We are not aware of a body of literaturethat specifically documents this assertion,but several examples will illustrate the intui-tive logic of this position First, consider thecurrent plight of the automobile industry.Beginnmg in approximately 1974, the declin-ing demand for American automobilesbecame increasingly recognized. One resultwas that the market for replacement automo-biles began to decline as consumers chose toretain the vehicles they owned rather thanpay higher prices for less satisfactory vehi-cles This, in turn, reduced the effectivenessof pollution control devices because non-equipped automobiles stayed on the roadlonger. It also reduced the contribution of theauto industry to the economy and was a largefactor in the turn toward imported autos,which further eroded the economy within theUnited States. Ten years later the conse-quences are still strongly felt.

A second example applies to a servicearea Suppose a child from a low socioeco-nomic status family is having serious achieve-ment trouble in school The mother visits thechild's teacher to see what can be done. Boththe mother and the teacher are sincerelyinterested in the child's welfare. But theteacher does not know how to empathize andcommunicate with the mother, and themother is made to feel inferior and inade-quate. She VIerefore avoids continuingschool visits, and, as a result, the real contri-bution she zould make is lost The child'seducation continues to suffer.

A third example, in an entirely differentarea, concerns a hypothetical situation in thedelivery of medical care. A patient goes to aphysician's office with an annoying but nottoo painful ailment, as yet undiagnosed. Thepatient is treated with perfunctory courtesyMore attention is given to the ability to paythan to diagnosis of the ailment, and the phy-sician appears to be hurried and not veryinterested. The next time the symptomsappear, the patient puts off seeking medicalservice until a serious and perhaps Irreversi-ble condition develops.

21

We have emphasized in these examplesthe service products more than the tangibleproducts. Deterioration in the quality ofeither cordributes to the disruption of societythrough disillusion with its institutions,through displacement of people (witness theMichigan-Arizona migration phenomenon asdisplaced auto workers sought jobs in theSouthweet), and through more extreme dis-ruptions such as riots, hijackings, and theorganizing of private armies.

These results are, of course, not attribut-able to lowered product quality alone, butsuch a phenomenon increases the potentialfor serious societal consequences.

The third outcome suggested in figure 1as relating to social stability is the accumula-tion of the necessary rsources for self-support by the substantial majority of themembers of a society. The proportion of thesociety engaged in productive work that maybe exchanged for other goods and services isa function of both the standard of living andthe productivity of the society.

If individual productivity is relatively lowand the desired standard of living is high,then proportionately more persons will needto be employed in a self-supporting way inorder to carry the remaining members of thesociety at a similar level. Conversely, asmaller proportion of more highly productiveself-supporting people can carry the remain-ing members assuming the standard of livingstays constant. If, however, the standard ofliving is lower, less individual productivity isnecessary to maintain the living standard. Inimmediate and practical terms, the actuarialdifficulties of the Social Security system illus-trate this problem. As the proportion ofretired * okers who are no longer productiverise. in relation to those paying SocialSecurity taxes, the Nation is forced to choosesome combination of four alternatives:

Accept a lower standard of living dueto higher taxes and inflation

11

Impose a lower standard of living onthe retired group by maintaining afixed income level despite inflation

increase sharply the productivity ofthe working group

Encourage continued productivity ofthe retired group

It is again difficult to specify just howprogress toward this goal should be mea-sured. However, a reasonable goal to whichvocational education should contribute is theincrease or maintenance of such a productivemajority.

A society in which every individual con-tributes to his or her own betterment alsocontributes to the betterment of the lives ofothers in that society. This is what is meantby societal improvementworking as indi-viduals to improve the quality of life for theaggregate. This type of positive growth isconsidered necessary in order for a societyto sustain itself rather than stagnate. Thus.societal improvement Ici the final critical ele-ment supporting sc..ial stability.

Societal Outcomes Specific to Employers

A subset of the societal outcomes arethose that impact employers directly. To beworth the investment from the employer'spoint of view, vocational education shouldcontribute to ensuring that needed skills areavailable, that on-the-job training time forgraduates of vocational programs is less thanfor those without such training, and that theproductivity of these trained workers ishigher than that of untrained workers. Todate, the evaluation of vocational educationhas been largely limited to factors such asthese, while ignoring the numerous otheroutcomes suggested in figure 1. It is ourjudgment that although these outcomes arenecessary in the paid work market, they arenot sufficient for assessing the quality ofvocational education there or in the otherdefined outcome areas.

22

The Place of Vocational Education

We began this discussion by consideringthe role of vocational education as a part ofthe broad concept of education in generalWe have not been willing to accept a narrowdefinition that divorces the training in spe-cific, employment-related skills from theeducation of a person for all aspects of livingThus, our definition of vocational educationincludes, but is not limited to, employment-related skills. Development of those skills is anecessary, though not sufficient, componentof vocational education as we see it. Thisleads to a recognition that vocational educa-tion of a positive quality must developemployment-related skills; it may not sub-stantially interfere with the development ofdesirable or even essential skills or disposi-tions that are not directly emp'oymentrelated; and it should, to the extent possible,enhance those dispositions. (For a succinct,normative discussion of some of these dispo-sitions, see Thimmesh 1984 )

At the secondary level, vocational educa-tion competes for class time with other elec-tives and with required courses that reflectAmerican society's collective judgment aboutwhat is basic and essential. At the postsec-ondary level, a similar competition is presentin the planned sequences but is less enforcedbecause of the greater legitimacy of takingone or two courses for immediate personalgoals and then discontinuing the program.*These conditions require that vocationaleducation share responsibility for the kindsof goals shown in figure 1, although it cannotbe held responsible for their completeattainment, nor can the larger subset of for-mal education. It logically follows that thequality of vocational education can only beassessed by taking into account its role in thelarger context.

'In addition some people assume that some 01 the goals have already been attained by the time of secondary school com-pletion and consequently need not be readdressed at the postsecondary levsl

12

CHAPTER 3

THE EDUCATIONAL EXPERIENCE

General Definition of theEducational Experience

The vocational educational experienceinvolves a network of interrelated and inter-dependent people, procasses, and structures.A complex environment comprising anumber of interacting parts, it functionswithin the general education system andwithin a much larger environmental context.In order to determine whether the environ-ment is operating effectively in achievingdesired individual and societal outcomes andin order to make improvements in the educa-tional experience as a whole, it is essential tounderstand how each part functions, bothindependently and interdependently. Thus,defining in detail the elements of quality inthe educational experience is a critical steptoward achieving these ends.

The conceptual framework proposed infigure 1 provides a starting point for discus-sion. The educational experience repre-sented by the central panel of the model issituated within the larger context of environ-mental inputs and outcomes shown in theleft-hand and right-hand panels. The centralpanel is divided into "School Context," dis-played in the uppermost cell, and "SchoolProcess," represented by the three lowercells of the panel. The components fisted*thin all four cells define the boundaries ofthe school context and school processdomains.

School Context

School context deals with both organiza-tional and physical aspects of the schoolunit. It is represented in figure 1 as the firstcell in the central panel. Personnel, facilities,

13

equipment, policies, and lormal organiza-tional structure are general eIaments ofschool context.

School context sets the stage for schoolprocess and can best be described throughillustration. The following two fictitious "con-text" scenarios help to show the place ofschool context within the overall educationalexperience.

The development cf Middletown Techni-cal Institute (MTI), a postsecondary voca-tional education school, was a directoutgrowth of current and projected needs ofbusiness and industry in Middletown, U.S.A.Goals of the school and its individual pro-grams were developed by a committee thatincluded educators, administrators, students.parents, and representatives of business andindustry. Consideration of school and pro-gram goals influenced the design of the phys-ical structure of the school as well as thepurchase of equipment. Local business andindustry demonstrated support of MTIthrough generous donations of equipmentand material. Selection criteria for instructorsincluded technical experience and knowl-edge, formal training in education methods.enthusiasm for teaching, and other selectedvariables. An active business-industry advi-sory board helped MTI stay abreast of chang-ing needs. Qualified counselors werecharged with tailoring student programs toindividual needs and assisting in job-seekingefforts. MTI enjoyed a favorable image in thecommunity. There were more applicants toMTI than its programs could handle; studentselection criteria were established to managethe demand overload.

Steeltown Vocational School (SVS). oursecond fictitious program, presents a much

4.° '1

different picture of school context. SVS is apostsecondary institution that was opened inthe early 1950s to train skilled laborers forthe local steel and machine tool industries.School and program goals have not beenmodified since its inception. The schoolbuilding is a converted warehouse and theinstructional staff is mainly composed offormer supervisors of skilled laborers.Instructors have litde or no formal educationin teaching methods. The physical facility isin desperate need of repair having crackedwalls and ceilings and antiquated plumbing,heating, and lighting systems. However, withenrollment and revenues down, there are noplans for making the needed repairs. Place-ment rates for SVS graduates are at en all-time low. Support from the community isvirtually nonexistent; it is only a matter oftime before SVS will be closed.

These scenarios help to illustrate theimportance of school context in "setting thestage" for the educational experience. Per-sonnel, facilities, equipment, and organiza-tional structure are all part of the "context" ofan educational facility and relate to the qual-ity of a facility. The contrast between theschool context of MTI and SVS was exagger-ated in order to demonstrate the significanceof context in the educational experience. Amore detailed discussion of the componentsand elements of school context will appearlater in this chapter.

School Process

If school context provides the setting,school process provides the action. Referringagain to figure 1, school process is repre-sented by the second, third, and fourth cellsof the central panel. Cell 2, entitled "StudentUtilization," relates to patterns of participa-tion that are influenced by school contextand school process (Campbell, Gardner, andSeitz 1982). Cell 3, "Teacher, Student, Mate-rial Interaction," combined with Cell 4,"Instructional Content and Equipment,"defines the teaching-learning process.

14

Descriptions of school process includereferences to course content, course-specificequipment, and materials. Discussions ofstudent-teacher, student-student, student-material interface, and level of formal studentinvolvement in the vocational educationschool unit or program are also involved in acomprehensive description and analysis ofschool process quality.

The following description of a fictitious,comprehensive high school may help to illus-trate the general boundaries of the schoolprocess domain.

Jefferson Comprehensive High School isviewed mainly as a college preparatory insti-tution. Students are discouraged from for-mally entering the vocational educationprogram, so those students that take voca-tional education coursework do so mainly asa diversion from purely academic courseloads. Concentrators are generally held inlow esteem by their peers and by the majorityof academic program instructional staff.Vocational education instructors have littleinfluence on purchase of instructionalequipment and are forced to use out-datedtextbooks and equipment. Students havelimited opportunity for "hands-on" expe-rience These restrictions and limitations arealso reflected in episodes of student-teacher,student-student, and student-material inter-action. Instruction in vocational educationcourses appears to be a one-way process.Teachers dispense information and studentsabsorb information; a feedback loop is virtu-ally nonexistent between teacher and stu-dents Typically, student-student interfaceconsists of rubber band shooting, note pass-ing, and joke telling Because of the lack ofsupport and resources for vocetional educa-tion programs, there is a high rate of turnoverof 'good' vocational education teachers. Itseems that those who stay are the type whocare little for the job and the well-being of thestudent.

This description touches upon the majorcomponents of quality in the school process

25

portion of the model. It provides a general"feel" for the school process domain and thepotential effect of process on the overalleducational experience.

Interrelationship between theEducational Experience and Outcome

The processes and structures that definethe elements of quality within the vocationaleducation experientte should relate tointended individual and societal outcomesThe educational experience should increasethe likelihood that desired individual andsocietal outcomes are achieved. In order todo so, a systematic understanding is requiredof the relationships between components ofthe educational experience and outcomesThis will allow the school unit to maintainand replicate effective progrAms and willpromote systematic program modification asoutcome indicators change.

For example, with the current emphasison and movement toward electronic dataprocessing both at home and on the job,computer literacy may eventually be includedalong with basic math, English, and so forth6S part of basic skills. In turn, the addition rifcomputer literacy should be reflected at theoutcome indicator level and also in the formof school context and school process indica-tors. In other words, there.should be a sys-tematic linkage between desired outcomeand experience; the educational experienceshould be responsive to changes in desiredoutcomes.

In order for the linkage to be systematic,the educational experience must be dis-sected into observable components or ele-ments of quality that relate to intermediateobjectives. Achievement of thtse objectivesshould support the achievement of desiredoutcomes. Thus, outcomes are considered asgoals, and the educational experience as theway to move closer to the goals.

15

Elements of Schoof Context

A number of components related to qual-ity of education are classified within theschool context and process domains of Ourmodel. For example, as figure 1 indicates, thecomponents of school context are these:

School settingClassroom settingTeacher skillsTeacher characteristicsPeer group characteristics

Each component consists of a set of ele-ments defining its parameters. A thoroughunderstanding of elements of quality is pre-requisite to developing specific indicators ofquality. Therefore, the elements of qualityrelated to school context and school processwill now be identified and discussed in detail.

School Setting

Our model proposes five components ofquality within the school context domain (seef gure 2). The first of these is school settingSchool setting encompasses both phyaicaland organizational aspects of the educationalexperience. The physical elements of schoolsetting include the school building, theschool property, and general equipment asopposed to program-specific equipment.Integrity of basic systems such as structuralsupport systems, plumbing and heating sys-tems, ventilation and lighting systems, andexterior and interior design features would beconsidered as part of the "school building."The second element, school property, dealswith the adequacy of adjunct facilities, suchas parking lots. The category of general sup-port equipment is the third physical elementrelated to quality of school setting. Generalsupport equipment is defined as equipmentrelated to the functioning of the school unitas a whole and does not include equipmentpurchased for individual programs. Copymachines, telephones, file cabinets, andmanagement related hardware and softwareare examples of general support equipment.

26

SCHOOLCONTEXT

Clauroom Setting Teacher Skills Teacher Characteristics IPeer Group Characteristics

Physical Aspects Physical Aspectsof Instructional

Technical Knowledge Teaching Approach Attitude toward Education

whool building Setting Technical Experience Teaching Style Academic Performanceschool property(lateral equipment systems

furnishingsTeaching Methods,

Knowledge, andBehaviors in Instructional

SettingOrganizational design features SkillsAspects

organization

generalinstructionalequipment

strategypolicy

Figure 2. Elements of quality of school context 28

A lathe is considered program-specificequipment and would not be included here.

Elements within the organizationalaspect of school setting include formalorganizational structure, strategies, and poli-cies. Organizational structure relates to theformal organizational chart and related man-agement philosophy. It plays a major role indefining school setting since it directlyaffects personnel networks and informationflow. For instance, one school unit mighthave a rigid, hierarchical organization chartwith an autocratic management approach, atop-down flow of information, and a principalwho is primarily concerned with manage-ment. A second school might also have ahierarchical structure but one that is definedby a participative management style (e.g.,collegiality at the community college level),an interactive communication network, andan administrator who is primarily concernedwith instruction.

Strategies relate to long-range goals anddirections of the school as a unit. Researchindicates that schools with clearly statedmission statements are more effective (AASA1983). The basic strategic alternative ofgrowth vs. no-growth is an example of a stra-tegic direction that would affect school set-ting. An example of a strategic goal would beminimization of capital expenditures forequipment through development of andemphasis on a strong industry donation pro-gram. This goal could potentially affect theactivities and responsibilities of administra-tors, instructors, and students and wouldtherefore need to be considered in a descrip-tion of school setting.

Organizational policy is the final elementwithin the organizational aspects of schoolsetting. Policy affects the daily operation ofthe organizaton. Dress code is one exampleof a policy decision. Or, consider a 2-yearcollege policy regarding credit for nontradi-tional learning experiences.

Thorough consideration of the physicaland organizational aspects of school setting

should result in a comprehensive descriptionof the subelements of quality within thatdomain. Only then can judgments be made ofadequacy, appropriateness, and effectivenessand indicators of quality be developed.

Classroom Setting

Classroom setting is the second compo-nent of school context. It is much narrower infocus than school setting. Classroom settingdeals with physical aspects of the classroomas they relate to general instruction. Class-room setting deals with the overall physicalcomfort, safety, and adequacy of the generalinstructional sett ....g (Westbrook 1982).Aspects to be considered would include ade-quacy of safety equipment; adequacy of light-ing, ventitation and heating; adequacy ofdesks and work areas; and availability ofbasic teaching equipment such as black-boards, pointers, flip charts, and more tech-nologically advanced teaching equipmentsuch as videotapes, computers, closed circuitTVs, and so forth. Classroom setting is ofparticular importance in a vocational educa-tion setting. For example, in a course such asc welding lab where open flames are neces-sary, ventilation and traffic flow patterns areof extreme importance. Availability and ade-quacy of safety equipment and proceduresare also critical. Elements of classroom set-ting are important to general instructionalsettings too. Most of us know what is it like totake a test in a cold, poorly lit classroom sowe can understand why classroom settinghas an effect on the overall educationalexperience. The underlying premise is thatcertain standards should be met in order toenhance learning and increase the probabil-ity of meeting course objectives.

Teacher Skills

The third component of school context isteacher skills. Three basic elements of qualityare included in our definition of teacherskills. First is technical experienceHowmuch direct work experience do teachers

*Administrator skills and CharaCteristics are subsumed within the elements of the organizational aspect of school setting

17

29

have in the fields in which they are teaching?Second is teaching methods, knowledge, andskillsHas the teacher completed postsec-ondary course work in history of education,teaching methods, approaches, and tech-niques? Does the teacher apply this knowl-edge to the classroom setting? Does theteacher present subject matter in a form ormanner that makes it more meaningful andsignificant to the learner (National Commis-sion on Secondary Vocational Education1984)? Third is technical knowledgels theteacher technologically up-to-date (NationalResearch Council 1983)? Has the teachercontinued formal course work related to theteaching field? Does the teacher have recentfield experience? These are the elements tobe considered and some general questionsaddressed when assessing the teacher skillscomponent of school context.

Teacher Characteristics

The fourth component of school contextis teacher characteristics, defined by twoquality-related elements: teaching approachand teaching style. Teacher characteristics isa much "softer" component than teacherskills, but it is no less important in terms ofits effect on the educational experience.

Teaching approach is related to teachingphilosophy and is manifested as attitudetoward the teaching-learning process asdemonstrated through daily behavior both inand out of the classroom. Is learning a one-way process with the teacher dispensinginformation and the student absorbing infor-mation, or is the student-teacher feedbackloop a critical element of process? Is learningto be dreaded or enjoyed? Does learninghave a definite start and finish or is it a life-long process? Are the teacher and student onthe same team or opposing teams? Theanswers to these types of questions help todefine teaching approach.

Teaching style is the result of a numberof factors. Attitude toward job, attitudetoward students, subject and content area,

18

and individual personality all affect teachingstyle Teaching style is exoressed throughteacher classroom behavior Is the teacherenthusiastic and innovative? Is humor incor-porated, as appropriate, into the teachingprocess? Does the teacher show empathywith students? Is the teacher concerned withindividual students? The teacher behaviorsthat provide the answers to these and similartypes of questions constitute teaching style

In summery, the teacher characteristicscategory is the combination of a range of var-iables falling within the elements of style andapproach. There is undoubtedly a range ofeffective teaching styles and approaches.However, in order to develop indicators ofquality teacher characteristics, the assump-tion must be made that there are identifiablecharacteristics that are common to effectivestyles and approaches. Most of us havememories of specific teachers whose stylesand approaches were either particularlyaffective or ineffective. One can thereforeeasily understand why teacher characteris-tics play an important part of the overalllearning experience.

Peer Group Characteristics

The fifth and final component of schoolcontext is peer group characteristics. Thiscan generally be defined as the attitudetoward education, academic performance,and behavior of the student body as a wholeOr of a homogeneous group within the stu-dent body For exrnple, secondary educa-tion could be synonymous with collegepreparatory education for one student bodyAnother student group may view high schoolas a setting for courtship and socialization, athird may view secondary education as astepping stone to the world of work, or afourth group might view it as a setting fororganized amateur athletic competition Thisaggregate view pervades the group and typi-cally affects students' behavior and perfor-mance at the individual level Peer pressureplays a significant part in the formulation andmaintenance of peer group characteristics

,'3 0

Consider the following example. A homo-genous group of very bright students decidesthat "grades" are meaningless symbols.They claim that course content is not chal-lenging and "anyone can get an A." There-fore, grades do not differentiate the brightstudents from the marginal students. Thispear group holds the learning process and=hoot programs in very low esteem. Theydecide to express this attitude through poorattendance records, nonparticipatory behav-ior in the classroom, and failing grades in alltheir courses. In doing so, they are making agroup statement about the educational expe-rience that affects group members academicperformance and behavior at the individuallevel.

Thus, peer group characteristics maysignificantly affect the student-learning inter-face and can be perpetuated or modifiedthrough peer pressure systems. An under-standing of peer group characteristics is pre-requisite to a comprehensive definition ofschool context.

School context "sets the stage" forschool process, which will provide the"action.' Context relates to the quality of thephysical and organizational setting throughthe school setting and classroom settingcomponents and provides characterization ofthe main actors through the components ofteacher skills, teacher characteristics, peergroup characteristics, and school settingsince it includes administrative skills andcharacteristics.

Elements of School Process

Scho Ol process is the remaining portionof our conceptualization of elements of qual-ity in the educational experience. Theteaching-learning process and patterns ofparticipation are its major subdivisions.Components and elements were identifiedand defined as part of the model fnrmulationprocess described in chapter 1. The results ofthat effort as they relate to school processfollow.

19

Teaching-Learning Process

The teaching-learning process includestwo components related to quality: teacher,student, end material interaction and instruc-tional content and equipment. These compo-nents are represented as cells three and fourof the central panel in our conceptual frame-work (figure 1) and are further detailed infigure 3.

Instructional content and equipment.Instructional content and equipment dealswith course-specific information and equip-ment. Even though there is a wide range ofcourse offerings within any given school unit.there are certain common elements related toquality of content and equipment that shouldbe considered regardless of the course.These elements include but are not limited tothe following:

Clear course objectives (Westbrook1982)Business- and industry-relatedcourse content (National ResearchCouncil 1983; Benson 1982)Instructional content related tocourse objectives (Westbrook 1982;Miller 1984)Course objectives consistent withprogram goals (Westbrook 1982;Miller 1984)Appropriate equipment availableUtilization of multiple media forinstructionBalance between materials requiringcontent knowledge. understanding,and application (Benson 1982)Balance between knowledge, skill,and attitude development (NationalResearch Council 1983)

These elements and others like themserve as a starting point for the developmentof indicators of quality content and equip-ment. The underlying premise is that certainstandards indicative of quality can be identi-fied for each element.

31

1

I

SCHOOL PROCESS

1

Teaching-Learning Process

ZinstructionalContent andEquipment

Clear Course Objectives

Business- and Industry-related Course Content_

Instructional ContentRelated to CourseObjectives

Course ObjectivesConsistent withProgram Goals

Appropriate EquipmentAvailable

Use of Multimediafor Instruction

...Teacher, Student,

Material Interactioni

Balance betweeo MaterialsRequiring ContentKnowledge, Understanding,and Application

Balance between Knowledge,Skill, and Attitude Development

Type

Class

Frequency

Dumtion

Nature

Figure 3. Elements of quality of school process

---,Student Utilization ofVocational Education

Patterns of Participation

Concentrators

Limited Concentrators

Concentrator/Explorers

Explorers

Incidental/Personals

Teacher, student, material Interaction.The second component of the teaching/learning process is labeled teacher, student,material interaction. This component cangenerally be described in terms of five de-scriptive elements: type, class, frequency,duration, and nature.

Type refers to the specific, interactivecombination (e.g., student with stu-dent, student with teacher, etc.).

Class refers to the basic classificationof the interaction. Examplesinclude-

-teacher monitors student,teacher questions student,teacher reinforces student,student reads material,student handles equipment,student questions teacher.

Frequency involves an actual countof interactions by type or class. Forinstance, the number of student-with-teacher interactions occurring withina class period might be noted.

Duration involves a time measure ofthe length of each interaction. This isusually recorded by type of interac-tion (e.g., How long did the studentwith student interface last?).

Nature involves a general assessmentof the quality of the interface, such aspositive, neutral, or negative.

A considerable number of referencesaided in the development of this classifica-tion scheme. For example, Halasz, Behm, andFisch (1984), Miller (1984), and Westbrook

(1982) suggested an increase of "time ontask" in effective programs. This wouldinvolve development of indicators thatinclude type, class, and duration elements.Hunt and Woods (1979) recommended fre-quent positive reinforcement of student byteacher that would include the elements oftype, class, and nature. Westbrook (1982)

21

and Miller (1984) suggested that teachersshould monitor student performance andprovide qualitative feedback. This sort ofinteraction would be described by a combi-nation of all five elements in our scheme.Therefore, references helped to identify ele-ments related to quality of the teacher, stu-dent, material interaction component.

Thus, a description of student, teacher,and material interaction in the classroomwould include a statement of type, class, fre-quency, duration, and nature. Developmentof indicators of quality for these interactionswould then be made in consideration ofinstructional content and equipment. Forinstance, it may be determined that a lab set-ting requires a greater frequency and dura-tion of student-with-student interection thandoes a lecture class. Or, there may be ahigher need for teachers to monitor individ-ual students in a lab setting than in a class-room setting.

Therefore, a comprehensive analysis ofthe quality of the teaching-learning processcomponent would include references to bothinstructional content and equipment andteacher, student, material interaction. Thisprocess provides much of the "action" in theschool unit and is clearly a major part of theoverall quality of the educational experience.

Student utilization of Instructional pro-cess. The balance of school process dealswith the formal relationship between the stu-dent and the vocational education schoolunit or program. Campbell et al. (1982) iden-tified this interface es patterns of participa-tion: it has been incorporated into our modelas a component of school process (figure 3),but it is also accurately described as anintermediate outcome of the interactionbetween school context and school process(figure 1).

Campbell conceptualized a five-dimensional matrix for describing possiblepatterns of participation. The dimensions ofthe matrix were intensity, diversity, conti-nuity, supportive diversity, and proximity

33

..

Five patterns were hypothesized and empiri-cally verified. These patterns were i0entifiedby Campbell, Gardner, and Seitz (1982) asfollows:

ConcentratorsLimited ConcentratorsConcentrator/ExplorersExplorersIncidental/Personals

Briefly, the descriptive concepts include:(1) the number of credits received invocational courses in the program areaof oPecialization; (t) the number of pro-gram areas in which the specialty waspursued; (3) the number of vocationalcredits in the program area that weredetermin vl to be supportive of the spe-cialty area: (4) a scaled measure ofwhether the specialty was pursued in the11th or 12th grade; and (5) the number ofyears in which the specialty was taken. Astudent's area of specialization wasdefined as a program area (for example.distributive education, home economics)in which at least six-tenths of the totalnumber of vocational credits werereceived.

For example, Concentrators take anaverage of 8 vocational credits over a 3-year period. Limited Concentratorsgenerally take about half the number ofvocational credits that Concentratorstake, usually within a 2-year period.Concentrator/Explorer is similar to theLimited Concentrator pattern, exceptthat the vocational course work is usuallycompleted early in the high school years.

Students classified in the Explorer pat-terns pursue courses in three or moreprogram areas but do not achieve anylevel of specialization. In comparison,Incidental/Personal students averageless than a full credit and generally com-plete the work in a semester. These pat-terns were used in the analyses in placeof the traditional curriculum descriptorsof vocational, general, and collegepreparatory. (p. 16)

Although detailed discussion of thesepatterns goes beyond the scope of thisreport, it is important to be aware that theextent and kind of participation in vocationaleducation is critical to a comprehensive eval-uation of quality of the educational expe-rience. Patterns of participation seem to beinfluenced by a number of aspects of theeducational experience, such as programstructure, use of facilities and equipment,assignments of students, course objectives,and elements of teacher, student, and mate-rial interface. In other words, patterns ofparticipation are interrelated with theteaching-learning process and school con-text and may therefore be described as anintermediate outcome of their interaction.

Our conceptualization of elements ofquality in the educational experience pro-vides a framework for developing indicatorsof quality. The model may not be compre-hensive, but a continuation of the process bywhich it was devs!oped should lead to acomplete definition of the concept. A generaldiscussion regarding development of indica-tors of quality Will be presented in chapter 4.

CHAPTER 4

ASSESSING QUALITY OF VOCATIONAL EDUCATION

Introduction

It is no longer sufficient to say Johnnycan read. One must also state how well hecan read, why he can read, and what can bedone to improve Johnny's reading ability.Consequently, evaluation has become amajor activity within the field of educationtoday. The individual, Federal, State, andlocal agencies and society as a whole haveconcerns about the quality of educationalprocesses, structures, and outcomes. There-fore, educators are being asked to assess acomplex educational system by means of acomplex evaluation system that is quite acomplex undertaking in itself.

Evaluation is a major area of emphasis inthe Carl D. Perkins Vocational Education Actof 194. In addition to the requirement forplacement and employment data for pro-grams assisted under the act, each State isrequired to evaluate program effectivenessand "quality and availability of instructionalofferings." Consequently, within vocationaleducation today, the evaluation of quality isemphasized through its ties to legislation andfunding.

Chapter 4 presents some general, -itimportant, issues and guidelines related toassessing quality of vocational education. Itis not intended to be a comprehensive reviewof program evaluation techniques, models,and methods; the literature on the topic isvoluminous end could only be superficiallytreated within the scope of this writing.Rather, general evaluation approaches, spe-cific models, and so forth are discussed asthey relate to developing indicators for theelements of quality vocational education thatwere discussed in chapters 2 and 3.

23

What Is Quality?

Chapters 1, 2, and 3 presented within asystems framework the components andelements that affect quality of vocationaleducation. These components and elementsare generalizable across school units andprograms. But the specific indicators of qual-ity gssociated with them may vary withschool unit and program and are likely tochange over time. This is because specificdefinitions of what indicates quality are rela-tive, judgmental, and subject to change asthe scope of the individual provider respon-sibility changes.

The definition of quality education isstrongly influenced by conceptualapproaches operant within the field at a par-ticular point in time. Therefore, the definitionof "good" depends on present knowledgeand opinion. For example, in the early 1970$,the "open" classroom approach was adoptedby many educators. Programs were modifiedand facilities were tailored to this newapproach with the underlying assumptionthat the "open" classroom was positivelyrelated to "quality" education.

Consequently, although the componentsand elements of school context, school pro-cess, and particularly the educational out-comes remained intact, specific standards ofquality related to those components probablychanged. Consider, for example, classroomsetting. One indicator of a quality classroomsetting prior to the open classroom approachmight have been this:

A closed, well-sound-proofed instruc-tional setting allowing X square feetper student and one desk perstudent.

35

For those schools that subsequentlyaccepted the open classroom concept theparallel indicator would probably have beenthis:

An open instructional setting with nostructural barriers between fourindependent classes and 'work tables'designed for simultaneous use byfour to six students (no individualdesks for students or teachers).

Of course, this is just one example of how aspecific indicator of quality might change asapproaches within the field change. In actual-ity, the open classroom approach undoubt-edly affected indicators associated with mostof the elements of the school context andschool process portion of our framework.

The definition of quality also changes asscope of provider responsibility changes.Consider the growing demand for and roleof day-care services today. Many childrenfrom the Vine they are infants are placed withday-care providers for 45 hours per week.These providers spend considerably morewaking hours with these children than dotheir parents. The scope of the providers' rolehas potentially expanded from one of basiccare provider to that of primary role model.The overall societal and individual assess-ment of the importance or substance of childraising has not changed. However, the defini-tion of day-care has expanded and theindicators of quality day-care should theoret-ically reflect this change. Specific indicatorsof quality are expected to change eventhough the basic conceptual framework forelements of quality education will remain theUMW

Consequently, the nature of qualitymakes assessment difficult but not impossi-ble. The difficulty in no way reduces thenecessity. Popham and Grasso (cited inGrasso 1979) have stated that assessing theworth or merit of educational phenomena isan essential component of systematic educa-tional evaluation. In additit. n, section 112(b)(1)(13) of the 1976 Amendments to the

24

Vocational Education Act of 1963 as inter-pretod by Federal regulations explicitly statesthe need for evaluating "quality and availabil-ity of instructional offerings" (Federal Regis-ter, 3 October 1977; section 104.402). Thus,the assessment of quality in vocational edu-cation is not only generally recognized andaccepted as important, but it is mandated bylaw.

Approaches to Asussing Ouallty

Considerable effort has been expendedin developing evaluative strategies to facili-tate the assessment of quality. Much of thiswork has been drawn from the areas oforganizational behavior and systems theoryand adapted to specific field evaluation mod-els. Three predominant approaches haveemerged that are relevant to assessments ofquality within the field of vocational educa-tion. We will generally refer to them as struc-tural, process, and outcome approaches.

Structural Approaches

Structural approaches focus on aspectsof the organization that impact on quality ofeducation. This would include such variablesas personnel, facilities, equipment, formalorganizational structure, policies, and soforth. These variables primarily relate tocomponents and elements of quality withinthe school context domain of our model (fig-ures 1 and 2). For example, in a structuralapproach to the issue of human resourcedevelopment, a judgment might be made thatcertain teacher qualifications are necessaryto ensure quality education. These qualifica-tions would be expressed as indicators withinperhaps the teacher skills and teacher char-acteristics components of school context. Or,in terms of classroom setting, indicatorsmight be developed that specify teacher-student ratios and student-square feet ratios,and so forth. These would serve as standardsof quality for instructional setting.

36

Two major assumptions support a struc-tural approach to assessing quality. First, it ispossible to identify "good" in terms of staff,physical facilities, formal organization, andso forth. And second, the quality of educa-tion will improve as a result of such things asbetter qualified staff, improved physical facili-ties and equipment, and sounder organiza-tional structure.

Process Approaches

Process approaches utilize criteria-oriented methods and individual studies toexamine activities involving teachers, stu-dents, materials and their interaction, andother elements that are included within ourdefinition of school process. Therefore, pro-cess approaches would primarily focus onthe school process portion of our model (fig-ure 3). A process approach might examinesuch things as "hands-on" equipment time ina welding class, frequency of "teacher ques-tioning student" during a lecture class, orclarity of course objectives against selectedindicators of quality.

Process approaches to the assessment ofquality assume that it is possible-

1. to set intermediate course of programgoals and evaluate whether goalshave been met,

OR2. to establish quantitative and qualita-

tive standards of quality, based onexpert opinion and applied to indi-vidual case review,

OR3. to directly observe critical behaviors

that permit determination of quality,AND

4. for school educators and administra-tors to agree upon what constitutesquality without continually monitor-ing educational outcome.

Process evaluations can be utilized to detectproblems in program design and implementa-tion, to provide information for formative

25

decision making, and to develop databasesfor long-range planning (Stufflebeam et al.cited in Grasso 1979).

Outccm Approachu

Outcome approaches to assessing qual-ity of education assume that certain directmeasures of results are related to quality ofeducation. Theoretically, within a pureoutcome framework, the qualifications ofeducators, the structure of the organization,facilities and equipment, and school processcomponents and their interactions can bedisregarded in favor of assessing whether ornot desired results are achieved. One exam-ple of an outcome approach is Scriven's con-ceptualization of payoff evaluation thatappears to include analyses of both outcomeand impact (cited in Grasso 1979).

The underlying assumptions of an out-come approach are the following:

1. There is a high level of agreementbetween societal, individual, and pro-fessional estimations as to what endresults are desirable.

2. Desired outcomes can be expressedas quantitative and qualitative indicesof success.

3. Desired outcomes are brought aboutto a great extent by qualityeducation.

4. There is a high level of agreementbetween experts and practitionersregarding validity of selected out-come indicators

An outcome approach would focus primarilyon the elements of quality vocational educa-tion outcomes depicted in the right handpanel of our model (figure 1). Outcome indi-cators or sets of outcome indicators wouldbe developed for each component in thisoutcome panel and a determination ofattainment or lack of attainment would

3?

subsequently be made. In some cases, theassessment could be immediate; in others,there may be a considerable time lag. Forexample, reading ability might be assessedby means of a reading achievement test bat-tery just prior to graduation. In some profes-sions, licensing exams are taken shortly attergraduation and serve as indicators of attain-ment of job-specific skills. But in the case ofpositive interpersonal interactions, a consid-erable period of time on the job may berequired prior to assessment and the methodof measurement may not be so readilyavailable.

A major difficulty with a pure outcomeapproach is that it requires no systematic link-age between desired outcomes and com-ponents and elements of the educationalexperience. Program modifications resultingfrom a pure outcome approach may not besystematic. The problem is due to the limitedscope of the analysis. The same problemexists for pure structural and processapproaches.

An evaluation system designed to assessquality of vocational education as defined byour conceptual framework must considerschool context, school process, and out-comes and therefore requires a synthesis ofthese approaches. An integrated approach tothe evaluation of quality assumes thatappropriate structure (school context)increases the probability of quality programs(school process), which, in turn, increasesthe likelihood of successful outcome.

The CIPP (Context, Input. Process, Prod-uct) model designed for institutional decisionmaking (Stuffitbeam et al. cited in Grasso1979) is one example of a integrated evalua-tion approach that could be applied to theassessment of quality. Another would beDrewes' expansion of the CIPP model toinclude impact evaluation (Drowse et al.1975). Drewes' comprehensive model incor-porates structural, process, and outcomeapproaches within the following evaluationmethodologies:

26

Context evaluationidentifies unmetneeds and unused opportunities

Input evaluationidentifies andassesses merits of various structuralvariables such as staff, organization.and physical facilities

Process evaluationreviews opera-tions of programs and the teaching-learning process as a whole

Product evaluationmeasures andinterprets attainments at end ofcourses and programs to assessintermediate goal attainment

Impact evaluationexaminesintended and unintended results ofvocational education, such asemployee satisfaction and post-program placement

Pairwise *valuationexamines pai r-wise interrelationships of evaluationtypes listed previously

The primary point is that a comprehensiveand systematic evaluation of the elements ofvocational education quality requires a sys-tem of evaluation that includes structural,process, and outcome approaches.

Developing Indicators of Quality

Indicators are standards klainst whichsomething, in this case, quality, :an be mea-sured. In orJer to assess quality )f vocationaleducation comprehensively, sets of indica-tors need to be developed for the elements ofquality presented in chapter 2 (outcomes)and chapter 3 (educational experience) andincorporated into an evaluation system suchas Drewes' model. Assessing quality withinthese constraints imposes formidable difficul-ties in terms of time and cost. It is not practi-cal to tabulate exhaustively all of theindicators related to the elements of ourmodel, nor is it financially nractical to

38

.

support an evaluation system capable of col-lecting and analyzing such data in a com-plete form.

Our approach to this difficulty involvestwo initial steps:

1. Prepare a representative list of sub-elements. These should be selectedto encompass the range of eachmajor component and element com-bination of our model as described inchapters 2 and 3.

2. Secure a consensual judgment aboutthe relative priority of time orresource investment in the subele-ments. These priorities might serveas proxies for the overall value of thesubelements. (Allow the essentialityof specific subelements to influencethis priority. Some may be essentiala* preconditions or in their own right:others may be alternatives.) Thisexercise will also provide the oppor-tunity to suggest new subelements orto eliminate inappropriate ones.

These steps should lead to the prioritizationof elements within our framework. This order-ing may vary from school unit to school unitand within school units over time. But afterthese two steps are reasonably complete, itwill be possible to develop a set of indicatorsfor at least the high-priority subelementswithin D.3 outcome and educational expe-rience portions of our model.

A set of underlying assumptions isneeded to support the development of quaInyindicators. The following assumptions havebeen extrapolated from Zusman andLawson's (cited in Riedel et ai. 1974) workrelated to assessing the quality of mentalhealth patient care.

Indicators of "quality" education canbe characterized and rated.

Definition of quality education asexpressed through indicators

27

depends on present knowledge andopinion.

Definition of quality education asexpressed through indicatorschanges as scope-of-providerresponsibility changes.

Decisions regarding adequacy of par-ticular educational structures andappropriateness of particular pro-cesses and programs are influencedby decision makers' value systemsand judgments and are thereforeultimately subjective.

These assumptions take into account the rel-ative. judgmental, and changing nature ofdefinitions of quality as expressed at theindicator level. For the purposes of the ensu-ing discussion, these assumptions will beconsidered valid.

Normative versus Empirical

Both normative and empiricalapproaches should be drawn upc- for devel-oping indicators. Normative indicators wouldbe formulated on the basis of an ideal of whatrepresents excellence. Methods used todevelop normative criteria include judgmen3of experts or highly qualified panels andstate-of-the-art literature and research. Nor-mative standards are based more on a bodyof knowledge than actual practice. One prob-lem with a purely normative approach is thatpractitioners and educators feel that it is notpractical from a time, cost, and other per-spective to apply idealized standards toactual practice. Empirical standards. on theother hand, are based on actual practiceeducational programs and courses. A majorcriticism of purely empirical approaches isthat standards may appear adequate relativeto other situations (school units, programs,courses) yet fall far short of what is achiev-able through application of current knowl-edge. A blend of the normative with theempirical is the recommended approach todeveloping indicators of quality. Empiricalobservation should provide a baseline that is

39

then modified by normative judgment; schoolunits and programs should not lose sight ofthe ideal.

Quantitative versus Qualitative

There exists a long-standing debate con-cerning the advantages ot qualitative versusquantitative methods of evaluation. However,a growing body of evaluation researchersholds that the two methods can be utilizedquite profitably :nether. Each method hasits biases, so selection of the method shouldbe based on purpose of the evaluation andspecific constraints connnected with theproject.

There are a number of reasons why acombination of qualitative and quantitativemethods is suited to a comprehensive evalu-ation of quality. First, a comprehensiveevaluation requires analyses of quality ofetructural, process, and outcome elements.This variety of analyses typically requires avariety of methods. Second, when usedtogether for the same purpose, the two-method-type indicators can support oneanother and often lead to insights thatneither one could provide independently.And third, utilizing a combination of quaNta-tive and quantitative methods helps toaccount for the biases that are associatedwith each approach (Cook and Reichert1979). Evaluators need to fit the method andthe indicators to the demands and purposesof the particular evaluation: restricting one-self to only one of the method types willprobaoly restrict a comprehensive evaluationof quality.

Examples of indictors

As previously noted, specific indicatorsof quality are typically not generalizableacross school units or even school programs,so it is not practically possible to tabulate allpotential indicators of .14, elements of qualityin our model exhaustively. Therefore, just afew examples will be mentioned in order tcclarify the relationship between indicators

28

and model components and elements and todemonstrate the level of specificity involvedat the indicator level. These examples areshown in figure 4.

Of course, these are only a few examplesof indicators that might be associated withcomponents and elements of quality voca-tional education. The key point is that indica-tors must be measurable or observable andconsidered as good proxies for the context,process, or outcome elements that they areintended to measure. It is realized that con-siderable effort is involved in developingindicators of quality. However, if the problemis attacked in a systematic and phasedmanner, it is possible for individual schoolunits and programs to establish some valu-able indicators of quality.

Summary

The elements of quality proposed in thispaper have been derived from a diverse bodyof literature and experience. We have attempt-ed to incorporate various findings related toattributes of quality education and programsinto a general yet comprehensive conceptualframework in order to broaden our under-standing of the elements of quality vocationaleducation. We hope it will contribute to anincreased awareness of the major compo-nents influencing the quality of educationThis report is intended to inform policy-makers of the breadth of the elements ofquality and to suggest an integrated frame-work for assessing quality to policymakersand vocational education evaluationtheoreticians.

For purposes of this inquiry, vocationaleducation was defined as school-relatedactivities that occur in secondary schoolsand 1-and 2-year programs in postsecondaryschools: vocational education was consid-ered to be a subset of general, formal educa-tion, which in turn is a subset of educationcoming from a variety of life experiences.

,1 0

I

COMPONENT ELEMENTS INDICATOR

Teacher Skills Technical Knowledge Credentials, licenses

Technical Experience Teacher must have minimum of3 years of full-time work experi-ence in field prior to hire

Teaching Methods

Teacher must accumulate X num-ber of hours of field experiencethrough co-op program or summeremployment ior each 5-year teach-ing Period

Prior to hire, teacher must have20 quarter hours in teachingmethods and 10 quarter hoursin history-of-education course workfrom an accredited college

Teacher must take minimum of4 quarter hours in tesching meth-ods course work from an accreditedcollege for each 5 years of teaching

Competency Development Job-specific Skills Program graduate receives a passinggrade on licensing exam

Student Utilization of Patterns of A minimum of 65% of vocationalInstructional Process Participation program graduates are classified as

Concentrators and Limited Concen-trators

Basic Skills Reading Program graduate score ranks above50% percentile on standardizedreading comprehension test

Tenho", Student, and Student, MaterialMaterial Interaction

Over a period of 4 lecture classes,student should spend no morethan 25% of class tirne readingmaterial in class

In metal-working lab, 2-personstudent teams should be workingtogether between 40% and 60%of class time

Figure 4. Examples of quality indicators

29

41

Our framework for the elements of qual-ity grew out of a multiphased process thatincluded reviews of related literature, expertinput, and key informant interviews. Theliterature provided preliminary structure tothe framework which was further refined anddefined by expert opinion and key informantcommentaries.

Some vocational education has beendirected by law to be evaivated in terms thatinclude job placement in training-relatedemployment, employer satisfaction, and qual-ity and availability of instructional offerings.This project, however, initially explored a va-riety of desirable outcomes that may help todefine high quality vocational education pro-grams and subsequently identified the com-ponents of the educational environment thatmay support attainment of those outcomes.The outcomes were therefore considered asgoals and the structures and processeswithin the educational environment wereperceived as means for achieving the goals.The resulting framework for the elements ofvocatinnal education quality (figure 1) wasconceptualized as a complex environmentconverging on the educational experience,driven by desired individual and societal out-comes, and influenced by environmentalinputs such as community setting and labormarket conditions.

The Framework: Elements of OualityVocational Education

Elements of Ouality Outcome

Two general types of outcomes areincluded in our framework: individual andsocietal with a subset of societal outcomesspecific to employers. Four elements of qual-ity relating to individual uutcomes were iden-tified and defined. Competency developmentencompasses the development of work-related skills (both occupational and job spe-cific), ba;ic skills such as verbal andmathematical, and general problem-solvingability. Positive interpersonal interaction

30

involves the ability of people to have positiveand mutually supportive relations with othersin the home, the workplace, and the com-munity. A sense of efficacy is an attitude ofconfidence in one's ability to utilize skills toachieve goals. The ability to manage a largearray of tasks without having to depend uponsomeone else is designated as a sense ofindependence. These are the four generalelements of quality individual outcomesincluded within our conceptual framework

Social stability and employer needs arethe primary elements of quality within thesocietal outcome portion of our framework.Social stability does not imply social stagna-tion; rather, it means gradual, orderlychange. Provision of essential services, qual-ity of work products, resources for self-support, and societal improvement are thesubelements that support social stability.

We also examined a subset of societaloutcomes specific to employers. To be worththe investment from the employer's point ofview, vocational education should contributeto available skills, reduced training time, andhigh productivity. These are the three ele-ments of quality related to employers needs.Our definition of quality vocational educationoutcomes includes, but is not limited to,employment-related skills. Rather, vocationaleducation of a positive quality must alsoenhance the development of desirable oreven essential skills or dispositions that arenot directly employment related.

Elements of Ouality withinthe Educational Experience

Taking a systems perspective, theprocesses and structures that define the ele-ments of quality within the vocational educa-tion e Ape, :ence should relate to intendedindividual and societal outcomes. Our modeldivides the educational experience into twodomains, school context and school processSchool context deals with both physical andorganizational aspects of the school unit.Personnel, facilities, equipment, policies, and

4 2

formal organizational structure all fall withinschOol context. The elements of qualitywithin school Context that are formallyincluded in our model are school setting,classroom setting, tachr skills, teachercharacteristics, and peer group characteris-tics. Figuratively speaking, school contextprovideS the setting for school process,which provides the "action." Estimations ofqtiality of school process would considersuch things as cOurse content; course-specific equipment; interaction betweenteacher, student, and materials; and level offormal student involvement in the vocationaleducation program. The elements of qualityformally included within school process aretacher, student, and material interaction;instructional content and material; and stu-dnt utilization of instructional process.

These elements of quality within theeducational experience should relate tointermediate objectives that, in turn, shouldsupport attainment of desired outcomes; thelinkage should be systematic between qualityof experience and quality of outcome inorder to maintain and replicate effective pro-grams and make modifications as necessary.

Assessing the Quality ofVocational Education

The elements of quality within ourframework are generalizable across schoolunit and program. However, the specific indi-cators of quality associated with them mayvary across school and program and arelikely to vary over time.

31

Considerable effort has teen expendedin developing evaluative strategies to facili-tate the assessment of quality. Evaluationmodels should combine structural, process,and outcome approaches in order to assessthe range of elements of quality within ourframework.

Specific indicators of quality need to bedeveloped at the individual school unit level.It is not practically possible to tabulateexhaustively all the indicators related to theelements of our model, nor is it financiallypractical to support an evaluation systemcapable of gathering and analyzing such datain a complete form. A process for identifyingand prioritizing subelements within themodel is suggested as a practical startingpoint for developing indicators. A combina-tion of quantitative and qualitative methods issuited to a comprehensive evaluation of qual-ity, and both normative and empiricalapproaches should be drawn upon for devel-oping indicators.

In summary, this discussion presents ajustification and set of procedures that willpermit an integrated consideration of theelements of vocational education quality. Wehave advocated considering a broad set ofelements of quality rather than a narrowlydefined or segregated set, so that tne evalua-tion of vocational education reflects its con-tribution to a set of living experiences inaddition to employment.

43

REFERENCES

Abramson, Lyn Y.; Seligman, Martin E. P.; Teesdale, John D. "Learned Helplessness in Humans:Critique and Reformulation." Journal of Abnormal Psychology 87, no. 1 (1978). 49-74.

American Association of School Administrators, The Role of the Principal in Effective Schools:Problems & Solutions. The AASA Critical Issues Series. Sacramento, CA: Education NewsService, 1983.

Benson, Charles. "The Ouestion of Quality." VocEd 57, no. 6. (September 1982): 27-29, 55.

Bleuner, Robert. Alienation and Freedom. Chicago: The University of Chicago Press, 1964.

Bowles, Samuel, and Gintis, Herbert. Schooling in Capitalist America. New York: Basic Books,1976.

Campbell, Paul B.; Gardner, John A.; and Seitz, Patricia. Portsecondary Experiences of Studentswith Varying ParticipaUon in Secondary Vocational Education. Columbus: The National Cen-ter for Research in Vocational Education, The Ohio State University, 1982.

Campbell, Paul B.; Mertens, Donna M.; Seitz, Patricia; and Cox, Sterling. Job Satisfaction: Antece-dents and Associations. Columbus: The National Center for Research in Vocational Educa-tion, The Ohio State University, 1982.

Cook, Thomas D., and Reichert, Charles, S., eds. Qualitative and Quantitative Methods in Evalua-tion Research. Beverly Hills, CA: Sage Publications, 1979.

Dewey, John. Democracy and Education: An Introduction to the Philosophy of Education. NewYork: Macmillan Company, 1916.

Drewes, D. W.; Nerden, J. T.; Lawrence, J. E. C.; and Oglesby, E. H. Questions in Vocational Edu-cation Career and Vocational Education Professional Development Report No. 18. Raleigh:North Carolina State University, 1975.

Federal Register. (3 October 1977): Section 104.402.

Frankena, William K. "A Model for Analyzing a Philosophy of Education." In Toward a PhilosophicPerspective, edited by Joe Park, pp. 139-144. New York: Macmillan Publishing Co., 1974.

Gardner, John W. Excellence. New York: Harper and Row, 1961.

Grasso, John T. Impact Evaluation in Vocational Education: The State of the Art. Columbus: TheNational Center for Research in Vocational Education, The Ohio State 1979

33

4 4

Ha lasz, I. M.; Behm. K. S.; and Fisch, M. R. Influences on Secondary and PostsecondaryVocational-Technical Student Time on Task. Columbus: The National Center for Research inVocational Education, The Ohio State University, 1984.

Hills, Stephen M., and Crowley, Joan E. "The Quality of Youth Employment." In Pathways to theFuture, Vol. 3, edited by Michael E. Borus, pp. +-24. Columbus: Center for Human ResourceResearch, The Ohio State University, 1983.

Hunt, Neon, and Woods, Jerry. Interrupted Education: Students Who Drop Out. Brooklyn: Office ofEducation, New York City, 1979.

Katzell, Raymond A. "Changing Attitudes toward Work." In Work in America the Decade Ahead.edited by Clark Kerr and Jerome M. Rosow, pp. 35-57. New York: Van Nostrand ReinholdCompany, 1979.

Kelly, Harold H. "Attribution Theory in Social Psychology." In Nebraska Symposium on Motiva-tion. Vol. 15, edited by D. Levine. Lincoln: University of Nebraska Press, 1967.

McKinney, F.; Franchak, S. J.; Halau-Salster, I.; Morrison, I.; and McElwain, D. Factors Relating tothe Job Placement of Former Secondary Vocational Education Students. Columbus: TheNational Center for Research in Vocational Education, The Ohio State University, 1981. (ERICDocument Reproduction)

Miller, Juliet "Effective Schools Research and Vocational Education." ERIC Digest, no. 31.Columbus: The National Center for Research in Vocational Education, The Ohio State Uni-versity, 1984.

National Commission on Secondary Vocational Education. The Unfinished Agenda: The Role ofVocational Education in the High School. Columbus: The National Center for Research inVocational Education, The Ohio State University, 1984.

National Research Council, Commission on Behavioral and Social Sciences and Education. Edu-cation for Tomorrow's Jobs. Washington DC: National Academy Press, 1983.

Riedel, Donald D. C.; Tisch ler, Gary L.; and Myers, Jerome K., eds. Patient Care Evaluation in Men-tal Health Programs. Cambridge, MA: Ballinger Publishing Company. 1974.

Sternberg, Robert J. "What Should Intelligence Tests Test? Implications of a Triarchic Theory ofIntelligence for Intelligence Testing." Educational Researcher 13, no. 1 (January 1984): 5-15.

Taylor. Charles. "Neutrality in Political Science." In Social Structure and Political Theory, editedby William E. Connolly and Glen Gordan. Lexington, MA: D. C. Heath and Co., 1974.

Taylor, Shelley E. "Adjustment to Threatening Events: A Theory of Cognative Adaption." AmericanPsychologist 38, no. 11 (November 1983): 1161-1173

Tennyson, W. W.; Hansen, L. S.; raurens, M. K.; and Amtholz, M. B. Career Development Educa-tion: A Program Approach for Teachers and Counselors. Alexandria. VA: National VocationalGuidance Association, 1980.

34

45

Thimmesh. Hilary, O.S.B. "Education Is about Civilization: Lose Sight of That and You Lose Sightof Humanity." The Chronicle of Highar Education 28, no 17 (20 June 1984). 64

Thurow, Lester C. Generating Inequality: Mechanisms of Oistribution in the U.S. Economy. NewYork: Basic Books, 1975.

Westbrook, John D. Considering the Research: What Makes an Effective School? Austin, TX.Southwest Educational Development Laboratory, 1982.

Wong, Paul T. P., and Weiner, Bernard. "When People Ask 'Why' Ouestions, and the Heuristics ofAttributional Search." Journal of Personality and Social Psychology 40, no 4 (1981): 650-663.

35

'1 6

"

,',111

riV

1.44,

1,,Tiop