DOCUMENT RESUME

ED 093 812 95 SP 008 176

AUTHOR Burdin, Joel L., Comp.; Cruickshank*, Donald R.,Comp.

TITLE Protocol Materials: Training Materials for UnitingTheory and Practice.

INSTITUTION ERIC Clearinghouse on Teacher Education, Washington,D.C.

SPONS AGENCY National Inst. of Education (DHEW), Washington,D.C.

PUB DATE Aug 74NOTE 114p.

EDRS PRICEDESCRIPTORS

MF-$0.75 HC-35.40 PLUS POSTAGE*Behavioral Objectives; *Program Development;*Program Evaluation; *Protocol Materials; *TeacherBehavior; Teacher Education

ABSTRACTThis publication brings together seven papers by

writers who have been extensively involved in the preparation and useof protocol materials in teacher education. These papers are: (ayProtocol MaterialS: Historical Notes on Protocols Development, byDoris V. Gunderson; (b) The Protocol Materials Movement: An Exemplarof Efforts To Wed Theory and Practice in Teacher Education, by DonaldR. Cruickshank; (c) The University of Colorado Protocol Project: ACase Study, by Celeste P. Woodley and Laura A. Driscoll; (d) A

CatalOgue of Concepts in the Pedagogical Domain of Teacher Education,by Bryce B. Hudgins; (e) The Protocol Materials Program, by Dopald E.Orlosky; (f) A Protocol Materials Evaluation: The Language ofChildren, by Victor M. Rentel; and (g) b Survey of Protocol MaterialsEvaluation, by John E. Cooper. (MBM)

1

PROTOCOL MATERIALS: TRAININGMATERIALS FOR UNITING THEORY AND

PRACTICE

Compiled byJoel L. Burdin

and Donald R. Cruickshank

U.S DEPARTMENT OF HL ALTH,EDUCATION & WELFARE

NATIONAL INSTITUTE OFEDUCATION

THIS DOCUMENT HAS BEEN REPRODUCED EXACTLY AS RECEIVED FROMTHE PERSON OR ORGANIZATION ORIGINACING IT POINTS OF VIEW OR OPINIONSSTATED DO NOT NECESSARILY REPRESENT OFFICIAL NATIONAL INSTITUTE OFEDUCATION POSITION OR 0OLICY

Published byERIC Clearinghouse on Teacher Education

Number One Dupont Circle, N.W.Washirgton, D.C. 20036

cN\N Sponsored by: American Association ofColleges for Teacher Education (fiscal

agent); Association of Teacher Educators;Instruction and Professional Development,

National Education Association

.11101 August 1974

SP 008 176

CONTENTS

FOREWORD

ABSTRACT AND ERIC DESCRIPTORS vii

THEME INTRODUCTION ixProtocols: A Reality Approach to Vitalizing Teacher Education

PAPERS"Protocol Materials: Historical Notes on Protocols Develop-

ment," by Doris V. Gunderson"The Protocol Materials Movement: An Exemplar of Efforts

To Wed Theory and Practice in Teacher Education," byDonald R. Cruickshank

"The University of Colorado Protocol Project: A Case Study,"by Celeste P. Woodley and Laura A. Driscoll

"A Catalogue of Concepts in the Pedagogical Domain of TeacherEducation," by Bryce B. Hudgins 51

"The Protocol Materials Program," by Donald E. 67

"A Protocol Materials Evaluation: The Lahguage of Children,"by Victor M. Rentel

"A Survey of Protocol Materials Evaluation," by John E.Cooper

1

7

33

81

95

ABOUT ERIC 115

FOREWORD

An examination of the educational data base reveals relatively fewcitations on protocol materials. A very large number of ERIC-processeddocuments indexed under "protocol materials" are primarily about micro-teaching. Undoubtedly there are some documents which hive been circulatedamong those actively producing protocol materials, but, for whateverreason, materials on protocols available to the general educationalcommunity are limited. The Clearinghouse is pleased to present a seriesof papers which provide some suggestions on how to move from a con-ceptualization on integrating theory and practice; to production,field testing, and revising; and finally to publicizing'and marketing.

Special credit is due Donald Cruickshank for his help inconceptualizing the contents of this publication. Recognition is alsodue to the writers who contributed their articles.

It is appropriate to note the unique role played by Lawrence Lipsitz,editor of Educational Technology. The possibility that this series ofarticles would be published in that journal provided a stimulus to eachof us. The ERIC system benefits from the cooperation of editorswho provide a valued supplement to the dissemination capabilitiesof the ERIC Document Reproduction Service.

It is hoped that this document will provide a solid addition tothe educational literature on an important topic: the creation oftraining materials which have considerable potential for transformingthe abstract study of education into something more vital, dynamic,arid meaningful. When this happens, it will be a just reward for theinventor of the protocol idea in education--B. Othanel Smith.

You may do further research on this topic by checking issues ofResearch in Education ERIE) and Current Index to Journals in Education(CIJE). Both RIE and CIJE use the sime descriptors (index terms).Documents in RIE are listed in blocks according to the clearinghousecode letters which processed them, beginning with the ERIC Clearinghouseon Career Education (CE) and ending with the ERIC Clearinghouse on theDisadvantaged (UD). The clearinghouse code letters, which are listed atthe beginning of RIE, appear opposite the ED number at the beginning ofeach entry. "SP" (School Personnel) designates documents processed bythe ERIC Clearinghouse on Teacher Education. For readers uncertainhow to use ERIC capabilities effectively, we recommend How To Conducta Search Through ERIC, ED 036 499, microfiche $.75; hardcopy $1.85.It is available from the ERIC Document Reproduction Service, P. O.

Box 190, Arlington, Virginia 22210.

--Joel L. Burdin, Director

February 1974

ABSTRACT

This publication brings together seven papers by writers whonave been extensively involved in the preparation and use of protocolmaterials in teacher education; These papers are a) Protocol Materials:Historical Notes on Protocols Development, by Doris V. Gunderson;b) The Protocol Materials Movement: An Exemplar of Efforts ToWed Theory and Practice in Teacher Education, by uonald R. Cruickshank;c) The University of Colorado Protocol Project: A Case Study, byCeleste P. Woodley and Laura A. Driscoll; d) A Catalogue of Conceptsin the Pedagogical Demain of Teacher Education, by Bryce B. Hudgins;e) The Protocol Materials Program, by Donald E. Orlosky; f) AProtocol Materials Evaluation' The Language of Children, by VictorM. Rentel; and g) A Surve of Protocol Materials Evaluation, byJohn E. Cooper. (MBM)

ERIC DESCRIPTORS

To expand a bibliography using ERIC, descriptors or search termsare used. To use a descriptor: (1) Look up the descriptor in theSUBJECT INDEX of monthly, semi-annual, or annual issue of Research inEducation (RIE). (2) Beneath the descriptors you will find title(s)of documents. Decide which title(s) you wish to pursue. (3) Note the"ED" number beside the title. (4) Look up the "ED" number in the"DOCUMENT RESUME SECTION" of the appropriate issue of RIE. With thenumber you will find a summary of the document and often the document'scost in micrcfiche and/or hardcopy. (5) Repeat the above procedure,if desired, fog other issues of RIE and for other descriptors. (6) Forinformation about how to order ERIC documents, turn to the back pagesof RIE. (fl Indexes and annotations of journal article5 can be foundn Current Index to Journals in Education by following the same pro -

cedure. Periodical articles cannot be secured through ERIC.

TOPIC: Protocol Materials: Training Materials For Uniting Theory andPractice

DESCRIPTORS TO USE IN CONTINUING SEARCH OF RIE AND CIJE:

*Behavioral Objectives*Program Development*Program Evaluation

*Protocol Materials*Teacher Behavior*Teacher Education

*Asterisk(s) indicate major descriptors.

vii

Vol

lt

THEME INTRODUCTIONProtocols: A Reality Approach toVitalizing Teacher Education

Since Smith introduced the term "protocols" into the literature, *there has been a continuing search for the meaning and use of suchmaterials. Protocol materials are commonly recognized a a high -

potential approach to vitalizing pre- and in-service teacher education.These materials supplement others used in teacher education and providea creative way to meld theory and practice into effective learning.

The literature on protocol materials tends to be somewhat limited.The authors of these papers hope that they will increase understandingof an important concept, encourage experiment with effective use, andstimulate assessment of that use and continual efforts to improve protocolsutilization.

Gunderson delineates the need for developing understandings aboutlearning and gives an overview of how protocols conceptualization anddevelopment were stimulated by U. S. Office of Education (USOE) efforts.Her perspective is that of a former USOE project monitor for theprotocols project.

Cruickshank defines protocols as an original record of an eventof educational significance, utilized in order to permit learners tointerpret the event or solve the problem depicted in the event, withappropriate concepts from related fields of study. For clarity, heexplains the term "events of educational significance" and distinguishesbetween protocols and field-based education. lie discusses very brieflythe development of protocols, then examines how the initiation of USOEsupport modified the original definition. (The USOE Master CoordinateSystem is appended.) Finally he questions whether the powerful notionof protccols as originally conceived will become lost as the individualprojects move toward a less empirically based, more prescriptive concept.

Woodley and Driscoll describe the University of Colorado ProtocCProject and show how an educational idea is translated.into an instruc-tional product. They detail the steps taken in developing, testing,and disseminating the protocol-materials relevant to teacher-pupilclassroom interaction and share the insights, methods, problems,mistakes, and successes of their project.

Hudgins responds to the need felt since the beginning of the protocolmaterials program for a conceptual map of the pedagogi-..al domain of teachereducation. This should aid in deciding which concepts, or at least, whichgroup of concepts, should be developed into protocols and should havetwo functions: (a) to identify appropriate concepts in the literatureof teacher education and (b) to.give salient and significant informationto help portray the concept. Such a map has been under development underthe direction of the author. While it features concepts pertaining to inter-active teaching, the author recognizes that other concepts of teachereducation will require future development. An extensive example ofseeking out and. classifying one concept is given, the format of thecatalogue Ls explained, samples from the catalogue are shown.

* B. Othanel Smith and others, Teachers for the :Real World (Washington,D.C.: American Association of Colleges for Teacher Education, 1969)pp.52-53, 62-64, and 158.

xi

Orlosky provides a history of the protocol materials and thendiscusses major developments, including the definition of protocols,development procedures, selection of concepts, protocols vs. observation,the master plan, the technical and educational criteria for the protocols,supplementary materials included in the packages, field testing, anddissemination. He concludes with a consideration of the current statusof the program and summarizes the possible future for protocols.

Rentel analyzes protocol materials developed by Ohio State UniversityCollege of Education to illustrate concepts of oral language. The authordescribes how the concepts were selected and briefly defines thosedeveloped as protocols. He discusses the method and procedure for revisingthe materials, based on the results of a questionnaire answered by thoseinvolved in the revisions. The author discusses the recommendationsmade by participants and instructors. In conclusion, he describes themethod, procedure, instruments, and results of the product evaluationand discusses their meaning.

Cooper summarizes evidence of the effectiveness of protocol materialsprepared in a number of institutions, including Utah State University,Michigan Statc University, the Far West Laboratory for Educational

Research and Development, Indiana University, and Southern IllinoisUniversity at Edwardsville. He places this evaluation in the contextof a brief review of protocol materials and concludes with somerecommendations for the future.

xii

Protocol Materials: Historical Notes onProtocols Development\

'./

by Doris V. Gunderson

Universities and.school systems, the inst*tutions that train edu-cational personnel, recognize the need'for-mAerial that will helpteachers acquire the understanding and skills necessary to teaching.Although innumerable written materials exist in education and alliedfields, little or no evidence indicates that. these materials are toolscapable of inducing desired behaviors in pre- or in-service teachers.

Traditionally, the teacher's primary function at any Level hasbeen to help students learn. If teachers are to help students learn,they must know how to teach so'that teachers will. learn. Teachersmust know the subject matter, and they must know how.to teach. Theteacher must understand the student and understand the learning pro-cess. Individuals differ in learning rates and respond to teaching-learniag situations.in various ways.

Developing understandings about learning is not easy. Extensiveobservation of students is essential, yet observation without focusaccomplishes little.- The observation must be directed; the teachermust look for particular kinds of behaviors. There is no guaranteethat a given stimulus will result in the behavior theteacher wishesto observe. Even if a particular behavior is exhibited, the instancemay be fleeting, and the observer has only his recollection for con-sidered study.

The observer's preparation for interpretinebehavior usuallyconsists of little more than courses presented in the traditionalreading-lecture-discussion manner.' The instructionis divorced fromreality. Onemeans-of bridging the gap between theory and realityis to reproduce a variety of behaviors of students, teachers, andothers in a permanently recallable form. A p'articular segment of.behavior can thus be produced again and again to be studied, analyzed,and the concepts appropriate to its interpretation spotlighted, ex-plained, learned, and reviewed. The concepts can be applied to theunderstanding of other behaViors. Instructional materials of thistype are referred to as protocol materials.

Protocol materials constitute one of two eneral categories ofinstructional materials for teachers, namelyithe-materials that directthe pre- or in-service teacher in studying 'his -and others' behavior.Materials that guide him in systematic practice of the skills he -mustacquire are called training materials and are designed to help theteacher in the acquisition of skills. They provide for (a) identi-fication of skills, (b) description of behavior entailed by the skills,(c) performance of the behavior, and (d) feedback to the performer andfurther performance by him.

Cognitions are developed primarily through the use of prot6colmaterials which provide (a) segments of behavior categorized forthe purpose of teaching concepts and principles used in interpretingbehavior as well as the social context in which the teacher works, (b)

segments of behavior categorized for the purpose of teaching knowledgeabout knowledge, and (c) ,segments of behavior categorized for thepurpose of teaching self-understanding.

3

Protocol materials are designed to bridge the gap between theoryand the teaching-learning situation. Since they provide reproductionsof behavior; they foster the teacher's interpretive and diagnosticcompetency.

A basic problem in preparing protocol materials is precision indefining a concept; a concept must be defined exactly and its attributesspecified in terms which permit no misunderstanding. Concepts, accordingto Carroll, are properties of organismic experience, the "abstracted andoften cognitively structured classes of mental experience learned byorganisms in the course.of their life histories." As the complexit., ofthe concepts increases:the necessity for an appropriate sequencing ofpositive -and negative.instances to assure adequate learning c.r the con-cept becomes greater.1

Recognizing ;he demand for materials for teaching concepts inteacher education programs, the Bureau of Educional Personnel Devel-opment (USOE) in 1970 initiated an effort tr train educational personnelto develop and use protocol materials. NItterials developed for trainingteachers generally have been prepared in isolation with no field testingduring the development stages. Si,Ice protocol materials should beattuned to the problems teach:.s encounter, the Office of Educationinsisted that people shoul0 be trained to develop protocol materials,with field testing and - -edification based on the field testing an inte-gral part of the trrCiing program. In addition to the field testingcarried out by eat:, project director, the Florida State Department ofEducation, under a grant from the Office of Education, is conductingfield tests of protoccl materials developed in the various projects withpre- and in- service teachers in school systems and institutions ofhigher education in Florida.

The protocol materials effort is essentially a training program.Project directors are trained to develop and use protocol materials,'The effort is under the direction of the Leadership Training Institute,a group of consultants outsile the Office of Education, headed by B.Othanel Smith of the University of South Florida. The group is respon-sible for providing technical assistance to the project directors. Thetraining'involves several stages'. The concepts to be exemplified inprotocol materials must be seleCted and analyzed. The concepts must'be critical to teacher education; that is, they must be concepts thatteachers need to know. A paramount consideration is utility; the con-cepts to be portrayed in the materials will he useful at any institutionor agency which trains or retrains teachers.

The materials being produced, primarily films, cover diversesubject matter areas such as educational psychology, social psychology,reading, literature, language acquisition, Black English, teachinganalysis, and social studies. The important result of this projectto the future of teacher training lies in th9 fact that these leadersin the field of teacher education will them<elves have acquired skillsin the development of protocol materials.

NOTES

1. John B. Carroll, "Words, Meanings and Concepts," Harvard EducationalReview, September 1964, 'pp. 178-20,2.

I

.)

O

The Protocol Materials Movement: An Exemplar ofEfforts To Wed Theory and Practice in Teacher

Education

by Donald R. Cruickshank

PROTOCOLS

My initial encounter with the term "protocol" in an educationalcontext was confusing since I knew it only in the limited sense ofdescribing a rigid, long-established deferential code practiced ingovernment, especially diplomatic circles. Consequently, I brought anunsuitable meaning to the printed page which had to be corrected. In

the sense in which it is used herein, at least initially, a protocol'is an original record of an event of educational significance.

The rationale for creating and using protocols (or protocol mate-rials) probably was first brought to the attention of the educationalcommunity at large by Smith and others.' This group inquired into theeducation of disadvantaged youth and teacher education and subsequentlymade several recommendations and proposals calling for change in thelatter. Their report states:

Teachers fail because they have not been trained calmly to analyzenew situations against a firm background of relevant theory. Typ-ically they base their interpretations of behavior on intuitionand common sense. . . . If the teacher is incapable of understand-ing classroom situations, the actions he takes will often increasehis difficulties.2

This initial assertion that teachers need to be-trained to analyzeclassroom situations against a firm background of relevant theory isamplified and supported as theoretical knowledge is defined,3 and itis noted that such knowledge is becoming more abundant in the behavioralsciences.4 Further, the Smith task force asserts that "the basic ele-ments of theoretical knowledge are concepts."5 Now we can begin tooperationalize the earlier definition of protocols from an originalrecord of an event of educational significance to an original recordof an event of educational significance utilized in order to permitlearners to interpret the event or solve the problem depicted in theev-mt using appropriate concepts fr.:5m related fields of study includingpsychology, sociology, anthropology, philosophy, or others.

It seems clear from the extended definition that at least twodecisions are paramount for the developer and/or user of protocols.First, he must answer the question, "What is an event of educationalsignificance?" Second, he must determine which available concepts arelikely to increase the subject's capacity to interpret and understandthe event recorded in the protocol. This one-two procedure is amplifiedelsewhere() and referred to as the "situation-first strategy."

Events of Educational Significance

/Smith equates an event of educational significance with situationsthe teacher most frequently encounters but notes that since no systematicanalysis of the teacher's frequent work exists, the event must beselected from practical experience.i The following categories ofevents are suggested:

1. Classroom situations

9

a. Instructional situationsb. Situations of classroom management and control

2. Extraclassroom situationsa. Situations that arise in planning school programs,

working with peers and the administrationb. Situations that occur in working with parents and other

members of the communityc. Situations that occur in working in professional organi-

zations.8

Following this "situations-first strategy," once events of educa-tional significance are determined, they must be inspected to considerwhat theoretical notionsconceptsare needed to understand and inter-pret them. Indeed, Smith cautions, "No instruction should begin untilthe instructor has studied the [event] with extreme cale."9

The following will help to clarify. It describes an example ofwhat inner-city teachers consider to be frequent and bothersome class-room behavior, one criterion which could be applied in selecting eventsof educational significance. It probably falls in category lb above.

The teacher has broken the sixth grade class into small groupseach of which is experimenting with a pendulum. In one of thegroups, one student is swinging the pendulum, another is keepingtime using a stopwatch while a third is recording the time.Others are in the group but these three are prominent. Duringan interval between timing the period of the pendulum therecorder, Wesley, reaches over and abruptly takes the stopwatchfrom that student using it, Bradley. The teacher mildly admon-ishes Wesley and orders him to return the watch. Wesley doesso reluctantly and the experiment continues. A child from an-other group approaches the teacher and a conversation ensues.Soon scuffling noises occur and a fight breaks out betweenBradley and Wesley. The teacher intervenes verbally and thephysical aggression changes to name-calling and accusations.Bradley holds up his watch which is broken and claims thatWesley is responsible. Wesley responds that "It was an accident."Bradley counters callirg Wesley a liar. Wesley charges thatBradley stole the watch anyway and tells Bradley to just stealanother one as the incident closes."

If the above incident were used as a protocol, it follows that theteacher, alone or with the assistance of colleagues from the behavioralsciences, must inspect the event and consider what theoretical notions--concepts--are needed to understand and interpret it. To begin, onemight identify or describe the phenomena observed. These seem to in-clude sixth-grade class, grouping for instruction, experiment, pendulum,differentiation of labor,,stopwatch, keeping time, recording, seizure,disapproval-reprimand, acquiescence, interruption, fighting, inter-vention, name-calling, accusation, destruction, informing-tattling,and so forth. It is significant that the teacher educator be able tohelp students select from among these phenomena those which may be ofspecial interest and power in explaining, understanding, and inter-preting the event. Which phenomena and/or related concepts do you seeas most germane?

10

Up to this point it is clear that the protocol movement is con-ce7ned with finding out what teachers need to know in order to functionmore effectively in educationally significant situations. This isquite a departure from the usual method of teacher education whichexposes the undergraduate to knowledge organized seemingly logicallyinto courses in education! The usual method requires the learner toapply theory, explaining human behavior, for example, to the classroom.Conversely, the protocol way forces the teacher educator to begin withclassroom events anO. show how certain concepts make those events moreunderstandable.

Protocols and/versus Field-based Education

Since recently there has been a big push toward field-based teachereducation, a word of caution may be in order. Promoters of field-basedprograms are convinced that you learn about teaching by teaching and sonaturally professional education should ocLur where teaching takesplace--in the schools. The Smith task force makes this notion suspect,stating:

Teaching behavior is complex, involving interactions with bothpupils and materials of instruction. It cannot be studied in theclassroom because behavior perishes as it happens and nothing isleft to analyze except the memory or a check sheet. The fidelityof the memory is questionable and not detailed enough. The infor-mation contained on check sheets is almost no record at all. Tolearn to interpret situations they must be held in situ or repro-duced at will approximately as they occurred. It is then possibleto study the situations at length and use concepts . . . to

interpret them.11

If the protocol movement--an exemplar of theory into practice--istenable, it behooves the field-based movement to explain how it is, orwill be, taken into account therein. Quality of experience in a class-room may not be an adequate replacement for the kind of quality ofexperience the study of protocols coIld provide.

Development of Protocols

What is the progression of steps suggested in developing a proto-col program? As mentioned earlier, the Smith report equates an eventof educational significance with situations the teacher most frequentlyencounters. Consequently "the first step in the formulation of a[protocol materials program] is to identify the most general categoriesof situations teachers face."12 These situations become the objects -

of study and are recorded in some manner. They are then placed in pre-determined categories such as classroom instructional situations,classroom management situations, extraclassroom situations, or others.Next, the situations are studied intensely to determine what theoreticalknowledge (especially concepts) is required in order to understand andinterpret the phenomenon displayed in the situation. Finally, the pro-tocols are arranged in courses of instruction. 13

11

SUPPORT AND REDIRECTION VIA WASHINGTON

The Smith task force recognized that building new teacher educa-tion programs based upon protocols would be a massive and expensiveundertaking. It therefore recommended that "a number of institutions,federal and state agencies, and professional organizations could co-operate" in the endeavor.14

Since the U.S. Office of Education through the National DefenseEducation Act (NDEA) had financed the work of the Smith task force, itcame as no great surprise that it would be interested in consideringand supporting selected task force recommendations.

That support manifested itself in April 1970 when the formerBureau of Educational Personnel Development invited several selectedinstitutions to submit proposals for the development of protocol mate-rials. With USOE intervention and support, the protocol movement wasgiven sanction by Washington and thereby given prominence for a largesection of the education community which until that time probably wasnot well acquainted with the Smith report and its recommendations.(See "The Protocol Materials Program" by Orlosky, p. 69 for detailsof Washington's involvement. Herein, attention will be given only tothe request for proposals (RFP) which was the universities' officialinvitation to partake in the protocol movement.)

Guidelines

The RFP and especially the accompanying guidelines were to rede-fine the original notion of protocols and redirect materials development.Most agree that the redefined notion of protocols and the new directionwere laudatory. In fact, recipients of protocol grants used less of theSmith report and more of the Washington guidelines in their work. Afew argued and still maintain that although the Washington approach wasacceptable and appropriate, Smith's original notion of protocols asexemplars of classroom situation-based teacher education has never beenunderstood fully and certainly has not been implemented in the Wash-ington-directed movemc.71t.

The RFP cover letter from Don Davies, then Associate Commissionerof Education, made clear (a) that USOE was supporting the notion ofprotocols per the Smith task force, Oh) that the 40 persons receivingrequests for proposals had been recommended by an ad hoc advisorycommittee, (c) that not more than a half million dollars of USOE first-year money would be involved, and (d) that the USOE protocol materialsprogram was an "attempt to prepare and use such materials in order tohelp make the training of educational personnel more exact and real."Of the four enclosures accompanying Davies' letter, Attachment A whichexplained protocol materials and delineated the guidelines for sub-mitting proposals is probably most germane to the program's history.

In Attachment A the following points were made:

1. Although teachers in training take courses intended to help them

12

-interpret school-rP-ated pi'enomenon, this instruction is notmeaningful nor a successful as it should be because typicallyit occurs divorced from reality.

2. As presently constituted, the brie" periods of observation, par-ticipation, or other laboratory expe-jence do not cause teachersin training to see the relationship of educational theory and class-room practice.

3. Educational technology is now available which would permit thereproduction of school-related events which cln be used in teachereducation programs wherein appropriate theory can be learned andits relationship to reality seen. (For example, a particularsegment of classroom behavior can be reproduced and, as it isviewed, made more understandable through the study of specificrelated concepts.)

Further, a distinction was drawn between protocol materials used tointerpret and understand classroom life and training materials intendedto teach the prospective teacher skills.

The guidelines permitted significant alteration of Smith's defini-tion for protocols, eliminating the requirement that proposals be"original records" and permitting them to be staged representations ofreality,; more precisely, simulations, although the term is not mentioned.Henceforth, the concept of 5rotocol could be attached to any authenticrepresentation of an educationally significant event which is utilizedin order to permit 1,earners to interpret or solve the problem depictedusing appropriate concepts from related fields of study. Clearly theemphasis turned more toward interpreting the event rather than capturingvalid and correct ones, a problem which has eluded persons interestedin research on teaching for some time.15 Consequently, as pointed outelsewhere, already existing simulations or other forms of teaching mate-rials .hich are used to increase the learner's analytic and interpretivepowers through the use of theoretical knowledge suddenly became proto-cols, and the newer concept seemed to lose some of its newness andpunch. 16

The Master Coordinate System

The guidelines further contained an intricate Master CoordinateSystem (see Appendix A) which was to be used to "develop protocol mate-rials in an orderly way." In contrast to the Smith report whereinprotocols were defined as original records of events of educationalsignificance which would be made more understandable by the applicationof theory/concepts to them, the Washington guidelines and the MasterCoordinate System operationally defined protocols as illustrations ofconcepts. (See "The University of Colorado Protocol Project: A CaseStudy" p. 36 and "A Protocol Materials Evaldation: The Language ofChildren" p. 83.) The Woodley group chose to illustrate concepts inthe "pedagogical domain" of the Master Coordinate System. The Rentelgroup on the other hand worked in the "basic fields of knowledge." So,

following the guidelines, "original records" were replaced by "authenticones" and "events of educational signfi,7ancc by "concepts of educational

13

significance." (Hudgins, on p. 69 describes how he is attempting toprepare a catalog of the latter.) Naturally the concepts must beselected with some educational event in mind but that event is givenshort shrift. Some reviewers of the protocol movement, as this one,fault it for failing to identify its proposed teacher education cur-riculum (concepts) after a careful analysis of the real world ofteaching (educationally significant events). Instead, in a manner notwell-suited to pedagogy, protocol developers were asked to behave asif there were a thoroughly explicated discipline of education whichpresented concepts or ideas of major magnitude and accepted validity.17All the developer had to do was to select an ornament (concept) from theChristmas tree (education discipline). Naturally only the most "illumi-nating" would be chosen. What happened in most cases was that developersfell back to looking through related disciplines for concepts. In theusual tradition, concepts were lifted from psychology, for example,and many protocol projects simply became projects to develop visualmaterials which illustrate concepts from the behavioral sciences. Theconcepts became the ends rather than the means, primary rather thansecondary, as the Smith task force seemed to advocate.

Concept Overkill?

Further, many projects seemed to forget that concepts often canbe gained very quickly through definition. Projects seemed to runamok visually illustrating concepts. Concept overkill may have re-sulted. The above would suggest that although the Smith task force(directly or inadvertently) argued for the study of school life andthe selection or development of theory to understand it, the protocolmovement resorted to looking for some "neat" concepts which could bemade visual and "colorful" via a protocol. Theory building and theoryrelating simply have not reached their potential yet in the movement.

On June 1, 1970 the winners of the protocol competition were an-nounced in a letter to members of Congress, and eleven projects in tenstates were funded for the first year.

The projects were to be overseen by a Leadership Training Insti-tute (LTI)18 led by Smith. Orlosky (p.XXX) describes LTI, its member-ship, and function. A thorough description of the individual protocolprojects and the materials produced through 1971 is found in Kincaid.'9

A MAGNIFICANT OBSESSION?

Probably when the protocol program is recorded in American educa-tional history, it will be remembered as a magnificant attempt (perhapsan obsession) to relate theory to practice. Since at least anotheryear of federal sponsorship remains, it is not appropriate to considerwhether or not the movement will achieve this hoped-for result. Atthis writing, however, several things are becoming clearer and areworthy of comment. Individual protocol projects have moved away fromthe situation-first, analytic, interpretive strategy suggested by theSmith task force toward a less empirically based and more prescriptive

14

concept-first.approach based upon the Master Coordinate System. In

effect, there seems to be a considerable difference between the originalnotion and hoped-for result of protocols and what seems to be evolving.

Additionally, in their effort to identify concepts to undergirdmaterial development, project personnel seem to have either selectedsome which are not as acceptable to teacher educators or, as suggestedearlier, concept overkill has occurred. Since the concepts which havebeen visualized are not new or uncommon, most likely the latter hasoccurred. Developers may have treated the concepts to be illustratedtoo similarly, assuming that all learners had no advanced organizers,preinformation, or experience which would facilitate learning theconcept. Certainly some concepts are learned more easily than othersand must be learned differently by different people.

If a concept-first approach continues to be followed, each conceptconsidered for illustration should be placed. in juxtaposition with thevery useful "cone of experience" to determine whether it is best learnedand necessarily learned in the context of an elaborate visual experi-ence. Films illustrating concepts should be used only when somewhatmore direct, concrete forms of learning are required. There is littlereason to develop elaborate and extensive visuals for use in conceptformation when simpler, less expensive, or less time-consumingapproaches are available and justifiable. Teacher educators simplywill not spend more time or money to teach a concept than is necessary.All things being equal, given the choice of defining a concept such as"peer group" or having preservice teachers learn the concept by seeingan illustrative film or filmstrip, the teacher is likely to use themore efficient, parsimonious means. In several instances, in orderto learn a very few concepts the learner must engage in protocol studyfor a disproportionate amount of time. Consequently, as noted above,publishers do not seem to be excited about the possibility of the adoptionof protocols into the professional curriculum. Of course, cost isanother deterrent not to be underestimated.

POINT OF VIEW

In conclusion, the Smith task force suggested a powerful notionwhich, if we arc not careful, may become lost. It is possible thatthe sole difference between the way educational theory is learned nowand the way it would be learned utilizing a protocol is that in thelatter the concepts are illustrated more elaborately. Elaborate ill-ustration may not he related to the teacher's ability to recall thatconcept when necessary in order to better analyze and interpret aclassroom event. I tend to agree that the concept must be learnedin situ. To me the protocol .should illustrate the significant educa-tional event rather than the concepts used to interpret it. It seems'somehow we have gotten the cart before the horse. Once we have recordedthese events we have, perhaps for the first time, an opportunity empir-ically to develop a classroom situation-based curriculum For teachers.Concepts, skills, and attitudes will he found to understand and interpretthoe s i tuat ions. They will he learned or di seoVered in and used inrelation to these .ovcLs. These concepts can he illustrated as nec-essary. It is onl; from events that theory can he derived and to them

15

that it takes on utility. It seems we may end up with concepts but noeducationally significant events to which they can be readily applied.Are the concepts educationally significant events in themselves? Whereis the real world of teaching called for in Teachers for the Real World?

16

APPENDIX A

MASTER COORDINATE SYSTEM

COMPONENTS OF THE GENERAL PLAN

In order to develop protocol materials in an orderly way, it isnecessary to follow a general plan. The plan should be as comprehensiveas possible and free of doctrines about what teachers are to be preparedto do and how they are to be prepared to do it. If these conditions aresatisfied, those who are engaged in teacher education would be able totake part in a national effort to develop materials without being framedby theories and doctrines to which they cannot subscribe.

The general plan consists Of two basic interrelated components orsub-plans: one for the pedagogical domain and one for the basic fieldsof knowledge. Each sub-plan will be presented in the form of a coordi-nate system comprised of three dimensions. Essentially, protocol mate-rials to be developed in the pedagogical domain should be concerned withthe act of teaching and of learning, with the behavior or teachers,learners, and teachers and learners in interaction. In contrast, pro-tocol materials in the domain of basic fields of knowledge should beconcerned with the content of what is taught--or, more specifically,with the knowledge about the knowledge that is taught.

The protocol materials to be developed in either plan should beseen initially in terms of the master coordinate system for that plan(see Figures 1 and 5). That is, in the pedagogical plan, a decisionshould be made about the levels, behaviors, and settings to be portrayedin the protocol materials. In the basic-fields-of-knowledge plan, deci-sions must be made about the levels, types of knowledge about knowledge,and areas of knowledge to be portrayed in the protocol material. Follow-ing this procedure will help "chart" the kinds of protocol materialsneeded as part of a large-scale effort in developing a variety of suchmaterials.

PEDAGOGICAL PLAN

The pedagogical plan is a coordinate system comprised of threegeneric categories: setting, level, and behavior. These terms willbe defined later, but for immediate purposes "setting" refers to thecontext in which behavior occurs, "level" stands for the stage of aperson's general development, and "behavior" for observable activities.Figure 1 depicts the system which is described in detail in the pagesfollowing.

In order to indicate the various types of protocol materials thatcan be developed in accordance with the master plan, it is necessaryto derive subordinate systems. These will be set forth after the masterplan has been presented. The categories and sub-categories in thesystem below are purposely general. The lack of specificity under thecategory "Teacher-Pupil" or sub-category "Instructional: Skill," forexample, allows the developer a high degree of flexibility in deciding

17 .

18

Figure 1

Master Coordinate System of.Settings, Behaviors, and Levels

(Co) Community

Settings (F) Family

(PG) Peer Group

(S) School

(C) Classroom

(T) Teacher

(Tp) Teacher -pupil

(P) Pupil

Levels

upon the specific behaviors he wishes to record. It should be rememberedthat the categories are intended to help assure an adequate coverage ofsettings, behaviors, and levels in the protocol materials to be devel-oped in all projects taken collectively. They are not intended to be aframework for retrieval or for dissemination of materials.

1. Definitions.

The major categories are named in Figure 1. We shall first definethe kinds of settings, and,then the kinds of levels, and finallythe sorts of behaviors.

A. Settings. The-context from which the protocol material istaken.

(1) Classroom. Any room in a school building in which theactivities carried on are intended to promote learning,or more generally, any place where the activities areconducted with that intention.

(2) School. Any establishment for teaching and learning.

(3) Peer Group. A number of individuals of approximately thesame age forming a recognizable unit either in school orout.

(4) Family. Any group made up of parents and their children.

(5) Community. The people who live in a district or cityunder the same laws and institutions.

B. Levels. The periods or phases of a person's growth from birthto adulthood.

(1) Early Childhood. The period from infancy to the time thechild begins school.

(2) Middle Childhood. The period between early childhood andthe beginning of adolescence. Roughly, the elementaryschool years.

(3) Pre-Adolescence. The transition period between middlechildhood and the adolescent period. Roughly, the juniorhigh school years.

(4) Adolescence. The period immediately preceding adulthood.Roughly, the high school and early college years.

(5) .Adult. Post-adolescent years.

C. Sources of Behavior. The observable actions of a person, verbaland non-verbal.

19

(1) . Teacher Behavior. Any behavior that a person exhibits ashe engages in performing the tasks of a teacher, such asquestioning, explaining, assigning, conferring, and managinga classroom or as he takes part in extra-classroom activities.

(2)'. Teacher-Pupil Behavior. Any behavior that involves inter-action between a pupil and a teacher.

(3) Pupil. Behavior. Any behavior that a child or adolescentexhibits as he attempts to meet the situations that facehim from moment-to-moment throughout the day:

2. How To Interpret the Master System.

By identifying each category with code letters, we can indicateeach three-dimensional cell. In Figure 1, letters CAT identify acell. The first of the three letters (C), stands for the kind ofsetting, the second (A) for the level of.development of the pupil,and the third (T) for the source of the 'behavior. The protocolmaterials for this cell would consist of reproductions of teacherbehavior in classrooms at the high school level. By the same token,the protocol materials that depict pupil behavior at the adolescentlevel in a family setting can be indicated by the letters FAP.

SUBORDINATE SYSTEMS

1. Hou the Sub-Cells Were Derived.

The subordinate systems in Figures 2, 3, and 4 result from expandingeach of the behavior categories in Figure 1, while leaving the "set-ting categories" and the "level categories" unanalyzed. In Figure2, category P has been expanded into behaviors that allow the devel-opment of pupils. In Figures 3 and 4 respectively, categories Tpand T have been expanded into kinds of teacher behavior. Figure 3represents a model in which teacher-pupil behavior is included,while Figure 4 represents the behavior of the teacher as he interactswith peers and others.

2. Definitions.

The definitions of "levels" and "setting.' .and their sub-terms forFigure.; 2, 3, and 4 are the same as the definitions of these termsand sub-terms for Figure 1.

A. Types of Behavior in Figure 2.

(1) Cognitive Behavior. The behavior of pupils that entailsthe acquisition of knowledge: perceiving, conceptualizing,inferring, classifying, etc.

(2) Affective Behavior. Pupil behavior in the areas of moti-vation, valuing, commitment, personal choice, etc.

20

(3) Social Behavior. Pupil behavior characterized by suchprocesses as social cooperation, ceatpetition, authorityrelationships,..etc.

(4) Skills. Such cognitive or psycho-motor behaviors as.spelling, penmanship, typing, woodworking, gymnastics,etc.

B. Types of Teacher-Pupil Interaction in Figure 3.

(1) Instructional: Cognitive--Interactions in the cognitiverealm, such as conveying tnforMation, building concepts;explaining, diagnosing of difficulty in understanding.

\

(2) Instructional: Affective--Interactions in the aff6ctiverealm, such as motivating, attitude formation and refor-mation, influencing techniques.

(3) Instructional: Skill--Interactions in the realm of cog-nitive skills (such as word analysis, spelling)- and .phy-sical-coordinate skills (such as typing, woodworking,gymnastics).

(4) Classroom Control and Management--Interactions involvingclassroom control and discipline, especially social and,physical control.

(5) Personal-Social Development--Interactions involving per- .

sonal development of puPils in such areas as responsibi-lity, personal concerns, social relationships.

C. Types of Teacher Behavior in Figure 4.

(1) Teacher-Civic Groups. The behavior of one or more teachersin community groups where educational policies, programs,etc., are being considered: parent-teacher groups, opensessions of boards of education, special interest groups,etc.

(2) Teacher-Adults. The behavior of a teacher in conferencewith a parent or other adults o: the community about theeducation and welfare of pupils.

(3) Teacher-Colleagues. Teacher behavior in situations in-volving other teachers, school psychologists, socialworkers, or counselors where pupil problems, school pro-grams, etc., are being considered.

(4) Teacher-Administrators. Teacher behavior in conference withprincipals, supervisors, department heads, and the like.

21

_01

22

Figure 2

Expansion of P Category for All Settings and Levels,

Settings

(PG) Peer Group(S) School

(Co) Community

(F) Family

.(C) Classroom

(S1) Social

(Af) Affective

(Ce) Cognitive

(Sk) Skill

Levels

C.)

Figure 3

Expansion of Tp Category for All Settings and Levels

(Co) Community

Settings (F) Family

(PG) Peer Group(S) School,

(C) Classroom

(Ic) Instructional:

Cognitive

(Ia)Instructional:Affettive

(Is) Instructional:Skill

(Cm) Classroom Con-trol and Man-agement

(Pe) Psycho-Educa-tional

^J.I-4

1 .-4

Levels

O

23

24

Settings

Figure 4

Expansion of T Category for All Settings and Levels

(Co) community IIPAI/v/Pridalla(PG) Peer Group

4411PAPPAPPAMI(F) Family

_

(S) School

0d

P

0

Classroom AAMA(Tc) Teacher-

Civic Groups

(Ta) TeacherAdults

(T1) Teacher-

PAO

r400

00

TCoalclhea

gues

(Td)

Aemin.r

Admin.

00°

Levels

3. How to Interpret the Subordinate Systems.

Using the code letters of each category, we can see by referencesto Figure 2 that protocol materials that depict cognitive behaviorat the pre-adolescent level in a classroom can be indicated by theletters CPaCe. In Figure 3, protocol materials showing conceptbuilding at the pre-adolescent level can be designated by CPalc.In Figure 4, protocol materials that depict teacher behavior in acivic group concerned with high school pupils in the community canbe designated by CoATc.

It should be noted that there are no empty cells in category P,Figure 2, because each type of pupil behavior can be studiedevery setting for each level of development. Nevertheless, it islikely that some types of behavior are more appropriately studiedin some settings than in others. For example, cognitive behaviorscan be studied perhaps more appropriately in the school room orfamily than in the broader community setting.

The Teacher-Pupil category, Figure 3, may have a number of emptycells. This is so because teacher-pupil interaction is typicallylimited to classroom and school settings. All the cells boundedby peer group, family, and community dimensions are likely to beempty.

Likewise, there are empty cells in category T, Figure 4. Forexample, no protocol materials are likely to be developed in cellPgETc because Teacher-Civic group behavior is not ordinarily ex-hibited in pupil peer groups. It is easy to identify other emptycells in Figure 4 by the same criterion.

PLAN FOR THE BASIC FIELDS OF KNOWLEDGF

The term "basic fields of knowledge" refers to all subjects of in-struction except those in pedagogy. It covers vocational and techno-logical

history,developm

ubjects, arts, and the conventional disciplines such as physics,and mathematics. The plan set forth here provides for thent of protocol materials to teach certain things, not ordinarily

taught ii4 courses, aboUt the content of these subjects. For example,

1

the cont nt of a course in history may be biased against a minoritygroup or it may be composed of certain elements of knowledge--concepts,values, generalizations--and yet the teacher-in-training may miss theseaspects. \ The development of protocol materials will help to provideinstruction in these neglected aspects of the subjects of instruction.

. The basic- skills -of- knowledge plan is a system of coordinatesmade of three generic categories: Areas of Knowledge, Levels, and Typesof Knowledge about Knowledge. Figure 5 represents this system's co-ordinates. The expressions used in Figure 5 are defined below.

1. Definitions.

We shall give the uses of the expressions "Areas of Knowledge" and"Types of Knowledge about Knowledge" and then define the termsdesignating the sub-categories.

25

A. Areas of Knowledge. This expression refers to the variousclasses of arts and sciences. (Over 350 subjects are offeredin the public schools. To represent these in a three-dimensional grid requires that they be reduced to a few cate-gories, -as is attempted in the following defintions adaptedfrom TyKociner's Zetetics.)

(1) Symbolics. kn area of knowledge that includes languagearts, mathematics, and logic.

(2) Arts. The area that includes dramatics, graphic arts,music, painting, sculpture, literature, industrial design,choreography, and architecture.

(3) Science of Matter and Energy. The group made up principallyof physics and chemistry, but also including astronomy,geology (earth sciences), and mineralogy, all of which areunified by the principle of equivalence of matter and energy.

(4) Biological Sciences. The group treating living things, asexhibited in growth and reproduction including botany,zoology, morphology, genetics, and cytology, and related to3 above by biophysics and biochemistry, and to psychologicalsciences by physiology.

(5) Psychological Sciences. The sciences of behavior of livingthings including principally individual psychology, grouppsychology, and social psychology.

(6) Sociological Sciences. The sciences which deal with thevarious facts about, and forms of, group life--demography,geography, social institutions and ethnology, related tothe historical studies by anthropology.

(7) Developmental Sciences. Those disciplines which are con-cerned with the past, including cosmic evolution and historyand pre-history of cultures. Among these are cosmogony,certain aspects of geology and biology, and the history ofmankind.

(8) Sustaining Sciences. The studies aimed at maintaining lifeand advancing its welfare including health, physical educa-tion, public hygiene, and, at a more sophisticated level,agriculture, medicine, engineering, and technologies (manualarts, home economics, metal working, and other vocationalsubjects).

(9) Regulative Sciences. The areas of knowledge concerned withsustaining cooperation among men as they strive to satisfytheir individual and collective needs. They attempt to keepthe various elements of society adjusted to one another.Among these sciences are economics, political science,jurisprudence, and management (Business Education).

26

(10) Disseminative Sciences. Thosesciences which are concernedwith the task of transmitting information accumulated fromgeneration to generation. Among these are library science,pedagogy, and journalism.

(11) Integrative Sciences. The studies attempting to bringknowledge to bear upcn such questions as consistency ofknowledge, man's purpose, and his destiny. Among thesedisciplines are philosophy, theology, and ideologies.

B. Levels. This term has been defined on p. 19.

C. Types of Knowledge about Knowledge.

(1) Kinds of Knowledge. The epistemological elements of in-structional content such as laws, concepts, rules, values,procedures and facts.

(2) Logical Operation. Logical relations and such operationsas defining, explaining, classifying, and valuing.

(3) Types of Biases. Adherence to a point of view in theselection and interpretaton of the content of instruction.(In some cases, the point of view may be socially neutralas in the so-called new math, or in the different approachesto biology, chemistry, and physics. In other cases, itmay be socially deleterious. For example, biases aboutraces, social classes, labor, management, and the like, canbe injurious to the well-being of certain groups or to thepublic welfare.)

(4) Types of Utility. The uses to which the various kinds ofknowledge and logical operations can be put by pupils andadults as they carry on the normal activities of life.For example, knowledge may be used in repetitive, asso-ciative, applicative,or interpretive ways or in making deci-sions or in justifying actions.

2. Settings. It should be noted that this category has been replacedin the basic fields schema by the category "Areas of Knowledge."This means that no settings are indicated for the development ofprotocol materials in Figure 5, for "Areas of Knowledge" are notsettings but realms of content.

In what settings may materials concerned with the content of in-struction be developed? A few settings are suggested here as exam-ples. Some may to taken from the classroom. Others from publicschool textbooks, and still others from newspapers, magazines, andradio and television broadcasts. For example, protocol materials.to show the utility of certain elements of scientific knowledgemay be taken from articles in newspapers where such knowledge isbeing used. The analysis of the protocols would attempt to showhow the knowledge is used in an article as well as how the reader

27

Figure

Master Coordinate System of Areas of Knowledge, Knowledgeabout Knowledge, and Levels

(In) Int. Sci.

(Di) Dis. Sci.

(Rs) Reg. Sci.

(Su) Sustain. Sci.

(Ds) Develop. Sci.

(So) Socio. Sci.

(Ps) Psych. Sci.

(Bi) Biol. Sci.

(Me) Sci. of Mat.

Oi & Energy

^c)0 (Ar) Arts

O(Sy)

Symbolics

4-40

O(U) Typesof Utility

(K) Kinds ofKnowledge

(L) LogicalOperations

(B) Typesof Biases

28

11-.

Levels

uses his own knowledge as he reads it. Similarly, protocol mate-rials to depict elements of knowledge--concepts, values, general-izations, etc.--in history may be taken from classroom discoursewhere the content of history is being discussed. The analysis ofthe protocols would help the teacher-in-training to identify theelements as they appear in teacher-pupil interaction.

3. How To Interpret the Basic-Fields-of-Knowledge Plan; Protocol mate-rials developed to show how economic knowledge studied in highschool is used in out-of-school life are placed (Figure 5) in cellRsAU. Similarly, materials that are to be used to show the racialbiases of classroom discourse in a junior high school course inhistory are placed in cell DsPaB. If protocol materials are devel-oped to show the elements of knowledge in elementary school geog-raphy, they would be located in cell SoChK. Other materials canbe distributed in their proper cell by reference to the variouscategories in Figure 5.

ILLUSTRATIONS CF THE USE OF THE MASTER COORDINATE SYSTEMS

It might be helpful to illustrate the use of each of the two mastercoordinate systems, that in the pedagogical domain and that in the basic-fields-of-knowledge domain, by citing the development of two fictitiousprotocol projects, one for each of the domains above.

1. A Proposed Project in the Pedagogical Plan. A developer decides interms of his own criteria, that protocol materLi is needed to pro-vide experience in the interpretation of group relationships in theclassroom. He must first decide upon the precise group conceptsthat he wishes to have reflected in his protocol material. He nextdefines the concepts of reference group, group norm, and group rein-forcement. He must then decide upon the level at which his protocolmaterial will be made: Since group relationships are so significantat the adolescent level, he decides to develop material portrayingstudents in interaction at the high school level. Thus, in Figure1, he has chosen to work at the adolescent level (A). He has, ofcourse, already decided upon the source of behavior in Figure 1pupil behavior (P) since it involves relationships among adolescentstudents. He next must decide upon the setting for this interaction.He vacillates between showing this interaction in the peer groupoutside of school or in the classroom. Because he is anxious tohave this interaction directly related to teaching, he decides uponthe classroom as a setting. He has chosen the classroom category(C) in Figure 1.

He next must decide more specifically on the kind of behavior thathe wants to occur. Since he is dealing with pupil behavior, Figure2, in which pupil behavior is further specified, is most pertinentfor him.* In considering the possible behaviors, he decides that

*Figures 3 and 4 arc not pertinent because they deal with refinementsof teacher behavior and of teacher-pupil interaction.

29

both the affective (AF) and social (S1) aspects of behavior arelikely to be involved in the type of group interactions he intendsto portray. These, then, are the types of behaviors which will besignificant in his protocol material.

In this process of planning by the coordinate system, the developerof protocol materials has become highly specific about the concepts,the levels, the settings,*and the types of behavior that his mate-rial will portray. In turn, such specification will aid in thenational effort to develop a variety of protocol materials withoutextensive overlapping and repetition.

2. A Proposed Project in the Basic-Fields-of-Knowledge Plan. A pro-fessor of education becomes concerned about the kinds of biases(racial, social class, etc.) that are often reinforced in thepublic schools. He particularly wants his teachers-in-trainingto become aware of the frequency with which teachers unintentionallyconvey biases to students in their classes. Referring to Figure 5,he may first decide upon the area of knowl&ge in which such biasesare likely to be conveyed. The sociological area would be an ob-vious choice,but the systematic presentation of certain biasesthrough the selection and discussion of literary material interestshim more. Thus, he has decided on the symbolic category (Sy) asthe area of knowledge the teaching of which will be portrayed inhis protocol material. He next decides that-he would like to showthis at the elementary level when many biases are being formed.Thus, he selects the level of middle childhood (Ch). He has, ofcourse, already determined that types of biases (B) will be theknowledge about knowledge that will be portrayed in his protocolmaterials.

Once again, the developer of materials has necessarily and desirablybecome highly specific about the dimensions that will be portrayedin his protocol material. Once again, this dill help future devel-opers to determine what areas in the basic-fields-of-knowledge planstill have not been sufficiently developed in the form of protocolmaterials.

PRIORITIES IN THE PEDAGOGICAL PLAN

Since the cells that include teacher-pupil interaction in the class-room at all levels are primarily those in which most teaching problemsfall, these cells shall be given top priority for the next year (1971-72). The second priority should be given to any of the remaining cellsin the pedagogical plan, but withsthe proVision that not more than 20percent of the new projects can be assigned to the second level.

PriorityLevel Setting Behavior Level

1 Classroom (C) Teacher-pupil (Tp) All levels

2 Peer-group (Pg) Pupil (P) Any (E,Ch,Pa,A,Adt)

2 Family (F) Pupil (P) Any (E,Ch,Pa,A,Adt)

30

PriorityLevel Setting Behavior Level

2 School (S) Teacher (T) Any (E,Ch,Pa,A,Adt)

2 Community (Co) Teacher (T) Any (E,Ch,Pa,A,Adt)

PRIORITIES IN THE PLAN FOR BASIC FIELDS OF KNOWLEDGE

In the first phase of this program to develop protocol materialson a nationwide scale, it appears that some arbitrarily determined dis-tributionfor the two plans would be appropriate. Accordingly, it is pro-posed that twenty-five percent of the funded projects be allocated tothe basic-fields-of-knowledge plan, and that proposals in any cell ofthe plan for the basic fields he considered.

31

NOTES

1. B. Othanel Smith and others, Teachers for the Real World .(Washing-ton, D.C.: American Association of Colleges for Teacher Education,1969).

2. Ibid., PP. 28-29

3. Ibid. , p. 42.

4. Ibid., p. 43.

5. Ibid., p. 44.

6. Donald R. Cruickshank, "The Developing Notion of Protocol Mate-rials," Journal of Teacher Education 23, no. 3 (Fall 1972): 281-85.

7. Smith and others, op. cit., p. 48.

8. Ibid., p. 52.

9. Ibid., p. 53.

10. Donald R."Cruickshank, The Inner-City Simulation Laboratory(Chicago: Science Research Associates, 1969). A description offilmed incident 7.

11. Smith and others, op. cit., p. 54.

12. Ibid., p. 52.

13. Ibid., ,pp. 64-6).

14. Ibid., p. 53. Smith also supports the need for a national store-house of protocol materials. Presently one exists at Indiana Universityunder the supervision of David Gleissman.

15. Robert Howsam, Who's A Good Teacher? (Burlingame, Calif.:California School Boards Association, 1960).

16. Cruickshank, "The Developing Notion of Protocol Materials,"p. 282.

17. Philip II. Fhenix, Realms of Meaning: Philosophy of the Curriculumfor General Education (New York: McGraw-Hill, 1964).

18. For more information on LTIs see James W. Kelley, "LeadershipTraining Institutes," Journal of Teacher Education 23, no. 3 (Fall1972): 277-80.

19. George H. Kincaid, "A Catalogue of 'Protocol' Materials in TeacherEducation" (Tallahassee: Department of Education, State of Florida,July 1971). Mimeographed.

32

The University of Colorado Protocol Project: ACase Study

bl Celeste P. Woodley and Laura A. Driscoll

33/34

INTRODUCTION

The Romans casc grain for birds I() scratch to guide their moves inpublic affairs or to predict outcomes of an undertaking. The patternsof grain left by the birds were interpreted as. good or bad omens. Hadwe followed that old Raman custom, the hen-scratchings might haveaugured three years of very interesting, challenging, and often frus-trating work in the development of multi-media protocol materials.

The justification for writing a case study of the University ofColorado Protocol Project is not that the work was interesting or thatwe finally produced something with 3 years of federal money. Whatwarrants reporting are the ways in which the project staff came tounderstand a new and sophisticated idea in teacher education and theexperiences of the staff in'developing, testing. and disseminatingprotocol material.s. We will share the insights, the methods, the prob-lems, mistakes, and successes of the University of Colorado ProtocolProject with readers who might wish to develop similar materials andwith those who are curious about how an educational idea gets translatedinto an instructional product.

Soiue brief mention of who we are and what resources we had to workwith may be helpful since the backgrounds and experience of the staffand the amount of the U.S. Office of Education grant were to influencedecisions during the life of the project. The academic interests ofthe staff ,.suggested the general conceptual framework within which weworked and the specific concepts we chose to develop. Our inexperiencewith media techniques and technicians accounted partly for the manyfumbles with videotape, 35mm slides, and 16mm color sound film.

The total grant for the 42 months, July 1, 1970 through December1973, was about $165,000, averaging just under $4,000 a month. No morethan two persons ever worked full-time at once on the project, althoughpart-time clerical and professional assistance came from several personsduring the 3 1/2 years. In the long run, a relatively low budget,somepoor luck with contract technicians, and a resolve to complete the mate-rials and get them out turned the Colorado project into a "do-it-your-self" operation with the project staff doing much of the technical workand all of ele dissemination.

The director of the project, Celeste Woodley, and the assistantdirectors, Ina Mullis (1970-72) and Laura Driscoll (1972-73) wereresponsible for the development, testing, and dissemination of theUniversity of Colorado protocol materials. Woodley is a member'of thefaculty of the School of Education; Mullis is currently in the Researchand Analysis Division of National Assessment of Educ,It4.onal Progress,Denver; and Driscoll is a doctoral student in Educational Research andEvaluation at the University of Colorado. The academic backgrounds ofthe three are in social science education or research and evaluation;their working experiences are in teaching, the analysis of teaching,curriculum development, product and program evaluation, and adminis-tration of teacher education programs. When the project began, thestaff's direct experience with media was limited .t.o clasSroom video-taping and threading old movie projectors.

35

The university agency which holds the USOE grant is the Centerfor Education in the Social Sciences, University of Colorado, an orga-nization interested in improving education at all levels, particularlysocial science education, by strengthening the ties and the dialoguebetween the School of Education and academic departments in the Collegeof Arts and Sciences. The background, experience, and organizationalaffiliations of the staff contributed to the fraMe of reference and tothe content of Colorado's protocol materials and to the style and for-mat of the final products.

BEGINNINGS

To begin, we had to understand protocol materials. From there wechose our concepts and finally learned about technical quality the hardway.

Scrambling for a Referent

The USOE request for proposals (RFP) that came to the Center inApril 1970 contained ideas and phrases that were not common in ourrepertoire: protocol, behavioral specimens, segments of behavior,

.

protocol materials, aut:Lentic representations of educationally rele-vant behaviors, the distinction between protocol materials and trainingmaterials. Uncertainty about the nature of the materials described inthe RFP caused us to read many times those few pages of description anddefinition. We studied Teachers for the Real Wcrld in which B. OthanelSmith had described protocol materials and set in the context of amuch larger instructional scheme, and we spent a lot of time discussingand refining our own interpretations of the nature and meaning of pro-tocol materials. The initial struggle to understand the concept"protocol materials" gave us an appreciation of the task of conceptanalysis that was to become central in product development. To beginto understand protocol materials for ourselves, we needed a definitionor rough idea of the concept. We profited from knowing what protocolmaterials were not; more exaMples would have been helpful, and delin-eation of specific characteristics or attributes might have providedgood clues as to the nature of protocol materials. By the time webegan to develop materials in the summer of 1970 we knew that protocolmaterials linked concepts with their behavioral referentl as repre-sented in filmed or video vignettes. It was clear that the two criticalaspeCts of development were concept selection and analysis and Zile re-production and representation of descriptive behaviors in an appropriatemedia.

Conceptual Dimension. From the several areas suggested for proto-col development, we chose to'work with concepts relevant to teacher-pupil classroom interaction. Initially we had proposed to illustrateconcepts such as "motivation," "involvement," "ethnocentricity,". "cog-nitive style," "value formation," "classroom climate," "concept attain-ment," "hypothesizing," "resolution of conflict," and "evaluation."These concepts were chosen because they were referred to frequently inteacher education texts. This smorgasbord seemed plausible at the time,but by the fall of 1970 it was difficult to justify the shot-gun

36

approach. It was also apparent that we would have to drop plans touse, :In final product, vignettes from our large collection of 1/2-inchvideo tapes of classroom teaching. Earlier it had seemed, reasonablethat behaviors exemplifying all of the concepts we chose to portraywould occur somewhere in the 60 hours of video footage. That ideawas not inconsi1stent with the notion of protocols as "slices of life"tha,. was pervasive early in the program. It seemed practical to selectthose slices from existing resources. We had begun to catalogue thetapes by segments according to conceptual content, noting quality ofsound and quality of picture. The next step, we thought, would be'editing the segments into a flow of examples emphasizing the behavioralattributes of,a particular concept, adding some titles and music, con-verting to kinescope, writing a guide, and Voila! protocol materials.It was not quite that simple. Figuring out just what we were supposedto be doing and how we were to do it took a while.

The quarterly meetings of the protocol directors and members ofthe Leadership TrainingInstitute (LTI) provided the feedback necessaryfor us to find out, if somewhat indirectly, how we were doing with thetwo major criteria of protocol materials production, i.e., the con-ceptual validity and technical quality. These "show and tell" sessionsprovided important verbal and non-verbal clues. A viewer's question- -"What is the concept?"--immediately following a presentation, was apretty good indication that conceptual quality was doubtful. Afterthree or four wrong guesses from others in the audience, one mightprudently consider beginning all over again.

Technical Dimension. The real meaning of technical quality grad-ually emerged frcm the reactions of the directors and LTI audience tojumpy cameras, sterotyped acting, blurred visuals, and slurred sound.All these flaws wera tolerable. when the products were viewed as roughdraft films or video tapes, but whatever delusions we had about pre-serving the home-made look in a final product were laughed away. Thecriterion "Commercial Broadcast Quality" became the word of the day bylate 1970, and the implications of what that meant in/terms of time,money, and precision also prompted a reassessment of our production /plans.

Steps in Development. The trial and error proicess of the earlymonths of the project indicated the following stepi as basic in proto-col material development:

.

1. Developing a conceptual framework that has utility for understandingthe teaching-learning process;

2. Selecting from the framework and further defining concepts forprotocol development;

3. Identifying the properties and behavioral attributes of the con-cepts;

4. Deciding on the appropriate media;5. Loosely scripting classroom episodes exemplifying both instances

and non-instances of the concepts;6. Filming or taping the vignettes;7. Editing the footage;8. Adding appropriate titles, -credits, music definitions, questions,

and models;9. Writing supplementary teacher's and student's guides.

37

REASSESSMENT: STRATEGIES AND TACTICS OF SURVIVAL

The skirmishes of 1970 gave us a better look at our target andsuggested some strategies for attack. .The target, it turned out, wasa formidable task, more complex and sophisticated than we had realized.Precise exemplification of well-defined concepts in various media formsof high technical quality was required.

The Conceptual Framework and Selection of Concepts

The selection of concepts was the first concern in our assessment.We wanted to stay with teacher -pupil interaction in the classroombecause we were comfortable in that area and we felt we could success-fully portray concept-behavior linkages there.' It was also an area ofhigh priority for the entire protocol program. We were convinced,however, that producing protocols for concepts like "questioning" or"punishment and reward" or "probing" did not get at what is basicallyand crucially'missing in a teacher's ability to interpret classroominteraction. The rejection of specific, skill-related concepts was nota denial of the significance or 'validity of such concepts, just a dis-agreement with the level and scope of conceptual focus. Furthermore,other projects were dealing successfully with many of those concepts.There was no need to be redundant.

We were interested in equipping teachers with the conceptual andanalytic tools to change their own behaviors. It seemed to us thatteachers first needed an understanding of a theoretical framework thatmade sense of the instructional process itself. Then the meaning ofary particular specific concept and the interpretation of associatedbehaviors could be placed in a theoretical, as well as in a situational,context. There would be much greater opportunity for teachers to com-prehend what was happening in teacher-pupil interaction and to go beyondthe specific behaviors to some idea of why they were happening. Refer-ring to a conceptual model, teachers could modify their decisions atthe point where most impact on consequent interactional behaviors mightbe made.

We derived two models to guide our thinking and selection of con-cepts. One, derived primarily from the work of Stakel and C A.,2 isa decision-making model of the process of instruction (Figur ) whichindicates major conceptual components of a theory of instructionaldecision making and suggests the sequential and functional interrela-tionships of these components. The concepts we choSe to exemplify arerelated to one another functionally and sequentially in this conceptualmodel as generally indicated by the arrows between components. We hadwished to analyze and illustrate at least one concept identifiable ineach of the major components, but we had neither the time, the money,nor the talent for so large an undertaking. We chose to illustrate sixconcerts, one related to the entire process of instruction, one toantecedent conditions, one to selection of instructional alternatives,and three to interaction. Crwo were later combined in one unit.)

38

Figure 1

Basic Model of Instruction GuidingDevelopment of Protocol Materials

University of Colorado

The Specificationof Outcomes: Objectives

IDiagnosis of Antecedent

Conditions

r------1 EvAuation of Instruction

Modification

Selection of InstructionalAlternatives

Lutcome71

39

Before we selected specific concepts and defined thot, concepts,we had to resolve the question of the level of specificity of conceptswith which we would work. Concepts can be arranged in a hierarchy ofcomplexity depending upon the number of discrete attributes that char-acterize them and the logical linkages and dependencies among theattributes. There are large and complex concepts, middle-level concepts,and specific and simple concepts pertinent to instructional theory. Atwhat level in the conceptual hierarchy was it useful and practical forus to work? Our frame of reference dictated a focus on middle-levelconcepts that provided, we believed, context for understanding and in-terpreting the more specific behaviors that rhaxacterize more narrowlydefined concepts. We were-not unaware of the risk of ambiguity and thedifficulty of representing specific behavioral attributes when wedecided to portray these more general middle-level concepts. The trun-cated outline in Figure 2 indicates, in relation to some other concept-ual categories, the concepts portrayed in the set of products developedin 1971 and 1972 by the Colorado project. The specific titles of thefive products and the concepts exemplified follow.

Title Concept

1. Conceptualizirg the Process of Instruction Process of Instruction

2. Learners and Their Characteristics: Lnpli- Attitudes toward Schoolcations for Instructiclal Decision-Making Aptitudes

Personological Variables

3. Verbal Interaction in the Cognitive Verbal InteractionDimension (Cognitive)

4. Fair Verbal Behavior Fairness

5. Prganizing Facts To Teach Meaningful Concept TeachingRelationships

The middle-level designation of these concepts is relative, e.g., "Pro-cess of Instruction" is a less general, less complex concept than "Edu-cation" and is subsum-d under it. "Fair Verbal BehaVior" is moregeneral, more complex than "Initiating and Reacting Behaviors" and en-compasses them and the even more specific "Accepting," "Encouraging,"and "Task-Oriented Behaviors," for example.

Linking Concepts to Observable Behaviors

We discovered that middle-level concepts become quite a mouthfulwhen verbalized--"process of instruction," "attitudes toward school,""verbal interaction in the cognitive dimension," "fair verbal behavior,""concept teaching." They are also unwieldy and hard to pin down tospecific, unambiguous indicators.

The process of analyzing concepts, defining them, and listingtheir behavioral attributes was the most difficult and critical taskof protocol materials development. It is the essential step between

40

Figure 2

A Part'_al Model of Conceptual Categories inEducation Arranged in a Hierarchy of Complexity

EDUCATION

SOURCES OF BEHAVIOR

)thei71 fraTents1 I Teacher

I--I Inputs

10biective:1

Student

ti)

O.

z0U1 x

L.1.1

1 THE SETTINGS Ja....

--- i

1 I

r.

PeerJ

LGroupl FamilyI 00

Classroom [School] k 1

ommunity 1

The Process of Instruction

AntecedentConditions

Student

n.s. Variables/

'Attitudes

'Teacher

Attitudes

IInteraction' Outcomes 1

Change in Change inTeacher Behavior Pupil Behavior

verbal

Teacher

'Materials Environment

Non- verbal I Verbal

Affective t ISocial

I 1

Initiating71, -

ExtendingEncouragingAcceptinOrganizingClarifyingShapingTask-orientedRole expectantEtc.

Verbal

5on -Verbal I

LStudentt

LNon-yverbal _11

Higher order thinkingProductiveReproductiveSelf-depreciativeAdaptiveRationalizationRigidEtc.

NOTE: The heavy outlines indicate categories in which protocol mate-rials were developed by the Colorado Project.

selection of the concepts and the technical work with media. Analysisof the selected concepts must yield definitions and characterizingbehavioral attributes. (See "A Catalogue of Concepts in the PedagogicalDomain of Teacher Education,' p. 54.) It is the behavioral attributesor indicators of the concept that are observable and able to be cap-tured on film or tape. We defined "Fair Verbal Behavior" for example,as teacher verbal behavior that follows a consistent standard towardall students and is free from bias, dishonesty, and injustice. Thatdefinition was derived from student descriptions of teachers and class-room experiences. available in general and research literature on class-room teaching. Students have a concept of fairness, and how a teachermeasures up as fair or unfair seems to affect student behaviors. Totreat "Fair Verbal Behavior" as an aspect of classroom interaction andas a significant concept seemed justified. The behavioral indicatorsof "Fair Verbal Behavior" were identified from the definition and fromspecific student perceptions. A fair teacher:

1. Praises students' ideas rather than students personally.

2. Praises students when they deserve it and corrects them when theymake a mistake or misbehave.

3. Explains the mistake or misbehavior rather than criticizes thestu:lent personally.

4. Asks students to clarify statements or defend actions in order todetermine whether to praise or correct students.

5. Is not capricious and arbitrary in dispensing praise or correctingstudents.

It was helpful to us to list also negative instances of "Fair VerbalBehavior," e.g., praises students who don't deserve it. This analyticalaspect of development required a heavy investment of time.

Once the observable behavioral characteristics of any concept weredelineated, we had to decide on the appropriate vehicle for media repre-sentation. Our deliberations were not terribly complex. They werebased in some cases on LTI advice and sometimes on experience, Thethinking went something like this: If continous flow of dialogue andphysical movement are both necessary to identify the behavioral indi-cators of the concept, use video tape or film. If non-verbal expressionor symbolic records convey the attributes of a concept, use slides (orfilmstrip). If the subjects' verbal behaviors add a necessary dimen-sion, use cassette or audio tape, but add slides for interest orclarity. If the behaviors are difficult to distinguish initially with-out some help, use overprint, arrows, or other focusing aids. Since 1/2-illch video converted to kinescope looks like reruns of "Our Gang" com-edies, use 2-inch tape for video. Since good sound is critical, use astudio set-up and plenty of microphones when filming. Since there aredifficulties with the compatibility of the various video formats, 16mmfilm is preferred. Since color film is usually more attractive to theviewer, use color. Since teacher's guides are often a drag, keep them

42

brief and put instructions on the film where feasible. What actuallyhappened on the technical end of protocol materials production atColorado is another story.

The Root of All Evil: Technical Incompetence

All great dreams have sub-themes tha... run under the main plot. Inour case, the sub-plot almost overwhelmed the serious theme of producingconceptually sound protocol materials. The struggle in the wings with avaried assortment of incompetent media specialists almost closed theshow. The judgmental errors in selecting these persons were ours, andwe have vowed to give a lie detector test in the future to every appli-cant who claims to know anything about media. Some highlights from ourexperiences may forewarn the reader who contemplates .a multi-mediaventure.

Where Is the Picture? A Video Vignette

The VTR is on and reels are turning; camera is on; twoplugs are in the wall. Students and teacher are engaging inbehaviors perfect for the protocol.

"Everything set? Are you getting a good picture?""Don't worry about a thing." (Peers into camera.)

"Picture is just great!"Twenty minutes go by. Kids can hardly wait to see them-

selves on the monitor. Rewind. Switch to play."Where's the picture? Where's the picture?""Ooops. Forgot to plug the camera to the VTR."

A Study in Gray and Gray

"What this kine needs is some sharp title cards and musicunder the introduction. I'll pick olt the mood music for you.I know the beat to look for. You just take your titles overto Murph; he'll shoot them with his Boley."

Murph is stuck away in' the basement of the oldest buildingin town. The "Boley" has no tripod, but "My hand is steady."The poor light and the steadily shaking hand produced a pieceof film that the processor tried to expose twice.

But the viewer hardly notices it go by at all; he is in .

shock from the variation of "Babes in Toyland" that "sets themood" for "Conceptualizing the Process of Instruction."

The Sound of Speed and All That

"Glad you decided to go to 16 mill. Queen of the media. . . I'll get on the phone to my contacts in the morning andput in the order for everything we need . . . Ship it by AirCargo the next day . . . Let's see: positive-negative, doublereversal, optical sound and a double-sync projector, edgemarkings and the Movieola, crystal sync recorder and threeNagras, Bolex, Eclair, and Aeroflex, booms and wireless . .

43

"Wait a minute. You just lost us.""You've got to remember. With me as producer-director

you move with the sound of speed."("Sound of speed? Is that -- CRASH?")

Lest you be afraid to buy a roll of film from now on, be advisedthat we were trying to keep our expenses down and had not taken ourbusiness to well-established film companies., Their prices were in therange of $1,000 a minute for 16mm color film and $20.00 an hour for35mm work. Much later we did find some very able professional peopleto help us whose charges for high quality work were minimal.

"Do-It-Yourself": The Last Resort

There were many times in the last 3 years that the three of usfelt we had embarked on new careers as graphic artists, paste-upexperts, script writers, film editors, sound women, and recordingartists. We made almost all of the title and credit_cards for copystand photography and for revision of black and white films. Weused press-on letters, 9x12 negatives and colored gel or acrylic, en-larged color prints, and poster paints. The selection and sequencingof slides for filmstrips, the scripting of the dialogue for t'he film-strips, the selection of music and special effects, the locating ofsubjects to be in the films or filmstrips were all part of the job.The sessions with the Movieola, sync'ing lips, with sound, lasted farinto many nights. We had 6,000 feet of color film and 8,000 feet ofsound tape that we had to see and hear together in order to selectthe most illustrative instances for two of the products. Once we hadidentified the segments, we searched for and found a competent filmeditor who didn't mind us working right with him until we were satisfiedthat we had what we wanted.

"Do-it-yourself" is not quite right. There were many persons whodonated time and talent to help us out. Cooperation and interest weredemonstrated by students, teachers, school administrators, professionalsand employees of film sound studios, amateur actors in a local theatregroup, and many persons whom we talked with but never saw. The gener-osity of the Boulder Valley District Re -2 and Adams County District#50 schools in allowing us to use their classroom facilities for filmingmade much of the work possible. With all of this help, our color filmscost not more than $500 per minute instead of the usually quoted priceof $1,000 per minute.

TESTING, EVALUATION, REVISION

Testing, evaluation, and revision of the protocol materials wereongoing processes that began as early as the fall of 1970.

Pilot Testing

Pilot tests provided information on student and instructor percep-tions of technical quality and on the clarity and significance of concepts.

44

These early trials were instrumental in revising the films and inwriting the instructor guides and student materials.

In late 1970, the first film unit, "Conceptualizing the Processof Instruction," was piloted in education classes at the University ofColorado, Colorado State University, and at sites in Florida. The 200students who used the materials at that time reacted positively to theidea of protocols, but were negative about the quality of the film andthe structured presentations. "You're forcing us to see something inthis that we just can't (or don't) see" was a typical comment. Thiskind of student reaction was given much weight in revising-the film andinstructor's guide.

The revised unit on the "Process of Instruction" was again pilottested under similar circumstances in the spring of 1971 along withtwo new protocol materials units, "Verbal Interaction in the CognitiveDimension" and Part I of "Learners and Their Characteristics," a film-strip on attitudes toward school. We learned that instructors did notalways understand the suggested uses and often had different ideas aboutthe most advantageous way to present the materials. Their comments andreactions led us to'write a more detailed rationale for the units, moreexplicit directions, and suggest alternative teaching procedures foreach unit. All five products were readied for the major field test inthis way.

Field Testing

In the spring of 1972 our five multi-media protocol products werefield tested at 15 colleges and-universities around the country.

The Subjects. Five social science methods classes served as acontrol group; the ten experimental classes consisted of five generalmethods and five social science methods classes. The 15 different.test sites represented a diversity of urban and rural regions. Theinstructors were selected from among willing persons suggested byfriends around the country. ,

The Instruments. The evaluation instruments were, constructed byIna Mullis, with assistance from Gene V Glass of the Laboratory ofEducational Research at the University of Colorado. The instrumentswere drawn up according to the objectives of the project and in orderto answer two evaluative questions: "How effective did the participantsjudge each product to be?" and "Is the protocol materials approach moreinteresting then conventional approaches in teacher education-courses?"The questionnaires included a pre- and post-test, izItended to measureconcept acquisition; instructor and student background informationforms, which yielded such data as years of teaching experience, degreesobtained it specific areas, age, sex, number of hours taken in differ-ent subject areas, previous non-teaching experiences, and future plans;separate criterion tests for each product; and instructor and studentevaluation forms for each of the five products. The purpose of thecriterion tests was to assess whether or not the objectives of eachproduct had been met. They differed in form from one product toanother, depending upon the nature of the activities prescribed for

45

each unit. The evaluation questionnaires were similar for all fiveproducts. The 15 Likert scale items on each form were intended tomeasure the interest and perception of the significance, clarity, suf-ficiency, and effectiveness of each unit.

The Instructions. All of the field test materials and instructionswere air expressed to the instructors during the last week in March1972. Complete instructions and an inventory of contents of the testingbox were included with the shipment. For every instrument administereda large, stamped return envelope was provided for immediate return ofquestionnaires or tests. Instructors in the experimental classes hadthe prerogative to decide when and in what order to teach the protocolunits.

Problems. The field testing procedure could have been improved.We were fortunate that we found instructors to participate at all con-sidering the late date. We should have sent the materials a monthearlier than we'did. Some instructors did not have enough time to com-plete all the units before the end of the semester. Confusion aboutthe nature of protocol materials might hve been avoided had we con-ducted an orientation session for the instructors before they used thematerials.

Evaluation

Returns of the data were disappointing. Two instructors returnedthe box of materials unopened. Only two instructors returned all ofthe data; there were partial returns from the others. More than halfof the students did not take all the criterion tests, mostly becausethe tests were too long and too difficult. Incomplete returns preventedus from analyzing the relationships between the background informationdata and achievement or attitude scores. We were able to compute ananalysis of co-variance on post-test scores using pre-test scores asa covariate for experimental and control groups and to obtain meanscores for criterion tests for the two experimental groups and meanscores for instructor and student groups by product.

The staff worked with an outside. evaluator, Mary Lee Smith ofthe University of Colorado, on the scoring and tabulation of the data.Smith analyzed the data and prepared the evaluation report.

The results of the field testing can best be summarized by examiningthe responses, generally speaking, to the two primary evaluative ques-tions which were embodied in the instruments. In answer to "Howeffective did the participants judge each product to be?" it can besaid that both the concepts and the activities used in the units wereconsidered worthwhile and significant by both students and instructorsfor all five products. The interest level of all units but one wasrated as "more" or "much more" interesting than what "normally takesplace in my education classes." Two units suffered somewhat from alack of clarity, according to the respondents.

46

In order to measure the difference between the prrtocol approachand conventional approaches to teacher education, the responses.of thetwo experimental groups and the responses of the control group tothe post -Lest were statistically compared. The questionnaire wasdivided into five sub-sets (groups of three or four items) which relatedto different instructional objectives. Also used to judge the differ-ence were some of the items on the evaluation questionnaires. Bothstudents and instructors apparently perceived that "Verbal Interactionin the Cognitive Dimension" and "Fair Verbal Behavior" taught thestudents more than they would have ordinarily learned in a teachereducation course. Instructors thought that "Learners and Their Char-acteristics" offered a better approach than the traditional, butstudents did not. Students thought that "Organizing Facts To TeachMeaningful Relationships" (on concept teaching) was an improvement onconventional approaches, but instructors thought it was no better.

Judgments

The responses were studied by the evaluator, the project director,and a social studies methods instructor. Each wrote an independentjudgment of the implications of the data for revising, disseminating,and using the five products. The judgments reflected agreement thatrevisions had to be made, but also indicated that the materials wereessentially useful and should be disseminated as widely as possible.

Revision

Final revision of the materials according to the directions in-ferred from the data was undertaken in the fall of 1972. All blackand white films, for example, were converted to color effect by tintingthe kinescope in printing and by adding new color titles. Some of theepisodes were shortened, and some Aarifying questions or instructionswere added. All of the instructor's guides and student handouts werere-written and shortened. The main emphasis there was on revising thecriterion tests, included in the guide to make them less time consuming,easier to understand and use, and more uniform.

The field test results showed the materials to be generally worth-while and interesting. Better products had resulted from revision, and'there was no reason not to disseminate the materials as quickly and asinexpensively as possible.

DISSEMINATION

It was a great feeling to survey a shelf full of finished mate-rials that were apparently useful and wanted. By the spring of 1973,the materials were ready for a publisher and distributor, and contactswere made with the publishing industry. The responses of the commercialpublishers were less enthusiastic than anticipated and probably forgood reason. 'Although publishers responded favorably to the contentand technical quality of the materials, they had concerns and questionsabout the size of the market, cost and time problems of reproducing

/47

and marketing multi-media materials, the packaging and warehousingproblems, and the high mark-up price which would prohibit many potentialcustomers from purchasing the materials. The prospect of any commercialpublisher marketing the products looked extremely bleak. Although themarket was small by commercial standards, materials like these weremuch in demand. To spend more time trying to find a commercial pub-lisher seemed ill-advised. We wantc:d to get the protocols out whilethey were still relevant and up-to-date. This looked like a time forthe "try-it-yourself" strategy.

Authorization to secure final copyright and to make the Universityof Colorado the publisher and disseminator of the materials was appliedfor from USUE. Such authorization was granted for a 5-year period, withpermission to sell the materials on a non-profit basis.

It Pays To Advertise

An advertiiing campaign was launched on May 1, 1973 by sending out10,000 e-page brochures announcing the availability and cost of thematerial. These were sent to persons on the mailing lists of theAmerican Association of Colleges for 'Teacher Education, the OlIC Clear-inghouse on Social Studies/Social Science Education, and the SocialScience Education Consortium. The cost of printing, mailing liSts andmailing was about $650.

In anticipation of requests for preview, rental, and-isale, weordered 25 prints made of each film and filmstrip. The.tOtal expensefor the prints and the appropriate sets of written materials came toabout $5,500. The.sellinvprice of each protocol materials twit wasset to cover expenses, includin; mailing and packaging costs, and toprovide for replacement of the unit.

The response from advertising has been very gratifying. Requestshave come from agencies at all levels of education. As of the end ofDecember 1973, there had been about 500 mail and telephone contacts.There'had_been over 100 requests for more information; 200 requestsfor previews; 50 rental orders (at $5.00 per unit); and 75 sale orders.The toal dollar value of orders by the end of December Was just over$7,000.

Producticil Goes On , .

With the dissemination problem temporarily under control, thestaff turned attention back to development of protocol materials in anew area in the third year of the grant from USOE.

In the fall of 1972, while still evaluating and revising theearlier products, we began to plan production of four filmstrips deal,ing with social science concepts. The goal for 1973 was to select andportray(concepts significant for developing an understanding of socialbehaviot as it related to the phenomena 6f social stability and socialchange. The search for a theoretical and integrating framework withinwhich to locate and explore concepts related to social conduct directedus to role theory as the general area in which to work. The role

48

orientation appealed because it offers an interdisciplinary approachto the several social sciences. The concepts selected for protocoldevelopment are "role conflict," "conf6rMity," "consensus," and"position, identity, status, and role." These'concepts are useful inexploring the "fit" between individual behavior and group behavior,and they have important implications for examining the nature of socialchange or social stability.

At this writing "Role Conflict" and "Conformity" have been completed.e Conflict" is a filmstrip and cassette unit, using six vignettes to

illustrate expectations, positions, and behavitirs and their relationshipsin the dilemma of conflict. "Conformity" is also presented on filmstripand'ta-iette. The concept is treated according to its various defini-tions, each of which is illustrated by one or more episodes in thefilmstrip. There are no instructor's guides for the materials; .direc-ti.ons and definitions are part of the filmstrip. Research has beencompleted for "Consensus" and the script is in preparation.: The lastunit on "Position, Identity, Status, and Role" will.be prepared inFebruary. Field' testing on all four units will be carried out in thespring of 1974. In the preparation of the four units on concepts. fromrole theory, the Colorado project has been fortunate to have had'asconsultants Bruce J. Biddle and Barbara J. Bank of the Center forResearch in Social Behavior, University. of Missouri.

APPRECIATION

No project has been more interesting and more challenging for thisstaff than the Protocol Materials Development Project. The opportunityto be part of the USOE Protocol Materials Development Program is muchappreciated.

49

NOTES

1. Robert Stake, "The Countenance of Educational Evaluation;" TeachersCollege Record 68, no. 7 (April 1967).

2. Robert Glaser, Evaluation of Instruction and Changing EducationalModels (Los Angeles: Center for the Study of Evaluation, UCLA,1968), p. 5.

50

A Catalogueof Concepts

in the PedagogicalDomainof reacher

Education

by BryceB. Hudgins

INTRODUCTION

If protocol materials are to make a programmatic contributi_on tothe improvement of teacher education on a nationwide basis, certainobjectives must be realized. Other papers in this publication dealwith some of those objectives; for example, technical critieria for pro-duction of materials must be specified and demanded. Individualprotocols must teach concepts. The pedagogical and technical knowl-edge necessary to achieve criteria associated with both of thesegeneral goals has been growing during the past 4 years. Protocolmaterials which are being developed at the present time appear tobe far superior on both these iimensions than were the prototypes of1970.

Despite these real and sizeable advances in the program, however,much must be accomplished to reach the original goals concerning.national impact on teacher education programs. From the earliestplanning stages of the protocol program, the need for conceptualmapping of the pedagogical domain of teacher education was recognized.Such a. map would presumably aid in making decisions about which con-cepts, or at least which groups of concepts, should be developed intoprotocols. Through that mapping effort and, it is hoped, through

--other similar efforts still to be planned, protocol materials devel-opers would receive guidance about undeveloped concepts in thepedagogical domain. From the outset, this conceptual mapping wasconceived as a catalogue of concepts about teacher education thatwould contribute to the execution of two principal functions. First,

it should identify appropriate concepts in the literature of teachereducation, estimating the overlap or redundancy among such concepts.and identifying ostensible gaps and the kinds of concepts needed tofill those gaps. Secondly, statements about individual conceptsshould give the developer salient and significant information tohelp portray the concept. These statements must define the concepts,specify their essential attributes, and give brief examples 'thatmight serve as preliminary sketches for concept portrayal.

During 1973, a project to develop this catalogue was housed inthe Graduate Institute of Education, Washington University, underthe directibn of the author. This paper is a preliminary statement ofthe rationale developed to execute the two primary, catalogue functions.

CONCEPTS IN TEACHER EDUCATION

Concepts, are the building blocks of thought. They outline thecontent of distiplines and undergird their structure. Thus the psy-chologist thinks in terms of "stimuli" and "responses "; of "rein-forcement" and "extinction"; of "learning," "forgetting," "retention,"and "interference." For the sociologist, his thinking and work aredelineated by concepts such as "group," "norm," "interaction," and soforth. It is the specialized concepts a group employs to draw atten-tion to its issues and phenomena that distinguish its discourse fromthat of anothr:- group or from laymen. Of course much of the potency ofa shared and specialized vocabulary lies not only in the individual

53

concepts themselves, but in the increased analytical power obtainedwhen relationships among sets of those concepts can be established.Relationships of this order are to be found in all areas of humaninquiry that involve logical, propositional, and theoretical thinking.Nor is this generalization restricted to the provinces of the purelyacademic and theoretical. Great strides in the technological base ofprofessions such

inmedicine and engineering have been accompanied by

vast, increments in their level of conceptual and relational sophisti-cation. Practitioners are thus able to identify and solve complexproblems which were perhaps incapable even of formulation prior toappropriate conceptual and theoretical development.

Shifting to the field of teaching, and particularly to teachereducation, we discover that relatively little is available by way ofa specialized vocabulary or set of concepts about teaching and thatvirtually no propositional knowledge has been generated. Furthermore,individual concepts about teaching are not widely shared among prac-titioners. Finally, numerous education scholars have pointed to thepaucity of technical language available to teachers for discussingissues that occur in the classroom (and therefore have some chanceof defining and solving problems). In short, no language of teachingexists that provides a set of intellectual tools by which practitionerscan isolate areas of difficulty and subsequently comprehend and inter-pret them.- Most concepts that teachers possess are not universal andmay be quite personal. Those teaching concepts that are available areoften so limited in scope that the individual teacher is left withnothing but his own powers of insight and innovation to survive in theclassroom and to try to improve the instruction he gives his students.

The author believes this characterization of teacher education asa field in conceptual disarray to be substantially accurate. The prin-cipal question that we must confront, then, concerns the nature andextent of the contribution to be made by a catalogue of concepts inteacher education. Clearly, no catalogue of concepts can altogethereliminate the conceptual problems of teacher education. It canfacilitate the process of conceptual clarity, however, if two majorobjectives can be achieved. First, it must specify and delineate thedefinition of the "pedagogical domain of teacher education." Secondly,it must identify, define, and exemplify appropriate concepts (organizedaccording to their functions) within the broader domain of pedagogy.

PEDAGOGICAL DOMAIN

An analysis of the phrase "pedagogical domain" was significant inthe preliminary effort to organize the catalogue. No clear and widelyshared referents exist for the term.

Definition

We construed it broadly, suggesting that the pedagogical domainencompasses those settings and those people that bear upon the contentwhich is to be taught to pupils, and the methods or procedures employedfor purposes of conveying content. The domain includes decisions about

54

the choice and communication of content. Thus, not only are conceptsabout the teacher, teaching, or the pupils included in the definition ofpedagogical domain, but also concepts about relationships betweenteachers and their peers, teachers and administrators, and teachersand boards of education and other community groups. Figure 1 portraysthe major layers of concepts required to exhaust this conceptualdomain.

Selection and Delineation

The pedagogical domain is much more extensive than had been anti-cipated prior to our analysis of it, and priorities had to be set toguide the selection and delineation of some portion of the domain asopposed to its totality. Since teaching practice is more likely to beguided by concepts with an ostensible immediate application to thedaily tasks of teaching than by more remote concepts, the decision wasmade to develop a catalogue of concepts pertaining to interactiveteaching. This decision was based upon the belief that the preponder-ance of concepts most central to the day-to-day activity of teachersand pupils are found in that ring. Thus one criterion established forthe selection of concepts in the catalogue was that they should haveproximity to the ongoing classroom activities both of teachers andpupils.

Regardless of this decision, concepts that lie within the otherrings will also, in the future, require conceptual development-. Forexample, the ring jabelled "teacher collegial relationships" might beexpected to contain relatively few concepts, but they are concepts ofconsiderable importance and explanatory power. To our knowledge,these concepts have not been systematically collected and analyzed.At a guess, however, the concepts in that ring should clarify relat ion -ships among teachers and their peers. Teachers manifestly or subtlycommunicate to one another their ideas on treatment of children,maintenance of discipline, amounts and kinds of content, and communi-cating content. The intriguing analyses of groups other than teachersstrongly. imply that the use of sanctions and rewards occurs in allsorts of peer working relationships to control both quality and quantityof work accomplished and to govern relationships among workers, clien-tele, and superiors. Books such as Waller's classic Sociology ofTeachingl and the recent monograph Small Town Teacher2 by MacPhersonsuggest that these principles and their component concepts are applicableto teaching as well.

The point of this discussion is to illustrate that concepts aboutinteractive teaching, as crucial and as central as they are to the domainof pedagogical concepts, do not exhaust it. Nor can any such cataloguedeveloped around a fraction of the domain begin to show the integratedrichness of the domain, as all concept levels influence and are influencedby each other. However, with the present catalogue cf concepts aboutinteractive teaching to serve as a prototype, other areas of the domainshould be more easily and more quickly developed.

55

Table 1

A Sample Concept To Appear in the Catalogue: From the Cognitive Section ofCategories of Content, Reception Learning

Comparison

Definition

A pupil statement indicating the similarities and/or differencesbetween two or mr,re items. The statement is recalled or reconstructedfrom a prior instructional presentation.

Indicators

1. The verbal content of a comparison is similar to that of theconcept "comparing" (q.v.), which is subclassified underproductive thinking. The distinction between the twoconcepts is that in reception learning the pupil does.not produce the comparison himself.

2 The term "comparison" is used to represent statements by pupilsthat contain only comparisons, or only contrasts, or whichmay contain both.

Illustration

As a response to the teacher's question, following a presentation bythe teacher on closed plane figures, a pupil says, "A triangle hasonly three sides, but a square has four." Any illustration of theconcert must include the fact that the original comparison wasexpressed in the teacher's presentation.

Source

Smith, B. Othanel, and Milton Meux. A Study of the Logic of Teaching.Urbana: University of Illinois Press, 1970. pp. 116-21.

56

Interactive Teaching

The decision to use concepts pertaining to interactive teachingdemanded that we analyze the term "interactive teaching." From thatanalysis ultimately emerged the basic categories of pupil outcomes andteacher behaviors that guided the development of the catalogue. Inter-active teaching denotes all the events and transactions that our inthe classroom during the time that the teacher and his pupils are rrutall yengaged in implementing the educational program. These transactionsmay in part be based upon plans and objectives that have been worked

. out in advance, but interactive teaching refers only to the transactionsthemselves. Conceived in this way, interactive teaching occurs withina social system composed of four mutually dependent elements: (a) theteacher, (b) the content Which represents the curriculum or the mate-rialsto be learned, (c) the pupils as a social group or series ofsocial subgroups, and (d) the pupils considered as individuals operatingwithin a social contex-. Of course, pupils as a'group or series of sub-groups and pupils as i lividuals are simply different perspectivesadopted for thinking about the same people, but alternative perspectivesof this sort are quite indisperisible for the teacher. Another qual-ification in our delineation of the elements as constituting a socialSystem needs to-be made. When we say that the elements are mutuallydependent, we do not meant that they are of equal weight in the system.Common observation as well as the weight of research evidence aboutclassroom functioning both testify to the major domination of the class-room that is Wilt into the teacher's role. The proportion of conceptsin the catalogue that reflect what teachers do and how they influencepupil outcomes is great. Nonetheless, the teacher's behavior is dependentupon the other elements. If group morale sags, the teacher must restoreit. If an explanation of a scientific principle is not clear to thepupils, the teacher must communicate more meaningfully. If a particularbody of content bores the pupils, the teacher must omit it, make it morestimulating, compel the pupils to endure the dullness, etc. All thespecific ways in which the teacher's interactive rola with pupils ismodified by them are virtually countless.

Although it is appropriate to picture interactive teaching as em-bedded in a social system of the kind described here, the interrelation-ships and the mutual dependencies among concepts and variables impliedby such a system far outdistance our current ability to identify anddefine them or to specify their empirical values. The store of pro-positional knowledge about interactive teaching is meager. Therefore,in developing the catalogue, we disregarded issues of propositionalknowledge and the question of Whether the behavior represented by oneconcept is a function of another. Instead we concentrated upon iden-tifying and developing concepts about the behavior of the teacher ininteractive teaching and about the behavior of pupils, either as givensor as outcomes, in three major subdivisions: content, groups, and in-dividuals in a classroom context. The fourth element, the teacher,pervades all three of the major subdivisions. How this scheme generatesthe broad rubrics of the catalogue can be seen in Figu '-e 2.

57

Table 2

A Sample Concept To Appear in the Catalogue: From the Cognitive Sectionof Categories of Content, Productive Thinking

Comparing

Definition

A cognitive process in which pupils note and indicate similaritiesand/or differences between two or more items on the basis of thepupils' original analysis of the item.

Indicators

1. Comparing often involves a twofold process and results in twokinds of products: comparisons and contrasts.

2. The outcomes of comparing may be very similar to those of"comparison" (q.v.).The fundamental differences reside inthe source of the comparison. In comparing, the product isthe result of the pupils' productive thinking. Comparison,on the other hand, simply involves restatement by the pupil'of an intellectual product that he has heard or read.

3. The character and complexity of the comparison that resultsfrom comparing may vary in several ways, depending uponthe structure of the pedagogical control that brought aboutthe comparing behavior. For example, such controls, whetherthose of the teacher or of the textbook, may

a. provide one or more bases of comparison,

b. provide no basis of comparison, or

c. provide one item and a basis for comparison andrequest that the pupil provide a comparable orcontrasting item.

Illustiations

1. Sixth graders have constructed a table to compare twoexplorations. The columns of the table are headed "Columbus"and "Apollo 11" respectively. Rows are labeled "size of crew,""type of vehicle," "source of energy," "average speed," "distancetraveled," "cost of voyage," and "significant discoveries." ,Tlepupils are formulating comparisons and contrasts, using the data.

58

2. Two of the variations indicated in Indicator #3 above areillustrated here.

a. When asked whether Chicago or Detroit is larger, a pupilrefers to a population table and responds, "Chicago."The control in this case provides a basis of comparison:size. Note that the pupil's single word response is theend product of the process of comparison..

b. When asked to compare Hamlet and Macbeth, a studentresponds: "They were both royalty. Macbeth was motivatedby ambition, Hamlet by revenge. Both were troubled by

. their consciences. Each was killed, but in differentways. . ." Here no basis of comparison is provided.

Sources

Smith, B. Othanel, and Milton Meux. A Study of the Logic of Teaching.Urbana: University of Illinois Press, 1970. pp. 116-21.

Aschner, Mary Jane, and J. J. Gallagher. "A System fox ClassifyingThought Processes in the Context of Classroom Verbal Interaction."Champaign, Ill.: Institute for Research on Exceptional Children, 1965.

Taba, Hilda. "Teaching Strategies and Cognitive Functioning inElementary School Children." Washington, D.C.'7 U.S. Departmentof Health, Education and Welfare; San Francisco: San FranciscoState College, 1966. p. 39.

The major source is Smith and Meux. Aschner and Gallagher classifythe concept under "Association." Taba discusses "Identifying CommonProperties" as a process involved in "Grouping."

The material that appears in Tables 1 and 2 was developed for the catalogueby Joseph Gore.

59

The bottom branches of Figure 2 indicate the major categorical sub-divisions of the catalogue, but in most instances those categories arefurther elaborated. Without suggesting that the catalogue has taxonomicproperties, we tried to group concepts within the broader categories.For example, concepts about pupil groups are organized, with someexceptions, according to whether they deal with,issues of group morale,group productivity, or the control (in a social seise) of groups.

Concepts pertaining to how teachers control content in interactiveteaching are grouped depending upon whether they refer to broad approachesor styles of controlling content; whether their function is principallyone of regulating rate, amount, or intellectual level of the contentflow; or whether the concept involves evaluation of ideas in the con-tent, or as expressed by participants.

COGNITIVE CONTENT CATEGORIES

A similar principle has been utilized in organizing concepts thatdeal with pupil content outcomes. Following is the rationale developedfor seeking out and classifying one major dimension of such concepts,those having to do with cognitive outcomes that may be achieved bypupils.

Concepts pertaining to cognitive content are construed as indicativeof objectives about content or as representing pupil content outcomes.It is hoped that these concepts will have two functions: first, to suggestto the teacher or prospective teacher asrange of cognitive outcomesthat appears possible for pupils to achieve and, second, to define andillustrate the outcomes carefully in the context of interactive teach-ing. In this way, it is hoped, teachers may extend and deepen theircomprehension of what various cognitive outcomes may look like in pupilbehavior in the classroom.

Concepts about content have been subdivided into two principalcomponents: reception learning and productive thinking. The receptionlearning component is further subdivided into reproductive learningand comprehension. We are indebted to the writings of at least threeindividuals, groups, or schools: Ausubel (1968),3 Bloom and his col-leagues (1956),4 and Wertheimer (1959),5 for these labels and the con-ceptual structure that guided the development of the component.

Reception Learning'

Reception learning outcomes result principally from transactionsby the learner with instructional materials, including communicationsfrom the teacher. Ordinarily there is not a great demand that thepupil add his original thinking to the ideas or information impartedby these materials. On the contrary, the objectives typically requirehim to remember what he has learned. (Ausubel's conception of receptionlearning and the category identified as "knowledge" in the "Bloomtaxonomy" are instructive on this score.) The learner may either para-phrase the central elements of what he has been told or he may respond

60

more faithfully to the original information. This type of pupil out-comes or responses is called "reproductive" and is included as a firstclassificLtion under "reception learning." "Comprehension," on theother hand, requires the learner to demonstrate something in additionto the simple ability to reproduce a communique previously learned.

The term "understanding," except for all the ambiguities with whichit has been charged, begins to approximate our use of comprehension.We need to add to it, however, the stipulation by Bloom that the learneris acting upon the material. That is, in the language of the Taxonomyof Educational Objectives: Cognitive Domain,6 comprehension is reflectedby the student's ability to translate, interpret, or extrapolate from agiven body of material. We have chosen to classify comprehension underthe broader rubric of reception learning because of the dependence ofsuch outcomes upon a given message, passage, or communication, Thelearner is not asked to create what is essentially a new product. In-stead his actions reflect his ability to develop in a fuller sensewhat the meaning and implications of the message may be.

The importance of concepts about reception learning classifed underinteractive teaching is obvious. Without wishing to guess at percentages,it is apparent that a very large proportion of interactions in the class-room that deal with content do so at the level of reception learning aswe have defined it. Thus, the frequency with which teachers will gen-erate and encounter behaviors of these types and their pervasivenessin classroom interaction constitute important criteria for their in-clusion.

Productive Thinking

Productive thinking occurs, no doubt, with significantly less fre-quency than does reception learning. The principal criterion used toinclude concepts about this area in the catalogue is that pupil out-comes of productive thinking are vital in the achievement of mosteducational objectives pertaining to the improvement of students'intellectual skills and abilities, and the more teachers learn aboutsuch outcomes as they are manifested through interactive teaching, themore likely they are to identify appropriate pupil behaviors when theyoccur, and tc- arrange a classroom that is condUcive to increasedproductive chinking by their students. Although we do not wish tostretch the similarity too far, there are some parallels betweenreception learning and productive thinking. For example, receptionlearning includes the concept of "explanation." According to thismeaning, the learner would be able to explain an event, process, orphenomenon when the explanation had been given to him. Although notoriginal in any fundamental sense, the explanation would presumablybe complete and accurate. One kind of behavior involved in produc-tive thinking is "explaining." In'this case the student is confronted

illilwith an event, and although detaabout how much direction he isgiven may vary, he is given something less than a full explanation.The construction and communication of an explanation become hisprincipal intellectual tasks. L;uchmann7 provides excellent illustrationsof explaining. Pupils, in this case in intermediate grades, are shownsimple physics demonstrations. These brief films. end with the queStion,

5

61

"Why [did such an event occur]l" The process through which childrenlearn to provide explanations is a lengthy and complex one, but the pointis that in the. end, the learner must construct the explanation. Oneis not given to him. This is exactly the case for what we call pro-ductive thinking.

Pr9duririg the Concepts

We mentioned earlier that much of the conceptual argument for theorganization of this section of the catalogue came from the writings of'Ausubel, Bloom, and perhaps others. Similarly, for the concepts that weidentifed and defined in the body of the section, we are indebted to agroup of investigators who have studied interactive teaching in ongoingclassrooms and whose fundamental attention has been to what we call thecognitive components of such interaction. The scholars to whose workwe refer include, but are not restricted to, .Smith (1962, 1967),8Bellack and others (1966),9 Taba (1964, 1966),10 and Aschner andGallagher (1963, 1965).11 These works have provided us with validmodels of cognitive teacher-pupil interaction.

Our task in producing concepts for this section consisted ofseveral elements: First was the need to construct a conceptual frame-work to help identify concepts to be included in the category andorganize the presentation of those concepts. Secondly, we had to com-pare concepts across systems (e.g., between Smith and Bellack) and makean initial effort at integrating these concepts rather than simply col-lecting them. The latter tactic would.have contributed little to anintegrated view of concepts about teaching. The tactic we chose, ofattempting a preliminary integration of concepts with an eye towardreducing the number to be developed, has been successful only to alimited degree. Such efforts, however, seem to be desperately neededin discussions of education, and we would hope that subsequent workerswould follow the spirit of our intention and surpass the failings ofour execution. Finally, our effort included atteLipts to define, de-lineate, and illustrate the various concepts settled upon: We intendedto provide a maximum amount of beginninv assistance to the materialsdeveloper or the general teacher educator who wishes to expand uponthese concepts and incorporate them into the curriculum of teachereducation. This latter effort has been materially aided by the exis-tence of countless examples and illustrations included in the mono-graphs about interactive teaching to which we earlier alluded. Wehave drawn freely from these materials, and in each case we have citedthe sources, partly as an acknowledgement, but also as a reference forthe user to consult the original material for additional examples ormeanings beyond those used in the limited confines of the catalogue.

FORMAT OF CONCEPTS WITHIN THE CATALOGUE

The preceding discussion has focused on the chain of reasoningused to limit and define the conceptual domain represented in thecatalogue. By defining the domain, we included the specification ofthe concepts which it contains. We now turn to the descriptions of

62

concept in the catalogue. Once the developer of protocol materialshas located the concept(s) he wishesito develop, the dimensions andmagnitude of the problems confronting hjm undergo a shift. He mustbe concerned with the production of protocol materials or what hasearlier been referred to as "the portrayal of concepts" (Hudgins,1972).12 In that monograph, three basic issues that the developermust contend with as he translates abstract labels into ,vital proto-col materials for teacher education were established. The first twowere salient to the format adopted for descriptions of concepts in thecatalogue; the third remains essentially a matter to be determined byindividual developers. The issues referred to tare (a) analytic issues,(b) didactic issues, and (c) issues of outcomes.

Analytic Issues

Analytic issues pertain principally to the clarity and comprehen-siveness of statements of the concept. Attention to concept analysis isa first and critical stage in the production process. Little or noattention is paid to arranging conceptual displays for purposes ofteaching; rather, emphasis is on explaining as fully and as clearlyas possible the meaning of the concept. and including the elements orattributes necessary to ensure its comprehensiveness. The definitionand the presentation of indicators contained in the descriptions ofeach concept included in the catalogue attend to the analytic issues.Tables 1 and 2 illustrate two concepts in the Content dimension ofthe catalogue. The first concept, "comparison," assumes only recallby the pupil of what he has previously heard or read. In "comparing,"pupils must act upon a given body or item of consent in order to drawfrom it explicit statements about similarities and differences betweenit and some other event or phenomenon.

Didactic Issues

Didactic issues involve the developer in considerations of por-traying a concept in order to maximize its effectiveess as a teachingmaterial. At a minimum,. didactic issues demand that she developerpresent scenes or episodes that are clean and relatively simple examplesof the concept. Since the concepts with which teachers have to dealin class are seldom so simple, the developer ptdinarily needs to workout strategies for producing lifelike protocol materials. A rationalefor this procedure and suggestions for achieving didactic success aretreated extensively in Hudgins (1972).13 In the cataglogue, didacticissues are treated by one or more brief illustrations of the conceptin a classroom setting. When confusion with closely related conceptsappears imminent, examples have been added to aid discriminationbetween the two. These examples are not intended to be prescriptiveor in'any way limit the conceptual or artistic freedom of the devel-oper. Presumably most of these concepts could he illustrated in almostlimitless ways. The intention is simply to portray faithfully theconcept in one or two ways.

Analytic issues and didactic issues go far beyond tie introductionto them included in our descriptions within the catalogue. They playimportant and continuing roles throughout the developmental process.On the other hand, the descriptions that we have provided ought to

63

assist the developer in his preliminary consideration of these issuesand in his first attempts at portraying the concept.

Finally, in most instances, we have cited the major references orsources consulted in the development of illustrations. The developershould be materially assisted by having a brief but highly pertinentset of references for More extensive analytic and didactic guides.

ASPIRATIONS FOR THE CATALOGUE: ELABORATING THE CONCEPTUAL MLPS OFTEACHERS

We hold two major aspirations for the utilization of this catalogueof concepts about interactive teaching. First, if protocols of theapproximately one hundred concepts in the catalogue are eventually produced,a rew, systematic, and potentially powerful set of instructional mate-rials will be available to teacher educators on a wide scale. Tf theconcepts that our analysis of the domain of interactive teaching hasled to are productive in that they help teachers in training or in-service to more fully understand and interpret significant events asthey bccur in the classroom, then the basic purpose for which thecatalogue project---Vas originated will have been adequately achieved.The catalogue4it-Self is an inert object, and the viability of thataspiration can be tested only when a sufficiently large proportion ofthe concep_s identified therein have been developed and utilized in.ongoing programs of teacher education. Whether such development isto occur must, of coursebe decided by others. We would hope thatthe catalogue will. stimulate rather than inhibit such development. Ifit does, the catalogue may make some contribution toward helping toextend and deepen the conceptual base that teachers have for the ex-planation and interpretation of classroom events.

The second aspiration hais a broader scope and involves two ele-ments: one pertaining to the utilization of the catalogue and theother to the interpretation of the.caXalogue as a stimulus to the gen-eration of concepts. Throughout this discussion we have attempted tocomment upon the applicability of the .catalcgue to the work of protocoldevelopers. However, the conceptual framework of the catalogue as wellas the contemi: of individual concepts,may also have applicability formany teacher educators who are not formal materials developers, .but whoare responsible for the construction of teacher education programs andcourses. Finally, the -Catalogue is not inteneed to be used in a purelyprescriptive fashion, despite the fact that great thought and efforthave been poured into btith the classificatory system and the developmentof individual concepts. The concepts identified are important and de-fensible ones for delineating the interactive teaching subdomain; how-ever, we have no basis to certify that the concepts so identifiedexhaust the subdomain or the categories which we have established withinit. Were our work to spur others on to extend what we have begun, oreven to supplant it with better reasoned and more productive formu-lations of the domain; we would be happy to conclude that our efforts.had contributed toward the more intellectually rigortus education ofteachers which is, after all, the fundamental objective which all ofus share.

64

ACKNOWLEDGEMENTS

We acknowledge those who have labored long and hard throughout the .

life span of this pro2,ect.to produce the catalogue of concepts. Con-tinuing consultants to the project are Richard Derr, Case WesternReserve University; Joseph Gore, Southern Illinois University atEdwardsville; Eugene Jabker, Illinois State University; and CelesteWoodley, University of Colorado. Staf2 members at Washington Univer-sity are Deborah Berman and Helen Wait.

65

NOTES

1. Willard Waller, The Sociology of Teaching (New York: J. Wiley &Sons, 1932).

2. Gertrude H. MacPherson, Small Town Teacher (Cambridge, Mass.:Harvard University Press, 1972).

3. David P. Ausubel, Educational Psychology: A Cognitive View (NewYork: Holt, Rinehart, and Winston, 1968).

4. Benjamin Bloom and others, Taxonomy of Educational Objectives:Cognitive Domain (New York: Longmans, Green, 1956).

5. Max Wertheimer, Productive Thinking (New York: Harper, 1959).

6. Bloom, op. cit.

7. J. Richard Suchthan, "The Elementary School Training Program inScientific Inquiry" (Washington, D.C.: Department of Health,Educa-tioL and Welfare; Urbana: University of Illinois, 1965).

8. B. Othanel Smith and Milton Meux, "A Study of the Logic ofTeaching" (Washington, D.C.: U.S. Office of Education, 1962).

9. Arno A. Bellack and others, The Language of the Classroom (NewYork: Teachers College Press, ColUmbia University, 1966).

10. Hilda Taba and Freeman F. Elzey, "Teaching Strategies and ThoughtProcesses," Teachers College Record 65 (March 1964):524-34.

11. Mary Jane Aschner and J. J. Gallagher, "A Preliminary Report onAnalyses of Classroom Interaction," Merrill-Palmer Quarterly ofBehavior and Development 9 (July 1963):183-94; and Mary Jane Aschnerand J. J. Gallagher, "A System for Classifying Thought Processes inthe Content of ClaSsroom Verbal Interaction" (Champaign, Ill.:Institute for ResearCh on Exceptional Children, 1965).

12. Bryce B. Hudgins; "The Portrayal of Concepts: An Issue inthe Development of Protocol Materials," in Acquiring Teacher Competencies(Bloomington: Indiana University, School of Education, National Centerfor the Development of Training Materials in Teacher Education, 1972).

13. Ibid.

66

The ProtocOl Materials Program

by Donald E. Orlosky

HISTORY

\personnel training for the production of protocol materials inteacher education began in 1970 with support from the Bureau of Educa-tional Personnel Development (BEPD) of the U.S. Office of Education(USOE). The first meeting with nrotocol project directors was heldin Chicago during the summer of that year and chaired by B. OthanelSmith whose description of protocol materials in Teachers for the RealWorldl stimulated interest and support that enabled the program tobegin. Don Davies and Don Bigelow from USOE made presentations atthis meeting and added the weight ofjheir convictions favoring thedevelopment of protocol materials to the initial plans in the program.Doris Gunderson, BEPD program director, assisted in establishingprocedures to be followed by project directors.

Three groups were combined to provide the support necessary todevelop protocol materials. These three groups were BEPD, the Leader-ship Training Institute on Protocol Materials (LTI), and the projectdirectors. The interdependence of these three groups proved to beessential to the success of the program. A brief description of eachsupplies the foundation and context within which the program operates.

BEPD supported the training of personnel in the production ofprotocol materials as part of a comprehensive design to improve theperformance of educational personnel. Teacher performance is dependenton .a teacher's knowledge and his ability to employ his knowledge. Pro-viding materials that decrease the dissonance between vicarious learningand contact with pupils aids in the acquisition of knowledge. Textbooksprovide instruction about teaching, but they cannot actualize behaviorfor study. Classroom behavior provides actual behavior for study, butit is unpredictable, vanishes after it occurs, and is impossible tostudy completely from memory. A compromise between these two approachesis offered by protocol materials.

During the spring of 1970, BEPD screened responses to requests forproposals. The 11 sites designated to train personnel and develop pro-tocol materials began their work that fall. Since the first year of theprogram, support from BEPD and, more recently, the National Center forthe Improvement of Educational Systems (NCIES) has continued; briefingson protocol materials have been given to personnel at USOE. LTI hasreceived continued support to service the program, promote the trainingof personnel, and disseminate and publicize prdtocol materials to advancepersonnel training throughout the country. As LTI expanded its respon-sibilities, significant support for the program continued from theprogram director, Doris Gunderson, and from other USOE officials, par-ticularly William Smith and Allen Schmieder.

LTI was formulated to give technical assistance to the protocolprojects, provide liaison between projects and BEPD, coordinate theprotocol projects, and relate protocol materials to other federallyfunded training programs. It was created when the projects were fundedand is directed by B. Othanel Smith. Based at the University of SouthFlorida in Tampa, it utilizes personnel from throughout the nation whosetraining and experience qualify them to.give technical assistance.2

69

The major activities 01 LTI have been to plan the overall developmentof the program; meet with project directors to discuss and solve pro-duction problems; arrange consultant services for project directors;and assume, responsibility for standards of production, publicity,training of additional personnel, field testing, and development ofdissemination procedures.

One of the original 11 sites designed to train personnel was theFlorida State Department of Education. Its responsibility was differentfrom that of the other 10 locations.3 During the first year, the Floridaproject was responsible for producing protocol materials and forming aconsortium of institutions of higher education whose combined effortswould produce and field test the materials. The production of protocolmaterials through a consortium proved to be unrealistic, but the Floridaproject continued to serve as a field test site for the other projects.The project also trained personnel in the public schools and the insti-tutions of higher education in field test procedures and the use of pro-tocol materials.

MAJOR DEVELOPMENTS

Considerable progress has been made since-the beginning of theprogram. The issues and answers that arose can be understood best bythose who were engaged in the program. However, a description of themajor concerns and their solutions can give some idea of the progressin development of techniques to resolve production and training problemsand the procedures that grew out of the experience of those involved inthe program. Following is a discussion of the major developments in theprogram including (a) the definition of protocol materials and theselection of concepts, (b) technical requirements and criteria, (c)

supplementary materials, and (d) field testing and dissemination.

Definition

A general definition of protocol materials served to guide thosewho submitted proposals to BEPD. The basis on which the program begancan be found in the following statement:

Until the development of educational technology, it was dif-ficult to reproduce teaching behavior. But today audio and videorecordings of behavior can be made and studied in detail. Theymake it possible to teach theoretical knowledge of pedagogy inthe context of its use as well as in formal courses.

To follow this mode of instruction it is necessary to haveavailable an extensive supply of audio and video recordings ofhome, street, playground and classroom situations, of committeemeetings, and interviews. These recordings will be referred tohere as protocol materials, behavioral situations, or simplysituations. Protocol materials should represent the most poverty-stricken and most affluent rural and urban communities, as wellas all minority groups. They should also represent all gradelevels and teaching procedures such as problem-solving, question-and-answer, and group discussion.4

70

Refinements in this description were needed before production couldactually begin. Based on the definition that a protocol is an originaldraft, minute, or record of a document or transaction,5 protocol mate-rials are records of behavior portraying school events. The developershad to obtain a record of behavior on film or some other appropriatemedium that would give the viewer an episode to further his conceptualknowledge. This task introduced numerous questions requiring preciseanswers.

Development Procedures

The developers and LTI followed certain procedures. The behav-ior illustrating the concept in each protocol occurred frequentlyenough to enable the observer to see numerous behavioral indicators ofthe concept. The number of unrelated behaviors that might distract theviewer from seeing the intended behavior was held to a minimum. Eachepisode lasted 5 to 10 minutes to economize the production and instruc-tional and viewing time. Although the episodes were to be a "slice ofreality," some staging and scripting was allowed to guarantee that thebehavior illustrating the concept would occur. It Was initially regardedas inappropriate for protocol films tO-,include any pointers or cues tohelp the viewer with his observation. This requirement was eventuallyrelaxed to allow experimentation in determining how much help should begiven to a viewer. In some cases, numbered frames were allowed asreferences to parts of a film. Under no circumstances was it acceptableto give the viewer a descriptive narrative on the behavior depicted. Anarrative would remove the burden of interpretation from the viewer andviolate this fundamental principle of protocol materials. Within thesegeneral operating procedures, the protocol developers were able toproceed with planning and producing protocol films.

Interpretive Role of Teacher

A problem arose in respect to the concepts to be exemplified inprotocol films. A teacher should be able to make accurate diagnosesif he is to cope effectively with the events he faces. Concepts arethe organizing tools that enable the teacher to classify behavior orevents according to a given type.

Suppose that a student throws an object and subsequently wandersaround the room. How are these actions to be understood? Theycould be indications of either attention seeking or "critical dis-sension." The teacher will not understand these acts correctlyunless he is able to classify them correctly. If these actionsare associated with others that are clearly in the category ofattention seeking, the teacher is apt to be correct if he clas-sifies these actions in that category also. On the other hand,they are apt to be signs of critical dissension if they occurin a context of other acts that clearly belong to this particularclass. In any event, the teacher understands behavior byclassifying it, and the accuracy of his categorizing will deter-mine the correctness of his understanding and partly the adequacyof the subsequent treatment.b

71

This emphasizes the interpretive role of the teacher and points outthe advantage the teacher has if he can diagnose and classify behavior.Protocol materials should provide the observer with examples of behav-ior that will help him acquire the conceptual knowledge necessary tointerpret pedagogical events.

Most preservice instruction on concepts has been conducted in thelecture hall or through discussions describing instances to illustratea concept. Students who learn about concepts this way may do very wellon paper-pencil tests; however, when confronted with the behavior in theclassroom, they are often unable to classify and accurately interpretit. This inability of the beginning or student teacher to interpretaccurately results in the complaint that there is little relationshipbetween theory and practice. Actually, there is a close relationshipbetween the two; the usual methods of relating them have simply beenineffective. Words alone are inadequate to describe behavior.

Protocols vs. Observation

The protocol materials program supports the premise that if behav-ior could be captured on film, students in training could see thebehavior exemplifying a given concept and be better prepared to rec-ognize and classify behavior. Why.bother filming behavior when thesame behavior can be observed first hand in the classroom? Protocolmaterials offer some advantages not found indirect classroom obser-vation. Anyone who has faced the logistical problems in arranging andsupervising classroom observation experiences must ask if it is a worth-while use of the time and efforts of the university and public schoolpersonnel.

Apart from the administrative details that accompany such arrange-ments, there are other even more important factors. If an observervisits a classroom to observe specific activities, there is littleassurance that the given activities will take place. It is not uncom-mon for students who visit a school not to see a typical class becauseof scheduled standardized testing, a school assembly, or a disruptionin the expected lesson of the day. The time spent in travel, sittingthrough a school assembly, or watching students complete an examinationis all too frequent to rely on observation as a predictable part of theteacher training program. Additionally, even though the school may beoperating normally, events that are productive for the observer may notactually occur. The phenomena an observer wishes to see in the behav-ior of the teacher or pupils may not be portrayed during the time ofthe visit. And even if the observer is in the classroom when certainbehavior occurs, being untrained, he may miss it. If he does see it,the behavior vanishes as soon as It happens; it cannot be restaged forfurther study and analysis.

It is apparent that theory taught in the college or university isnot very useful'to most students who attempt to use their knowledge inthe classroom. Student placement as observers or student teachersoffers no assurance that the events they encounter will serve any pur-pose for their training and eventual teaching responsibilities. Today's

72

teachers capable of making proper diagnoses have acquired their abilitythrough time, exposure to the events in the classroom, an#finterpretationof behavior by trial and error. Protpcol materials are designed toenable the beginning teacher to be better prepared to interpret anddiagnose behavior. In addition, protocol materials are beneficial tothe experienced teacher who has not been sufficiently analytical orsufficiently schooled in interpreting behavior. Protocol materialsshould enable preservice and in-service teachers to acquire a betterorganizing network of concepts, the vocabulary necessary to describebehavior, aLd the observational and diagnostic ability to make accurateinterpretations of behavior.

Master Plan

A master plan was developed to guide the protocol program (seeAppendix in Cruickshank's paper). The plan provided for concepts tobe selected in the pedagogical domain and in the basic fields of study.About one-third of the first projects were in the basic fields, theremainder in the pedagogical domain. The master plan provided for con-cepts to be selected,in different settings such as the classroom, school,or community; differept age levels such as early childhood or adolescence;and different sourcestof behavior such as the teacher, pupil, or teacher-pupil interaction. It was recommended that the classroom settingreceive highest priority and that the developer be free to select thesource of behavior and the level he preferred.

The master plan gave broad direction to the program; the specificconcepts to be portrayed remained to be chosen. Shortly after theprogram began, it became evident that the literature on concepts isvague; more precision and rigor were needed in naming and defining con-cepts. The definition of concepts being prerequisite to the productionof materials, LTI concentrated on recommendations to help in the devel-opment of workable definitions.

Concepts are not all of the same order. Different definitionalforms are necessary to account for the variety of concepts that can beportrayed in protocol materials. In developing protocol materials, itwas suggested that concepts be expressed in one of four possible forms:classification, equivalent-expression, open-context, and conditional.The classification form of a definition provides for the associationof a concept with a category and discriminating criteria to distinguishone concept from others_ In ,the equivalent-expression form, a conceptis set forth by an expression that is equivalent to the word or wordsused to name that concept. Tice open- context form expresses conceptswhose definitions are imprecise because the boundaries of the term arenot rigidly limited. The interpretation of behavior may vary accordingto the conditions that pr.acede the behavior; the conditions are a partof the definition of a term, and the conditional form then becomesappropriate.

Selection of Concepts

The developer was confronted with the question of whether he shouldfirst film behavior and then determine the concepts that would provide

for the interpretation of the recording or if he should choose the con-cepts first and then control situations to elicit the desired behavior.The developer chose to select the concepts first to assure success intheir recordings. (See "The Protocol Materials Movement: An Exemplarof Efforts To Wed Theory and Practice in Teacher Education" by Cruick- .

shank, p, 9, for another point of view.)

In selecting concepts, the developer first entered the master planand selected a level, setting, and source of behavior in which he wishedto portray behavior. His second step was to name the concept plannedto exemplify. Finally, he defined the concept according to one.of thelinguistic forms provided. Subsequent steps remained for the developerto specify the behavior required for the protocol, communicate hisdecisions to the technical staff, and arrange all the logisticS andediting that remained. All of his later decisions were related to thework done in selecting and defining concepts.

The project directors were given considerable latitude in selectingthe concepts which they wished to portray. Each project had listed theconcepts it intended to illustrate, but the'level of understanding atthe time the proposals were prepared and the improvement in that under-standing after the projects were underway necessitated revision of theoriginal proposals in many instances. In some cases different conceptswere chosen; in other cases the selected concepts were modified eitherin respect to their definition or in the number of concepts to be in-cluded in the final products. Project directors were understandablyambitious in estimating the amount of work they could complet; morerealistic plans emerged as they gained more experience in the productionof materials. The concepts selected were originally tLose concepts thatmost nearly coincided with the developers' own background and knowledge.Eventually the project directors related the concepts they chose to ataxonomic or nomothetic system and sharpened their definitions to attaina clearer understanding and distinction between the concepts they wereillustrating and other concepts that were related. As work progressed,the initial choices held up well, but the amount of clarifyi andsupport information required increased considerably. After severalyears in the project, developers have learned to identify conceptsthat are part of a system rather than isolated ones.

Standards

The LTI committee on field testing, utilization, and disseminationmet with four project directors selected by their peers to determinethe standards for protocol materials. The committee was to supply stan-dards of acceptability to guide directors and to protect the users ofprotocol materials from inferior products. Requirements developed bythe committee included technical and educational standards. The tech-nical criteria selected were related to audio quality, visual quality,instruction, format, and copyright procedures. The educational criteriaapplied to standards of concept attainment, learner reactions, and usersatisfaction.

As far as technical quality was concerned, the developer wasadvised to use a double-sound system and to produce audio quality at

74

such a standard that fewer than 10 percent of the viewers would reportunfavorable perceptions. The visual quality was required to have suchsharpness of detail that virtually any viewer under normal classroomprojection conditions would be able to distinguish the critical featuresof the images presented. The producer was advised to protect his firstanswer print by preparing a master from the conformed original forduplication of subsequent prints. The format recommended depended canthe nature of the concept, but the acceptable media listed were 16mmfilm or EBR transfer film, slides in 35mm or filmstrip, and cassettesrather than reel tapes for audio protocols. Each package was to in-clude a rationale for the use of the media and guides for instructorsand students. The guides were to contain suggestions` for using pro-tocols, ways for viewers to record observations, means to measureacquisition of concepts, and recommendations for relating material toprior learning and practical situations. The instructional package wasto be organized and packed for easy storage and shipping.

The recommendations about copyright provisions were that writtenreleases should be obtained from all persons and for all materials usedin the protocols in addition to written releases for background .'usic,sound effects, lyrics, and other audio materials. The usual copyrightapplication and notice was listed as a requirement for protection ofthe overall design and treatment of the material and its copyrightablecomponents.

The educational criteria specified that 80 percent of the learnerscompleting the unit be able to identify 80 percent or more of theexamples of the concept. Since it is difficult to establish criterion-referenced standards with new materials, an alternate standard was given.A statistically significant difference in favor of the performance ofgroups who studied concepts with protocols was required in comparisonwith groups who studied the concept under conventional methods,. Learnerreactions should be shown through field testing to be positive to theextent that fewer than 10 percent of the learners would report unfavor-able reaction to the protocol unit. Learner reactions were to beobtained on the interest level of the materials, perceived relevanceto teaching, desire for additional protocol materials, and willingnessto recommend protocol materials to other students. The standard of usersatisfaction was aimed at getting information from the instructor usingprotocol materials in his work with pupils. Field test data were re-quested particularly the users' reaction to the importance of the concepts,the ease of using protocols in the instructional program, and the appro-priateness of the concept to be taught.

As a consequence of the work of the committee on protocol standards,the developers had guidelines at their disposal that would enable them totailor their production to criteria for acceptability. The users wereassured field test information about the protocol materials that wouldhelp them make a judgment about selection and utilization of protocolmaterials.

75

Supplementary Materials

In addition to the record of behavior, the protocol package containswritten material: teacher guides, pupil guides, and general information.One issue at stake in the preparation of supplementary materials waswhether the producer could impose on the eventual user the instructionalmethod to use with the protocol. The user can ignore the written mate-rials and use a film or audio tape according to his own preference. In

the case of protocol materials, the user may regard behavior as exemplaryand use a film to instruct in teaching skills. Or, since the behaviorrecorded may be considered good teaching technique, the user may employthe protocol materials as training materials. There is no guaranteethat the user will not use the materials to improve behavioral abilitiesrather than to improve interpretive abilities. Another considerationin the use of supplementary materials was whether a given instructionalmethod should be recommended. Some instructors may prefer a didacticapproach; others may favor an inductive or inquiry approach; and tillothers may f.el that a heuristic approach should be utilized. lhc developer may provide materials according to one bias, and the user mayuse the materials according to another. A third issue was how to assurethat the instructor would read the printed materials, benefit from theexperience of the developer, and use the approaches recommended for in-structing with protocol materials. None of these issues were adequatelyresolved.

It was recommended that the developer supply written materialsgiving the evidence from field testing and present the procedures usedand recommended by the developer. In some cases, the developer mightsuggest several approaches for the user to employ. For example, hemight recommend that the protocol materials could be used in large-group, small-group, or individual instruction according to the dis-cretion of the instructor. He may have field test data to show thatany organization of the students will yield about the same results;the user should not feel bound to a given approach. In other cases,a varied sequence might be appropriate. However, the only obligationthe developer can have for the eventual use of his materials is topresent the informatio about procedures he used, present the fieldtest data from that procedure, and suggest to the user that anydeviation from the recommended procedure might yield different results.The developer must make supplementary materials readable, brief, andcomprehensive. Brevity should encourage the user to read them andcomprehensive coverage should assist in answering questions about everyaspect of the use of the materials.

Field Testing

A requirement imposed on project directors was that all materialsshould be field tested at two levels before they were made availablefor general consumption. One level of field testing, completed underthe direction of the producer, was primarily to assist in the productionof the materials. If local field testing reveal,d that the criteriaestablished for acceptable materials were not being met, then the devel-oper could revise his product until it met the standards. The second

76

level of field testing, under the direction of the project in theFlorida State Department of Education, provided trials aL neutral sitesand in a variety of settings giving an objective report on the standardsin universities or public schools at the preservice and in-service level.After completion of field testingoin Florida, changes in the productwere completed prior to release. After all standards are met, the pro-ducts are ready for dissemination.

Dissemination

The dissemination of products is an unfinished task Pt the moment.Part of this effort is to inform the market of the existence of protocolmaterials. LTI and the developers have shared responsibility for pub-licizing their work. Presentations about protocol materials have beenmade at the annual meeting of the American Association of Colleges forTeacher Education and the American Educational Research Association.Several workshops have been conducted by LTI or project directors at thestate, university, and local level. Until projects were ready for use,publicity about their production was premature. Thus, the first twoyears of production were completed without much publicity. Now thatthe program has been underway long enough for some products to be com-pleted, the mechanics of dissemination are be'Lng deve-loped.

ACURRENT STATUS

During the 1973-74 academic year, the project directors have threemajor tasks. First is to complete the remaining details so that allproducts underway are finished. Some products are in final form andare being distributed; others are too inadequate to salvage; and othersare nearly completed but require some additional work to meet standards.The second task is the establishment and implementation Jf proceduresto disseminate products. The dissemination process is just beginning.It may be necessary to publicize the work further in order to instructusers in the employment of protocol materials and to contact profes-sional organizations, state departments, universities, local schools,and other federally funded projects. Finally, developers rust continuewith the production of the additional materials they have contracted toproduce.

Most of the protocol materials have been developed in the peda-gogical field, and the concepts chosen have been selected by the indi-vidual developer. An expansion of both of these efforts is takingplace. Nine additional trainees have been chosen from the fields ofmathematics, social sciences, and English to learn how to produce protocolmaterials and to compile'a list of appropriate concepts for protocol mate-rials production. At the end of the current year, tese trainees shouldbe able to carry on the development of protocol materials In their basicfields of study. They should also produce a catalog of appropriateconcepts for the preparation of teachers in their fields. A subcontracthas been used by LTI to organize the concepts in the pedagogical field.Consequently, the selection of concepts for protocol materials productionshould become more systematic and be based on concepts of greatest need

77

and importance to the teacher. This is not to suggest that conceptsthat have been developed are not especially usefvil; it only raans thata systematic study of the field was lacking at the inception of theprogram.

Cons'derable improvement in production techniques has occured.Future developers will benefit from these early experiences. A listof concepts in the pedagogical domain will increase the usefulness ofthe concepts selected for protocol development. The training ofadditional personnel and the organization of the concepts in the basicfields of study will provide the foundation for expansion of pro-tocol development across a wider spectrum of the total field of teacherpreparation. The efforts at dissemination that will be carried forthshould yield information that will open effective channels for distri-bution and use of protocol materials.

A

SUMARY

The protocol materials program has come on the scene at a fortu-itous time. The schools have failed to capitalize on the technologyand research potential that would improve them, but there appears to bea receptive climate for remedying this lack. The emergence of theNational.Institute of Education (NIE) is one concrete illustration ofrecent change in promotion of research in teacher,edUcation. The ade-quacy of NIE remains to be seen, but clearly the potential to start acomprehensive national effort in educational research is at hand. Theminiaturization and the refinements in recording behavior make thepotential of technology in the schools plausible and effective. Pro-tocol materials provide the means to capitalize on these research andtechnical developments and employ them in the development of new mate-rials for the instruction of preservite and in-service teachers.

\ Another illustration of the timeliness of the protocol program isthat the schools are faced with demands for accountability such asthe movement towards competency-based certification of classrom teachers.One of the critical areas of competence for the teacher is his inter-pi\etive ability to diagnose situations. Protocol materials providethe means to assess teacher competence in the conceptual knowledge hepossesses.

When the protocol materials program began, it,was impossible toaAticipate the numerous problems that would arise. The project direc-tors and LTI have combined forces with federal support to solve most ofthe major ones. Protocol materials meet standards that protect theuser, And guide the developer. The analysis of concepts and the rigorousdemands placed on the developer to clarify his work is bringing betterorder to the pedagogical and basic fields of study. Materials that pro-vide research opportunities and also respond to the demands for teachercompetence are now being developed in the program. The training ofteachers can be carried out with protocol materials as a bridge betweentheory and practice heretofore not possible.

:78

The future of protocol materials development is dependent on ade-quate financing, successful use of materials, and a cadre of people whoare willing to-devote their time and energies to this difficult assign-ment of recording behavior under rigorous standards. The program hasproven that it is possible to produce protocol materials. Field testdata reveal that the materials are effective. The additional knowledgeabout the preparation of teachers is a substantial contribution. Thetime is ripe for building on the foundation that has been laid and de-veloping a collection of protocol materials that will elevate thetraining of teachers to the higher level that the current state ofknowledge makes possible.

79

ti

NOTES

1. B. Othanel Smith and others, Teachers for the Real World (Wash-ington, D.C.: American Association of Colleges for Teacher Education,1969).

2. LTI Membership: B. Othanel Smith, director; Donald E. Orlosky,associate director. Panel Members: James A. Banks, James M. Cooper,,William Drummond, Ned Flanders, Vern Haubrich, Charles J. Modiste,Irving Morrissett, Allen C. Purves, Frederick A. Rodgers, H. DelSchalock, Charles Schuller, Charles Scruggs, Roger Shuy, and RichardL. Turner.

3. The locations and directors who assumed responsib'lity during thefirst year of production were David Berliner, Far West Laboratory;Paul Twelker, Teaching Research, Oregon State System of Higher Education;Celeste Woodley; University of Colorado; Bryce Hudgins, WashingtonUniversity; David Gliessman, Indiana University; Judith Henderson and J.Bruce Burke, Michigan State University; Donald Cruickshank, Ohio StateUniversity; William Heiner, Bucknell University; Peter Dow and AnitaMishler, Education Development Center; Richard W. Lid, San FernandoValley State College (now California State University at Northridge);and Pauline Masterson,.Florida State Department of Education.During the second. year, Walter Borg at Utah State University, TheresaLove at Southern Illinois University, and Patricia Heffernan-Cabreraand William Tikunoff at the University of Southern California wereadded. Washington University of St. Louis discohtinued after oneyear due to the difficuqty in acquiring adequate media productionpersonnel. Throughout the program some of the original personnel havebeen replaced; currently, the directors listed above have continuedwith the following exceptions: R. E. Myers at Teaching Research atOregon, Frank Zidonis at Ohio State University, Edward Martin andSusan Bernstein at Educational Development Center, Greta Morine andGloria Golden at the Far West Laboratory, and G. Michael Kuhn at theFlorida State Department of Education Project.

4. Smith, op. cit., pp. 52-53.

5. Webster's New Collegiate Dictionary (Springfield, Mass.: G. F, C.

Merriam Co., 1973), p. D27.

6. B. Othanel Smith and Donald E. Orlosky, fhe Development of Pro-tocol Materials, Acquiring Teaching Competencies Report No. 3 (Bloom-ington: Indiana University, School of Education, January 1973), p. 12.

80

A Protocol Materials Evaluation: TheLanguage of Children

by Victor M. Rentel

4

INTRODUCTION

In April 1970, several faculty members from the Ohio State Univer-sity College of Education discussed the possibility of responding to aU.S. Office of Education (USOE) Request for Proposal for dseties ofprotocol materials.1 Few of us had more than a sliM notion of whatprotocol materials were, our only knowledge coming mainly from Teachersfor-the Real World.2 According to this, protocol materialsare visual,auditory, or printed reproductions of relevant situations occurring inan educational context.3 These episodic records provide teachers intraining with the opportunity to observe and:study classroom situationsagainst a backdrop of educational theory: Rppeated observation andinterpretation of these slices of classroom life, according to Smith,should lead to the acquisition of concepts which serve ^s a basis forinterpreting new teaching experiences.4 The Request for,Proposalfurther defined and described these protocol materials, and the projectstaff4tas not without experience in developing materials roughly sim-ilar to them. Cruickshank had done extensive work in creatingsimulation materials for teacher education while other members ofthe project group had been involved in various aspects of researchin language development. This combination of experience led the groupto propose a series of protocol materials to illustrate concepts ofchildren's oral language acquisition. (For a detailed discussion ofprotocol materials and the theoretical issues surrounding their devel-opment, see Cruickshank's and Orlosky's papers).

The rationale for illustrating concepts of oral language acquisitionwas rather straightforward. The staff reasoned that teachers wouldmore readily accept children's oral language if given opportunities toobserve language development and discover underlying linguistic con-cepts. As with most decisions, however, this one had its problems.Implicit in the decision was the presumption that teachers are unawareof the developmental implications of children's oral language even thoughimmersed in it daily. Additionally, linguistic theory applicable todevelopmental interpretation of children's everyday talk has been andremains in a state of flux. Linguists themselves are unable to agreeon the analysis of English let alone schooling and language develop-ment. The decision to illustrate oral language development put, thestaff in a position of choosing among a variety of language models avail-able to us (tagmemic; transformational-generative in its MIT, Pennsyl-vania, and other forms; stratificational; and so on), or of picking andchoosing on the basis of needs and disposition.

These two problems presented our first evaluation questions. Towhat extent are practicing teachers aware of the developmental impli-cations of children's oral language? How could the concepts to beselected be authenticated? Finally, in a technical sense, how effectivewere the materials? These questions were examined in what we labeled aprocess evaluation (or formative evaluation).

83

PROCESS EVALUATION

In most respects the protocol materials we had chosen to developbore more directly on the theoretical knowledge teachers need to inter-pret classroom settings rather than on-samples of teaching itself. Assuch, the selection of concepts to be illustrated had to be made withinthe domain of linguistic and psycholinguistic science. A major liter-ature search was conducted, and a bibliography of over 200 e%tries wasassembled. This literature search produced. 115 major findings which,when organized in a taxonomy, clustered logically around eight categories.As might be expected, arranging our findings in a taxonomy suggestedseveral important relationships among the categories. We referred tothese relationships as a "soft theory" which may be briefly stated asfollows: Children process linguistic information they encounter, dis-covering and revising implicit rules and regularities within theirlanguage, passing through progressive but overlapping .stages of linguisticcomplexity. Order of acquisition appears to be the only invariant presentlyobservable within this process of development.

It is important to understand that none of the literature reviewedby the staff stated or implied in a direct way the conclusions we drewregarding this view of oral language development. While we owe a greatdebt to the intellectual work of a variety of-linguists and psycho-linguists, our understandings and conclusions should not be taken as adirect reflection of current theory nor as a reflection of the individualworks we drew, upon. Responsibility rests with the project staff.Indeed, it is probably immodest to referto these relationships as asoft theory for we intended no contribution of this sort, but were inter-ested mainly in describing a coherent set of principles and assumptionswhich might give direction and dimension to our efforts. Most gener-ously, David Stampe, a linguist w om we consulted, referred to ourconclusions as a soft theory, anti lore out of convenience than anythingelse, the designation stuck.5

Clearly evident across a wide variety of studies reviewed in thefirst phase of the project was the notion that children acquired lin-guistic structures in a particular order. Berko's work on the acquisitionof morphology, Carol Chomsky's on the acquisition of selected syntacticstructures, and particularly, the rich naturalistic work of Brown andhis associates at Harvard strongly supported the position that childrenmay vary in the rate at which they acquire particular structures, butthe order in which they do so will be invariant. 6,7,8,9,10,11 Variationof another sort stemming from socioeconomic, group, or racial status ofchildren was also well-documented in the literature.12,15 Known tech-nically as sociolects or social class dialects, these variations derivefrom group membership and should be regarded as distinct from regionaldialects. Finally, varying social contexts appear related to variin grammatical and lexical features as well as differences in field,mode, and style of discourse.14 These variations are known as registers.

This brief summary of our exhaustive literature review highlightsseveral of the major findings out of which a theoretical framework wascreated. Ultimately, from this framework, concepts were selected and

84

protocol materials developed. But, before that happened, the staffdecided that an independent authoritative review of our soft theory wasnecessary if subsequent materials were to be regarded as valid andauthentic.

AUTHENTICATION

Four subject-matter consultants were invited at intervals tovalidate both the theory and later the concepts under consideration.Roger Shuy, Fred Williams, and Courtney Cazden each visited the projectand not only suggested content revisions in the conceptual frameworkbut identified language situations they felt would illustrate particularconcepts as well. David Stampe from the Ohio State University Depart-ment of Linguistics reviewed and basically agreed with the softtheoretical account of how these concepts related to one another. Cazdenpaid a second visit to the project during the period when video tapeswere edited and helped to remove extraneous material and sharpeL conceptsunder consideration. This phase of the process evaluation provided cor-rective feedback which was used to validate and sharpen thf.: conceptualor theoretical framework for the project.

CONCEPT SELECTION

As noted above, our literature search produced a bibliography ofover 200 entries from which 70 were considered. The original taxonomyof 115 major findings was reduced to eight categories: performance-competence, selection-production, variability, sequential acquisition,innateness, role, culture specific, and systematic. Reclassificationyielded the final two categories and five subcategories. The secondphase of the process evaluation thus completed provided essential infor-mation that not only facilitated the selection of concepts for videotaping but suggested as well that these concepts comprised an importantsubject-matter for teacher education. These categories were then repro-duced and a panel of teachers, subject matter specialists, and teachereducators designated the concepts most salient to their field of activity.The panel was instructed to apply the following criteria to the selec-tion of a concept for inclusion in the taping matrix:

1. Universality of acceptance

2. Support of theory and/or research

3. Utility

4. applicability

5. Learnability

6. Timeliness

7. Power (importance)

8. Subject's lack of prior knowledge

85

86

9. Technical feasibility of taping

10. Potential for obtaining natural language

11. Potential for changing teacher behavior

12. Interpretability

13. Current effectiveness in teacher education

14. Fulfillment of central purpose of project

THE CONCEPTS

Following is a brief definition of each of the concepts selectedfor development as protocol materials. The major concepts identifiedare divided into two main categories: the "process of acquisition"and "variations within acquisition."

I. THE PROCESS OF ACQUISITION

Children progressively develop and revise sets of rules abouttheir language unconsciously by drawing upon their linguistic en-vironment.

A. Sequence

The process of acquisition is evident in the predictableorder of acquisition of sets of rules with variations in rateof acquisition.

1. Concept of specific syntactic structures:

a. Promise/tell. Structures in which "promise" is folloWedby a noun and an infinitive will be more difficult tointerpret than those in which "tell" is followed by anoun and an infinitive. The "promise" structures departfrom the minimal distance principle in English, whichpredicts that the noun closer to the infinitive willserve as its. subject. In the "promise" structures, the \

subject of "promise" also serves as the subject of thefollowing infinitive.

b. Easy/hard to see. In the sentence,

The clown is easy to see

the relatiorsh_p between "to see" and "clown" is thatof verb an object rather than subject and verb. The

."easy to see" structure somewhat obscures this rela-tionship whereas in a sentence like

The clown is eager to see

the relationship between "clown" and "co see" is thatof subject and verb. Consequently, the "easy/hardto see" structure .s more likely to be misunderstoodby young children until they learn to sort out thegrammatical relationships of subject and verb.

2. Concept of morphological development:

a. Noun inflections and derived forms. Children pro-gressively develop and revise secs of morphologicalrules in the following predictable order: (a)

occasional use of a rule for an inflected form, (b)

overgeneralization of the rule to other forms, and(c) refined and accurate application of the rule toappropriate forms.

b. Verb inflections and adjective order. Children pro-gressively develop and revise sets of morphologicalrules in the following predictable order: (a)

occasional use of a rule for an inflected form, (b)

overgeneralization of the rule to other forms, and(c) refined and accurate application of the rule toappropriate forms.

B. Complexity

There are developmental increases in the number as wellas the kinds of syntactic structures produced or understood.

1. Syntactic Acquisition

The complexity level of a sentence is indicated bythe number and kinds of syntactic structures used in it;that is, the more structures and the more kinds of struc-tures used in a sentence, the more complex that sentenceis.

2. Concept of embedding as revealed in T-unit

The complexity level of a response can be reflectedthrough T-unit word length. A longer average T-unit 'engthindicates syntactic maturity. T-units by definition con-sist of a main clause plus all subordinate clauses. Gar-bles, words which do not add information to the utterance,are deleted from T-unit word count. Aa exception to thisare those words used to claim attention such as "well,""see," and "you see."

C. Fluency

There are progressive increases in the ease and the full-ness with which children speak. The fewer the number ofhesitations, garbles, and fillers per utterance, and thegreater the amount of language in each individual's response,the more fluent the speaker.

II. THE VARIATIONS WITHIN ACQUISITION

Language acquisition involves not only the common sets ofrules developed by speakers ofthe language, but variations thatoccur within both individuals and social situations.

87

A. Individual Variability

Individuals vary in their use of specific structures.

1. Variation in acquisition

Morphology. Children of the same age differ in the kindand -lumber of basic morphological structures they learnto use. While there appears to be a highly predictablesequence of acquisition, the rate at which children acquirea specific structure varies considerably.

2. Variation in acquisition

Derived forms and adjective order. Children of the sameage differ in the kind and number of basic morphologicalstructures they learn to use. While there appears to bea highly predictable sequence of acquisition, the rate atwhich given children acquire a specific structure variesconsiderably.

B. Social Variability

Individuals and groups develop registers of language- --or language appropriate for particular social situations--whks4 may vary in phonology, syntax, lexicon, and para-linguistics.

PROTOCOL MATERIALS REVISION

Before final versions of video tapes, guide materials, and instruc-tor's manual were completed, prototypes of each were field tested with

ir in-service teachers in a nearby school district to provide correctivefeedbazk regarding the quality of each protocol episode.

Method

One hundred teachers from the Franklin County (Ohio) School Dis-trict taking a course for graduate or undergraduate credit at OhioState participated in the study. They were told that all studentswould receive a passing grade in order to obtain candid opinions re-garding the method of instructim employed in the course. Eight 3-hour classes were held, during which two protocol episodes were shownand related activities completed. After the completion of each proto-col episode, a questionnaire was administered to each subject.

Procedure

oFor each protocol episode, subjects read a short introductorypassage summarizing conclusions from research undergirding the concept.A short discussion followed. Subjects were then asked to examine thebehavioral objectives for the episode and were allowed to raise oues-tions related to the objectives. They were instruct, I how to view the

88

videotaped episode, what data to record in their participant's guide,and how to record and summarize the data. A discussion followed eachviewing, and summarized data from the protocol were described withlabels, categories, generalizations, and principles. After completingactivities in the guide materials, subjects responded to a question-naire. These results are summarized in Figure 1.

RECOMMENDATIONS AND REVISIONS

Recommendations came from buth participants and instructors.

Participants

The majority of protocol episodes were seen as both interestingand informative. Visually, the protocols slicited few negative re-actions; however, several of the episodes were plagued by deficienciesin audio quality, particularly those dealing with the promise/tellstructure, pronominalization, and the easy/hard-to-see structure inchildren's language. Audio deterioration stemmed largely from tran-scribing these episodes from film to one-inch video tape. Somereduction of audio distortion was possible, however, through an atten-uation technique that was developed subsequent to the process evalua-tion and resulted in considerable improvement.

Minor revision was recommended for one of the protocol episodes," 'Play-Talk' in Kindergarten." Specifically, discussion questions wererevised to state the intent of the questions more explicitly, and twoworksheets were combined. The protocols dealing with the acquisitionof morphology were edited to eliminate unnecessary instances, andinconsistencies were eliminated.

Instructor

Generally, guide materials were in need of slight revision. Thosedealing with morphology required further coordination with the videoportions of the protocol episodes. In nearly every instance, objectiveswere considered clear, and the materials appeared to relate directly tothe objectives. New learnings were presented at an appropriate rate,and, with the exception of episodes dealing with individual variationsin the acquisition of derived formm and adi

rrtiveorder, tapes and guide

materials seemed to match the participant' learning capabilities. Ex-pository material was -asily grasped, and content' was seen to be closelyintegrated with videotaped illustrations. Difficulties experiencdwith ne acquisition of syntax were noted by all instructors. Instructorratings were basically' positive regarding content, but again some tech-nical , Difficulties were encountered.

PRODUCT EVALUATION

Only a painfully small sample of protocol materials as been studiedto date; thus, greet caution should be observed in generalizing from theresults of this and other early attempts to evaluate the protocol notion.

S9

Figure 1

Summary of Process Evaluation for Protocol Episodes

4'24

te.2,L

'c'0,....,

or-I

44

. iri

<`-t'

4,

go,-44-3CI

C.) NH r-I4-3 --U CI:LF-i

,E

tip g

4-10 t.._.,

C-

e,C.1 .--4

3-I ). .4-1 cu

U E-C3?) 6

'41

(1) c9

(,)4 k..._,

c

.,a)0=CC)

U 4.31-1 Cl)

4-3 .3-1C.) CI

>, gv) .E4

t ".4..,c

C.) r--,,4 >1.4-, 13C-) ',,11.--',.

.003

(f) .1

t`-'0 ;.....

c

U.-I ,--,4-3 N4_4 .4-3

La 740 a)>, -)(1) E"

48 8,..,c

--,tdC..)

o-Ib+0

0,0'fa.)4 ,--,0 to

"$.9

18 '..._.,

=

.--IalU

11DOo

3-Io.00.;-4 )--,0 in7. S48 2'm

,.4a)4-)

VI.1-4b4)a)

C:G''c

. P4

td

=0

.1-I43

.`,.1

;,-4

=0I4

.`1

',II

0-I4)

.`r1

1.4

00-I4..)

.',1

tg

rI4-'

!3',-4

0 0 V)' .-I ci)

0 V).,I a)

O'I ca

0 tr)..-I a)

0 )r)1 cu

0 In--, a)

O'-,-. a)

..-1P

ico

1--1

cd

.ticc)

--1ct

.",ct

<Li 4-1 f-i 4-4 $-4 4-4 F-4 4.) I-I 4-4 f-) 4-) $-I 44 II CC 0 0 0U )-I = -,-) 0 .41 0 + 0 -) = ..-I 0 )-I 0 > .0 ":1 11 nz) MIo 4r) 4-1 tr, 4.1 VI 4.3 , 4-3 CA 4-' Ul 4-1 to 4-, .r-3 .1-1 -I I ,^4

C) .,-c 4..) ..-1 U H C.) 1, u .-3 U -I U P-I C..) a, > > > >, >

g- '8- g- i 9 g - 4 2 , F r PI FT gr00 1-1

,:f14'0

-Iial.

r-I^0

',411

Questions (1) 6-)' ;4) 6) Q) C+) 'I Q) (+/-)1 t) (4/) .(7)' Q eh' t; I -4 1 51 / El 4 i.

1. Objecties were clear. 90 60 93 97 76 97 80 70 73 60 57 100 97 100

'. Materials were attractiveand interesting. . 70 30 87 76 64 89 64 60 80 47 57 83 93 73

3. Materials built on my pre-vious knowledge. 80 53 70 77 74 89 63 50 70 50 53 i3 86 90

4. Content was well organized. 87 67 94 87 80 87 83 60 73 37 76 90 97 90

S. Learnings were clearlypresented. 94 40 86 73 54 86 67 43 60 44 53 90 94 74

6. Repetition of important.

content was adequate. 84 SO 93 73 64 89 63 57 57 67 53 83 96 74

Learning tasks were easy. 84 50 87 90 64 93 80 40 73 33 46 90 90 83G.

8. Learning tasks were at mylevel of understanding. 84 60 94 87 t-.3

p8690 46 76 57 60 97 90 70

9. Protocol was visully clear. 60 43 77 70 76 80 83 64 70 80 53 80 87 70

10. Sound track was clearlyaudible. 63 3 73 3 57 33 66 7 54 70 36 100 57 26

11. Guide materials were easyto use. 86 77 87 90 94 85 90 57 63 3') 33 100 96 93

12. Ideas presented were w-,rthlearning. 87 43 97 77 80 93 63 63 60 43 56 80 93 74

Figure 1. Percent of Subjects Agreeing with Statements.

Method

Two groups of 61 subjects each were drawn at random from a reading-language arts methods course ari randomly assigned to equal treatments.Two instructors were assigned in random order to 9! equal number ofclass sessions for both conditions. The'protocol materials conditionwas compared with a treatment in which subjects were taught the same con-: ,ts through the lecture method coupled with classroom obseFations ofchildren's oral language. Posttest means were compared using a t testwith significance set at the .05 level in a simple randomized design.

Procedure

Subjects in the protocol materials condition completed 14 episodesin which five concepts were illustrated in a va iety of oral languageactivities depicting aspects of the acquisition ,f syntax, register,and morphology as well as individual difference.. in development. Asstated previously, during each protocol segment, subjects read a shortsummary of research conclusions about the concept being illustrated.Following a short discussion, subjects read behavioral objectives forthe protocol segment and asked any questions about the objectives.Subjects were then given directions on viewing the prctocol, instruc-tions on how to record data in their guidebooks, and opportunities towork through examples before viewing the video tape. After viewingeach tape, subjects discussed their observations. Finally, each subjectdescribed the data collected in terms or labels, categories, general-izations and principles.

Subjects in the lecture condition were provided with similar infor-mation regarding each concept illustrated in the protocol materials.They were also provided with written materials but had no opportunityto observe films or video tapes of children in various stages of lan-guage acquisition. Instead, subjects in this treatment observedchildren in public school classrooms and in informal play situationsindependent of the lecture class itself.

Both groups received approximately the same number of hours ofinstruction. The lecture condition group received slightly over i0hours of instruction avid 2 ho,irs of observation wi-Cle t7.e protocoltreatment group received 12 heirs of instruction.

Instrument

A 46-item test was constructed an equal number of items forall bu.c. one of the concepts illustrated in the protocol materials:Items were constructed from behavioral objectives for each protocolsegment and judged for their concept validity by a 5-member panel fromthe project staff. Items that did not receive full acceptance by thepanel were rewritten or rejected. Two reliability coerFficients wereobta. ?,c1 using the Kudor-Richardl,on methods 20 (.77) and 21 (.70). Thestaru d error of measurement for the instrument was 3.04 while themean item difficulty coefficient obtained was .30, and thr, mean itemdiscrimination coefficient obtained was .28 Tests of skewness revealed

91

only a slight negatively skewed distribution (.86) while tests ofkurtosis indicated some peakedness in the distribution (1.21).

Results

Posttest means between the protocol treatment (33.76; S.D.=4.71)and the lecture treatment (33.08; S.D.=4.40) were not significantlydifferent (t=.75).

Discussion

Protocol materials have been suggested as a means of overcomingthe weakness of unstructured observation and many theoretical inade-quacies characteristic of teacher education. Yet, protocol materialsdemonstrated no advantage over lecture and classroom observation wherethe criterion was a written achievement test administered 7 days fol-lowing the completion of all protocol episodes. It can be argued, andperhaps convincingly, that, for protocol materials, a straightforwardachievement test lacks construct validity. If, indeed, protoccl mate-rials have as their ultimate end improving a student's ability toabsti -t and transform raw data from a classroom into salient conceptsabout, in this case, language condition, then a different sort ofcriterion measure should be contemplated. A more desirable measurewould ct least make use of protocol-like instances of the specifiedbehavior as the criterion task. This sort of measure would more nearlyapproach the logic of protocol materials as well as the sort of learnerresponse anticipated in the notion. Hypothetically, at least, it hasbeen suggested that protocols may engender a learning-to-learn set aswell as numerous specific learnings that may transfer readily to newresponses or stimuli. Before firm conclusions are drawn regarding theeffectiveness of protocol materials as medium of instruction, theseestimates of both specific and general transfer should be obtained.

The finding of no difference between treatments, while not exactlywhat a development team would prefer, loses much of its sting whenviewed in the context of subjects' perceptions of these materials. Gen-erally enthusiastic responses from subjects who were asked to compareprotocols with other instructional techniques indicate that protocolmaterials are interesting and attractive, that they are well-organizedand clear in their presentation of concepts, and that they incorporatea body of learnings that students consider worth learning. To thedeveloper, the no-difference result should suggest evaluation proceduresthat are as interesting, precise, and clear as the materials themselves,and a component within the development plan that is, if not as sizable,at least as significant as the materials package.

92

NOTES

1. At its outset the project staff included Donald Cruickshank(direc'...or), Robert Emans, Carol Fisher, Martha King, Victor Rentel,and Frank Zidonis. Sharon Fox and Johanna DeStefano have since joinelthe staff as have several graduate assistants, Pat Lyons, Alice Swinger,Tony Goeree, and Mary Kuhner. Technical development has been the respon-sibility of The Ohio State University Telecommunications Center and theDepartment of Cinema and Photography. Frank Zidonis now directs theproject. Emans, Fisher, Lyons, and Swinger are no longer with theproject.

2. B. Othanel Smith and others, Teachers for the Real World (Wash-ington, D.C.: American Associ_cion of Colleges for Teacher Education,1966).

3. Smith and others, Ibid., pp. 52-53.

4. Smith and others, Ibid., p. 52.

5. David Stampe, a member of the Department of Linguistics at OhioState, served as a consultant during the project's first year.

6. Jean Berko, "The Child's Learning of English Morphology," Word14 (August l958):150-77.

7. Carol Chomsky, The Acquisition of Syntax in Children from Five toTcn (Cambridge, Mass: MIT Press, 1970).

8. Roger Brown and Colin Fraser, "The Acquisition of Syntax," inVerbal Behavior and Learning: Problems and Processes, eds. CharlesN. Gofer and Barbara Musgrave (New York: McGraw-Hill, 1963), pp. 158=201.

9. Roger Brown and Camille Hanion, "Derivalional Complexity and Orderof 'Acquisition of Chiba Speech,' in Cognition and the Development ofLanguage, ed. John R. Hayes (New York: John Wiley and Sons, 1970),pp. 11-53.

10. Courtney B. Cazden, "The Acquisition of Noun and Verb Inflections,"Child Development 39 (June 1968):433-48.

11. Colin Fraser and others, "Corrol of Grammar in Imitation, Com-prehension, and Production," Journal of Verbal Learning and VerbalBehavior 2 (September 1963):I21-35.

12. William Labov, The Social Stratification of English in New YorkCity (Washington, D.C.: Center for Applied Linguistics, 1966).

13. ---, "Contraction, Deletion, and Inherent Variobility of theEnglish Copila," Language 4 (December 1969):115-62.

14. M. A. K. Halliday and others, The Linguistic Science and LanguageTeaching (Bloomington, Ind.: Indiana University Press, 1969).

A Survey of Protocol Materials Evaluation

by John E. Cooper

0

9S/96,

A BRIEF HISTORY OF PROTOCOL MATERIALS

There is no shonage cf instructional materials in teacher education.Even a cursory review of Books in Print, recent Fublications announced bybook companies, and audiovisual catalogs will impress the reader with theproliferation'of instructional materials for teacher education. Probingbeyond the facts of volume productions, however; will reveal that mostof these materials ,re informational in nature-1 They address subjectmatter such as educational anthropology and, child development; methodology,e.g., reading instruction and pupil control; concepts such as transferof learning and motivation;, operations like constructing a sociogramor finding the median; and educational issues such as federal aid toparochir.1 schools and national testing.

While there is an impressive store of informational materials inteacher, education, little can be found which is based on organized theory.If teaching is to rise above the level of a craft, teachers must be ableto respond in some way other than through dependence on trial and error,common sense, and the practical. Theoretical knowledge is required forinterpreting and solving problems.

Concepts provide the basic elements of theoretical knowledge. Conceptacquisition is a sine qua non for the exercise of expertise in any learn'dprofession. Therefore, the preparation of the professional should providesufficient attention to concepts crucial to the nature of the profession.

The need for placing fundamental concepts at the center of teachereducation programs was emphasized in Teachers for the:Real World2 whichcalled for the development of protocol materials to illustrate key conceptsdrawn from psychology, sociology, and philosophy which would reproducebehaviors in life-like situations. In this connection, Smith differentiatedbetween protocol materials and training materials. While the former dealwith theoretical or conceptual elements, the latter address methodologicalor skill components.

By January 1974,rapproximately 140 protocol products had beendeveloped as part of the Protocol Materials Project of the U. S. Officeof Education. With the support of the National Center for the Improvementof Educational Systems (NCIES), approximately nine units of trainingmaterialS had been prod,,ced at Indiana University un. r the auspicesof the National Center for the Development of Train, ng Materials inTeacher Educat ion.

97

STATEMENT OF THE PROBLEM

The primary purpose of this report is to summarize evidence of theeffectiveness of protocol materials in the improvement of teaching. Asecondary objective is to place this evaluation in the context of a briefreview of protocol materials as an educational innovation. It shouldbe pointed out that the greatest proportion of funds and energies hasbeen directed at training personnel in the development of protocolmaterials rather than at the evaluation of this effort.

LIMITATIONS OF PAST EVALUATIONS

Even though Protocol Materials Project directors have attempted tofield test their products before distribution, relatively little attentionhas been given to large-scale evaluation, dissemination, and preparationof trainers of preservice and in-service teachers for using the materials.

There are several explanations for the relative neglect of theseessential consideratiOns. Foremost is that projects are funded for abrief term, generally one year. This forces project directors toconcentrate on developing materials, to the neglect of those elementswhich come after production, e.g., try-out, evaluation, training, anddissemination. Secondly, the resources required for executing thosephases arc not always available to protocol producers.

METHODS OF ANALYSIS

In reviewing evaluation, 73 pieces of evidence were examined,inventoried, and classified (see Appendix A). These items .game from12 protocol locations. They included professional articles, manuscriptsof addresses, summaries of studies, project reports, and letters andinstruments for collecting data. The investigator limited his reviewto material in his possession at the time of the study. Some protocolproject directors may have collected evidence of the effectiveness oftheir products that was not available to the investigator at the timethis report was written.

A matrix was devised for classifying the kinds of evidencerepresented by the documents (see Figure 1 and Appendix A). Twenty-one of the reviewed items did not contain information directly relatedto instructional effectiveness; they are listed in Appendix B. Certaincells which are not applicable or which have little if any possibilityof revealing evidence are blocked out in the matrix.

The most powerful evidence of the effectiveness of protocol materialswould reveal chcnges in the behavior of students. Since protocols aredesigned for use in teacher training rather than for the education ofchildren, it is not surprising to find that to date no project hastried to gather this kind of evidence. To do this it would be necessaryto demonstrate that children who were taught by teachers trained with

98

Na

A.

B.

C.

D.

F.

G.

1.

Figure 1. Protocol Evaluation: Evidence and Materials

1

1 No evidence ,

V0

k0km

.. g1) CD

,fl ;.."0

'0 ,--40 ,-4

s 0i

V 0 k 0.=4-

(1) LH..0

1 a (.0 >,-..4-., co

;-4 CO ,0 0 PA0 0 0LH Z tp.5 ()I ,o'0 k k Z V 0

C.) a.) CI r- CD r-1(u

U.40 04-J

M Mg tt, ,0 0 0-e-I V 04-) g 0

4-0 4-1 0

0 4-,M VIg (0 Li) ,--10 k 0 0

r-I 0.) CL) (..)

0

4-, Vin sA0 00- t4-10 0 (.()

k4- 0

14-4 U_o co 0..

4.4C.) X 0E a) S-1

Not uappltca---J

able -4>

W

I 1,1 Sources 4-40

listed incn

Appendix A -.J

zii

:.e

ture of the Materials

(u),(.-.:0(a

.c 4-U 0

.41111111Prir

All&m.

tat g., s---

C 4-,m .J=Z 0 4-1U rTh 0

CV

14111r4p141441KirAllillbAdihk

g g;-4 4.1 Um ct a g0 ti -.0 0

.- 4-, 4-, E

t.,)

.1111111111r.11111111V

.411166..

(3 k 4-)m m o0 k kc4 4-) R.

":1-

4-, .0 P:0 (..) .- 4-)ammo0 0 k /.-4

c4 4J 4-' fa

Lt)

R..-, -0

0 r:' 4-'1> cd 0

111111"

Films, tests, diaries,etc. to measure #1

Films,panel judgments,anecdotal records etc.

to measure 42.---Test results tomeasure 43

Alli11111111°'

3,1,5,6,7,8,9,1

1

2.0c4

21.27

.11111111111V

3,4,5,7,8,9,17,18,19,20,21,22,25,26,27,44,50

Questionnaires, inter-views, commentaries,etc.to measure #42.

Questionnaires, inter-views, commentaries, etc.to measury #5 .14 4P 4VLetters from users,lists of users, testi-monials, etc.

2, 27

9,45,47

Professional articles,manuscripts of :pooches,tapes, etc.

52 5, 49 5, 49 48

Summaries of studiesincluding field tests

32 3,4,15,19,20,21,22,44,50

3,4,17,1920,21,22,25,26,44,50

17, 44

Project reports 27, 51

4 __

27,51

-4,

1, 27 46

-A,-J. Materials designed to

facilitate acquisitionof concepts for #2

K. Materials designed rofacilitate acquisition

of concepts for #3 or44

L. Evaluative instruments

1. 6, 7,3, 9, 6,7,.3,9,23,24.28,32

Nf3,4,5,6,7,8, 4,5,6,7,8,9A 11,13,34,9,15,16,18, ; 10,12,14,17, 36,38,40,42j22,25,33,44

. 1S?21,22,27,: 44

301,32,35,37,

39;,41,43,44,-

51

protocol materials performed significantly better than those taughtby teachers who did not have access to these materials. It would alsobe necessary to show that there were no other significant differencesbetween groups of pupils, teachers, or learning settings than the isolates:variable of use of the protocol materials.

A second level of evidence would show differences in teaching behaviorresulting from teacher exposure to protocol materials. Obviously this, too,requires considerable effort to obtain, necessitating evidence of favorabledifferences in teacher performance before and after the introduction ofprotocols. Despite the difficulty of obtaining this information, oneitem of such evidence is revealed in this report.

Other types of evidence of the impact of protocols are easier toacquire. They include concept acquisition, reactions to the-materials bytrainees, and volume of demand for protocols. These kinds of evidencewere collected for this report.

FINDINGS

Evidence of Changed Teacher Behavior

Borg and Stone (52) * selected two of six protocol modules--"Encourage-ment" and "Extension"--developed at Utah State University during 1971-72'nd tested their impact on the classroom behavior of 19 in-serviceelementary teachers of the Weber (Utah) School District. These teachershad received approximately 16 hours of training over a period of 2 weeks.Training involved reading descriptions of the concepts; completingpractice lessons; viewing protocol films; and completing recognition tests,application practice lessons, application tests, and self-practice lessonsusing audio tape recorders.

Specific behaviors accompanying "Encouragement" included generalpraise, specific praise, and use of student ideas. Those relating to"Extension" were prompting, seeking further clarification, refocusing,and redirection.

Pre- and post-training audio tapes of 20 minutes duration were madeof each teacher performing in her classroom. These tapes were coded andscored by trained raters who did not know whether a particular tape wasrecorded prior to or after training.

Virtually no difference was found in the amount of general praisegiven before and after training; however, 17 of the 19 teachers increasedtheir use of specific praise. The average use of student ideas approximatelydoubled.

Prompting, which is a strategy for improving pupil response to teacherquestions, more than doubled following training. The average teacher alsonearly doubled the use of further clarification.

*Numbers in this section of the report refer to those items listedin the Protocol laterials Evaluation Exhibit Inventory (see Appendix A).

100

Refocusing is an attempt to enable the student to generalize hisknowledge through questioning that locates common elements drawn fromdiverse subjects. It is a relatively uncommon teaching technique.Eleven of the 19 subjects showed gains on this variable.

There was no significant change in the use of redirection, a strategyemployed to increase the number of discussion participants. This variablewas found to be a common teaching behavior at the pretraining level.

Evidence of Concept Acquisition

Utah State University tested six protocol modules dealing withimportant concepts related to teacher language (49,50,51). Theyincluded extension, encouragement, clarity, emphasis, feedback,and organization. A criterion level calling for 80 percent mastery by80 percent of the subjects for each module was established. Threecriteria were selected for evaluating each module: (a) recognitionof teacher use of the concept on film, (b) recognition of teacher useof the concept in typed manuscripts of class discussions, and (c) applicationof the concept to typed transcripts of classroom discussion lessons. Onthe final field test more than 80 percent of the learners reachedthe criterion level of mastery on all 18 of the criterion measures usedto evaluate the six modules.

Protocol films and guides entitled "Tasks of Teaching" have beenproduced at Michigan State University. They cover assessment, goalsetting, strategies, and evaluation. The protocol materials were evaluatedby using 429 Michigan State l iversity undergraduate education majors (20).They were randomly divided into` an experimental group of 215 subjectsand a control group of 214. The experimental group received instructionusing the protocols while students in the control group received parallelinstruction without the use of these materials.

Two measures of concept acquisition were selected, one of conceptrecall and the other, a measure of the ability to identify the conceptsas part of a teaching vignette. The results showed the clear superiorityin concept growth of the experimental grour.

Michigan State University has also been involved in the developmentof protocol materials, including filmed and written aids, for respondentlearning. Concept acquisition and transfer of the concept to a simulatedteaching situation were tested, using more than 600 Michigan StateUniversity education majors (22). Six different treatment conditionswere established in the experiment. The results produced strong evidenceof the effectiveness of protocol materials on concept acquisition andits transfer.

Four video tape protocols at the Far West Laboratory for EducationalResearch and Development were dsigned to aid in interpreting group processin the classroom. Acquisition tests were administered on three of theprotocols: task roles, unifying roles, and antigroup roles to instructedand noninstructed groups (4,5). The results showed the ability of theprotocols to teach concepts. Differences in:concept attainment favoredinstructed groups over the noninstructed group at the .01 level. The

fourth protocol, stages of group growth, waf not fully evaluated in thefield test.

Recently, protocols were completed and field tested by the Far WestLaboratory on using.student ideas, questioning, praise and correctivefeedback, and lesson organization (6,7,8,9). Although differentmeasurement techniques were employed to field test concept attainmentfor each of the four protocols, early positive results of the effectivenessof the protocol materials were attained.

Indiana University produced foul protocols, on film and in print,on cognitive interaction, affective interaction, classroom management,and counseling. An early evaluation report of these materials showedthat significant learning occurred from use of each of the four units (18).Since this report, the Protocol Materials Project at Indiana Universityhas developed and tested more thoroughly additional protocols on teacher-pupil interaction. Studies conducted on learning outcomes have yieldedpre- and post-treatment data on a single group and post-treatment datafor comparison with results of an untrained group. Results documentsignificant growth on the acquisition of tested concepts (19).

The findings of one doctoral study at Indiana University, on theeffects of protocol and training materials on concept acquisition andskill acquisition on teacher trainees, suggest "that either materialsexpressly designed as protocol materials or materials expressly designedas training materials lead to the acquisition of both interpretive conceptsand teaching skills. The finding that viewing protocol films instancingconcepts about teaching behavior leads to a demonstrable acquisition ofthose be'laviors (as well as the acquisition of concepts about thosebehaviors) should be of interest to future investigators" (19).

Protocol materials developed at Southern Illinois University atEdwardsville were in audio tape and printed form and dealt with morphologicaland syntactic features of Black Dialect. These materials were field testedin several states, using both education majors at the preservice level and,in-service teachers (27). Twelve concept acquisition tests were administeredon various concepts related to the linguistic content. The criterion levelof 80 percent mastery by 80 percent of the subjects was satisfactorilyachieved on each of the 12 tests.

Evidence of Reaction to the Materials

Many Protocol Materials Project directors designed methods formeasuring the attitude of trainees to protocol materials. A smallernumber also designed instruments to ascertain the attitude of teachersof trainees to drotocols.

102

User reactions to protocols falls into two broad categories: (a)

impressions about the technical quality of the protocol, e.g., soundquality of a film, and (b) value judgments about the usefulness of thecontent for improving teaching. The most common instruments employedto collect both kinds of information were rating scales.

Summaries of student reactions to the technical quality of the protocolare found in the reports of Utah State University (51), Michigan StateUniversity (20,21), Indiana University (17,19), and Southern IllinoisUniversity (27).

Summaries of student reactions to the relevancy of the protocol toteaching can be obtained from Utah State University (51), Michigan StateUniversity (20,21,22), the Far West LabOratory (5,6,7,8,9), IndianaUniversity (17,19), and Southern Illinois University (27).

Instructor reactions were gathered at Indiana University (17) andSouthern Illinois University (27). Six instructors from several institu-tions tried out Indiana University protocols and gave positive responsesto their quality, appropriateness of content, and utility in promotingintended concepts.

Evaluation of the Southern Illinois University audio tapes in BlackDialect by eight specialists in fields such as speech, linguistics, andanthropology appears in narrative form (27).

p

Evidence of Demand for the Materials

The Protocol Materials Project director at the University of Coloradois giving considerable attention to the subject of dissemination of protocolsproduced at that institution. The decision to publish and disseminateUniversity of Colorado protocol materials was made in April 1973. Tenthousand brochures describing the materials were printed with the firstmailing on Maya1, 1973 (46). By Novemper 15, 1973, more than 300 requestshad been received for previewing, renting, or purchasing or for furtherinformation about the materials (45,48).

SUMMARY OF THE FINDINGS AND RECOMMENDATIONS

A review of research on the effectiveness of protocol materials inimproving teaching and learning shows that no attempt has been made asyet to discover the influence, if any, on the behavior of pupils. Onestudy reveals that protocols result in favorable changes of teachingbehavior. Positive results have been obtained on the acquisition ofconcepts by preservice and in-service teachers. There is also evidenceof the reactions of both trainees and their teachers to the technicalqualities and relevance of protocol materials. While little attentionhas been directed to dissemination, there is a growing evideh-e ofdemand for protocol materials by preservice and in-service educators.

The following recommendations relate to future evaluations ofprotocol materials:

1. As Lheory is related to practice, so are concepts ,related toskills. Concerted attention should be given to identifying and-searchingfor relationships between instructional concepts and skills whichinfluence teaching and learning. The work of Hudgins in catalogingconcepts and Turner in cataloging skills should be supported as animportant contribution to this work.

2. Protocol and training products are essential instructionalmaterials for preservice and in-service competency-based teachereducation programs. In view of the national transition to competency-based teacher education, financial support needed to assist thedevelopment of these materials.

3. Ultimately, the question of whether or not protocol and trainingmaterials used in competency-based teacher education programs make anydifference to pupil performance should be researched. This question cannotbe answered finally and fairly, however, until materials have been developedwhich exemplify the full range of essential concepts and skills, untilteacher educators have been properly trained in their use, and untiltrainees have completed competency-based programs that have relied onthese tools.

4. In the meantime, the field testing and consequent revision ofnew protocol materials should be encouraged. While collecting userreactions to the materials serves a legitimate purpose, it cannot takethe place of tests of concept attainment. These should be conducted asrealistically aF possible. Audio- and video-taped evidence of traineelearning, when feasible, is superior to the evidence produced by paper-and-pencil tests. Similarly, information gathered from responses tofilmed testing should be a more accurate indicator of conceptualpower than information collected from a written simulation.

104

NOTES

1. David Chessman, "What Are Protocols: Their Nature and Purpose?"in Handbook on the Development and Use of Protocol Materials forTeacher Education (Chipley, Fla.: Panhandle Area EducationalCooperative, 1973).

B. Othanel Smith and others, Teachers for the Real World (Washington,D.C.: American Association of Colleges for Teacher Education, 1969).

1 o5

APPENDIX A

Source

Buckrell University

California StateUniversity at North-ridge

Protocol Materials Evaluation ExhibitInventory

Number Classification Title

1 5 1 ,protocols in DevelopmentalReading: May 1970, September1973

2 F

Far West Laboratory

Florida Departmentof Education

5

6

7

8

9

10

3C,

4D,

3C,

4D,

3C,

4D,

3C,

4D,

3C,

4D,

3C,

4D,

3C,

4D,

4L

3H,

4H

3H,

4H,

3G,

4G,

3K,

4K,

3K,

4K,

3K,

4K,

3K,

4K,

3L

3L

4L

3L

4L

3L

4L

3L

4L

3L

4L

3L

4L

11 5L

Letter to John Cooper,Dec. 12, 1973

Protocols on Group Process,.Instructor' s Manual

Learning Concepts a outGroup Process: An valuatioof Protocol Ma al s

-The Group Process Protocols:Tlie 1971-72 Protocol ProjectReport for AERA MeetingFeb. 25-Mar. 2, 1973

Lesson Organization: ProtocolMaterials for Teachers, 1973

Praise and Corrective Feed-back: Protocol Materialsfor Teachers, 1973

Questioning: Protocol Mater-ials for Teachers, 1973

Using Student Ideas: Proto-col Materials for Teachers, 1973.

Student Analysis Form forField Trial Evaluation ofProtocol ;Materials forTeacher Education

Analysis Form for .Instructionand Specialists for FieldTrial Evaluation of ProtocolMaterials for Teacher Educa-tion

10

Source .Number Classification

a

,Title

Florida Departmentof Education

Indiana University

12

13

4L

5L

Trainee Analysi3 Form forProtocol Materials forTeacher Education

Instructor'S Evaluation Fotin:,Concepts and Patterns inTeacher-Pupil Interaction.

14 4L Student's Evaluation Form:Concepts and Patterns inTeacher-Pupil Interaction

15 3L, Inventory IC ( Revision 5/17/73)

16 3L Inventory D. part I (Revision9/17/73)

17 4D, 4H, 4L RP,ction to Protocol Materials:5H A Survey of Students and

Faculty Users

18 3C,

4P. 4H,

3L4L

A Preliminary EvaluationReport on the Developmentand Use of Filmed ProtocolMaterials within Two InstructionalStrategies

19 3C,

4D,

3H

411

An Evaluation Summary andDissertation Abstract onthe Effectiveness of ProtocolMaterials

Michigan StateUniversity

20 3C,

4D,

311

4HExperimental and FieldEvaluation of Protocol Mater-ials Developed To Teach"Tasks of Teaching" Concepts,Report #2

21 3C,

4D,

3H

4H, 4LUniversity of South FloridaField Test of the MichiganState University ProtocolMaterials on Learning, Jan.197 3

22 3C, 3H, 3L MSU Research and Evaluation,

23

4D,

3K

4H,.4L Report #1, May 1972

Carrel Lesson One: The Tasksaw.f. Teaching

108

Source Number Classification

Michigan State 24 3KUniversity

Ohio State University 25 4D, 4H

26 4D, 4H

Southern IllinoisUniversity

27

28

3C,

4D,

SF,

3K,

31

41,

SI

3L

4L

29 6F

'SUNY at Buffalo 30 4L

Teaching Research 31 4L, 5L

University of Colorado 33

43K, 4L

3L

Title

Education 200, Unit IV,Teaching Task #3: TheProcess of Strategy Selection

A Protocol MaterialsEvaluation: The Languageof Children

Field Trial Report: TheLanguage of Children

Final Report: ProtocolMaterials Development Pro-ject, SIU at Edwardsville

Identifying the Morphologi-cal and Syntactic Featuresof Black Dialect,

Telephone conversation withTheresa Love, ProtocolMaterials Project Director,SIU, January 2, 1974

Field Test Evaluation Forms,Fredonia, N.Y.

Protocol Materials forTeacher Education, LearnerOutcomes, Field TrialEvaluation Guide, March 1971

Untitled (Important Defini-tion)

Student Background Informationand Questionnaires forConcepts about Teaching

34 SL Instructor Evaluation Ques-' tionnaire for Conceptualizing

the Process of Instruction

35 4L Student Evaluation Question-naire for Conceptualizing theProcess of Instruction

36 5L Instructor Evaluation-Ques-tionnaire for Learners andTheir Characteristics

1 09

Source Number Classification

University of Colorado 37

38

4L

5L

39 4L

40 5L

41 4L

42 5L

43 4L

44 3C, 31-f, 3L4D, 41-1, 4LSE, 5H, 5L

45 6F

46 61

47 6F

110

Title

Student Evaluation Question-naire for Learners and TheirCharacteristics

Instructor Evaluation Ques-tionnaire for Verbal Inter-action in the CognitiveDimension

Student Evaluation Question-naire for Verbal Interactionin the Cognitive Dimension

Instructor Evaluation Ques-tionnaire for Organizing FactsTo Teach Meaningful Relation-ships

Student Evaluation Cuestion-naire for Organizing Factslb Teach Meaningful Relationships

Instructor Evaluation Ques-tionnaire for Fair VerbalBehavior

Student Evaluation Question-naire for Fair Verbal Behavior

Evaluation Report of the1970-72 Protocol MaterialsUnits Developed by the Pro-tocols Materials DevelopmentProject, University ofColorado

Dissemination Report: Listof Persons Ordering Materialfor Preview, Rental or Sale,August 1-November 15, 1973

The Dissemination of ProtocolMaterials: One Project'sAnswer

Letters from Users of Univer-sity of Colorado ProtocolMaterials

Source Number Classification Title

University of Colorado 48 6G The University of ColoradoProtocol Project: A CaseStudy

Utah State University 49

50

3G, 4G

3C, 3H4D, 4H

51 3C, 314D, 41, 4L

Protocols: Competency BasedTeacher Education Modules,By Walter Borg in EducationalTechnology 12, no. 10 (October1973)

Field Testing and Evalua-tion in the Utah StateUniversity Protocol Project

The USU Protocol Project:Final Report, 1971-72

52 2G, 2H What are Protocol Materials?

111

APPENDIX B

Materials Reviewed but Not Applicable to the Study

Source

Far West Laboratory

Florida Departmentof Education

Indiana University

Michigan StateUniversity

Southern IllinoisUniversity

SUNY at Buffalo

Teaching Research

Number

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

Title

Introduction to Protocols

Field Test: Protocol Materials on GroupProcess

Florida Protocol Materials Project, Sept. 19,20, 21, 22

(Untitled paper) Review of 1971-72 Activitiesand a Summary of 1972-73 Activities

Memo; Subject: Agenda Item for LTI andDirectors Meeting in Tampa, January 23-25, '73

Protocol Materials Review Inventory

Protocol Materials Review Inventory andWritten Materials Form

Concepts and Patterns in Teacher-PupilInteraction: Categorizing Classroom Behavior.Filmed Version

Categorizing Teacher Behavior, Part 1

Protocol Materials Evaluation Plan forMichigan State University, 1973-74

Protocol Materials Development Project:Notes for Instructors

Project in Ethnography in Education

Project in Ethnography in Education Training.Materials: A Description

Some Specifics on Field Testing and TrainingActivities

Progress Report, March 21, 1973

Protocol Materials for Teacher. EduCation:Learner Outcomes User's Guide, April 1971

113

Source Number Title

University of Colorado 69 Progress Report: Development of Protocolsof Social Science Concepts and ProposedDesign for Testing and Evaluation of the1973 Products

70 Field Testing, Evaluation, Revision andDissemination of Protocol Materials Producedduring 1970-72 by the Protocol MaterialsDevelopment Project, University of Colorado

71 Protocol Materials Development Project:A Summary Report, 1970-1973

72 Instructor Background Information

73 Protocol Materials Development Project:A Summary Report, 1970-1972

111

ABOUT ER,

The Educational Resourc Information Cr, -,ter (ERIC) forms a nation-wide information system establ, shed by the U.S. Office of Education,designed to serve and advance'American education. Its basic objective isto provide ideas an(' information on significant current documents (e.g.,research reports, articles, theoretical papers, program descriptions,published and unpublished conference papers, newsletters, and curriculumguides or studies) and to publicize the availability of such documents.Central ERIC is the term given to the function of the U.S. Office of Edu-cation, which provides policy, coordination, training funds, and generalservices to the clearinghouses in the information system. Each clear-inghouse focuses its activities on a separate subject-matter area; acquires,evaluates, abstracts, and indexes documents; processes many significantdocuments into the ERIC system; and publicizes available ideas and infor-mation to the education community through its own publications, those ofCentral ERIC, and other educational media.

TEACHER EDUCATION AND ERIC

The ERIC Clearinghouse on Teacher Education, established June 20,1968, is sponsored by three professional groups--the American Associationof Colleges for Teacher Educktion (fiscal agent); the Association ofTeacher Educators; and. Instruction and Professional Development,'NationalEducation Association. It is located at One Dupont Circle, Washington,D.C. 20036.

SCOPE OF CLEARINGHOUSE ACTIVITIES

Users of this guide are encouraged to send to the ERIC Clearinghouseon Teacher Education documents related to its scope, a statement of whichfollows:

The Clearinghouse is responsible for research reports, curriculumdescriptions, theoretical papers, addresses, and other materialsrelative to the preparation of school personnel (nursery, elemen-tary, secondary, and supporting school personnel); the preparationand development of teacher educators; the profession of teaching;and the fields of health, physical education, and recreation. Thescope includes the preparation and continuing development of allinstructional personnel, their functions and roles. While themajor interest of the Clearinghouse is professional preparationand practice in America, it also is interested in internationalaspects of the field.

The scope also guides the Clearinghouse's Advisory and Policy Counciland staff in decision making relative to the commissioning of monographs,bibliographies, and directories. The scope is a flexible guide in theidea and information needs of those concerned with pre- and in-servicepreparation of school personnel and the profession of teaching.

115