ED 034 788

AUTHORTT criar

INSTITUTIONSP ONS AGENCY

P UPEAU NOPUB DATEGP ANTNOTE

EDES PRICEDESCRIPTORS

DOCUMENT RESUME

24 TE 001 670

Taylor, Vi MarieMichigan English Study of Structure for CurriculumEvaluation (MESSAGE) .

Michigan Council of Teachers of English, St. Joseph.Office of Education (DHEW) , Washington, D.C. Bureauof Research.BR-8-0813Apr 69OEG-0-9-320813-1374(010)76p.

EDRS Price MP -$0.50 HC-$3.90Curriculum Design, *Curriculum Evaluation,Curriculum Planning, Curriculum Problems, CurriculumResearch, Educational Objectives, *EnglishCurriculum, Essay Tests, Evaluation Criteria,*Evaluation Methods, Evaluation Techniques,*Inservice Teacher Education, Objective Tests,Supervisors

ABSTRACTThe Michigan English Study of Structure for

Curriculum Evaluation (MESSAGE) training session involved 40 leadersof English curriculum study in a 7-day intensive program. Theystudied objective evaluation in terms of behavior, instruction, andinstitution; approaches to the English curriculum throughinstructional objectives that include evaluation procedures; andimplementation of training by working with local curriculum groups.Through participation in the program, curriculum leaders should beable to develop criteria for evaluating English programs, Englishmaterials, and individual student progress; gather information aboutlocal programs; and exercise leadership in planning programs, writingobjectives, and evaluating materials and programs. Local schoolsshould be able to (1) find nearby curriculum leaders to help withbasic planning, (2) utilize program participants for training otherstaff members, and (3) evaluate present curriculum procedures andprepare changes based on rational methods of innovation. (Appendicesinclude the EPIC Evaluation Center's objectives for the ProjectMESSAGE program, a list of participants, suggestions for writingauestions, and feedback questionnaires.) (Author/LH)

U.S. DEPARTMENT OF HEALTH, EDUCATION & WELFARE

OFFICE OF EDUCATION

THIS DOCUMENT HAS BEEN REPRODUCED EXACTLY AS RECEIVED FROM THE

PERSON OR ORGANIZATION ORIGINATING IT. POINTS OF VIEW OR OPINIONS

STATED DO NOT NECESSARILY REPRESENT OFFICIAL OFFICE Of EDUCATION

POSITION OR POLICY.

03CO

COLaLU

vc

0

ll

FINAL REPORTProject No. 8-0813

Grant No. OEG-0-9-320813-1374 (010)

k- Ok/319/1- Y

Ci Ey/YR

MICHIGAN ENGLISH STUDY OF STRUCTURE FOR CURRICULUMEVALUATION(MESSAGE)

Vi Marie Taylor

Michigan Council of Teachers ofEnglish

St. Joseph High SchoolSt. Joseph, Michigan 49085

April 1969

The research reported herein was performed pursuant to agrant with the Office of Education, U. S. Department ofHealth, Education, and Welfare. Contractors undertakingsuch projects under Government sponsorship are encouragedto express freely their professional judgment in the con-duct of the project. Points of view or opinions stated donot, therefore, necessarily represent official Office ofEducation position or policy.

U. S. DEPARTMENT OFHEALTH, EDUCATION, AND WELFARE

Offica of EducationBureau of Research

FOREWORD

The MESSAGE (Michigan English Study of Structure for Curricu-lum Evaluation) training session involved forty curriculumleaders in English, selected from the state at large, whohave major responsibility for working with school staffs inthe development of curricula. The seven-day training pro-gram, described in this report, was used to introduce theprinciples of systematic evaluation, to provide practice inthe writing of relevant behavioral instructional objectives,and to prepare participants to apply long-range systematicevaluation procedures to the preparation of English programsfor schools.

The Michigan Council of Teachers of English sponsored MESSAGE;the U. S. Office of Education funded it; the Michigan Depart-ment of Education and administrators from throughout thestate supported it. All of these agencies may expect thatMESSAGE participants will provide state leadership in theareas of research, curriculum planning, and criticalevaluation.

1

it

SUMMARY

MICHIGAN ENGLISH STUDY OF STRUCTURE FOR CURRICULUMEVALUATION(MESSAGE)

The MESSAGE training session was a seven-day intensive pro7gram for Michigan curriculum leaders in English. Forty par-ticipants from over the state, who had been identified asleaders or potential leaders of curriculum studies inEnglish, were introduced to a three-dimensional model forevaluation. On this basis they studied objective evaluationin the context of the variables of behavior, instruction,and institution, Ways in which the English curriculum couldbe approached through the building of instructional objec-tives that would include evaluative procedures, and implemen-tation of the training: through work with local curriculumgroups. The emphasis throughout was on long-range evalu-ation of programs and curriculum building through a programof feedback of information about curricula.

As a result of participation of staff members in theprogram, schools in the State should be able to:

1. Find curriculum leaders in their own or nearby sys-tems to help with the basic planning for curriculum change.

2. Utilize the training of certain staff members whohave already been trained through Project English centers,summer institutes, or other formal or individualized pro-grams. With their backgrounds in subject-matter areas,these leaders will be able to lead the schools in combiningtechniques of evaluation and curriculum construction.

3. Evaluate present curriculum patterns and base anyproposed change on rational procedures of innovation.

As a result of participating in this training program,curriculum leaders should be able to:

1. Develop criteria for evaluating significant factorsin English programs.

2. Select, develop, and use a variety of methods forevaluating different English materials for comparativeeffectiveness.

3. Use a variety of resources for gathering infor-mation about local programs.

4. Discuss and develop a variety of models for evalu-ating individual progress.

5. Exert leadership in their own curriculum areas inthe planning of programs, the writing of instructional ob-jectives, the selection of procedures to evaluate materials,and the evaluation of the programs.

3

INTRODUCTION

The idea of the MESSAGE training program had its in--ception in the spring of 1967. The president-elect of theMichigan Council of Teachers of English, Dr. Vi Marie Taylor,began to talk with members of the executive board of theCouncil about ways in which the organization could helpMichigan schools and teachers to face some of the problemsin building sound English curricula for the schools of thestate. Teachers in the schools were asking questions whichapparently were going unanswered or being answered too quick-ly and without proper study, such as: Where should we startin selecting materials? Should we be adopting some of thesuggested materials being developed by the Project Englishcenters (which were then entering their final phases)? Whatshould we do about initiating innovations or putting intopractice some of the techniques being tried in variousschools of the state? These questions seemed to be surfacemanifestations of deeper problems, namely: how do depart-ments of English determine their goals; how competently dothey judge their achievements; how scientific is theirmeasurement of progress toward the goals and objectives theyhave set? Apparently, many systems were looking for peopleto help them to find answers to these questions; the uni-versities were frequently asked to send help, or to supplyshort-term answers, but many systems were still looking foranswers to questions that were symptomatic of problems thatwere not being reached.

In the fall of 1967, as president of the MichiganCouncil, Dr. Taylor brought to the Executive Board of theCouncil a plan for a training session which would helpschool systems to find among their own people the personnelthat could be developed to help to plan attacks on the basicproblem. With the approval of the Executive Board, shewrote a proposal for the Michigan English Study of Structurefor Curriculum Evaluation (MESSAGE) and secured a grant fromthe United States Office of Education, Research and TrainingDivision, for the training of curriculum leaders in the de-sign and writing of objectives for the teaching of English.Cooperation was pledged by the National Council of Teachersof English and by the Michigan Department of Education.

With the final approval of the grant, the trainingsession was scheduled for the Boyne Mountain Lodge, Boyne

Falls, Michigan, April 8 - 24, 1969. The seven days im-mediately preceded the annual conference of the MichiganCouncil of Teachers of English at the same location. Inthis isolated location, the participants, consultants, andstaff would work, eat, and play together for the period ofthe training session. They would also be able to end theirtraining program by making immediate reports of the programto the members attending the annual conference.

OBJECTIVES

The objectives of the training program, as announced inthe preliminary releases to the members of the Council, toadministrators, and through news media of the Department ofEducation, suggested ways in which school systems might bene-fit from participation in the MESSAGE project: These generalobjectives were stated as follows:

I. As a result of the participation of staff membersin the program, schools in the state will be able to

1. Find curriculum leaders in their own or nearbysystems to help with the basic planning for curriculumchange.

2. Utilize the training of certain staff memberswho have already been trained throcigh Project Englishcenters, summer institutes, or other formal or indi-vidualized programs. With their backgrounds in subject-matter areas, these leaders will be able to lead theschools in combining techniques of evaluation and cur-riculum construction.

3. Evaluate present curriculum patterns and baseany proposed change on rational procedures ofinnovation.

II. As a result of participating in this institute, cur-riculum leaders will be able to

1. Develop criteria for evaluating significantfactors in English programs.

2. Select, develop, and use a variety of methodsfor evaluating different English materials for com-parative effectiveness.

Use a variety of resources for gathering infor-mation about local programs.

4. Discuss and develop a variety of models forevaluating individual progress.

5. Exert leadership in their own curriculum areasin the planning of programs, the writing of instruction-al objectives, the selection of procedures to evaluatematerials, and the evaluation of the programs.

Specific behavioral objectives developed by the stafffor the training session are included in Appendix I in thesection outline for "Skill Development in Evaluation forTeachers of English."

METHODS

PRELIMINARY ANNOUNCEMENTS AND PARTICIPANT SELECTION

Several means of information dissemination were em-ployed to notify potential participants about the program.The complete mailing list of the Council, consisting of over2200 teachers, administrators, and curriculum leaders in thefield of English, as well as college and university person -.nel, was used for the distribution of a special four-pageedition of the Michigan English Teacher, which contained apreliminary application form. The mailing list of the StateDepartment of Education, which lists over 1300 intermediateand system superintendents and, principals, was used to sendannouncements and nomination forms. The State Department ofEducation also included announcements in its regular organs,which are sent to all administrators in the state.

From these sources, more than 170 preliminary appli-cations were returned. These people were supplied with thelonger application form, further information about require-ments, and the reminder that each application must be ac-companied by a statement from an administrator that thetrainee would be utilized by the school system to lead incurriculum evaluation studies. More than one hundred com-pleted applications were received. Applications were readby four different readers, and selections were made on thebasis of leadership potential; backgrounds in English, cur-riculum, and evaluation; geographic distribution; and availa-bility for service to schools other than their own. In ad-dition to the 40 participants selected for funding, fourmore were permitted to attend at their own or their school'sexpense. Every first choice appointee accepted the appoint-ment and attended the full session.

Participants included:

one consultant for 28 local school districts

eight K-l2 supervisors, coordinators, department heads, orcurriculum directors

four 7-12 supervisors, coordinators, department heads, orcurriculum directors

7

ten 9-12 supervisors, coordinators, department heads orcurriculum directors

four 10-12 supervisors, coordinators, department heads, orcurriculum directors

one 7-9 coordinator

eleven chairmen of curriculum committees of local districts

one university methods teacher (observer)

one college teacher-consultant

one elementary teacher, member of English curriculum com-mittee (observer)

two secondary classroom teachers, members of English cur-riculum committees (observers)

Administrators offered assurance that participants wouldbe assigned a major role in the areas of planning and evalu-ation and that they would be encouraged to assist nearbyschools in evaluation of programs and implementation of newideas for the improvement of the teaching of English in thestate.

Names, addresses, and positions of participants aregiven in Appendix 11.

PROGRAM PLANNING SESSIONS

Acting for the Executive Board of the Michigan Councilof Teachers of English, a committee met in January to es-tablish preliminary guidelines and plan staff. This com-mittee was composed of Dr. Stephen Dunning, 1968-69 presi-dent of the Council; Miss Mildred Webster, executivesecretary of the Council; Mr. Dwight Smith, English special-ist of the Michigan Department of Education; and Dr. ViMarie Taylor, director of the program.

This group set February 14-15 for a planning sessionand decided to invite Dr. Robert L. Hammond, Ohio State Uni-versity, and :Dr. Kellogg Hunt, Florida State University, tothe planning meeting. The committee learned that the repre-sentative of the National Council of Teachers of English atthe planning session would be Mr. William Scannell, of thenational office. At this meeting also, the dates April 18-24 were confirmed and the meeting place at Boyne MountainLodge, Boyne Falls, Michigan, was established. The time and

8

date were planned to precede the annual meeting of theMichigan Council of Teachers of English, scheduled forApril 25-27.

On February 14-15, specific elements of the programwere developed in the planning session. The division oftime among the various parts of the program was discussed,and approximately equal time segments were allocated togeneral evaluation procedures and the writing of objectives,on the one hand, and to the application of these techniquesto English programs. Dr. Hammond presented in some detailthe EPIC model for curriculum evaluation, which the projectdirector had suggested as a structure for the trainingsession. The English specialists present at the planningsession agreed that the EPIC model could be used for the ex-amination of English curricula in the context of dimensionsof behavior, instruction, and institution. It was decidedto extend an invitation to Dr. R. J. Armstrong to bring ateam from the EPIC Center, Tucson, to present the model totrainees and to lead tirr9 general evaluation segment.

It was decided that Dr. Hunt would use a day and a halfof the training session to apply the techniques of the pro,gram directly to programs in composition, syntax, and read,ing. Dr. Dunning would use one session of the training pro-gram for a demonstration of the teaching of poetry as a wayof arriving inductively at the construction of behavioralgoals. The methods employed by Dr. Alan C. Purves, Uni-versity of Illinois, were suggested for inclusion in theprogram as an additional objective instrument for measure-ment of English programs, and Dr. Purves was invited to par-ticipate in the training session.

9

PROGRAM

General Evaluation Segment

Throughout the training session, emphasis was placed onlong-range evaluation techniques and their application to thedesign of English programs. For this purpose, the staff in-cluded instructors capable of covering this material fromvaried approaches. Dr. Robert L. Hammond set the tone forthe entire session in his opening remarks. He enumeratedproblems faced by all curriculum designers, discussed presentsteps and processes available to correct these problems, andurged the development of evaluation procedures designed toutilize the capabilities of technological advances. Instressing the need for long-range evaluation programs, hepresented four specific components of long-range evaluation.These four components of a program of long-range evaluation,used as a basis for the total training session, are asfollows:

1. Program identification2. Collection of information about the program3. Organization of information about the program4. Analysis and reporting of the findings to the

decision makers.

Dr. Hammond also suggested desirable attributes for a systemof evaluation that would operate on a long-range basis.Such a program would provide:

1. An evaluation system that monitors the total in-structional program--that goes on year after year and pro-vides information to the teacher with respect to changes inthe program.

2. An evaluation system that provides information asto what new programs might be selected.

3. An evaluation program that monitors new programsand allows modification through evaluation of the program,not the students.

4. An evaluation system that gives information aboutthe year-long program.

Dr. Robert Armstrong, director of EPIC, and his staff,introduced the three-dimensional EPIC model for thestructure of evaluation and demonstrated how the structure

10

provides for the identification of variables in the cate-gories of behavior, instruction, and institution. (Thescheme is shown in detail and in diagram form in AppendixI, "Skill Development in Evaluation for Teachers of English.")After the presentation of the general scheme, participants,working in small groups, each with a leader from EPIC, andfollowing the plan for developing objectives, actuallypracticed the writing of specific, measurable objectives inboth cognitive and affective domains. Three small-groupsessions were devoted to these activities; in the generalevaluations made by individuals, these activities were citedas some of the most meaningful and practical experiences ofthe entire training program.

Additional sessions were devoted to the techniques ofconstructing test items to measure the objectives identifiedin the planning stages of curriculum. The role of validityand reliability in measurement was emphasized. Samples oftables of specifications and item banks were introduced (SeeAppendix III).

Models of test items of various kinds were presentedand discussed. Sample test questions measuring stated ob-jectives were prepared in practice sessions. Following thepractice sessions in the writing of test questions and ex-amples of objectives, a session was devoted to the organi-zation of information. This included the use of codingsystems and computers. Additional information regarding theanalysis and reporting of information to decision makers andthrough the proper channels was supplied.

Application to English Programs

At the conclusion of the three and one half days ofstudy of techniques of evaluation, the second phase of thetraining session began with the application of the model tothe English discipline. Dr. Kellogg Hunt discussed researchin how students write and presented specific techniques thatcould be applied to the evaluation of children's writing.These techniques included the use of the Cloze test, methodsof sentence analysis by means of the t-unit and clauselength, and imbedding or deep structure. In later sessionshe demonstrated the application of systems of analysis basedon syntax and transformational generative grammar, to facili-tate objective measurements. Measurement of systacticalgrowth in writing was illustrated.

The second instructor in the direct application ofspecific objectives to the teaching and measurement ofEnglish curricula was Dr. Alan Purves. Using the teaching

11

of literature as a basis for his discussion, this consultantled the participants through the actual performance of read-ing poetry and a short story. At first, participants playedthe role of students and were instructed in analysis and in-terpretation. In later periods, participants devised teach-ing questions usable with literature. Evaluation techniqueswere applied to teaching literature in a practical presen-tation. A "grid" in indicated areas to be measured was com-parable to the EPIC model and might be used to focus onspecific elements in defining further cube elements.

Having developed skills in the construction of ob-jectives and evaluation instruments for measuring the ac-complishment of these objectives, and having covered evalu-ation in the teaching of composition, language, andliterature, the participants were prepared for the nextphase--the application of this knowledge to specific cur-ricula in specific schools. Since this group is expected toserve schools other than their own, it was logical that allof the skills learned at this training program needed to becombined into a system of procedures that might be appliedto specific situations. The participants must not only beable to evaluate programs, but they must also know how toinstruct others and aid in their evaluation. The appli-cation of the learnings was the subject of the portion ofthe program designed to get the participants started ontheir own. At this point, Mr. William Scannell helped tosynthesize the heretofore disparate elements into a schemefor looking at specific English programs. Participants mettogether in small groups to establish procedures for imple-menting the program in their own geographic areas and to ar-range for cooperative work.

One additional training session was conducted on theteaching of poetry, the writing of objectives to cover theteaching of poetry, and the development of instruments forevaluation of specific goals in the teaching of poetry. Forthis presentation, Dr. Stephen Dunning used his own vastknowledge and interest in poetry to spark the interest ofparticipants in the application of the evaluative procedurespreviously experienced to a simulated teaching situation.

Activities

During the training session, many different activitieswere engaged in by the participants and the staff personnel.Most new information was presented in lecture to the totalgroup with the aid of numerous transparencies, programmedbooks, charts, and duplicated materials. For intensivelaboratory sessions on actual development of representative

12

materials for evaluation, small groups of 10-12 were used.Each group worked with a trained staff member. Some "home-work" assignments were produced as additional samples whichwere then evaluated by the staff. Many viewing sessionswere held, some planned as part of the activities, some asan independent kind of sharing activity. Several partici-pants brought examples of teacher-made and student-made ma-terials; viewing areas were arranged for showing these ma-terials. Each evening the "MESSAGE Media Theatre" showed avariety of films which might be useful background for Englishteachers and curriculum leaders. The British BroadcastingCompany films on improvised drama, for example, gave manyparticipants a first opportunity to observe the British ap-proach to oral English. The films from the National Councilof Teachers of English were useful both as an indication ofthe kinds of materials that might be obtained for local useand for the content of the films themselves. The humanitiesfilms, are, of course, familiar to many English teachers, butmost participants had not had an opportunity to see all ofthem, especially the most recent ones (Appendix IV) .

Committees were established early and meetings wereheld frequently in order to coordinate reporting to ad-ministrators and to the membership of the MCTE, to developevaluation forms for the sessions themselves, and to planprocedures for follow-up. Every participant was engaged insome reporting presentation. Besides helping the individualto crystallize his thought and synthesize the various ex-periences of the training session into a meaningful whole,these reporting sessions also gave practice in making presen-tations to groups just as each will be doing as he helps todisseminate information.

Availability of Resource Personnel

One distinct advantage of the manner in which thissession was conducted was the constant availability of thestaff members, for both formal and informal contacts. Dr.Hammond, Dr. Armstrong, and the EPIC staff were availablefor three and one-half days; Dr. Hunt was available forthree days, Dr. Purves for one day, and Dr. Dunning for twodays. Mr. Scannell was present for the full training period,as were the MCTE personnel: Miss Mildred Webster, MissThelma McAndless, Mrs. Helen Wagner, and Dr. Taylor. OtherMCTE executive board members were present for varying amountsof time. The availability of the staff and consultants gavemany times for questions, discussion, and, in some cases,differences of opinion, and participants used their availa-bility during meals, over coffee, and in impromptu recre-ational activities. Another bonus of the living-learning

13

situation of the training sessions was the opportunity forconstant interaction of participants. In their personalevaluations, most trainees mentioned the value of these con-tacts.

Follow -up Activities

Report to Administrators and to MCTE

In setting the Program Calendar for the MESSAGE train-ing program, the director included two sessions at which par-ticipants would report first to their administrators andsecond to the several hundred English teachers attending theMichigan Council of Teachers of English annual conference atBoyne Mountain, April 25-27.

Both presentations were planned by committees of par-ticipants. They appeared before the two audiences (1) toshare what they had been learning about effective ways toevaluate instructional programs, and (2) to show howknowledge, skills, and attitude in the behavioral domainscould effect improvement in teaching the discipline ofEnglish.

The audience of the administrative conference was com-posed of invited administrators, who had expressed aninterest in having a staff member participate in the train-ing program, or their representatives. The responses ofthese administrators indicated that they had gained new in-sights into the need for a systematic approach to evaluationand the part that the writing of behavioral objectives mayhave in program evaluation. Several indicated that theyplanned to start work immediately with participants in thetraining program to set up local studies.

The second general session of the annual meeting of theMCTE was devoted to the MESSAGE training program. In thepresentation the committee used duplicate overhead projectorsto make well-prepared and legible overlays visible in allparts of the large conference room. In the session, themain purpose of which was to generate enthusiasm for theproject and give basic understanding to many people of theimplications of long-range evaluation for English programs,the speakers introduced the EPIC model, gave a brief oppor-tunity for audience participation in building an objectivethat would incorporate the three dimensions plus that ofmeasurement, and indicated ways in which a local schoolmight begin to study the area of evaluation.

14

Evaluation

The evaluation planned for the MESSAGE program is to beof two types: immediate and long-range. The immediateevaluation was planned to cover two types of responses. Apre- and post-test was. administered by the EPIC team over theobjectives presented in the first part of the session. Re-sults, shown in Appendix V, are significant for each part ofthe instrument.

The two Michigan Council of Teachers of English evalu-ators, Mrs. Helen Wagner and Miss Thelma McAndless, preparedand administered a questionnaire, asking for response innine areas. The following quotations, chosen by the evalu-ators as presenting the complete range of responses to eachsection of the instrument, give an indication of the af-fective results of the session, as they are immediately ap-parent. (Procedures for gathering information about long-range results are discussed in the next segment.)

The organization of the week-long session

The careful planning of Project message was evidentfrom the first encounter with the week-long program. Lo-cated in a spot conducive to quiet contemplation, the fortyparticipants were introduced to a leisurely, yet firm de-velopment in the area of curriculum development, implemen-tation, and evaluation. It took about a day and a half forinitial frustrations to wear themselves out of teachers whohad left behind hectic schedules. But once they were re-laxed, it was evident that they were settled down to somereal thinking.

Advance knowledge of activities would have been helpfuland would have enabled me to do some background work beforecoming.

Variety in design and shifting focus on a range of con-cepts provided a tempo that established and maintained asense of involvement.

I think the week-long session has been well organized --not only have the speakers been informative, but there hasbeen time to relax and assimilate the ideas presented. Also,there has been opportunity for professional reading -- a realluxury!

The Message Program provided for a good balance of timebetween large-group instruction and small-group instruction.

15

The participants seem well chosen for ability, interest andconcern.

Although the organization was apparent, a pre-conferenceagenda outlining the objectives of the conference, identity-ing topics to be covered, and a schedule of activities wouldhave been most helpful.

A most valuable part of the week has been the time leftsomewhat free for exchange of ideas among participants. Somuch has been given to us that we have needed time to chewupon and digest, with others, the ideas we have dealt with.

.129.12 consultative help available

Completely satisfactory.

Consultants and participants eagerly, helpfully, andwillingly exchanged ideas all in the interest of and for theimprovement of the evaluation process. This was a thrillingexperience. To corner so much sincere dedication, enthusi-asm, and know-how is the epitome of achievement for Dr.Taylor and her committee.

The consultants couldn't have been better and the re-source personnel have been most eager to listen and help.

The consultants not only represented new and excitingideas in a number of fields but also communicated their in-tent with clarity and warmth.

EPIC section too much in too short a time--could havegone two more days. EPIC men very knowledgeable and helpful.

In a few days time with national experts in researchmethods to guide us, I believe all of us here are now ableto write our own program objectives with reasonably adequatemeasurement devices for reliability and validity, since theworkshop required us to demonstrate this ability.

The accessibility of resource personnel

The EPIC team had a tendency to stick together. Live-in consultants prompted and stimulated further development.Ideas flourished at every dining table, lounge chair, andcottage room.

All resource people were available; however, the use ofthese people might have been better planned. Evening

16

seminars could have been organized for the purpose ofsharing such things as new programs, successful old pro-grams, techniques being implemented, and problemscommon to the participants involved in the project. Apresentation and evaluation of the materials on displaywould have been more beneficial than just putting themon display.

Everyone connected with project Message seemedanxious to offer his services and knowledge, and to myknowledge was ready and waiting to help.

Good - consultants came here prepared to respondto questions and establish continuous communicationwith and among participants,

The interchange of ideas among _participants

Persons responsible for the selection of people forthis group are to be complimented. It was a dynamic col-lection of individuals willing and eager to share theirthoughts and experiences.

There was a fair amount of interchange of ideas goingon among participants, though after the first few days itseemed that many tried to avoid talking about curriculumduring informal chats, perhaps because of "overexposure."

It has been good, too, sharing at table, walking to-gether, gathering in small groups, with other people seekingsome of the same answers you are, learning what has workedfor them, contributing what you have done. The pace hasbeen about right for absorbing without tiring.

The interchange of ideas among participants has beenthe most valuable part of this program for me. I have neverbefore known such an opportunity to learn about the teachingof English.

The relevancy of content to needs of teachers and theirinstitutions

Some of the content was too time consuming for the busyclassroom teachers. The Cloze test from Dr. Hunt's presen-tation seems just the quick test of readability that weneed.

This is one study program from which I shall return homeconfident I have some workable knowledge which will be

17

quietly earth-shaking but will not cause great tremors tothe point of creating fear and hysteria.

Whether we adapt, adopt or discard, MESSAGE has pro-vided some guide lines for the decision-making which musttake place in curriculum development. More important, per-haps, than the specific messages is the attitude we've de-veloped: commitment to a specific program is possible.

Perhaps the most important message was the writing ofbehavioral objectives which included the method of measure-ment. The T-unit and the Cloze test offer some immediatepractical assessment of texts and student writing.

Everything has been on a Paul Bunyan scale in both ex-cellence and quantity: the food, the scenery, the friendli-ness of participants and staff and the ideas presented.

The relevancy of content proved itself against a highdegree of skepticism beforehand. The leaders helped and al-lowed participants to make the content meaningful.

The possibilities for implementation on local levels

The principles of evaluation learned here will probablybe implemented first in the classrooms of the participantsthen in other classrooms in the school, and on the otherschools in the system. This should help to up-date Englishprograms and, by means of proper evaluation, put them on asolid foundation.

In implementing these evaluations personal appearancesmight be more dynamic than just printed material. How aboutappearances in English department meetings in the highschools in the county -- as well as those in the city. Howabout going to junior high schools and elementary schools?

This program will provide a solid beginning for thestart of our evaluation of our new English program whichgoes into effect in September. I must say, though, that thefollow-up procedures of measurement were barely touched uponin these sessions and many questions remain unanswered abouttesting and measuring effectively the objectives of ourprogram.

If a list of the participants was made available toeach school along with a letter indicating that we are avail-able to discuss this program with any interested administra-tor or school staff, this study might have an impact onEnglish in Michigan by helping schools to become more

18

efficient and effective in gauging the value and effect oftheir programs.

Local implementation will be in direct proportion tothe degree of influence (and naked power) the MESSAGE par-ticipant has within his own system.

A close relationship should be established with eachtrainee by the director of the project. If this is done,there will be less opportunity for some of the members whoare experiencing difficulty to slip off and become lost tothe group.

Dr. Taylor's suggestions of area grouping led us to dis-covery of method of implementation. We are now firmly setin a path which can be anything we want it to be if we workfor it.

Implementation may be hampered because teachers whuhave not had the experience of MESSAGE will not have a totalcommitment.

Implementation is going to be difficult. I feel theonly way to do this is to start small. There will not bethe spread of MESSAGE that the directors hope for, but itwill be forthcoming in time.

The probable effects of MESSAGE upon the teaching_of Englishin Michigan

If behavioral objectives become a way of life in cur-riculum development, MESSAGE could become the miracle of thecentury.

The probable effects should not be just limited toEnglish but should involve all disciplines. If the EPICprinciples are implemented, a revolution in the discipline(teaching) could take place.

Long range effect will be felt in strengthened curriculaand less frustration in the area of curricula development be-cause forty teachers know a little more about a lot ofthings.

English teaching in the state can only profit from aproject such as this. It will have made the teachers moreaware of what others are doing, of what they might be doing,and, hopefully, of how they might go about effecting neces-sary changes in existing programs.

19

If we truly hope to affect state-wide English teaching,the next step may be intensive training of teams to work indifferent schools districts. Our MCTE regional districtsmight also be a possible organizational route.

The effects of MESSAGE may not be immediately apparent,but I am sure that the impact will be felt in the schoolsrepresented and hopefully will be talked about enough sothat other schools may seek similar help. Those people whowork on North Central evaluating teams should be able tomake other schools aware of MESSAGE concelts.

Superior-- has all kinds of possibilities.

The effects of this conference upon the teaching ofEnglish in Michigan will be dependent upon the enthusiasm ofmembers of this group and their ability to communicate withtheir administrators and their own local staffs. It seemsevident that this kind of improvement is not for Englishalong but has a necessary relationship with every otherdiscipline in the school.

The effects lie in a well-organized follow -up programand interest and encouragement from local school districts.

I believe MESSAGE will have an impact on the teachingof English in Michigan because:

the participants will be better teachers, more aware ofwhat they are doing anyway.

the participants as a group in MCTE can help MCTE havea continuing impact.

leadership and support from MCTE and NCTE can reinforceand extend the seeds planted here.

Specific recommendations as to follow -up on this the firstworkshop of this type

Have a second workshop to tabulate results in the field.Then have a third workshop for continued enlightenment.

Have this group intercommunicate; perhaps work in teamsof 2 or more to interpret MESSAGE to departments, admini-strators, perhaps school boards and community groups.

Plan a follow-up meeting in each of several geographi-cal areas, possibly next fall to monitor action that hasbeen an outgrowth of the conference.

Have at least two follow-up surveys and a presentationat next years MCTE.

20

Perhaps some regional meetings could be scheduled towrite out the nature of our successes as well as our ob-stacles or failures and what we consider to be the reasonsfor these various outcomes. The questionnaire which thefollow-up group is sending out may give some hint as to howgeneral our particular experiences have been and help us todraw up recommendations for future use.

Realizing that one of the values of this conference hasbeen the interchange of ideas, MESSAGE, or a like organi-zation, become an agency for gathering and disseminating ofinformation regarding English programs within the state ofMichigan. Funding for such an agency could be provided bylocal school districts making use of the services providedof a Title III project could be written.

Plan a second workshop for the 1969-70 school year,implementing a team approach. Selected school districtswould identify a team consisting of an English teacher, thedepartment chairman, an administrator, and a testing andmeasurement person to work together in a pilot program forthe development of an evaluation scheme.

Follow-ups are difficult to maintain, but we must meetagain next year to find out how we carried the MESSAGE.

How can we encourage future sessions?

Apply for more federal money for follow-up workshops.Secure additional financial support in order to involve moreteachers.

Use literature sent to members of MCTE to supply infor-mation about these meetings at Boyne.

Plan sessions to explore measurement - it was barelytouched upon.

Involve participants in planning for future sessionsthrough development of surveys, questionnaires, inventories.

Report this meeting in the Eng. Journal and othermedia. Let the school administration know who the speakersand consultants were and the implications of direct help be-ing given to those from their staff who attended.

Improve communications concerning the activities viapersonal contact, publicity. Giant strides have obviouslybeen made in the proper directions.

I felt that for many, it was "too much, too fast," andthat the relationships among the "Phases" of the conference

21

were not always clearly drawn. Plan for a slower pace forfuture sessions.

The state department of education representative under-stands the project and its implications for the improvementof teaching of English in Michigan. Obtain money from' thatagency -- by fall. Under the tension of producing, specificchanges could come about. If future sessions are plannedearly enough, local areas could earmark some monies for suchconferences.

General

This has been the most stimulating professional week Ihave ever spent.

I am thankful for those who had the vision and thedrive to organize this week's workshop, and for the oppor-tunity to be a participant.

Long-range Evaluation

Written into the MESSAGE Proposal were provisions forfollow-up. At the Boyne Mountain training session the pro-ject participants attacked the ways and means for carryingout the proposals. Headed by Dr. Robert Fichtenau, Directorof English, Intermediate District, Oakland Schools, Pontiac,a follow-up committee divided the areas of responsibility ona three-way basis:

I Distribution and codifying of a 5-part survey designed byMESSAGE staff in consultation with the Division of Syste-matic Studies of Oakland Schools

A. Purpose of Survey - Assessment of behavioral changesin MESSAGE Participants and schools represented

B. Description of the Survey instrument (see TentativeQuestionnaire, Appendix VII)

1. Part one to find how differently the participantsfeel as a result of Project MESSAGE

2. Part two to assess changes in behavior on thepart of the individual participants

3. Part three to gather information about specificchanges in evaluation techniques being used inclassrooms of the participants' schools and inschools in which the participants work

22

4. Part four to survey attitudes in the participants'own schools and the administrators' attitudes to-ward utilizing the special knowledge and skillsof the Project participant

5. Part five to survey the extent to which infor-mation about Project MESSAGE has been disseminated

C. Survey Schedule (September of 1969 and June 1970)

D. Administrators' Questionnaire

II Cooperation with the Michigan Department of Education(The following tentative commitment for assistance hasbeen obtained from the Department of Education)

A. Provide consultant service to those districts whichwish to develop objectives and evaluation devices fortheir English programs

B. Support additional workshops, on a limited scale, toenable English teachers to develop program objectivesand evaluative techniques

III Continuing Cooperation with MCTE

A. At each of the Fall (1969) Regional Conferences ofMCTE devote program time to "What's Happening inRegion " as a result of Project MESSAGE

B. At the annual spring meeting of MCTE (1970) schedulea session - Project MESSAGE - One Year Later. Thecommittee planning this program will communicate witheach of the MESSAGE participants on a scheduled basisthroughout the school year.

23

APPENDIX I

EPIC OBJECTIVES

FOR

PROJECT MESSAGE TRAINING SESSION

Boyne Mountain Lodge

April 18-21, 1969

Participants attending the Project Message/EPIC train-ing sessions, April 18-21, 1969, will:

1. increase their knowledge of the EPIC Scheme forEvaluation as measured by their scores on the EPICFeedback Questionnaire.

2. increase their knowledge of program objectivesterminology as measured by the EPIC FeedbackQuestionnaire and EIP tests.

3. be able to apply their understanding of program ob-jectives, by writing an objective of their choicethat includes the four EPIC criteria for objectives.

4. will respond positively to the EPIC TrainingSessions and staff as demonstrated by their re-sponses on the EPIC Feedback Questionnaire.

24

SKILL DEVELOPMENT IN EVALUATION FOR THE

MICHIGAN COUNCIL OF TEACHERS OF ENGLISH

EPIC Evaluation Center1034 East AdamsTucson, Arizona

INTRODUCTION

The nature of English programs will be quite variedfrom program to program; however, this does not alter thefact that a systematic scheme for evaluation is an absolutenecessity. The instruments and techniques employed in theactual evaluation of each program may be different, but thestep-by-step process should be consistent, or at least well-defined, for each program. This step-by-step process mustbe very understandable for personnel implementing Englishprograms, particularly if such personnel have had limitedtraining and experience in the area of evaluation. Thispaper presents a systematic approach to evaluation, theskills required of program personnel for its implementation,the technical assistance required to implement the system,and resources of the EPIC Evaluation Center available to pro-vide this training.

I. pescription of EPIC and its Funding Agency (Pima CountySuperintendents Office, 112 West Pennington, Tucson,Arizona)

The EPIC Evaluation Center, established July 1, 1967,is supported by the U.S. Office of Education under the Ele-mentary and Secondary Education Act of 1965. The primarypurpose of the Center has been that of developing and imple-menting a systematic scheme for the evaluation of instruction-al programs. Simultaneous with the development and implemen-tation of the EPIC approach, skill training workshops havebeen instituted by the Center as a part of its on-goingresponsibilities.

The EPIC Evaluation Center employs twenty-one staffmembers. These professionals each represent one or morespecialties in the total evaluation process as described byEPIC. The unique characteristic of the EPIC EvaluationCenter thus lies in its ability to facilitate the implemen-tation of the total evaluation process.

25

II. Evaluation A roach to be Presented in the Worksho

The EPIC Structure and Scheme provide for the implemen-tation of procedures that lead to "objective evaluation."There are two essential differences between objective evalu-ation and the other less valid forms, such as subjectiveevaluation and external evaluation.

The first difference is that of objectives. If a per-son is to make a valid decision based on the informationcollected in evaluation, he must know what he is evaluating.This is accomplished only when he is able to state his ob-jectives in behavioral terms. These objectives determinethe procedures and criteria to be used in evaluation. Thesecond difference is an outgrowth of the first. Objectiveevaluation guarantees that the people responsible fordecision-making will become involved in the evaluation be-cause they are the ones who have to write the behavioral ob-jectives to be evaluated.

The EPIC Structure of Variables Influencing In-structional Programs (Figure 1) facilitates the identif i-cation and description of the institutional and instruction-al variables affecting the program. Combining these vari-ables with the desired behavioral outcomes (cognitive,affective, or psychomotor) presents a simple, yet effective,method of developing program objectives.

Another important feature of the EPIC Scheme for Evalu-ation (Figure 2) is that it facilitates not only "product"but also "process" evaluation. That is to say, consider-ation is given to the need for continuous feedback to enableprogram personnel to evaluate the progress of a program andmake the necessary decisions to alter the program during itsimplementation, and not have to wait until the program isculminated.

Basically, the procedures developed by EPIC demon-strate the philosophy that effective "objective evaluation"must be a product of the local district, agency, or project.Each school, agency, project, etc. must decide the nature ofthe objectives to be met. EPIC then provides the technicalassistance, hardware, and in-service training programs be-yond the resources of the program to enable the personnel toimplement evaluation.

III. Evaluation of Enlish Pro. rams--General Comments

The degree to which English programs meet their antici-pated outcomes must be determined through sound evaluationprocedures implemented to measure specific objectives. This

26

SKILL DEVELOPMENTIN EVALUATION

Page 3

Imo

.44. Psychomotor DomainAffective Domain /

Cognitive Domain

Organization

Content

Method

Facilities

Cost

fiv iki° 0bej 1/4,47Y .1.4ZP

4;0' 20C) e$C)(se) 4?) coq

ofi1)N

ot"

OINSTITUTION

FIGURE 1

EPIC STRUCTUREOF VARIABLES

INFLUENCINGINSTRUCTIONAL

PROGRAIIIS.

27

EPIC SCHEME FOR EVALUATION OF INNOVATIONS

PLANNED PROGRAM

r 1IDENTIFY DESCRIPTIVE VARIABLES

BEHAVIOR. INSTRUCTION INSTITUTION

Cognitive Variable Organization ChildContent Teacher

Affective Variable Method AdministratorFacilities Specialist

Psychomotor Variable Cost FamilyCommunity

DEVELOP GOALS AND OBJECTIVES

STRUCTURE FOR EVALUATION

EVALUATION

FACTORS

DESIGN

PROGRAM CA LENDAR

PRODUCT

ACTUAL PROGRAMPROCESS

OUTPUTANALYSIS OF TERMINAL BEHAVIOR

RECYCLING PROCESS

28

III. Evaluation of English ProgramsGeneral Comments(Continued]

implies that prior to the initiation of each program, cer-tain steps in the evaluation process must be clearly defined.This would include adequate program description, objectivesstated in behavioral terms, and an evaluation design whichspells out the data to be collected, treatment descriptions,statistical analysis, and limitations of the evaluation pro-cedures. If we can accept the assumption that evaluation issimply a procedure for collecting pertinent information thatcan be used to facilitate decision-making, such objectiveevaluation seems to be valid.

The evaluation procedure employed by each program mustprovide for the recognition of the many variables that mayaffect the given program, but most important, provision forthe identification of outcomes in terms of cognitive, af-fective, and/or psychomotor behavior should be made - -in otherwords, a means of identifying and classifying behavioral out-comes through the use of some organized system, such as theTaxonomy of Educational Objectives (Bloom). In the trainingof personnel in the procedures of evaluation, it is impera-tive that:

1. the personnel responsible for evaluation withineach program be exposed to every phase of theevaluation process, and

2. the personnel responsible for evaluation withineach program develop the skills which they mustultimately employ in developing and implementingsystematic evaluation.

IV. The Evaluation Process and Related Skills

The approach in providing experiences that not only ex-pose the participants to the evaluation process, but also in-creases their skills in handling various aspects of theevaluation process, should include the following:

A.. Sequential Activities in B. Related Skill Developmentthe Evaluation Process and Orientation, and EPIC

Resources

1. Program Description

Sound evaluation pro-cedures .require that theprocess begin with a de-scription of the planned

29

1. Emphasis must be onthe development of skillsin identifying those vari-ables that are felt to beaffecting that portion ofthe instructional program

IV. The Evaluation Process and Related Skills (Continued)

program. The key to ef-fective evaluation is toidentify and select whatis to be evaluated.

2. Goals and Objectives

Evaluation as a pro-cess is best approachedthrough objectives statedin behavioral terms. Theobjectives of a given pro-gram are developed follow-ing the identification ofvariables.

under consideration. Todevelop these skills, asystematic, yet simpleapproach must be pro-vided to the programpersonnel.

The EPIC Structureof Variables InfluencingInstructional Programsis a three-dimensionalstructure which facili-tates the process ofidentifying and describ-ing the variables notonly within the edu-cational setting, but al-so outside of it. TheStructure considers thedimensions of instruction(organization, content,method, facilities, andcost), institution (stu-dent, teacher, admini-strator, educational spe-cialist, family, and com-munity) and behavior(cognitive, affective,and psychomotor).

2. Skill sessions shouldbe provided to programparticipants in the. de-velopment and writing.2ofbehavioral objectives.

The staff of theEPIC Evaluation Centerhas available program ma-terials in writing be-havioral objectives.These materials have beenespecially designed totrain educators who havehad little or no experi-ence in this area. TheEPIC Structure of Vari-ables Influencing In-structional Programs isutilized directly in

30

IV. The Evaluation Process and Related Skills (Continued"

3. Development of Evalu-ation Designs andFunctions of DataProcessing

Once the program ob-jectives have been stated,an evaluation design canbe developed. This designis basically a descriptionof how the data is to becollected and analyzed inorder to determine if theobjectives have been met.

To facilitate theevaluation process, moreoften than not, programpersonnel must avail them-selves of data processing.

31

teaching the skills ofbehavioral objectivewriting.

3. Program personnelmust have some orienta-tion to the cons ider-ations inherent in es-tablishing evaluationprocedures. Assistanceprovided during trainingsessions should be di-rected toward each pro-gram. Participantsshould be exposed to suchconsiderations as samplesize, treatment descrip-tions, statistical an-alysis, and limitations.(note the words exposedand assistance) Programpersonnel must be as-sisted in the developmentof "program calendars."A Program Calendar basi-cally consists of aschedule designating pre-cisely the evaluationprocedures to be employed.

The EPIC EvaluationCenter maintains a com-plete Evaluation Divisionwhich has the primaryfunctions of: (1) pro-viding technical as-sistance for the se-lection of treatments,measurement instruments,evaluation design, andstatistical analyses, (2)processing data, and (3)reporting the results ofthe evaluation process.

Program personnelmust receive instructionin the data processingfunction in evaluation.The intent is not to

IV. The Evaluation Process and Related Skills (Continued)

4. Description of ActualProgram

During the process ofevaluation, an on-going de-scription of what is actual-ly happening in the programis a necessity. The actualprogram description shouldprovide information as towhether the program is beingcarried out as planned.

32

provide them with me-chanical skills relatedto data processing, butto provide them with away of communicatingwith the evaluator andspecific orientation inthe most efficientmethods of collecting andanalyzing data. Exposureto such topics as theprincipal functions ofdata processing, storageof data, cost estimates,classifying data, andpreparing data for com-puter processing areintended.

The EPIC EvaluationCenter has, as a part ofits facility, the dataprocessing equipment andtechnical personnel todescribe and demonstratethe complete process ofinput, processing, andoutput.

4. Program implementorsneed training and as-sistance in the points tobe considered for con-tinual monitoring of theinstructional program.Such skill would enablethem to identify crucialpoints of reference inthe program's progressand the identificationof variations from theplanned program. Suchfeedback would enablethem to: (1) make thenecessary changes inevaluative procedures(2) compare the evalu-ation results with spe-cific techniques of in-struction during the

IV. The Evaluation Process and Related Skills (Continued)

5. Output Analysis ofTerminal Behavior

Following the comple-tion of the program, thenext step is that ofanalyzing the results ofthe evaluation to arriveat conclusions based onactual behavior.

6. Etesys12.in Process

With the programevaluated,. the school is,ready to consider changesin the program. Theevaluator and the imple-mentor come together. De-cisions must be made rela-tive to program develop-ment.

33

program, and (3) provideinsights which could per-mit the incorporation ofnew content and proceduresas the program progresses.

5. Program personnelmust have-actual experi-ence in the analyzing ofevaluation results as apart of their training.Such exposure would pro-vide them at least alimited skill in interpre-tation of findings.

The EPIC EvaluationCenter has, as one ofits major responsibili-ties, that of assistingprogram personnel in theinterpretation of evalu-ation results. The pro-gram personnel are goingto be making the decisionto change the directionof their respective pro-grams; therefore, theymust have the technicalassistance in the in-terpretation of the re-sults of evaluation.

6. Through the processof evaluation described,the recycling process en-tails the considerationof: (1) those objective '5not met through actualprogram activities, (2)

new or additional ob-jectives which mightfur ;.her improve the pro-gram, and (3) futureevaluation procedures.The recycling process de-mands both the knowledgeof the program on thepart of the implementorand the technical skillsof the evaluator.

IV. The Evaluation Process and Related Skills Continued

V. Workshop Leaders

Dr. Robert J. ArmstrongAssistant ProfessorCollege of EducationThe University of Arizona(Director, EPIC EvaluationCenter)

Dr. Robert E. KranerAssociate ProfessorCollege of EducationThe University of Arizona(Assistant Director, EPICEvaluation Center)

Special Consultants

Dr. Robert L. HammondAssociate ProfessorSchool of EducationOhio State University(Ohio State EvaluationCenter)

Dr. Terry D. CornellAssistant ProfessorCollege of EducationThe University of :Arizona

(Coordinator, Evalu-ation Services EPICEvaluation Center)

Dr. E. Wayne RobertsonAssistant ProfessorCollege of EducationThe University of

Arizona(Coordinator, FieldServices EPIC Evalu-ation Center)

PROJECT MESSAGE

Michigan Council of the Teachers of English

WORKSHOP SCHEDULE

(Participation by representatives of theEPIC Evaluation Center)

FRIDAY, APRIL 18, 1969

Morning General Session #1

"Introduction to the EPIC Evaluation Center andWorkshop Expectations"

Dr. Robert Armstrong

1:30 - 3:00 p.m. Small Group Session #1

"A Structure and Scheme for the Evaluationof Instructional Programs"

3:00 - 3:20 p.m. Coffee Break

3:30 - 4:30 p.m. Small Group Session #1 (Continued)

"Developing Program Calendars andProgram Monitoring Systems"

SATURDAY, APRIL 19z 1969

9:00 - 9:20 a.m. General Session #2

"Introduction to Behavioral Objectives"

[ 9:30 - 10:15 a.m.

Dr. E. Wayne Roberson

Small Group Session #21

"Developing and Writing Behavioral Objectives"

10:15 - 10:35 a.m. Coffee Break

35

Coffee Break

Coffee Break]

SATURDAY APRIL 19, 1969 Continued)

10:35 - 11:45 a.m. (Small Group Session #2 (Continued)

"Developing and Writing Behavioral Objectives"

1:30 - 3:00 p.m. Small Group Session #71

"Developing and Writing Behavioral Objectives"

3:00 - 3:20 p.m.

3:20 - 4:30 p.m. Small Group Session #3 (Continued)

"Critiquing Behavioral Objectives"

SUNDAY APRIL 20 1969

2:00 - 3:15 p.m. General Session #3

"Measuring Cognitive Behavior"

Dr. Terry D. Cornell

3:15 - 3:30 p.m.

3:30 - 4:45 p.m.-1

General Session #3 (Continued)

"Measuring Affective Behavior"

Dr. Robert E. Kraner

MONDAY, APRIL 21, 1969

9:00 - 10:45 a.m. Small Group Session #4

"Implementation of the EPIC Structure andScheme for Evaluation"

10:45 - 11:00 a.m. Coffee Break

11:00 - 11:45 a.m. General Session #5

"Final Summary Session"

EPIC Staff

37

Mr. Henry James BaronCalvin College1331 FranklinGrand Rapids, Mich. 49506

Mr. Theodore D. BarrisDouglas MacArthur High School505 N. Center Rd.Saginaw, Mich. 48603

Mrs. Donalda J. BenoitMuskegon Public Schools349 W. Webster Ave.Muskegon, Michigan 49440

Mrs. Gladys BowerWayne Memorial High School3001 Fourth StreetWayne, Michigan 48184

Mrs. Jean BradleyBig Rapids High School500 N. WarrenBig Rapids, Mich. 49307

Mrs. Elizabeth CalkinsNorthwestern High School2138 Carpenter Rd.Flint, Mich. 48505

Mrs. Carolyn CallendarNorthwest High School4200 Van Horn Rd.Jackson, Michigan

Mrs. Mary ChurchillBoyne City High SchoolBoyne City, Michigan 49712

Mr. Russell ConwayPetoskey Sneior High SchoolPetoskey, Michigan 49770

Mr. Neil B. DavisCheboygan High SchoolCheboygan Michigan 49721

APPENDIX II

PARTICIPANTS

PROJECT MESSAGE

Home: 1040 Alexander, S.E. Instructor inGrand Rapids, Mich. 49507 English and

English Education

2665 Cheyenne P1.Saginaw, Mich. 48603

2481 Wickham DriveMuskegon, Michigan 49441

35203 Ash StreetWayne, Michigan 48184

20609 Edgewood, Rt. 3Big Rapids, Mich. 49307

2721 HillcrestFlint, Michigan 48507

2024 Cascades DriveJackson, Michigan 49203

607 South Lake StreetBoyne City, Michigan

617 Harvey StreetPetoskey, Michigan 49770

Rt. 3, Box 163ACheboygan, Michigan 49721

38

Head, EnglishDepartment

L. A. Coordina-tor, TeamTeacher

Department Head

Department Head

Teacher

English Coordina-tor, NorthwestSchool District

Chairman,EnglishDepartment

Department Head

English coordina-tor, Teacher

Mrs. Lucille EastwoodAnn Arbor Pioneer H. S.601 West Stadium Blvd.Ann Arbor, Michigan 48103

Mrs. Ellen EvansTawas Area SchoolsTawas City, Michigan

Mrs. Bette J. FerencyAllen Park High School18401 ChampaignAllen Park, Michigan 48101

Dr. Robert L. FichtenauOakland SchoolsCampus DrivePontiac, Michigan 48053

Mr. Lewis FiferSault Ste. Marie High School438 E. Spruce StreetSault Ste. Marie, Michigan

Mrs. Ila FordWest Iron County HighIron River, Michigan 49935

Mrs. Dorothy FordyceBridgeport High SchoolBridgeport, Michigan 48722

Mr. Norman S.' FosterSouth Lake High School22000 Nine Mile RoadSt. Clair Shores, Mich. 48083

Mr. John E. FoxGrosse Pointe North HighGrosse Pointe, Mich. 48236

Sister Maria Julia Hawk S. C.O'Rafferty High School4300 W. SaginawLansing, Michigan 48917

Mr. Charles C. HemingwayFlint Community Schools923 E. Kearsley St.Flint, Michigan 48502

Mrs. Theone HughesEnglish nsiSgrtmentWestern Michigan UniversityKalamazoo, Michigan

2031 Medford RoadAnn Arbor, Michigan 48104

500 W. Bay StreetEast Tawas, Mich. 48739

Chairman,EnglishDepartment

Chairman,EnglishDepartment

20450 Canal Drive Teacher, Chair-Grosse Ile, Michigan 48138 man English Dept.

100 CherokeePontiac, Michigan 48053

360 DawsonSault Ste. Marie, Mich.

Director ofEnglish, Inter-mediate Dist.

H. S. CurriculumCoordinator,Dept. Head

Box 132, R.R. 1 Chairman,Iron River, Michigan 49935 English Dept.

3280 Dixie CourtSaginaw, Michigan 48601

28040 BohnRoseville, Michigan 48066

250 Mt. Vernon Rd.Grosse Pointe Farms, Mich.48236

4350 W. SaginawLansing, Michigan 48917

2236 Holly Tree DriveDavison, Michigan 48423

Route #1, County Rd. 358Paw Paw, Michigan 49079

39

Lang. Arts Dept.Chairman

Chairman, Depart-ment of English

Projects Di-rector/Coordina-tor

English Dept.Chairman

Coordinator,Humanities

College Teacher,English andMethods

Mrs. Julanne IsabelleMarquette Senior High1201 W. Fair Ave.Marquette, Michigan 48955

Mr. James Lee KraaiWest Middle School615 Turner N. W.Grand Rapids, Michigan 49504

Mrs. Reva Mae KristinIthaca High SchoolIthaca, Michigan 48847

Mrs. Clara H. LangEscanaba High School500 S. 23rd StreetEscanaba, Michigan 49829

Sister AlphonsusHoly Redeemer High School5678 EldredDetroit, Michigan 48209

Mr. Rudolph R. LeutzingerRiver Valley High SchoolBox 330Three Oaks, Michigan 49128

Mrs. Elsie LudeOtsego High School540 Washington StreetOtsego, Michigan

Miss Kathleen McDonaldThunder Bay Jr. High400 S. Second Ave.Alpena, Michigan 49707

Mr. Robert B. MacKayFerndale High School881 PinecrestFerndale, Michigan 48220

Miss Ruth L. MartenMidland Public Schools127 TownsendMidland, Michigan 48640

Mrs. Norda J. MassuchMiddle School26th StreetComstock, Michigan 49041

Miss Barbara M. MerrittOttawa Hills High School1050 IroquoisGrand Rapids, Michigan 49506

982 West College Ave.Marquette, Michigan 48955

1143 Thomas, S.E.Grand Rapids, Mich. 49506

125 S. St. Johns StreetIthaca, Michigan 48847

1106 South 15th StreetEscanaba, Michigan 49829

same

Box 337Sawyer, Michigan 49125

687 BenhoyPlainwell, Michigan

1161/2 W. DunbarAlpena, Michigan 49707

10305 OakhurstHolly, Michigan

1305 BooknessMidland, Michigan 48640

5224 FordhamKalamazoo, Michigan

2023 Paris S. E.Grand Rapids, Mich. 49507

40

Teacher, Chmn. K-12 L. A.Committee

Teacher; Com forL.A. Improvement

Teacher

Teacher, Chmn.Eng. Cur. Com.,K-12

Dept. Chmn.,English Com.Chairman forArchdiocese

English Dept.Chairman

Head of EnglishDept.

Department Head

Chairman,EnglishDepartment

Language ArtsCoordinator

Curriculum Dir.,English and Soc.Studies

Teacher, EnglishCur. Com.

IM,V11119.11.4.1,011._

Mr. Michael L. OliverBedford West Jr. HighTemperance, Michigan 48182

Mrs. Sharon OliverTemperance Rd. ElementaryTemperance, Michigan 48182

Mr. John W. OssenmacherHarvey H. Lowrey School6601 JonathonDearborn, Michigan 4.'126

Mrs. Avis A. SmithNorthwestern Junior High476 Limit StreetBattle Creek, Michigan 49017

Mrs. Shirley SpanielPlymouth Community Schools1024 S. Mill StreetPlymouth, Michigan 48170

Miss Isabel StruthersArthur Hill High SchoolSaginaw, Michigan 48602

Mr. Kenneth TaylorHolland High School600 Van RaalteHolland, Michigan 49423

Mrs. Nancy TelferMt. Pleasant High School1144 S. Elizabeth St.Mt. Pleasant, Michigan 48858

Mrs. Rosa L. WinchesterArthur Hill Tech. H. S.512 S. MichiganSaginaw, Michigan 48602

Sister Mary Clare Yates29300 Eleven Mile Rd.Farmington, Michigan 48024

Box 456230 Lewis AvenueToledo, Ohio

Box 456230 Lewis AvenueToledo, Ohio

6856 BinghamDearborn, Michigan

4764 M-66Athens, Michigan

1380 Linden

Teacher, 11thgrade English

3rd GradeTeacher

Teacher andDepartment Chmn.

Teacher

Coordinator ofPlymouth, Michigan 48170 LA

1212 HoughtonSaginaw, Michigan 48602

800 W. 24th

Department Head

Chairman,Holland, Michigan 49423 English K-12

621 E. High StreetMt. Pleasant, Michigan48858

326 South 24thSaginaw, Michigan 48601

same

41

English Dept.Chmn.

Department Head

Chairman, Eng.Dept.

MESSAGE CONSULTANTS AND STAFF

Dr. R. a. ArmstrongEPIC Evaluation CenterTucson, Arizona

Dr. Terry D. CornellEPIC Evaluation CenterTucson, Arizona

Dr. Stephen DunningUniversity of MichiganAnn Arbor, Michigan

Dr. R. L. HammondOhio State UniversityColumbus, Ohio

Dr. Kellogg HuntFlorida State UniversityTallahassee, Florida

Dr. Robert E. KranerEPIC Evaluation CenterTucson, Arizona

Miss Thelma McAndlessEastern Michigan UniversityYpsilanti, Michigan

Dr. Alan C. PurvesUniversity of IllinoisUrbana, Illinois

Dr. E. Wayne RobersonEPIC Evaluation CenterTucson, Arizona

Mr. William ScannelNational Council of Teachers

of EnglishChampaign, Illinois

Mr. Dwight SmithMichigan Department of

EducationLansing, Michigan

Dr. Vi Marie TaylorOkemos, Michigan, Public

Schools

Consultant, Evaluation

Consultant, Evaluation

Consultant, English

Consultant, Evaluation

Consultant, English

Consultant, Evaluation

Evaluator, Michigan Councilof Teachers of English

Consultant, English

Consultant, Evaluation

Observer-Consultant,National Council ofTeachers of English

Consultant, State Depart-ment of Education

Director

42

Mxs. Helen WagnerEast Lansing, Public SchoolsEast Lansing, Michigan

Miss Mildred WebsterSt. Joseph Public SchoolsSt. Joseph, Michigan

Evaluator, Michigan Councilof Teachers of English

Executive Secretary, Michi-gan Council of Teachers ofEnglishFiscal Officer

APPENDIX III

SOME SUGGESTIONS FOR WRITING TRUE-FALSE ITEMS

1. Base true or false items on propositions that are like-ly to be significant and useful in dealing with a widevariety of situations and problems.

2. Express the proposition as simply and as clearly aspossible. Use words whose meanings are definite andprecise.

3. Include enough background and information and qualifi-cations so that the truth or falsity of the statementdoes not depend on some special, uncommon assumption.

4. Make the true statement true enough and the false state-ment false enough so that experts would unanimouslyagree on the answer.

5. Choose and state propositions so that their truth orfalsity is not immediately obvious to every reasonableperson.

6. Let each test item express a single idea. Avoid com-plex statements which unnecessarily combine severalideas. If it is necessary to include explanatory orqualifying elements in the statement, indicate clearlywhich part is to be judged true or false.

7. Do not create false statements from true ones simply byinserting the word "not." The task of judging thetruth or falsity of a statement which says that some-thing is not so can become highly and needlesslycomplex.

8. Word the statements so that sheer memory of words,empty phrases, or meaningless sentences will ,not permita correct answer.

9. Guard against irrelevant clues, such as specific de-terminers, which would permit a test-wise but unpre-pared examinee to respond correctly. Specific de-terminersrefer to sweeping terms like "all," "always,""none," "never," "impossible," "inevitable," etc.Statements including such terms are likely to be false.

44

SOME SUGGESTIONS FOR WRITING TRUE-FALSE ITEMS (Continued)

On the other hand, statements including such qualifi-cations as "usually," "sometimes," "often," etc. arelikely to be true.

FAULTY TRUE-FALSE ITEMS

1. Two pitfalls should be avoided in writing true-falseitems. (F)

2. Camping has a good past, a better present, and an al-most unlimited future. (T)

3. The relation between a parasite and its host issignificant. (T)

4. When you see a highway with a marker which says "IowaU.S. 218" you know that the construction and upkeep ofthat road is built and maintained by the state andFederal governments. (T)

5. Rain clouds are light in weight. (F)

6. Merit is an important factor affecting a teacher'ssalary. (F)

7. Frozen foods of the highest quality may be ruined inthe kitchen. (T)

8. Insurance agencies may be either specialized orgeneral. (T)

9. As a result of mistakes in our foreign policy, the Com-munists won control of China after World War II. (T)

10. Life is a continuous process of choice-making, sacri-ficing one human value for another, which goes throughthe following steps: spontaneous mental selections re-garding everything we want, conflicting preferenceshold each other in check, hesitation becomes deliber-ation as we weigh and compare values, finally a choiceor preference emerges. (T)

11. Columbus did not make four voyages of exploration tothe Western hemisphere. (F)

12. For every action there is an equal and oppositereaction. (T)

45

FAULTY TRUE-FALSE ITEMS (Continued)

13. Objective tests are useful in measuring all aspects ofeducational achievement. (F)

14. Stars send out light that twinkles. (T)

SUGGESTIONS FOR PREPARING GOOD MULTIPLE-CHOICE TEST ITEMS

1. Develop multiple-choice test items on the basis of inde-pendently meaningful and demonstrably valid statementsof relevant, important ideas.

2. Choose item topics and ideas and write multiple-choicetest items with a view to maximizing the discriminatingpower of the items.

3. Arrange to write the initial item drafts so that subse-quent revision and assembly into the finished test willbe as convenient as possible.

4. Begin the item with a stem question or incomplete state-ment to which a reasonably adequate answer or completioncan be given concisely and for which plausible wronganswers can be found.

5. Phrase the intended correct response so that it isthoroughly correct or clearly adequate, expressed asclearly and concisely as possible, without providingclues which give it away to the clever but poorly pre-pared students.

6. Choose and phrase the incorrect alternatives (dis-tractors) so that they are thoroughly wrong or clearlyinadequate, yet plausible enough to appeal to sub-stantial.numbers of poorly prepared examinees.

7. Word the item as clearlyy, simply, and correctly aspossible.

8. Modify the item, if necessary, so that it is reasonableto expect about half of the examinees to answer theitem correctly.

9. Arrange for competent, independent review and revisionof the initial item draft.

ILLUSTRATIONS OF MULTIPLE-CHOICE TEST ITEMS

1. Item permitting a correct answer:

What happened in 1953 to the proposed act of Congressgranting statehood to Hawaii?

a. It was passed by both the House and the Senate.b. It was defeated in both the House and the Senate.

47

ILLUSTRATIONS OF MULTIPLE-CHOICE TEST ITEMS (Continued)

*c. It was passed by the House but not by the Senate.d . No act regarding statehood for Hawaii was intro-

duced in either the House or the Senate.

2. Item using citation of authority:

*

What does Ross say about the Kuder-Richardson methodfor calculating test reliability?

a. It involves assumptions which are likely to bedifficult to meet in the ordinary test situation.

b. It produces coefficients which are higher thanthose obtained by the split-halves procedure.

c. It is the simplest and generally most satisfactorymethod.

d . It takes account of both pupil errors in responseand sampling errors among the items.

3. Item requiring best answer:

Which statement best characterizes the man appointed byPresident Eisenhower to be Chief Justice of the UnitedStates Supreme Court?

a. An associate justice of the Supreme Court who hadonce been a professor of law at Harvard.

*b. A successful governor who had been an unsuccessfulcandidate for the Republican presidentialnomination.

c. A well-known New York attorney who successfullyprosecuted the leaders of the Communist Party inthe United States.

d . A Democratic senator from 'a Southern state who hadsupported' Eisenhower's campaign for the presidency.

4. Item admitting no best answer:

Which event in the following list has been of thegreatest importance in American history?

*a. Braddock's defeatb. Burr's conspiracyc. The Hayes-Tidden contestd . The Webthter-Hayne debate

ILLUSTRATIONS OF MULTIPLE-CHOICE TEST ITEMS (Continued)

5. Item dealing with an incidental detail:

This question is based on the advertising campaign ofMaumkeag Mills to retain the market leadership of Pequotbed linen. What was the competitive position of Pequotproducts in 1927?

a. Ahead of all competitors among all customers.*b. Strong with institutional buyers but weak with

household consumers.c. Second only to Wamsutta among all customers.e. Weak with all groups of consumers.

6. Item based on unique organization of subect matter:

The second principle of education is that the individual:

a. Gathers knowledge.b. Makes mistakes.c. Responds to situations.

*d. Resents domination.

7. Item requiring selective recall:

Which of the following was an important development inCanada during 1953?

a. Rapid business and industrial growth.b. A severe and widdspread economic depression.c. A marked trend toward Communism in provincial

governments.d. Appearance of a strong movement favoring unifi-

cation of the United States and Canada.

8. Item using descriptive responses:

What

a.

is monogamy?

Refusal to marry.b. Marriage of one woman to more than one husband.c. Marriage of one man to more than one wife.

*d. Marriage of one man to only one wife.

9. Item using label responses:

A marriage in which one woman marries one man is called

a. unicameral.b. dualism.c. monotheism.

*d. monogamy.

49

ILLUSTRATIONS OF MULTIPLE-CHOICE TEST ITEMS (Continued)

10. Item asking examinee his opinion:

What do you consider the most important objective ofthe staff meetings?

*a. to establish good working relations with your staff.b. to handle routine matters.c. to help teachers improve instruction.d . to practice and exemplify democracy in

administration.

11. Item combining a question with an explanation:

Has the average size of farms in the United Statestended to increase in recent years? Why?

a. Yes, because as the soil loses its natural fer-tility more land must be cultivated to maintain thesame output.

*b. Yes, because the use of farm machinery has made

large farms more efficient than small farms.C. No, because the difficulty in securing farm labor

has forced many farmers to limit their operations.d . No, because large family farms tend to be sub-

divided to provide smaller farms for the children.

12. Item using incomplete stem:

Physiology teaches us that

a. the development of vital organs is dependent uponmuscular activity.

b. strength is independent of muscle size.c. the mind and body are not influenced by each other.d . work is not exercise.

13. Item using negative stem:

In the definition of a mineral which of the followingis incorrect?

a. It was produced by geologic processes.b. It has distinctive physical properties.c. It contains one or more elements.

*d. Its chemical composition is variable.

ILLUSTRATIONS OF MULTIPLE-CHOICE TEST ITEMS (Continued)

14. Item asking no specific _question:

In comparing the period of heterosexual adjustment ofour culture with those of other cultures, it must beconcluded that

a. there are tremendous differences that can only beexplained on a cultural basis.

b. there are large differences which must be explainedby the interaction of biology and the more influen-tial culture.

c. although there are some differences, the biologicalfoundation of puberty is fundamental.

d. in most cultures puberty is the period of hetero-sexual adjustment.

15. Item including a necessary qualification:

What change occurs in the composition of the air in alighted airtight room in which the only living thingsare growing green plants?

a. Carbon dioxide increases and o:tygen decreases.*b. Carbon dioxide decreases and oxygen increases.c. Both carbon dioxide and oxygen increase.d . Both carbon dioxide and oxygen decrease.

16. Item involving an "instructional aside":

In purifying water for a city water supply, one processis to have the impure water seep through layers of sandand fine and course gravel. Here many impurities areleft behind. Below are four terms, one of which willdescribe this process better than the others. Selectthe correct one.

a. Sedimentation*b. Filtrationc. Chlorinationd . Aeration

17. Item showing nonparallel responses:

Slavery was first started

*a. at Jamestown settlement.b. at Plymouth settlement.c. at the settlement of Rhode Island.d . a decade before the Civil War.

51

ILLUSTRATIONS OF MULTIPLE-CHOICE TEST ITEMS (Continued)

18. Item including complicating elements in the responses:

Systematic geography differs from regional geographymainly in that

a. systematic geography deals, in the main, with physi-cal geography, while regional geography concerns it-self essentially with the field of human geography.

b. systematic geography studies a region systematical-ly, while regional geography is concerned only witha descriptive account of a region.

*c. systematic geography studies a single phenomenon in

its distribution over the earth in order to supplygeneralizations for regional geography, whichstudies the arrangement of phenomena in one givenarea.

d . systematic geography is the modern scientific wayof studying differentiation of the earth's surface,while regional geography is the traditional and de-scriptive way of studying distribution of phenomenain space.

19. Item using "none of these" appropriately:

Which word is misspelled?

a. Contrary*b. Tendancyc. Extremed . Variable

20. Item using stereotypes in distracters:

Which of these has affected the greatest change in do-mestic plants and animals?

a. Influence of environment on heredity.b. Organic evolution.

*c. Selective breeding.d . Survival of the fittest.

21. Item using obscure distracters:

A chaotic condition:

a. Asympotic*b. Confusedc. Gauched . Permutable

52

ILLUSTRATIONS OF MULTIPLE-CHOICE TEST ITEMS (Continued)

22. Item involving verbal trick:

Horace Greeley is known for his

a. advice to young men not to go west.b. discovery of anesthetics.*c. editorship of the New York Tribune.d . humorous anecdotes.

23. Item showing ineffective expression:

Among the factors listed which have contributed torapid expansion of cut-over land, the most importantduring recent years has been

a. confiscatory taxation of standing timber.4`. b. high prices for lumber.c. rapid growth of population.d . rising standards of living.

24. Items which interlock:

What, if anything, developed in 1955 with respect torelations between Egypt and Soviet Russia?

a. A Russian military force invaded Egypt.b. Egypt accused members of the Russian embassy of

spying.c. Soviet Russia supported Egypt's demand for inde-

pendence from Britain.*d. The Egyptian gove.mment arranged to buy arms for

Soviet Russia.

How did the Western powers react to Egypt's agreementto buy arms from the Soviet Russia?

a. They approved it as a gesture of good will.b. The pointed out that it was a private affair be-

tween Egypt and Russia.*c. They pointed out that it was a threat to peace in

the Eastern Mediterranean.d . They protested that Egypt was aiding a potential

enemy.

53

ILLUSTRATIONS OF MULTIPLE-CHOICE TEST ITEMS (Continued)

25. Item with response placed in tandem:

The balance sheet report for the Ajax Canning Companywould reveal

a. the company's*b. the amount ofc. the amount ofd. the amount of

profit for the previous fiscal year.money owed to its creditors.income tax paid.sales for the previous fiscal period.

26. Item with unnatural sequence of responses:

The population of Denmark is about:

a. 2 millionb. 15 million

*c. 4 milliond. 7 million

SOME TYPES OF ESSAY QUESTIONS

1. Selective recall--basis given:

Name three important developments in measurement thatoccurred during the first decade of the TwentiethCentury.

2. Evaluation recall--basis given:

Name three persons who have had the greatest influenceon the development of intelligence testing.

3. Comparison of two things - -on a single designated basis:

Compare essay tests and objective tests from the stand-point of their effect on the study procedures used bythe learner.

4. Comparison of two thingsin general:

Compare standardized and non-standardized tests.

5. Decision--for or against:

In which, in your opinion, can you do better--oral orwritten examinations? Why?

6. Cause or effects:

How do you account for the popularity of objectivetests during the last fifty years?

7. Explanation of the use or exact meaning of some word,phrase, or statement in a passage:

What is the meaning of "objective" in the abovequestion?

8. Summary of some unit of the text or of some articleread:

Summarize in not more than one page the advantages andlimitations of essay tests.

9. Analysis (The word itself seldom appears in the2uestion) :

Why are many so-called "progressive educators" sus-picious of standardized tests?

55

SOME TYPES OF ESSAY QUESTIONS (Continued)

10. Statement of relationship?

Why is it that nearly all essay tests, regardless ofthe school subject, tend to a considerable extent to bemeasures of the learner's mastery of English?

11. Illustrations or examples (the pupil's own) of princi-ples in science, construction in language, etc.

12. Classification:

What type of error appears in the following test item?(With what country did the United States fight duringWorld War II?)

13. Application of rules, laws, or principles to newsituations:

In the light of experience in the United States withexaminations fo-:" selecting college students, whatpublic-relations problems would you expect to arise inEngland because of the Age 11+ examinations there?

14. Discussion:

Discuss the role of Sir Francis Galton in the develop-ment of the Pearson Product-moment coefficient of corre-lation, r.

15. Statement of aim--author's purpose in his selection ororganization of material:

Why are individual mental tests not treated in greaterdetail in this book?

16. Criticism--as to the adequacy, correctness, or relevancyof a printed statement, or a classmate's answer to aquestion on the lesson:

Criticize or defend the statement, "The essay test over-rates the importance of knowing how to say a thing andunderrates the importance of having something to say.""To criticize" assumes a set of standards given orknown. Many persons are under the impression that itmeans "Tell something good about and tell something badabout, but mostly good, so that it will be construc-tive." Teachers should not foster this misconception.

56

SOME TYPES O1 ESSAY QUEST/ONS (Continuod)

17. Outlino:

Outline the principal steps in the construction of aninformal teacher-made test.

18. Reorganization of facts (a ,good type of review questionto give training in organization):

Name ten practical suggestions from this book that areparticularly applicable to the subject you teach orplan to teach.

19. Formulation of new questions--problems and_guestionsraised:

What are some problems relating to the use of essaytests that require further study?

20. New methods of procedure:

Suggest a plan for proving the truth or falsity of thecontention that exemption from semester examinationsfor the ablest students is a good policy in high school.

21. Inferential thinking:

Is the author of this book likely to use essay testsfrequently in his measurement classes?

57

Ui

0)

(SAMPLE)

SPECIFICATIONS FOR AN EIGHTH GRADE ENGLISH TEST

OBJECTIVES

Content Area

Knows

Common

Terms

Comprehends

Principles

and

Generalizations

Applies

Principles

and

Generalizations

Analyzes

Written ma-

terial and

Graphs

Evaluates

and Selects

Appropriate

Techniques

and Methods

1 i

Total{

fi

Parts of Speech

26

2/0

The Clause

26

2

._._,v

Correct Use of

Pronouns

42

22

515

Punctuation

42

22

515

Paragraph

55

10

Writing Letter

42

68

20

Dictionary

42

68

20

TOTAL

20

20

25

25

10

100

I

(SAMPLE)

TABLE OF SPECIFICATIONS FOR AN EIGHTH GRADE ENGLISH

AFFECTIVE BEHAVIOR INSTRUMENT

OBJECTIVES

Receiving

Responding

Valuing

Organization

Characterization

Total

Teacher

10

10

School

10

10

English

10

10

20

Peers

10

10

TOTAL

10

40

50

PROGRAM CALENDARS

A Program Calendar should contain the following information:

1. Objectives--the objectives of the given program shouldbe stated in behavioral terms, along with the specifi-cation of how these objectives are to be measured.

2. Program Description--the program description should be adetailed description of what is going to take place inthe program. Each variable along the instructional andinstitutional dimensions should be considered and thosevariables which are going to be directly affecting thegiven program should be expanded upon in the programcalendar. For example, each variable along the insti-tutional axis should be considered when the sample is de-scribed. Is the sample going to be of students,teachers, administrators, etc., or a combination ofthese? The size of the sample, grade level, ethnic back-ground, and other pertinent data should be included whenthe description of the treatment is considered. Whatcontent and materials are going to be used? Whatmethod(s) is/are going to be implemented? How about fa-cilities? Is there going to be a change in organi-zation? Next, an evaluation design should be specified.This could be a picture showing data collection pointsand the comparisons to be made. In addition, the sta-tistical analysis should be described.

3. Monitoring System--included in the program calendarshould be a description of the monitoring system that isgoing to be used to check to see if the planned programwent as planned. An example of a format which could beused is attached. Notice that the monitoring system de-scribes, by objective, what is going to take place.Then, when the actual program is implemented, it is amatter of comparing what was planned to what actuallytook place by using the same format.

4. Calendar of Events--a calendar of events should be drawnup to describe the sequence of events in terms of datacollection and other important responsibilities.

PROGRAM: Involvement of Teacher Effectiveness Through In-Service Training

=1114

Monitoring

Period

Example

Week ofSeptember18-22

=111.11MI

FORMAT FOR MONITORING

Related Program Objectives *Organization

Objective #1: Teachers par-ticipating in the in-serviceprogram will respond withincreased flexibility asmeasured by coded videotapes of their classroomteaching utilizing codingsystems of InteractionAnalysis and TeacherSelf-Appraisal.

Time: Two one-hour sessions

Space: Smallgroups, each in-volving fourteachers and oneresource person.Teachers random-ly selected forgroups.

Week ofSeptember25-29

Objective #1:Same as above.

Time: One-hoursession

Space: Largegroup structure,with no group-ing implied. Allparticipants inlarge groups.

61

.17....

Description of Instructional Variable

Content Method

1

Facilities Cost

Teaching SkillsDesigned to im-prove classroominteraction.Basic elementsof:

(1) listening,(2) questioning,(3) utilization

of studentideas,

(4) structuringresponses

Use of programmedmaterials

Teaching ActivitiesLeader acting onlyas a resource per-son.

Types of InteractionParticipants in eachgroup will assume apre-determined rolefor each exercise.

Equipmentand RelatedActivitiesVideo Tapingof teachersin their re-spectiveclassrooms(one-halfhour perteacher par-ticipant).

NotAppli-cable

Skill developmentin the coding ofteacher-pupilverbal interactionusing Flanders'Interaction Analy-sis System.

Teaching ActivitiesLecture and discus-sion method

Types of Inter-actionPrincipally resourceperson--teacherparticipants.

Equipmentand RelatedActivitiesVideo tapeequipmentand demon-strationtapes.

NotAppli-cable

*Time in the organization variable refers to the duration and/or sequence ofblocks of time devoted to the subject matter.

*Space refers to the general structure of the group to classify the parti-cipants.

61a

APPENDIX IV

MESSAGE MEDIA THEATRE

Friday Evening, April 18BBC Film: Improvised Drama IThe Meaning of Modern PaintingHuckleberry Finn I

Saturday EveningBBC Film: Improvised Drama IIThe OdysseyThe Themes of Macbeth

Sunday EveningThe Magic Prison (Emily Dickinson)The Age of ElizabethThe Poisoned Kingdom (Hamlet)Hamlet IIHamlet III

Monday EveningMCTE Elementary English Series: Word GeographyArt of Huckleberry FinnHuckleberry Finn and the American Experiment(check TV for the Spoon River Anthology)

Tuesday EveningPerspectives of the Imagination: Comparisons (NCTE)Toronto Board of Education, classroom movies

Wednesday EveningMarshall McLuhan I and II(poetry reading session, participants)

Thursday EveningThe Theatre--One of the HumanitiesOur Town IOur Town II

62

qb.

APPENDIX V

EPIC FEEDBACK QUESTIONNAIRE

PROJECT MESSAGE TRAINING SESSION

Boyne Mountain Lodge

April 18-21, 1969

SCHEME FOR EVALUATION

Place the letter in the correct response in each blank.

1. The EPIC Evaluation Center began operation in

a. 1961b. 1966

c. 1967d. 1964

2. The EPIC Evaluation Center is located in

a. Phoenix c. Washingtonb. Denver d. Tucson

3. The EPIC Evaluation Center is funded through

a. ESEA Title I c. NTLb. NDEA Title III d. ESEA Title III

4. The EPIC Evaluation Center has two divisions

a. Field Services and Computer Servicesb. Public Relations and Evaluationc. Field Services and Evaluationd. Measurement and Evaluation

5. Phase three of the Scheme for Evaluation is

a. feedback. c. measurement.b. planning. d. implementation.

6. The Program Calendar is part of the Phase.

a. Feedback c. Measurementd. Implementationb. Planning

63

EPIC 141DBACK WEST ,ONNAIREPROJECT MESSAGE TRAINING SESSIONApril 18-21, 1969

SCHEME FOR EVALIV.....__.1L41M,Jj:_a_...inued

7. The Recycling Process is part of the Phase.

a. Feedbackb. Planning

c. Measurementd . Xmplementation

8. Monitoriag is part of the Phase.

a. Feedback c. Measurementb. Planning d. Implementation

9. A Program Calendar does not contain

a. objectives. c. recycling procedures.b. data to be gathered. d. feedback dates.

10. The EPIC Structure of Variables Influencing In-structional Programs does not contain a(n)Dimension.

a. Behavior c. Institutionb. Measurement d. Instruction

OBJECTIVES

1. There are dimensions in the EPIC Structure ofVariables Influencing Instructional Programs.

a. threeb. four

C. fived . two

2. Utilizing the EPIC Structure of Variables InfluencingInstructional Programs, variables form a factor.

a. twob. four

c. sixd . three

3. is a variable of the Instructional Dimension.

a. Administrator c. Studentb. Facility d. Knowledge

64

EPIC FEEDBACK WE8TIONNAtHEPROJECT MESSA TR/UM:NO SLSSIONApril 18-21, 1969

OBJECTIVES (Continued)-

4. factors can be formed utilizing the EPIC Structureof Variables Influencing Instructional Programs.

a. 10b. 150

c. 325d. 420

5. Team teaching would be an example of which Instruction-al variable?

a. Content c. Methodb. Facility d. Organization

Match the following:

6. Application7. Respond8. Analysis9. Energy10. Receive11. Knowledge12. Comprehension

A. Affective

C. Cognitive

P. Psychomotor

13. In writing a program objective, utilizing the EPIC ap-proach, the following information is unnecessary.

a. Behavior variable.b. How behavior will be

measured.

c. Minimum acceptableperformance

d. Instructional variable.

14. Write a program objective of your choice in the spacebelow, underline the four EPIC criteria.

65

gl .11.1111

EPIC FEEDBACK QUESTIONNAIREPROJECT MESSAGE TRAINING. SESSIONApril 18-21, 1969

MEASUREMENT

1. Validity refers to

a. how much a test costs in relation to the acquiredinformation.

b. the degree a test measures what it purports tomeasure.

c. the degree of difficulty in interpreting testscores.

d . the number of equivalent forms accompanying thetest.

2. When one is concerned with reliability, he should lookat the

a. ease of interpretation of test scores.b. number of uses for the test.c. cost of the test.d . consistency of test scores.

3. The most common level of cognitive behavior measured bya standardized test item is

a. comprehension. c. knowledge.b. analysis. d. application.

4. The relationship of validity and reliability is

a. There is no relationship.b. Reliability is not a necessary, but a significant,

condition for validity.c. Reliability is a necessary and sufficient condition

for validity.d . Reliability is a necessary but not a sufficient

condition for validity.

5. When are the measuring instruments to be used in evalu-ation specified?

a. When the behavioral objectives are written.b. When the monitoring system is developed.c. During the output-analysis step.d. During the recycling phase.

66

c.t.thetmielailatokki.1 (MAW

EPIC FEEDBACK QUESTIONNAIREPROJECT MESSAGE TRAINING SESSIONApril 18-21, 1969

MEASUREMENT (Continued)

6. The highest level of measurement of Affective Behaviorin the classroom is usually on the level.

a. receiving c. valuingb. responding d. characterization

7. A table of specifications will help the teacher to de-termine the

a. amount of achievement. c. validity of the test.b. type of test items being d. standard divisions of

used. the measuringinstrument.

8. Which two dimensions from the EPIC Structure are uti-lized in developing a Table of Specifications formeasurement in the affective area?

a. Behavior--Instruction c. Institution--Behaviorb. Instruction--Institution d. Instruction--Measure-

ment

Circle One: SD = Strongly Disagree D = Disagree'Undecided = U A = Agree

SA = Strongly Agree Percentages

Omit1. As a result of this workshop,

I have more knowledge of 0 0 5 51 41 2

evaluation. SD D U A SA

2. When I hear the word "ob-jectives," I have a feeling 39 46 10 2 2 0of dislike. SD D U A SA

3. Members of the workshopstaff really knew their 0 0 5 41 54 0

subject. SD D U A SA

4. There was too much lecture 10 54 17 12 5 2and too little interaction. SD D U A SA

5. Competence in evaluationwill facilitate teaching 2 0 5 37 56 0procedures. SD D U A SA

67

EPIC FEEDBACK QUESTIONNAIREPROJECT MESSAGE TRAINING SESSIONApril 18-21, 1969

MEASUREMENT (Continued)Omit

6. Writing objectives is neces- 2 5 7 37 49 0sary for effective teaching. SD D U A SA

7. As a result of this work-shop, I have more knowledge 0 0 5 41 54 0of objective writing. SD D U A SA

8. As a result of this work-shop, I will be able to useteacher-made measuring 0 7 10 44 39 0instruments more effectively. SD D U A SA

9. The large group meetings inthis workshop were interest- 2 2 7 71 17 0ing and helpful. SD D U A SA

10. The small group meetings inthis workshop were interest- 0 2 2 32 61 2

ing and helpful. SD D U A SA

11. Generally speaking, thisworkshop provided me with 5 2 15 39 37 2

what I came for. SD D U A SA

EPIC FEEDBACK QUESTIONNAIRE

PROJECT MESSAGE TRAINING SESSION

Boyne Mountain Lodge

April 18-21, 1969

TOTAL TEST RESULTS

Pre-Test Mean

12.95

Post-Test Mean t value

22.35 10.00

SUB - PART RESULTS

1. Scheme for Evaluation

Pre-Test Post-Test t value

4.25 6.55 6.57

2. Objectives

Pre-Test Post-Test t value

4.90 10.64 10.07

3. Measurement

Pre-Test Post-Test t value

4.03 5.36 4.59

69

APPENDIX VI

NAME (Optional)

EVALUATION OF BEHAVIORAL OBJECTIVES OF MESSAGE

In compiling a report for the United States Office of Edu-cation, the staff needs a statement about your reactions toMESSAGE. GIVE YOUR EVALUATION TO HELEN WAGNER NOT LATERTHAN LUNCH TIME ON WEDNESDAY. Thanks.

I. Please submit 15 to 1 page covering the following items:

1) the organization of the week-long session

2) the quality and variety of consultative helpavailable

3) the accessibility of resource personnel

4) the interchange of ideas among participants

5) the relevancy of content to needs of teachers andtheir institutions

6) the possibilities for implementation on locallevels

7) the probable effects of MESSAGE upon the teachingof English in Michigan

II. PleaSe list your specific recommendations as to follow-up on this the first workshop of this type. Do youthink we should encourage future sessions of this type?

How can we encourage future sessions? (Be specific)

APPENDIX VII

TENTATIVE FOLLOW UP EVALUATION QUESTIONS

Please add suggestions and return Thrusday morning to BobFichtenau

1. Have you presented EPIC behavioral objectives to yourstaff?

2. Are you going on alone?

3. What measurement devices did you use, if any?

4. Which institutional variables did you write objectivesfor?

5. In your school, have you written, objectives for:

a. cognitive behaviorb. effective behaviorc. psychomotor behavior

6. Did you write objectives stressing - organization, con-tent, etc.?

7. Have you introduced the EPIC concept within

your buildingEnglish staffother teachers

8. Have you introduced the EPIC concept at the administra-tive level?

9. Did you talk about the EPIC concept beyond

your departmentyour schoolyour school district

10. Did you use a standardized test?

a.b.c.

72

11. Did you use teacher made tests?

a.b.

12. Did this test do what you wanted it to do?

13. What type of curriculum does your school have?

a.b.

14. What kind of program do you have in your school?

15. Would you be willing to share it?

16. What are you doing about writing behavioral objectives?

17. If responses are negative, why are they negative?

18. From whom did you get the most support?

19. Was in-service training involved in the introduction ofthis program?

20. Has there been any parent involvement?

21. Did any of the institutional variables aid you inwriting cognitive objectives?

22. Outside your own district personnel, from whom have youreceived help in utilizing your Project MESSAGEtraining?

a. other MESSAGE participantsb. Intermediate districtc. State Dept. of Ed.d. Universitye. Epic Evaluation Center

23. As a result of your training, are changes taking placein your school system?

a. no changeb. program changes are in the planning stagec. objectives are being developedd. both b & ce. b & c in addition to developed evaluation

73

24. From whom have you received the most encouragement toput to use the skills you developed by participating inMESSAGE?

a. your own departmentb. your principalc. curriculum director or specialistsd. superintendente. no support