Testing the Receptive Skills: So me Basic Co nsidera tio ns

Randall L. Jones Brigham Young University

Michael Canale has addressed some very in- teresting issues in his position paper. This reaction concentrates on three main areas: 1) the nature of the receptive skills and the requirements of a valid instrument to measure them; 2) the design features that are consistent with his test design principles; and 3) adaptive testing procedures.

As a preface to these remarks it is important to address two very basic questions relating to testing. The first question needs to be answered before any test development or selection is done: What is the purpose of the test? The question appears on the surface to be simplistic and obvious, but unfor- tunately, it is too often ignored. In the academic world we would be honest if we admitted that, in the majority of cases, the principal function of testing is to provide the necessary data on which we can determine grades. Criterion and content validity are certainly important, but the discrimina- tion value of the test is more important. However, if we are interested in measuring real language pro- ficiency and performance ability, we must be cer- tain that the tests we design have a close fit with these goals.

The second important question should be asked after the testing has been completed: How d o I in- terpret the results? What does a score of 87 on a reading test really mean, i.e. what is the predictive power of the test? The ACTFL proficiency guidelines include definitions for listening and reading. It would seem only logical that a test that is designed to measure proficiency on the basis of these definitions would be able to provide scores that reflect the degree of proficiency as it is defined.

Given these considerations, we can now turn to the nature of the receptive skills and the implica- tions for testing. It is important to keep in mind that proficiency in the use of receptive language skills cannot be observed directly. Any test of

Foreign Language Annals, 17, No. 4, 1984 365

reading or listening is not based on that skill direct- ly, but rather on some kind of response dependent on that skill. In traditional tests a passage is read or heard, after which the examinee performs a task which is dependent on the comprehension of the passage. For example, the examinee hears the sentence:

I would like to send this airmail.

He must then select from a test booklet the sentence that best describes where the speaker is.

A. In a hardware store. B. In a travel bureau. C . In a bus station. D. In a post office.

Although the examinee must be able to understand the sentence to be able to select the correct answer, the process of listening itself is not really observ- ed. The fact that the examinee selects D is really an indirect reflection of the listening process.

Three important questions must be raised at this point: 1) How closely d o the passages approximate experiences that are typical of real-life situations? 2) What level of abstraction is allowed between the comprehension task and the performance task, i.e. when does the performance task lose its relation to the original goal? 3) What variables other than comprehension can assist the examinee in ac- complishing the performance task successfully? For example, research has shown that on certain tradi- tional reading and listening tests clever examinees can score well without even seeing or hearing the stimulus passages. By using other clues in the struc- ture of the test they are able to determine what the correct alternative should be. In such cases we must seriously question what is actually being tested.

It would seem that not only the language material used in receptive skills tests should be authentic and

366 FOREIGN LANGUAGE ANNALS

well conceived, but that the types of performance tasks required of the examinees, i.e. what they must do after the stimulus is heard or read, should also be given careful consideration. What d o we really d o when we read or hear something? In many cases we merely process the information, store it if we feel that it will be important at a later time, and discard it if it seems irrelevant t o our needs. Fre- quently, however, we must make a decision based on the information. For example, a worker is driv- ing home and hears on the radio that construction is being done on a street that is normally on his route. He must immediately decide if he will con- tinue on the same route and put up with the delay or select an alternate route. In another instance, a prospective car buyer reads the classified ads to find a used car that she can afford and that will fit her needs. In still another context, an employer reads the application forms of several prospective employees to determine which one should be hired, and so on.

If we want tests of the receptive skills to reach the standard of quality that oral tests have achiev- ed in the past few years, we will need t o dedicate more resources to their design, development, and validation. We may have to move away from total- ly objective testing and experiment with some techniques that more closely approximate real-life situations. We may have t o give up such conve- niences as mass administration and scoring. We may have to develop training material to assist pro- spective test writers, scorers, and other personnel. It is likely that we will have to abandon old no- tions and allow our imaginations to assist us in developing new and more satisfying methods of testing reading and listening.

The second area that invites further comment is Canale’s suggestion that thematic organization should be considered in the development of recep- tive skills tests. As is well known, the majority of listening and reading test items have traditionally been isolated sentences that d o not relate to each other in any logical way. Each is designed to test a certain aspect of the language, e.g. vocabulary, grammar, sound discrimination, etc. A thematic organization, on the other hand, resembles more closely the types of listening and reading tasks we experience in reai life. The information in each sentence of the message relates t o information in the other sentences. The redundancy increases as clues about the intent and meaning of the message accumulate.

In the construction of performance listening and reading tests, thematically organized passages are a necessity. The purpose is to recreate a situation

similar to what a n examinee would find on the job or on the street. It is not common to hear or read single isolated sentences typical of conventional language tests. One should, however, be aware of certain problems inherent in thematic test stimuli. For example, thematic organization is generally less efficient than single items. Within a passage of natural text there is material for only a limited number of items, i.e. word-for-word, there is a higher ratio of items to text. The examinee must therefore either process more text or respond to fewer items in the same amount of time.

A second drawback is the fact that a thematic text usually limits the variety of language material that can be tested. Unlike a list of independent sentences, which can cover a wide range of gram- mar and numerous vocabulary items, an authen- tic text will have a limited number of grammatical structures and lexical items. Thematic texts and variety in language types seem almost to be mutual- ly exclusive.

A final potential problem with the use of a thematic text is that it can result in test contamina- tion. A question that relates to the first portion of the text may be answered unintentionally in a later section of the text or even in another question. A Spanish test in which many items could be answered correctly simply by reading the other questions in- cluded items such as: “Where did Juan go that morning?” “What did he want t o buy at the drug store?” “How much did the toothbrush cost?” Because the information is all integrated, it is sometimes difficult to develop items that are not also in some way integrated.

In spite of these problems, however, it seems clearly preferable that the text used in reading and listening tests be authentic, and that usually means thematically organized. The texts and the items bas- ed o n them must be carefully worked out t o be as efficient and valid as possible.

Finally, comment should be made on the topic of computer-adaptive testing. This is a very new area, and to our knowledge n o one has yet written a computer-adaptive language test. We are not yet certain that the results for language testing will be the same as they apparently have been for other types of tests. The prospects are intriguing, however, and it is important that we as a profes- sion keep abreast of this development and on our own investigate t h e possibilities as they apply to language testing.

It is important to point out that computer- adaptive tests have thus far been based on the multiple-choice format. There is, of course, nothing wrong with this, as most standardized tests used

SEPTEMBER 1984 367

on a wide scale are multiple choice. However, we are attempting in our profession to devise testing techniques that are more performance oriented, i.e. that more closely approximate real-life situations. It is therefore important that we do not resign ourselves to the position that computer-adaptive tests must include only items that can be objectively scored. Because of the processing power of the computer, an open-ended response-i.e. one that may differ slightly from examinee to examinee- can be evaluated relatively easily. We should be in-

novative in addressing this issue and explore a varie- ty of techniques.

The issues and challenges that Michael Canale has raised are important ones. It is evident that the receptive skills will play an increasingly important role in future language programs. As testing specialists, we are obligated to assist in the develop- ment of valid and reliable evaluation measures so that we can feel assured that our instruction in the receptive skills has been effective.

Arts Festival Integrates Curriculum The month of March was a lively one at King Philip Elementary School in West Hartford,

CT, as teachers of physical education, foreign language, music, art, library science, and health combined efforts to produce an activity-filled arts festival.

Weekly seminars in the beginning of the school year explored ways to better integrate ef- forts and curriculum. With the groundwork laid, specific events and joint lessons were planned. Flexibility was key. When March arrived, a calendar of day to day events was placed in the school foyer.

The festival’s activities were numerous and varied. The sixth grade class danced a Virginia Reel with calls in French. A Spanish play was performed depicting the “Legend of Popocapaetl” using colorful puppets made in art class. A third grade class learned to sing and perform the Hokey Pokey in French. Children from all grades learned about good nutri- tion in a skit entitled, “Vegetable Salad.” Foreign language and music were combined to produce ‘‘El Zapatero” and “0 Canada.”

Health took a global view as students exercised and described their movements in French. Special luncheon menus were planned by the students and the dietician, emphasizing good nutrition while including foods from other countries.

Several all-school activities were held as well. On “T-shirt Day” students and staff wore colorful silk-screened T-shirts with the logo “K.P.E. Celebrates The Arts.” “Fantasy Day” allowed students to impersonate a well-known person or character from the arts or sports. That day Van Gogh, Charlie Chaplin, Peter Pan, Casey at the Bat, and Michael Jackson, among others, attended King Philip Elementary School.

Highlights of the busy month were presented at a school assembly. Students presented por- tions of their experiences, including a fashion parade of the winning Fantasy Day characters. Participants represented all grade levels, and it was agreed by both students and faculty that the festival was a rewarding and enjoyable experience.