7/28/2019 gzuniga03

1/11

DESIGNING ACHIEVEMENT TESTS FOR SIXTH TO TENTH

GRADES IN STATE SCHOOLS

Gilma Zuiga, Viviana Mndez, Tania Paola Tinoco, Valentina Villanueva

Universidad Surcolombiana-Neiva

Two facts have affected the teaching of English in Colombia recently. They are: thecoming out of Lineamientos Curriculares en Idiomas Extranjeros (1999), and theinclusion of English as a compulsory component in the ICFES exam. These facts havemotivated teachers in state schools to express their desire to have tests that could be asource of feedback in order to improve teaching and learning in their schools. That is

why, we have decided to undertake this task and design achievement tests for sixth totenth grades that focus on the achievement indicators stated for those grades at stateschools. This is an ongoing project, and presently, we have designed, piloted andanalyzed the test designed for sixth grade. The next step is to continue to prepare and

pilot the tests for the next grades. The theoretical framework, and the steps followed todesign and validate the tests are presented in this document.

WHY TO TEST

Achievements tests assess what has been achieved or learned from what was taughtin a particular course. The purpose is to determine what the student has learned.

Achievement tests can provide assessment for teachers and learners. The assessmenttasks is nonthreating, allowing learners opportunities to demonstrate what they knowand do not know, and providing useful feedback both for the learners and for theirteachers. Therefore, the primary function of the achievement tests is instructional, eventhough their results could also have an administrative function.In an instructional function, achievement tests can help in diagnosing students learning,give evidence of progress, provide feedback to the respondent, or even giveinformation about the quality of teaching. This information can serve as a check for therelationship between what is taught and what is learned. It could, as well, be a valuablesource of information to find out weaknesses and strengths of English programs(Cohen, 1994; Bachman, 1990; Delgado, 1996; Estevez, 1996).The results of achievement tests could be useful for administrators, who may use theirresults for promotion, placement, or certification. Low results could find the causes instaff qualifications which may lead to teacher development schemes. Other causesmay include poor resources, lack of students motivation and poor administrativesupport.The purpose of designing the achievement test is to improve teaching and learning.This assessment can inform the teacher and influence instruction. Teachers can learnabout their students, about themselves as teachers, and about the impact of theinstructional program and then joint actions could be taken to tackle problems raised.Students could also take actions after recognizing problem areas and work on their

weaknesses.

7/28/2019 gzuniga03

2/11

WHAT TO TEST

Canale and Swain provided a theoretical framework for communicative competence that

has influenced the design of assessment instruments over the past years. Tests, thenbegan to focus on one or more of the four components making up the construct ofcommunicative competence, namely sociolinguistic, competence, discoursecompetence, grammatical competence, and strategic competence.For the purposes of these tests we will be using a version of the above theoreticalframework, proposed by Bachman (1990) and Cohen (1994). The term used iscommunicative language ability, as it provides a basis for the development and use oflanguage tests. It recognizes that the ability to use language communicatively involves

both knowledge or competence in the language, and the capacity fro implementing orusing this competence. Therefore what will be assessed is communicative ability. Partof communicative ability is sociocultural ability, sociolinguistic ability, grammaticalability, textual ability, and strategic ability.Sociocultural ability: It relates to the appropriateness of the strategies selected forrealizing communicative functions in given contexts, like (1)the culture involved, (29the age and sex of the speakers, (3) their social class and occupations, and (4) their rolesand status in the interaction. It also evaluates the amount of information, whether toomuch or too little and the relevance and clarity of the information supplied.Sociolingusitic ability. It deals with the linguistic forms that respondents use toexpress the intent of the communicative act. It assesses the respondents control over thelanguage forms used to realize (sorry, very sorry, excuse me). , their control of register(academic or casual), or the formality of the utterance.

Grammatical ability: Vocabulary, morphology, syntax, and phonology are assessed.The focus is on clear cases of error in form as well as style.Textual ability: It includes the knowledge of the conventions fro joining utterancestogether to from a text, which is essentially a unit of language structured according torules of cohesion and coherence.Strategic competence. That is strategies used to compensate for a lack in somelanguage area. This ability cannot be easily observed, however it will be broken downinto three components: respondents set communicative goals, plan their use, andexecute the plan.

HOW TO CONSTRUCT THE TESTS

According to Estevez (1996) and Cohen (1994), the place to start in designing anassessment instrument is by making an inventory of the objectives that are to beassessed. This involves distinguishing broad objectives from more specific ones. In ourcase we will be talking about terminal objectives, since state schools nowadays plan thesyllabus of the courses in terms of terminal and enabling objectives (Brown, 1994)(logros e indicadores de logro).The steps are1. Collect the syllabus of the courses from four state schools and make checklists

The teachers checked off terminal and enabling objectives that were common in

the four schools Important terminal objectives are distinguished from marginal ones for the task.

7/28/2019 gzuniga03

3/11

Objectives are weighed in order to obtain a representative sampling of alearners language behavior.

The most emphasized objectives are identified in the courses as well as those ofgreatest utility value for the students

2. Collect notebooks and portfolios of students used in classes

3. Collect textbooks and indicate material covered during the courses.4. Interview teachers in four state schools to find out the most common expectations ofteachers in relation to the achievement tests.

STATING THE SKILLS BEING ASSESSEDAccording to the Lineamientos Curriculares, a communicative methodology isexpected to be taught at schools in Colombia, therefore the tests are designed toassess the four linguistic skills: listening, speaking, reading, and writing.

STATING ITEM TYPES

Some item types stated for the ICFES exam will be developed. Other type ofquestions are added as the achievement tests developed in this project intends toassess listening and speaking as well.

The following were used as typesItem type Skill Communicative

Competence

Sentence construction Reading Grammatical

TextualUnderstanding pictures SpeakingListening

GrammaticalSociolinguistic

Incomplete texts reading SociolinguisticTextual

Paragraph Construction reading GrammaticalTextual

Incomplete dialogues reading SociolinguisticSociocultural

Understanding situations Speaking ScoiolinguisticSociocultural

Text comprehension Reading TextualPersonal questions Speaking Sociocultural

sociolinguisticFill in charts Listening SociolinguisticWrite texts Writing Sociolinguistic

Textual

7/28/2019 gzuniga03

4/11

ASSIGNING SCORES TO ITEMS

The first step will be to take the test the teachers themselves and then they determinethe value of scores rating according to how central to the course material or howdifficult they have perceived the item to be. Then scores will be given to each itemtaking into account that each item will be given the same weight, and each objectivewill be assessed with more than one item.

For writing and speaking, formats will be used according to the following scale :1-5 rating on the following:

SPEAKINGOverall fluency

Communicative efficacyGrammarVocabularyPronunciation

WRITINGCommunicative efficacyText organizationGrammarVocabularySpelling & Punctuation

ITEM ANALYSIS

Item difficulty

Refers to the ratio of correct responses to total responses for the tests. A normreferenced assessment instrument, which aims to differentiate among respondents,should have items that approximately 60% to 80% of the respondents answercorrectly (Cohen, 1994).

Piloting of the achievement tests

As the achievement tests are intended to perform an important function in aninstitution or in a Colombian region, they would undergo piloting on a smallsample of respondents similar to those for whom it is designed. In order to do thatafter the teachers themselves have answered the tests they would complete anevaluation questionnaire.Students as well be asked to answer questions that give information in relationshipwith :

How far they go when the time ran out then go ahead and complete the testso that there is feedback on all items in the test

Valuable insights about what part of the instrument needs to be revised.(See evaluation questionnaire)

7/28/2019 gzuniga03

5/11

ADMINISTERING THE ACHIEVEMNT TESTS

Cohen (1994), advises teachers to give the students a brief review of those elementsthat will appear on the instrument. Therefore, sample questions will be given to thestudents before taking the exam for the first time. Sample items usually help allay

test-taking anxieties. They can serve as a warm-up to get the learner into the task athand, a matter that should not be minimized. Poor performance may be frequentlytraced to inadequate warm-up.Giving instructions and a sample item or two before students receive the measureitself is advisable.The following checklist of administrative tips applies as suggestion given by Cohen(1994) The room should have adequate ventilation, heat, light, and acoustics. If a videotape is used, it should be set up and tested in advance to make sure

that it works. The test administrator should assume an affable but stern posture. Smiles can

help to put the respondents at ease. If the instructions are read aloud, they should be read slowly with no

departure from the established wording. The time that the exam begins and the total time remaining for the test and or

students subtests should be written on the chalkboard.

HOW TO EVALUATE THE TESTS

RELIABILITY

The reliability of a test concerns its precision as a measuring instrument. Reliabilityasks whether an assessment instrument administered to the same respondents asecond time would yield the same results. Three different types of factors contributeto the reliability of language assessment instruments: test factors, situational factors,and individual factorsTest factors include the extent of the sampling of the objective, the degree ofambiguity of the items and restrictions on freedom for response, the clarity andexplicitness of the instructions, the quality of the payout, the familiarity that therespondents have with the format, and the length of the total test.These test factors will give the test internal consistency. The reliability factor will

be measured with Alpha de Cronbach, software designed to measure reliability oftests.

VALIDITY

Validity refers to whether the achievement tests actually measures what they purportto measure. Thus the test must be reliable to be valid. Assuming that the test is

producing a reliable measure, the test must have content and construct validity.

Content validity: The adequacy with which the test samples the content forobjectives of the course or area being assessed. The content of each item and of the

7/28/2019 gzuniga03

6/11

test as a whole is evaluated. In order to find this out, the test will be evaluated bytest takers and teachers.Construct validity: Whether the instrument is true reflection of the theory of thetrait being measured: the language in this case. In order to show construct validity,the achievement indicators will be evaluated in different ways. That is to say one

achievement will be evaluated with different item elicitation and item format.Piloting and evaluation of the instrument given by students and teachers willcontribute to the validity of the achievement tests.

CUESTIONARIO PARA EVALUAR LOS EXAMENES DE RENDIMIENTO

ACADEMICO DE INGLES

EXAMEN CORRESPONDIENTE AL GRADO ________

NOMBRE DEL EVALUADOR: ___________________________________

GRUPO QUE RESPONDI :______________________________________

COLEGIO: ____________________________________________________

FECHA DEL EXAMEN: _________________________________________

Estimado (a) profesor (a) estudiante:Queremos conocer sus impresiones sobre el examen. Por favor responda las siguientes

preguntas. La informacin suministrada por usted es de mucho valor para nosotros puesnos permitir mejorar el instrumento de evaluacin.

A. SOBRE LAS INSTRUCCIONES1. Las instrucciones para cada seccin son claras?

Si___ No___ Por qu _____________________________________________

2. Los ejemplos de cada seccin son suficientes?Si_____ No_____ Por qu?

3. El tiempo es adecuado?

4. El mtodo de administrar el examen se estableci cuidadosamente? (otrapersona lo puede administrar exactamente de la misma manera?)

B. SOBRE EL CONTENIDO

1. El examen evala los logros propuestos para el grado?

2. Le parece conveniente evaluar todas las habilidades lingsticas?

7/28/2019 gzuniga03

7/11

De su opinin sobre la forma como se evalaHabla:

Escucha:

Lectura:Escritura:

3. El examen evala todos los componentes de la competencia comunicativasuficientemente? Hay algn nfasis innecesario o al contrario falta enfatizaralgn componente?

4. (Solo para profesores). Por favor tome el examen y califique cada pregunta deacuerdo con la importancia de ella. Escriba al lado de cada pregunta:

MI: Muy ImportanteI: ImportanteM: Media

5. (Solo para profesores). Marque en el examen cada pregunta, con relacin a ladificultad de la misma.

MF: Muy fcilF: FcilD: Difcil

C. SOBRE LA FORMA DE LAS PREGUNTAS Y EL DISEO

1. Cmo le parece la extensin del examen?Muy largo______ Muy corto_____ Adecuado_____

2. Cmo le parece la presentacin del examen?Es clara_____ Se presta a confusiones ____Comentarios

3. Cul es su opinin sobre el puntaje dado a cada pregunta y a cada habilidad.

7/28/2019 gzuniga03

8/11

PREPARING THE TEST FOR SIXTH GRADE

For sixth grade the following is the list of the common objectives found in the fourschools:

Students ask and answer greetings and farewells Students introduce people Students give personal information Students perform some commands in the classroom Students use numbers in different situations (1 to 1000) Students give information about family members Students ask and answer about occupations and nationalities Students spell the words known. Students describe a house Students tell and understand the time.

Each school planned to teach some objectives that were not shared in the four schools:They were: Students give information using parts of the body Students give information using the days of the weeks and the months of the

year Students inform about actions in progress Students give information about food Students describe pets animals Students express abilities with can Students inform about daily activities.

SAMPLE OF QUESTIONS

TEXTOS INCOMPLETOS

Juan Pablo Montoya (1)______from Bogota, he was born in 1975. He started racingwhen he was very (2)_______. Juan Pablo won the 500 miles in Indianapolis, USAin 2000. Now, (3)________ runs in formula one with the Williams team.

1. (A) was (B) is (C) am

2. (A) young (B) intelligent (C) old

3. (A) she (B) I (C) he

DIALOGOS INCOMPLETOS

A: Hello, Ben. How are you?

7/28/2019 gzuniga03

9/11

B: ____________________A: Ben this is my daughter. Lisa, this is my friend, Ben.B: Nice to meet youC: Nice to meet you too

(A) How are you? (B) Im fine, thanks (C) Good morning

RELACIONAR GRAFICAS CON TEXTOS

Lea la carta y relacione los prrafos con las grficas que describa cada uno

Picture 1. Picture 2 Picture 3

(1) Dear Jose, How are you? Im fine. Im sending these pictures of me in New York.I visited that city last week. It is a very big and exciting city. I loved Central park, it is

beautiful in summer. Many people play and run there.

(2) I have English classes in a Community College. Im in a class with twentystudents. They come from different countries: Japan, Brazil, India, and Ecuador.

(3) I live in an apartment with two Brazilian boys, we speak in English because I dontunderstand Portuguese, and they dont speak Spanish. They are brothers. Thy are veryfriendly.

Write to me soon.

Johan

QUESTIONS FOR LISTENING COMPREHENSION

1. Look at the pictures and choose the picture that corresponds to the descriptionyou hear.

2. Listen to the words and choose the word you hear

A. mother B brother C. soccerA. B. C.

3. Listen to the times you hear and choose the corresponding picture.

SPEAKING

7/28/2019 gzuniga03

10/11

1. Answer these PERSONAL QUESTIONS

Whats your name?How old are you ?Who is your English teacher?

Whats your address?Whats your telephone number?Do you have any brothers or sisters?What s your fathers occupation?Whats your mothers occupation?

2. Look at these family photos and describe the people in the pictures.

REFERENCES

Bachman, Lyle F. (1991). What does language testing have to offer? TESOLQuarterly, 25(4), 671-704.Bachman, Lyle F. (1990). Fundamental Considerations in Language Testing.Oxford: Oxford University Press.

Burns, Anne (1999). Collaborative action research for English language teachers.Cambridge: Cambridge University Press.

Cohen, Andrew. (1994) Assessing Language Ability in the classroom. Boston:Heinle & Heinle.

Colombia. (1999). Ministerio de Educacional Nacional. Lineamientos Curriculares.Idiomas Extranjeros.

Colombia. (1999). Ministerio de Educacin Nacional. Nuevo Examen de estadopara el ingreso a la educacin superior. Cambios para el siglo XXI. Idiomas.

Cmara de Comercio del Huila. (2001). El futuro Est Sealado. Corporacin Huila

Futuro. Prospectiva del Huila y Formulacin de la Vision al ao 2020. Neiva:Editora Surcolombiana S.A.

Delgado Santa Gadea, Kenneth. (1996). Evaluacin y Calidad en la Educacin:Nuevos aportes, procesos, y resultados. Santaf de Bogot: Coleccin MesaRedonda Magisterio.

Estvez, Solano, Cayetano (1996). Evaluacin integral pro procesos: unaexperiencia construida desde y en el aula. Santaf de Bogot: Mesa redondaMagisterio.

Jonz, J. (1990) Another turn in the conversation.: What does cloze measure? TESOLQuarterly, 24(1), 61-83.

7/28/2019 gzuniga03

11/11