Testing Writing

Testing for language teachers by Arthur Hughes

Reporter: Tumana, Wenlie Jean

The best way to test people’s

writing ability is to get them to write

1. We have to set writing tasks that are properly representative of the population of tasks that we should expect the students to be able to perform.

2. The tasks should elicit valid samples of writing (i.e. which truly represent the students’ ability).

3. It is essential that the samples of writing can and will be scored validly and reliably.

3 parts in testing problem

1. Representative tasks(i) Specify all possible content

–We have to be clear at the outset just what these tasks are that they should be able to perform. These should be identified in the test specifications.

- From the standpoint of content validity, the ideal test would be one which required candidates to perform all the relevant potential writing tasks. The total score obtained on that test (the

sum of the scores on each of the different tasks) would be our best estimate of a candidate’s ability.

(ii) Include a representative sample of the specified content

So if we aren’t able to include every tasks (that usually actually happens) and happen to choose a task or tasks that the students are good or bad at, then the outcome will be very different.

The more tasks (reasonable number of tasks) that we set, the more representative of a candidate’s ability will be the totality of the samples we obtain.

Testing problem1. Representative tasks

(i) Specify all possible content

(ii) Include a representative sample

of the specified content

2. Elicit a valid sample of writing ability(i) Set as many separate tasks as is feasible

–This requirement is closely related to the need to include a representative sample of the specified content.

(ii) Test only writing ability, and nothing else

– In language testing we are not normally interested in knowing whether students are creative, imaginative, or even intelligent, have wide general knowledge, or have good reasons for the opinions they have. Therefore, for the sake of validity, we should not set tasks which measure these abilities.

1. Write the conversation you have with a friend about the holiday you plan to have together.

2. You spend a year abroad. While you are there, you are asked to talk to a group of young people about life in your country. Write down what you would say to them.

3. ‘Envy is the sin which most harms the sinner.’ Discuss.

4. Discuss the advantages and disadvantages of being born into a wealthy family.

Another ability that at times interferes with the accurate measurement of writing ability is that of reading. While it is perfectly acceptable to expect the candidate to be able to read simple instructions, we have to ensure that these can be fully understood by everyone whose ability is of sufficiently high standard.

Testing problem2. Elicit a valid sample of writing ability

(i) Set as many separate tasks as is feasible

(ii) Test only writing ability, nothing else

(iii) Restrict candidates

There are so many significantly different ways of developing a response to the stimulus. Writing tasks should be well defined: candidates should know just what is required of them, and they should not be allowed to go too far.

- A useful devise is to provide information in the form of notes or pictures.

Testing problem (iii) Restrict candidates

Full sentences avoided

Tasks should not only fit well with the specifications, but they should also be made as authentic as possible. When thinking of authenticity, it is important to take into account the nature of the candidates and their relationship for whom the task requires to write.

Testing problem3. Ensure valid and reliable scoring

(i) Set tasks which can be reliably scored

Set as many tasks as possible

Restrict candidates

Ensure long enough samples

Give no choice of tasks

Suggestions

(i) Set tasks which can be reliably scored

Suggestions:

1. Set as many tasks as possible

2. Restrict candidates. The greater the restriction imposed on the candidates, the more directly comparable will be the performance of different candidates.

3. Give no choice of tasks. Making them perform all tasks also makes comparisons between candidates easier.

4. Ensure long enough samples. The samples of writing that are elicited have to be long enough for judgements to be made reliably.

Testing problem(ii) Create appropiate scales for scoring

Two approaches

Holistic scoring

•A single scoring to a piece of writing

Analytic scoring

•A method that requires a separate score for each aspect of a task

Holistic scoring

– has the advantage of being very rapid. This means that it is possible for each piece of work to be scored more than once because this is also necessary. Not every scoring system will give equally valid and reliable

results in every situation. The system has to be appropriate to the level of the candidates and the purpose of the test.

Testers have to be prepared to modify existing scales to suit their own purposes.

The purpose is purely to measure proficiency, regardless of how it has been achieved.

Analytic scoring

– Method of scoring which require a separate score for each of a number of aspects of a task.

Advantages of analytical scoring:

1. It disposes of the problem of uneven development of subskills in individuals.

2. Scores are compelled to consider aspects of performance which they might otherwise ignore.

3. The very fact that the scorer has to give a number of scores will tend to make the scoring more reliable.

Disadvantages:

1. The main disadvantage of the analytic method is the time that it takes. Even with practice, scoring will take longer than with the holistic method.

2. Concentration on the different aspects may divert attention from the overall effect of the piece of writing.

What will we choose

between holistic and

analytic scoring?

The choice between holistic and analytic scoring depends in part on the purpose of the testing. If diagnostic information is required directly from the ratings given, then analytic scoring is essential.

The choice also depends on the circumstances of scoring. If it is being carried out by a small, well-knit group at a single site, then holistic scoring, which is likely to be more economical of time, may be the most appropriate.

But if scoring is being conducted by a heterogeneous, possibly less well trained group, or in a number of different places analytic scoring is probably called for.

Testing problem(iii) Calibrate the scale to be used

Collect samples and members of the testing team cover the full range of the scales

(iv) Select and train scorers

Trainee scores should be native speakers, be sensitive to the languages, have had experience of teaching writing.

(iv) Follow acceptable scoring procedures

Once the test is completed, a search should be made to identify ‘benchmark’ scripts that typify key levels of ability on each writing task.

Copies of these should then be presented to the scorers for an initial scoring.

Only when there is agreement on these benchmark scripts should scoring begin.

Testing problem(v) Follow acceptable scoring procedures

If the differences are small the two scores can be averaged but if they are big senior members will decide the score. Two o more scorers to give a score independentlyA senior member of the them identify if there are discrepancies

Multiple scoring should ensure scorer reliability

It is important that scoring should take place in a quiet, well-lit environment.

Scorers should not be allowed to become too tired.

While holistic scoring can be very rapid, it is nevertheless extremely demanding if concentration is maintained.

Testing problem4. Feedback

In some situations feedback is very useful to the students

The content that appears in the feedback can be decided during calibration