1

Assessment Professional Learning

Module 2: Activity 2-4BDesigning Better Written Tests

2

Assessment OF learning:

Assessment OF learning occurs when teachers use evidence of student learning to make judgements on student achievement against goals and standards.

It is usually formal, frequently occurring at the end of units of work where it sums up student achievement at a particular point in time. It is often organised around themes or major projects and judgements may be based on student performance on multi-domain assessment tasks. It has a summative use, showing how students are progressing against the Standards, and a formative use, providing evidence to inform long term planning.

3

The Teacher’s Role:

• is to construct valid and reliable assessment tasks - including written tests - that fairly assess student achievement;

• to moderate with colleagues to ensure fair and consistent evidence of learning;

• to modify the learning and teaching program in light of the assessment evidence.

4

Assessment OF Learning

• Can involve extended integrative multi-domain tasks that are authentic, rich and focus on complex learning across domains.

• Can be more focussed, shorter tasks possibly completed under exam conditions, simple tests conventional or “live” performances.

• This presentation focuses ONLY on aspects of constructing better quality written tests.

5

Designing Conventional

Written Tests

6

How would you feel if you were told there will be a test or exam: right NOW?

30 second think:

7

Rule for designing pencil and paper test items:

The easier and quicker to design, the harder and longer to “mark” (assess)

(and vice versa).

8

Written Tests Open/closed book; timed; individual

• Multiple choice items

• Matching items

• True-False questions

• Completion items

• Short answer questions

• Extended response

or essay questions

Select a response

Supply or Construct a response

9

• Venn diagrams, concept maps, drawings, etc• Problem-solving, calculations,

(closed/open)• Creative/lateral thinking responses

Teacher-designed tests tend to be predominantly low order thinking

(in Bloom terms).

Construct a response

10

“Objective” test items

• Objective test items are assessment tasks for which there is general agreement about the “right” answer and a mark to be given to each item.

• However, in another sense, there is no such thing as a completely objective test item. Subjective judgements are involved in constructing the items and selecting them for inclusion in the test.

11

Advantages of objective tests

They allow a wide sampling across a lot of content, because you can usually ask more questions in a given time.

They generally give reliable marks luck being less important because of the large number of questions.

They can be efficiently marked (even by computer). Marking is quicker and easier than essays & projects.

Assessment doesn’t depend on the assessor. On a good objective test, it is clear to all assessors what the correct answers are.

12

Disadvantages of objective tests

They often provide an indirect assessment of learning (a ‘proxy’ for the real thing).

There are many worthwhile Standards and learning outcomes that cannot be assessed this way.

Constructing objective test items is very time-consuming.

Writing good objective items requires high level skills.

Some students don’t handle these questions well.

They can encourage guessing (but not successfully

if the test contains many questions).

13

Multiple Choice Items

The longest river in the world is theA Amazon.B Murray-Darling.C* Nile.D Mississippi.

• The stem should be complete, simple & meaningful.

• Each choice should state one idea, and plausibly flow from the stem.

• Put choices in sequence (especially numbers).

• Negative words should be avoided or capitalised.

• “All of the above”, “two or more of the above”, and “none of the above”, should not be used.

• Choices should be similar in length and detail.

14

Common errors in the construction of multiple-choice items

Teachers may legitimately disagree about the best answer (sometimes more than one is correct).

The level of reading skill required can be too high. Use of “textbook language” encourages rote learning. The stem contains irrelevant material. Negative terms are used unnecessarily, and are not emphasised. Words are repeated in each option when they would be better in

the stem. The question provides clues to the correct answer:

e.g. by the grammar.

15

Spot the design flaw:

Who were the first explorers of Australia?A. the Dutch.B. Dampier.C. the French.D. British.

16

Design flaws:

• Dampier is singular (question implies plural)

• “British” has no article (can’t be easily read with the stem)

• brings it down to choice of only two• options not in alphabetical order

17

Design flaw?

The Prime Minister of Australia isA. elected directly by the electorate.

B. elected by the Members of Parliament in the governing party or parties.

C. elected by the Parliament.

D. appointed by the Governor-General.

18

Flaw:This item has more than one correct answer

The Prime Minister of Australia isA. elected directly by the electorate.

B. elected by the Members of Parliament in the governing party or parties.

C. elected by the Parliament.

D. appointed by the Governor-General.

• Answers B and D

19

Design flaw?

Newton’s third law states thatA. when a body is acted on by a force, the body will move in

the direction of the force.

B. the work done by a force is the product of the force and the distance moved by the body.

C. for every action there is an equal and opposite reaction.

D. curriculum only changes if acted upon by a large force of assessment.

20

Flaw 1: Use of “textbook language” that encourages rote learning, rather than making meaning.Flaw 2: a nonsense distractor is easy to detect

(it is Sherborne’s law of curriculum motion).Newton’s third law states that

A. when a body is acted on by a force, the body will move in the direction of the force.

B. the work done by a force is the product of the force and the distance moved by the body.

C. for every action there is an equal and opposite reaction.

D. curriculum only changes if acted upon by a large force of assessment.

21

Design flaw?

Which of the following types of test item best measures outcomes at the level of recall?

A. Supply-type items.

B. Selection-type items.

C. Matching items.

D. Multiple-choice items.

22

Flaw: Overlapping distracters - more than one category: B, C or D

Which of the following types of test item best measures outcomes at the level of recall?

A. Supply-type items.

B. Selection-type items.

C. Matching items.

D. Multiple-choice items.

23

True-False items

Advantages Easy to write; quickly constructed Quick to administer Can reduce student anxiety at the start of a test

Disadvantages Hard to write worthwhile statements that are

unequivocally true or unequivocally false Susceptible to guessing.

24

Designing True-False Items

Sulfuric acid on a metal gives off hydrogen gas.

True* or False

• Only useful for facts, cause/effect, definitions, but not for opinions.• Keep language simple & clear.• Avoid negatives and qualifiers like always, never, some, sometimes, may, often.• Mix T/F answers, but not in a recognisable pattern.• T/F questions do encourage guessing.

25

Design Flaw?

John Marsden is Australia’s greatest author of fiction for young adults.

True of False?

26

Flaw:

Even experts might have good reasons for disagreeing about the right answer.

John Marsden is Australia’s greatest author of fiction for young adults.

27

Design Flaw?

Archimedes Principle states that when a body is wholly or partially submerged in a fluid, it is subject to an upthrust equal to the weight of fluid displaced.

28

Flaw:Use of textbook language encourages rote learning.

Archimedes Principle states that when a body is wholly or partially submerged in a fluid, it is subject to an upthrust equal to the weight of fluid displaced.

29

Designing Matching Items

Adelaide NSW

Brisbane Qld

Hobart SA

Melbourne Tas

Perth Vic

WA

• Only useful for facts, cause/effect, dates/events, people/events, symbols/concepts but not for opinions.

• Keep to one type of connection e.g. date/event.

• Keep language simple & clear.

• Make columns of unequal length to minimise guessing.

• Limit to 4-6 matches (keeps them plausible).

• Consistently use alphabetical or numerical order.

30

Completion (Cloze) Items

A verb is a …………… word and a …………… is a naming word.

• Easy to construct but take care to stick to facts not opinion.

• Keep language simple & clear.

• Make sure there is only one possible correct response for each blank.

• Place blanks in the middle or end of the sentence.

• Do not insert too many blanks - removes context.

• Make blanks the same length.

• Avoid ‘a’ or ‘an’ before the blank.

31

Completion and short-answer items

These can be in the form of:

Questions:Who was the first wife of the Prince of Wales?

Incomplete sentences:The Prime Minister of Australia is ______.

Simple computational problems:The mean of 3, 4, 5 and 6 is _______.

Directions asking for information in response to cues:Beside the name of each element, write its valence.

32

Short Answer

• Briefly describe four major functions of the Supreme Court.

• Students must know something to respond (can’t choose a supplied option) & minimises guessing.

• Allows for flexibility in student responses.

• Allows for “it depends” responses.

• Easier to construct than multiple choice questions.

• More difficult to mark - lower ‘inter-rater reliability’ (see Module 5).

• Needs care to keep the item clear and direct.

33

Problems with completion and short-answer items

Can only ask simple recall of facts, or simple computational tasks.

Completion items are often ambiguous and may leave students too much room to wriggle.

34

Design Flaw?

For example:

The capital city of South Australia is _________.

Australia was founded in ___________.

35

Design Flaw: too much “wiggle” roomFor example:

The capital city of South Australia is _________.

full of churches.populated by crazy Port and Crows

fans.

Australia was founded in ___________.the summer holidays.error.1901.

36

Essay items

• Compare and contrast the work of the Supreme Court and the High Court.

Give examples and evidence to support your views.

• Students must know something to respond (can’t select an option) and hence minimises guessing.

• Allows for flexibility and creativity in student responses.

• Allows for “it depends” responses.

• Very easy to construct.

• Much more difficult to mark - low ‘inter-rater reliability’.

• Needs care to keep the item clear and direct.

37

Advantages of Essay items

For many performance objectives, the essay is a direct measure of students’ achievement of the intended learning outcome.

Only the essay items measures students’ ability to communicate ideas in writing.

Essay items require students to supply the answer, not just to recognise it.

(Oosterhof, 1999, pp. 71-72)

38

Limitations of the essay items

Essay items often do not provide an adequate sampling of content.

Essay items are time-consuming to score. The scoring of essays is notoriously unreliable. “Bluffing” can have undue influence on students’

scores.

(Oosterhof, 1999, pp. 72-73)

39

What type of test item type do you perform best on?Do you tend to ask more of these type of questions?

You are now ready to take the online quiz about designing tests and quizzes.Good luck! Note: It is private: no one will know your result except you!