Item Writing TrainingDiann M. Brady, M.S | May 15, 2018

Agenda slide

1 Welcome and Introductions2 Item Writing Presentation3 Cognitive Level Activity4 Competency Definition Review5 Questions

2

3

• Does the test measure what it is intended to measure?

Validity

• Are test scores consistent, dependable and repeatable?

Reliability

• Are the questions clear and free from sensitive issues?

Fairness

Validity

• Does the test measure what is it intended to measure?~ Certified ScrumMaster® (CSM®) ~ Certified Scrum Product Owner® (CSPO®)

• How is the item linked to the validation documentation?~ Does the item have a reference?~ Does the item match the classification?~ Is the item format appropriate?

4

Reliability• Are the test scores consistent,

dependable, and repeatable?

• What features of items support reliability?~ Are the items clearly written?~ Are there enough items to measure each topic appropriately?~ Are the items distributed appropriately according to the content topics?

5

Fairness

• Why is fairness important? How can it be achieved?~ Are the questions clear and unambiguous?~ Is the wording of the items free from sensitive issues?~ Do the items represent the body of knowledge?

6

Multiple Choice Items (MCSR)How to Write/Review Them

7

• The initial part of the item in which the task is delineated ~ it may be a question or an incomplete statement

TERMINOLOGY

8

• All of the choices in an item• Four options: A, B, C, and D• All options need to respond to the stem

TERMINOLOGY

9

• The correct answer• There can be only one (MCSR)

TERMINOLOGY

10

• The incorrect options• Must be plausible but clearly incorrect• If numeric, must be calculable

TERMINOLOGY

11

Item Writing Terminology

The capital of California is (Stem)

Options (A) Los AngelesDistracters (B) San Diego

(C) San Francisco(D) Sacramento (Key)

12

Item Writing Terminology

What is the capital of California? (Stem)

Options (A) Los AngelesDistracters (B) San Diego

(C) San Francisco(D) Sacramento (Key)

13

Multiple Choice Items (MCSR)Polishing the Items

14

Rules for Writing/Reviewing the Stem • Incomplete statement or direct question• Complete enough to establish the test

taker's task• No longer than is necessary to make the

question clear• Positively-worded stems• Applicable to the job you do

15

Rules for Writing/Reviewing Options

• Distractors that are plausible to individuals with insufficient or limited knowledge

• Options that fit grammatically with the stem

• Options that are parallel in construction (length, format, complexity)

• Options that directly answer the question

16

Rules for Writing/Reviewing Options

What to AVOID• Distracters that may be “keyable” under

any plausible interpretation• Negative options• Repetition of words in stem and options• Addressing the candidate directly~ using

“you”

17

Rules for Writing/Reviewing Options

What to AVOID• Specific determiners ~ always, never,

most, least, best• Overlapping ranges or subsuming options• "None of the above" as an option • “all of the above” as an option

18

2 FUNDAMENTAL RULES OF ITEM WRITING

19

1. Do not confuse people who know the concept

2. Do not give away the answer to people who do not know the concept

Item Writing Supporting Information~ Classification~ Reference

20

Classification

• The topic areas to which the item is written

• Refer to the test blueprint (live link post Test Specifications Meeting)

21

Reference

• The citation to support the item and key (for legal defensibility purposes)

22

References

References for CSM:1. Scrum Guide2. Agile Manifesto3. CSM Learning Objectives

References for CSPO:1. Scrum Guide2. Agile Manifesto3. CSPO Learning Objectives

23

Cognitive Levels

• RECALL

• APPLICATION

• ANALYSIS

24

RECALL

• Remembering or recognizing appropriate terminology, facts, ideas, materials, trends, sequences, methodology, principles, and generalizations

• Question verbs: identify, label, define, who, when, what

25

APPLICATION

• Applying ideas, rules of procedure, methods, formulae, principles, and theories in job-related situations

• Question verbs: Show, demonstrate, modify, solve, use

26

ANALYSIS

• Breaking down material or information into its constituent parts and detecting the relationship of the parts and the way they are organized

• Question verbs: distinguish, differentiate, compare/contrast, why, how, classify, relate

27

Cognitive Level Activity

28

An ordinary object:

A Toothbrush

Recall Questions:• Who uses our object?• What color is our object?

Application Questions:• How is our object used?

Analysis Questions:• What are the features of our object?• What conclusions can be made about someone who uses our object?

29

Cognitive Level Activity

30

An ordinary object:

A Frying pan

Recall Questions:• Who is the primary type of person who would use a frying pan?• What type of material is used to make a frying pan?

Application Questions:• How would you use a frying pan?• In what situation should a frying pan be used versus another instrument?

Analysis Questions:• Compare a copper-bottom frying pan with a non-stick frying pan.• What does the use of a frying pan imply?

31

Moving Beyond the Blank Page

Beginning Steps

• Identify the knowledge specification area • Outline areas to write to in the Test

Specifications• Obtain your reference• Start writing what the candidate should

know, with the key in mind

33

Filling in the details

• Develop distracters that include different series of common misconceptions/mistakes

v Think about common mistakes you have encountered that are absolutely incorrect, but plausible to an entry-level professional

v All numeric distracters must be calculable

34

Next Steps

35

• Test Specifications Meeting June 4th

v Final recommendations used to guide examination development activities (item writing and exam assembly)

• Link for Item Writersv Will be sent after the June meetingv Test Blueprintv Question and answer opportunities

• Scrum Alliance’s Get Involved pagev New opportunities for volunteers

Thank you for attending!

36