1 Producing Your Assessment Question Mark “Software for creating and delivering assessments with powerful reports” Copyright 2000 QuestionMark. All

1

Producing Your Assessment

Question Mark “Software for creating

and delivering assessments with powerful reports”

Copyright 2000 QuestionMark. All rights reserved.

2

Objectives

204/20/23

Compute your passing score

Define your passing score

04/20/23Copyright 2000 QuestionMark. All rights reserved.

At the end of this presentation, you will understand how to:

Beta test

Alpha test

3

Best practices for developing tests


1.Analyze the job 2.Document the test blueprint 3.Survey the blueprint 4.Develop items 5.Alpha test and revise items6.Beta test the exam 7.Select items and set the cut score 8.Go live with the test

4

Costly to revise Inexperienced test writers New test content or program New test design

Why test your test?


5

What does a stepped evaluation process give you?

Verify technical accuracy Ensure implement-ability Test content effectiveness Gauge organizational or industry

acceptance


6

Stepped evaluations for alpha & beta testing


Self evaluation

Expert review

Small group review- alpha test

Large group review – beta test

7

Inside evaluation steps


SelfRough version

Expert Sm.Group Lg.GroupPolished version

High interaction

Low interaction

Content & clarity

Usability & performance

8

What do you want to know?


Reaction ? Customer or trainee satisfaction

Learning ?? Knowledge learnedSkills developedAttitudes changed

Results ?!??! Productivity, quality, profits increased

Certify !!!!!!! Mastery achieved

9

Types of measurements


Reaction ? “Smile” sheet Attitude survey

Learning ?? Self-assessmentSkill/knowledge check

Results ?!??! Training ROI Behavioral impact

Certify !!!!!!! Certification exams Legal defensibilityConfidentiality policies

Low Stakes

High Stake

s

10

Relative scores Absolute scores Compromised scores Failure rates Arbitrary, but not capricious

Scoring terminology


11

Contrasting groups method

Experts set score arbitrarily

Experts’ consensus opinion

Ways to set passing scores


12

Contrasting Group method Expert performers Poor performers Consider the overlap Pick a score in the zone of acceptance

Low stakes score setting


13

Contrasting group sample


Score Hi Group Low Group

24+ xxxxxx

24 x

23 x

22 x

21 x x

20 x

19 xxx

<+=18 xxxxx

Zone of Acceptance

14

Use your judgment to assign points to test items

Estimate a percentage to pass Variations on estimating a percentage Compromise on a passing score Some practical hints

High stakes score setting


15

Identify what it would take to “just” pass

Rate & categorize characteristics Set required number of characteristics

Compute minimum points

Assigning points


16

Agree on “borderline” characteristics

Estimate the percentage of candidates

Compute average

Estimating a percentage


17

Supplying correct answers Omitting result data Working individually

Common variations on estimating


18

Both relative and absolute Define acceptable standards Compute curves Compute cut score

Compromise on your passing score


19

Minimum failure rate Maximum failure rate Minimum passing point Maximum passing point

Acceptable standards for the compromise method


20

50

40

30

20

10

0<45 50 55 60 65 70

Compromise curve


%Correct Score

% Fail

20% acceptable failure rate

Minimum fail score

Maximum passing score

21

Multiple judges Provide correct answers to items Provide result data Collect relevant demographics Avoid abstraction Perform additional surveying

Common themes & practical hints


22

Thank you for your attention

Thank you for your time and attention. We hope you have enjoyed learning about how to write test blueprints and test items.

We at Question Mark strive to improve our products and services. To let us know how you felt about this presentation, visit our North American home page and complete the Learning Center Survey: http://www.questionmark.com/home.htm#Surveys

Question Mark provides numerous presentations to help our customers, and prospective customers, successfully deploy computerized assessments. We provide public “teleseminars” that use a combination of conference calls and web technology to discuss computerized assessment issues. You can register for a public seminar by going to: www.questionmark.com/seminars/index.asp

For other material that you might find useful, please visit our Learning Center at: www.questionmark.com/learningcenter/

Thank you once again. This is the end of the presentation.


Documents

1 Producing Your Assessment Question Mark “Software for creating and delivering assessments with powerful reports” Copyright 2000 QuestionMark. All