Tips for Providing Fair, Accessible, and Efficient Online Assessments

Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Tips for Providing Fair, Accessible, and Efficient

Online AssessmentsJeff Place, Questionmark

email: [email protected] www.questionmark.com

mailto:[email protected]

http://www.questionmark.com/


Role of assessments – IN Learning and OF Learning

Copyright © 1995-2009 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are

acknowledged.

Types of Assessments

… IN learning

Formative Assessments◦ Strengthen memory recall◦ Examples:

Quizzes

Diagnostic Assessments◦ Route you to learning◦ Learn what you know◦ Learn what you need to

know◦ Examples:

Learn Quizzes Tests

… OF Learning

Summative Assessments◦ Measure knowledge, skills

and competencies◦ Examples:

Exams and Certifications Compliance tests Pre-employment tests

Reaction Assessments◦ How did you feel

about..?◦ Examples:

Course evaluations Level 1s

Slide 3


acknowledged.Slide 4

Assessments IN Learning

Results Database – Aggregating results for all interactionsNeeds Analysis, Opinion/attitude surveys, Transcripts, Skills Gap Analysis, Certification Results

AppropriateLearningEvents

Pre-Course

Formative

Post-Course






Pre-Course

Formative

Post-Course

Purpose:• Create intrigue• Route to course• Establish benchmarks






Pre-Course

Formative

Post-CoursePurpose:• Stimulate and strengthen

memory• Correct misconceptions






Pre-Course

Formative

Post-Course

Purpose:• Determine if

knowledge transfer occurred

• Reduce Forgetting Curve


acknowledged.

Learning Curve

Slide 8

Know

ledg

e/Sk

ill

Time

Trai

ning Trai

ning

Ouch! The Forgetting Curve


acknowledged.

Questions IN the Learning Process

Slide 9

Know

ledg

e/Sk

ill

Time

Redu

ce F

orge

tting

Redu

ce F

orge

tting

Crea

tes

Intri

gue

Stre

ngth

ens M

emor

y

Questions….


acknowledged.

Assessments

… IN learning

Formative Assessments◦ Strengthen memory recall◦ Examples:

Quizzes

Diagnostic Assessments◦ Route you to learning◦ Learn what you know◦ Learn what you need to

know◦ Examples:

Learn Quizzes Tests

… OF Learning

Summative Assessments◦ Measure knowledge, skills

and competencies◦ Examples:

Exams and Certifications Compliance tests Pre-employment tests

Reaction Assessments◦ How did you feel

about..?◦ Examples:

Course evaluations Level 1s

Slide 10



Assessments After LearningKirkpatrick’s Levels of assessment

Level 1 - ReactionWas the environment suitable for learning?

Level 2 - KnowledgeDid they learn anything

Level 3 - BehaviorKSA being used on the job?

Level 4 - ResultsWas it worth it?

Assessment

Assessment

Assessment


acknowledged.

Assessments must: Provide the Right Stimulus to Gather Data

...then Provide the Right Analysis

with Right Presentationto the Right Person at the Right Time

Be Valid and Reliable Be Efficient and Cost Effective


acknowledged.

Tip: Accommodating Test Takers with

Disabilities Pay particular attention to the authoring and delivery of assessments


acknowledged.

People with disabilities deserve to be assessed fairly and sometimes Accommodations must be made

Following good practice in web accessibility not only helps those with disabilities but many others◦ Workers not used to computers can use touch

screens◦ Temporary injuries incapacitating participants

Increasingly important for government, academic and corporations to empower and harness the intellect and energy of those with disabilities

Accommodations and Accessibility


acknowledged.

International definitions of blindness and low vision / partial sightedness varies, but there are tens of 1,000s of people of working age impacted

Accommodations include:◦ Screens to be compatible

with text to speech screen readers

◦ Allow participant to: Select large text Change colors and contrast

Blindness and Partially Sighted


acknowledged.

Many types of color “blindness” People see colors differently to

the norm ~7% of men and ~0.5% of

women are color blind Accommodations include:

◦ Not using only color to distinguish information or navigation

◦ Not relying on color within the question or choices to get the right answer

Color Blindness

Colors of rainbow as viewed by a person with no color vision deficiencies.

Colors of rainbow as viewed by a person with protanopia.

Colors of rainbow as viewed by a person with deuteranopia.

Colors of rainbow as viewed by a person with tritanopia.


acknowledged.

Dyslexia challenges people to read and spell ~10% of people are impacted; 4% severely Most sufferers muddle through Accommodations include:

◦ Allow participant to: Select larger text Change colors and contrast Use (text to speech) screen readers Have extra time

◦ Not using only color to distinguish information or navigation

◦ Not relying on color within the question or choices to get the right answer

Dyslexia


acknowledged.

Some disabilities (multiple sclerosis, paralysis, muscle and joint problems) prevent use of mouse/keyboard and require special input devices

Number of people affected:◦ Quadriplegia (often caused by

vehicle or sports injury) ~100,000 people in US ~25,000 people in UK

Accommodations include:◦ Ensure assessment system

can be used by keyboard (not mouse) as such special input devices emulate keyboard

Motor Disabilities


acknowledged.

W3C (WCAG)http://www.w3.org/TR/WCAG10/ (WCAG 1, 1999)http://www.w3.org/TR/WCAG20/ (WCAG 2, Dec 2008)

Section 508 from the US government◦ Main document (web applications):

http://www.access-board.gov/sec508/guide/1194.22.htm

◦ Secondary document (software): http://www.access-board.gov/sec508/guide/1194.21.htm

◦ Section 508 similar to WCAG but not identicalwww.jimthatcher.com/sidebyside.htm

Guidance and Standards

http://www.w3.org/TR/WCAG10/

http://www.w3.org/TR/WCAG20/





http://www.jimthatcher.com/sidebyside.htm


acknowledged.

Optimize delivery for screen-reader compatibility

Provide text-sizing and contrast controls

Enable navigation without mouse Keyboard shortcuts Alternate pointing devices

Accessibility and Accommodations


acknowledged.

Text-to-Speech Screen readers Market leader text-to-speech screen reader

is JAWS (75% of market) www.freedomscientific.com/jaws-hq.asp

◦ Costs about US$1,000/seat◦ Demo version available at

http://www.freedomscientific.com/downloads/jaws/jaws-downloads.asp allows 40 mins use before restarting

http://www.freedomscientific.com/jaws-hq.asp

http://www.freedomscientific.com/downloads/jaws/jaws-downloads.asp

http://www.freedomscientific.com/downloads/jaws/jaws-downloads.asp


acknowledged.

Allowing Extra Time


acknowledged.

Provide Accessibility Review

Workflow feature ensure review of accessibility issues


acknowledged.

Provide scribes and proctors to help those with disabilities answer assessments


acknowledged.

Expand “Clickable” Regions

Select by clickingform field or choice text region


acknowledged.

Text Sizing Controls


acknowledged.

Contrast controls


acknowledged.

Tip: Consider how you plan to score a test or exam before

you begin writingRecommended reading:

Criterion Referenced Test Development: Technical and Legal Guidelines for Corporate Training

Sharon A. Shrock and William C. Coscarelli

Slide 28


acknowledged.

CRT vs. NRT

Criterion ReferencedHow well did the

participant master the content and/or skill

Typically used for certification tests,

licensing exams, end of course exams, etc.

Norm ReferencedHow well did the

participant perform compared to others in a

group

Typically used for standardized college

placement tests, ‘competition’ tests (e.g. advancement test for

limited number of openings)


acknowledged.

Informed Judgment◦ Poll SMEs to determine criteria for passing (no. of correct

answers, no. of attempts)◦ Not recommended for high-stakes assessments

Angoff Method◦ Poll SMEs◦ Consider marginal candidates and the probability of getting specific

questions right (0-100%)◦ Make a mathematical calculation to determine cut score

Borderline Group Method◦ Candidates are interviewed prior to an exam and borderline cases are

identified◦ After the exam, the scores of borderline candidates can be used to

determine the cut score Contrasting Group

◦ Compare SME to others

Setting cut scores for Criterion Referenced Tests

Slide 30

Angoff MethodWhat is the expected possibility of a marginal student getting question right?

Expert A

Expert B

Expert C Agreement

Question 1 70% 75% 60% 68%Question 2 55% 65% 60% 60%Question 3 60% 65% 60% 62%Question 4 70% 65% 60% 65%Question 5 25% 75% 80% 60%Question 6 60% 75% 75% 70%Question 7 55% 45% 80% 60%Question 8 80% 80% 85% 82%Question 9 85% 75% 75% 78%Question 10 70% 65% 70% 78%

63% 69% 71% 67%

Slide 31


acknowledged.

Norm Reference Testing (NRT): The “Normal Curve”

Slide 32


acknowledged.

NRT: Test designed to “spread” and rank participants within a group; so to easily identify top performers

Slide 33

Top 10% of scorers



Setting a Pass/Fail score?

Norm ReferencedIf used to determine a ‘cut’ or ‘pass/fail’,

the pass/fail score is determined after test

Criterion ReferencedPass/fail score determined before test

175 people

Only 50 peoplecan pass

175 people

1st group

55 pass

2nd group

35 pass

3rd group

71 pass


acknowledged.

Tip: Outline a “Test Blue Print” before you begin writing your

test or exam questionsEnsure your test meets your objectives

Improve efficiency of test writing

Slide 35


acknowledged.

Assessment Blue Print A detailed, written plan for an assessment Typically includes

descriptions of the purpose and target audience the topics and performance areas it will cover the types of items and number to be written for each

topic or performance area scoring timing the administration method (paper, browser, CD) desired reports, distribution of results decisions made based on results


acknowledged.

Blueprint for Nursing Licensure Exam


Tip: Consider options for gathering content from

globally dispersed SMEs


acknowledged.

Using a distributed authoring and centralized item management with globally dispersed authors

Internet


acknowledged.

Can you provide multilingual interfaces to SMEs?


acknowledged.

Tip: When delivering localized/multilingual

assessments, consider the workflow of your translation

process.


acknowledged.

Efficient Models for Translating Items, Quizzes and Tests

Translated Item

Base Language

Item

English

FrenchGerma

nDutch

Translated Test

Base Language

Test

English

FrenchGerma

nDutch


acknowledged.

Interactive Translations

3rd-party localisation using industry-standard format

Two Translation Methods

Export to XLIFF (XML)

Translate using 3rd

Party tool or service

Import from XLIFF (XML)

Translate withinan assessment authoring tool

Define translation

project

On-The-Fly


acknowledged.

Multilingual Item Banking


acknowledged.

Example of localization software: Trados Studio

Slide 45


Tip: Align delivery environment to the stakes of an assessment

Slide 46


acknowledged.

Higher Stakes

High

Medium

Low

Medium Stakes

Low Stakes

Low/High to High/High Stakes Tests

Slide 47

What’s at Stake?

Life and Limb

Promotion & Jobs & Legal Concern

Educational Exams

Tests

Elearning & Surveys


acknowledged.

Issues to Consider

Fraud Triangle Threats

Rationalization Opportunity Motivation

Impersonation Content Theft Cheating

http://blog.eric.info/wp-content/uploads/2009/11/image3.png





High stakes: Secure Browser Mitigates risks of:

◦ Cheating◦ Items being leaked/shared with

future test participants

How it works…◦ No menu or icons◦ Cannot switch to another

program◦ Disables printing◦ Can be customized to permit

certain functions, i.e. printing feedback page Secure Browser

Regular Browser



Other risks to mitigate Power Outage? Loss of Internet connection?

Ensure there is an “auto save” or “save as you go” mechanism:◦Answers auto-saved as the

participant takes test◦In the event of technical failure,

participants can resume an assessment where they left off


acknowledged.

Tip: True “Blended” delivery of assessments should

improve more than your delivery options

Blended delivery should also yield more efficiency in authoring, administration and reporting



Secure Browser

Blended Delivery

Web Browser Disconnected Delivery

LMS

Integration Windows / CDTest Centers

Printing andScanning

Mobile Devices

Social Networking Platforms Embedded in Wikis, Blogs, and Web pages


What is “disconnected delivery?

Slide 53


acknowledged.

Disconnected delivery – content downloaded before event, results synched after test

Participants

Thumb Drive

Assessment Content

AssessmentsResults

CD

SD Card

Synch with Central Assessment Server

Synch with Central Assessment Server


acknowledged.

Participant Answers on

Paper

Printing and Scanning – An option when workstations, Internet not available

Slide 55

Print Test Form

Assessment Preparation

Paper Scanned

Paper

Responses Uploaded to Central Server for

Scoring and Reporting


What is “auto sensing”?

Slide 56


acknowledged.

Auto Sense Device/Browser

Senses Device/Brows

er

Server packages the Assessment for Device/Browser/iFrame

Sends HTML

On-the-fly auto-sizing of skin and controls

http://images.google.com/imgres?imgurl=http://nexus404.com/Blog/wp-content/uploads2/2008/04/new-toshiba-satellite-pro-business-laptops-l300-l300d-m300-u300-l350.jpg&imgrefurl=http://nexus404.com/Blog/2008/04/24/toshiba-satellite-pro-business-laptop-range-expands-new-l300-l300d-m300-u300-l350-models/&usg=__yok_KR9K3b0j37XNli2vcyMbdNY=&h=384&w=400&sz=41&hl=en&start=9&tbnid=XEWQP6z97GqN7M:&tbnh=119&tbnw=124&prev=/images?q=laptops&gbv=2&hl=en


acknowledged.

Inexpensive, very portable Internet-connected mobile devices


acknowledged.

The benefits?

Slide 59

Deliver to any device or context

Schedule once

Author once

Single results set






acknowledged.

Investigate which tools support development/delivery for mobile devices

Review W3C’s Mobile Web Best Practices: Consider potential device limitations – e.g. screen size,

plug-in support (e.g. Flash) Avoid pop-ups, frames, image maps and use of tables

for formatting Ensure simple, consistent navigation See full best practices guide at:

http://www.w3.org/TR/mobile-bp/ Consider the benefits of “auto sensing” of

participant’s device/browser

Delivering assessments to mobile devices?

Slide 60

http://www.w3.org/TR/mobile-bp/


acknowledged.

Tip: Make effective use of corrective feedback

Enhances learningReduces forgetting

Slide 61


acknowledged.

What is feedback? Stimulus is provided to a participant

according to their responses within an assessment.

Feedback can be provided…◦ Based on how a participant scores and/or

answers… A single question A group or “topic” of questions

◦ Or, based on how a participant scores on an assessment

Slide 62


acknowledged.

Simple corrective question feedback

Slide 63

Question and

participant’s answer

Corrective feedback

given


acknowledged.

Feedback based topics of questions

Slide 64

Corrective feedback points participant to appropriate

learning resources

“Topics” of questions

answered in assessment

Score on

topic


acknowledged.

General guidelines on feedback… Improving performance

Give the participant guidance for improving their skill or knowledge

PrecisionBe precise enough that the participant knows what needs to be improved

Learning resourcesTell the participant how and where to go to acquire the information

Slide 65


acknowledged.

When is feedback useful? Dealing with difficult questions

Helps participant understand, learn from error

Correcting major errorsErrors that demonstrate a fundamental misunderstanding of a topic

Slide 66


acknowledged.

Times you might avoid giving feedback… When security is important

Where doing so may compromise the integrity of the test as a whole

When questions are relatedWhen it may provide clues to answer other questions

Slide 67


acknowledged.

Tip: Use item analysis and other techniques to ensure

valid, reliable measurementsHow do you know it’s a good test?

Slide 68


acknowledged.

Reliability and Validity Assessments must be reliable and valid

◦ Reliable = They work consistently

◦ Valid = They ‘look like the job’

If an assessment is not reliable it cannot be valid


acknowledged.

Reliable = ConsistentNot Valid = Not like the job


acknowledged.

Not Reliable = Not ConsistentNot Valid = Not like the job


acknowledged.

Reliable = ConsistentValid = Like the job


acknowledged.

Types of Validity Face Validity

◦ Do test-takers think the test measures what it’s supposed to measure?

Content Validity◦ Do SMEs agree that the test measures what it’s supposed

to measure? ◦ Based on review of objectives and items as specified in

assessment blueprint Concurrent Validity

◦ Does the test accurately classify known experts and known novices?

◦ Correlation calculation


acknowledged.

Assessment analytics for reliabilityType of analysis

What you aim to learn…

Analysis of Items (Questions)

•Is the question too easy? Too difficult?•How many results for this question?•Are there potential problems with this question (such as ambiguous wording)?•Has this question been over-exposed?

Test analysis Is the test reliably measuring what is was designed to measure? Are these assessment results “defensible” (will they withstand an appeal or legal challenge?)

Slide 74

Slide 75

Item Statistics- ChoicesChoices

# of Responden

ts%

Selected

Difficulty(also p-value)

0 = Too difficult1 = Too easy

Participant Mean

OutcomeDiscrimin

-ation

Outcome Correlatio

n

A

B

C

D

Slide 76

Item Statistics- # of respondentsChoices

# of Responden

ts%

Selected



Participant Mean

OutcomeDiscrimin

-ation

Outcome Correlatio

n

A 21

B 52

C 323

D 112

Slide 77

Item Statistics- % of respondentsChoices

# of Responden

ts%

Selected



Participant Mean

OutcomeDiscrimin

-ation

Outcome Correlatio

n

A 21 4%

B 52 10%

C 323 64%

D 112 22%

Slide 78

Item Statistics- Difficulty (also P-value)Choices

# of Responden

ts%

Selected



Participant Mean

OutcomeDiscrimin

-ation

Outcome Correlatio

n

A 21 4%

B 52 10%

C 323 64% .64

D 112 22%

Slide 79

Item Statistics- Participant MeanChoices

# of Responden

ts%

Selected



Participant Mean

OutcomeDiscrimin

-ation

Outcome Correlatio

n

A 21 4% 36%

B 52 10% 47%

C 323 64% .64 66%

D 112 22% 52%

Slide 80

Item Statistics- Outcome DiscriminationChoices

# of Responden

ts%

Selected



Participant Mean

OutcomeDiscrimin

-ation

Outcome Correlatio

n

A 21 4% 36%

B 52 10% 47%

C 323 64% .64 66% 0.49

D 112 22% 52%

Slide 81

Item Statistics- Outcome CorrelationChoices

# of Responden

ts%

Selected



Participant Mean

OutcomeDiscrimin

-ation

Outcome Correlatio

n

A 21 4% 36% -0.32

B 52 10% 47% -0.22

C 323 64% .64 66% 0.49 0.24

D 112 22% 52% -0.15



Other question and item statistics… Number of participants who missed a question

◦ Are an unexpected number of participants are missing a question (i.e., not responding)? Could be a problem with the question.

The number of responses for each question◦ The more results for a question, the more stable the

information Better assurance that the statistics generated are trustable

# Times item has been presented◦ The more times that a question is presented, the more

the question is “exposed.” ◦ Tracking item exposure vital to retiring questions to avoid

questions from being “over exposed”


acknowledged.

Item Analysis - Summary Items with zero or negative discrimination

are suspect Items with zero or negative correlation are

suspect Pay attention to distractor pattern

◦ Distractors with low p-values are probably unrealistic and are therefore discounted

Test Reliability: Cronbach’s Alpha Internal consistency test

reliability: A measure of how well the assessment questions relate to one another to measure the same topic

Crucial information for legal defensibility – is the test reliably measuring what is was designed to measure?

Speaks to the quality of assessment questions: Are they of high statistical quality?

)(1 2

22

T

pT

SSS

kk



Learn about assessments… Questionmark White Papers

◦ Assessments through the Learning Process◦ Delivering Assessments Safely and Securely ◦ Using Questionmark Perception to Help Ensure Compliance

with Regulatory Principles for e-Assessment ◦ Measuring Learning Results◦ Providing Feedback to Learners◦ Using Online Assessment for Compliance◦ The Learning Benefits of Questions

◦ www.questionmark.com/whitepapers

Questionmark Blog◦ blog.questionmark.com

http://www.questionmark.com/whitepapers

http://blog.questionmark.com/


What questions do you have?Jeff Place, Questionmark





Thank you!Jeff Place, Questionmark




Documents

Tips for Providing Fair, Accessible, and Efficient Online Assessments