Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark

Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Assessments Through The Learning ProcessOctober 2013Eric Shepherd. Questionmark, [email protected] www.questionmark.com

mailto:[email protected]

http://www.questionmark.com/

Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are

acknowledged.

Thank you

Slide 2


acknowledged.

You have colored cards on your chairs/tables Get one of each color, lettered A, B, C and D Let’s practice holding up cards

During this presentation we’ll be using questions to show you the power of questions

Please answer quickly and don’t be embarrassed – we are here to learn!

Using the cards

Slide 3

A B C D


acknowledged.

Learners generally prefer response cards to simple hand-raising

Response Cards Shown to Improve classroom engagement amount they learn amount they retain

-- Dr. Will Thalheimer, citing multiple examples of research on the use of response cards

Research Suggests…

Slide 4

ABC

D


acknowledged.

100 chests of tea

34 chests of tea

342 chests of tea

342 tea bags

How much tea was dumped into Boston Harbor at the Boston Tea Party?

Slide 5

B

C

A

D


acknowledged.

342 chests of tea

Over 46 tons of tea Worth (at the time) over £10,000

£678,827 using the retail price index £866,694 using the GDP deflator £7,422,564 using average earnings £10,025,165 using per capita GDP £25,350,172 using the GDP

Citation: Lawrence H. Officer, "Five Ways to Compute the Relative Value of a UK Pound Amount, 1830 - 2005" MeasuringWorth.Com, 2006.

Answer:

Slide 6

C


acknowledged.

Learning Without Questions

Slide 7

Content


acknowledged.

Learning Benefits of Questions

Slide 8

Content

Questions


acknowledged.

…expose poor presentation skills

…stimulate thinking and keep you awake

…frustrate our color blind friends

All of the above

We are using answer cards to…

Slide 9

B

C

A

D


acknowledged.

Provide Assessment Management Systems To create, deliver and report on assessments For exams, tests, quizzes, and surveys

More than 1,500 customers have purchased Questionmark Corporate, Government, and Education markets

More than 14,000 authoring systems installed

Last month users created 40,000+ questions in Questionmark

Offices in US, UK, Canada, Germany, Belgium

Available in numerous languages

Questionmark

Slide 10


acknowledged.

Worldwide Practical Experience

Slide 11

http://www.dell.com/downloads/global/corporate/imagebank/hq_japan2_300.jpg

http://www.dell.com/downloads/global/corporate/imagebank/hq_japan2_300.jpg


acknowledged.

Alberta Health Services Aluminerie Alouette Inc. Apache Corporation ATB Financial British Columbia Hydro Canada Border Services Agency - CBSA CMLTO - College of Medical Laboratory

Technologists of Ontario Department of National Defence Everest Colleges Canada Inc ExxonMobil Flakeboard Company Limited Fortis BC Inc Grant MacEwan University International Competency Assessment

Board Invesco Canada Ltd. Justice Institute of British Columbia

Loyalist College Manitoba Hydro Memorial University of Newfoundland Moneris Solutions National Association of Pharmacy Regulatory

Authorities Northern Ontario School of Medicine Omnivex Corporation Plains Midstream Safety Codes Council Scotiabank Telus : Telus Supporting Units University of Calgary : Faculty of Veterinary

Medicine - University of Calgary University of Manitoba - Faculty of Medicine University of Ottawa University of Ottawa - Faculty of Medicine University of Saskatchewan

Sample List of Canadian Customers

Slide 12


acknowledged.

Who Else Uses Questionmark?

Slide 13

Financial Services

Pharma / Health

High Tech / Manufacturing

Telcom Energy / Utilities

Public Sector EducationTransportation

Prof. Services

Retail

http://www.t-mobile.com/

http://en.wikipedia.org/wiki/File:Apollo_Group_Logo.png


acknowledged.

…brain dump; tell me everything

…I don’t know; say something interesting

…hurry up; I am only here for the networking

…why so many questions?

What would you like from this session?

Slide 14

B

C

A

D


acknowledged.

Things that Drive What-To-Do

Slide 15

Increase Revenue = € £ $ ¥

Reduce Costs = € £ $ ¥

Reduce Time to Market


acknowledged.

Practical Limitations for Planning

Slide 16

Mon

ey =

€ £

$ ¥

How Long Do We Have?

People or Talent

Resources

Laws

Health &

Safety


acknowledged.

Increase revenue, time-to-market, laws

Increase revenue, reduce costs, safety

Increase revenue, reduce costs, time-to-market

Money, time-to-market, laws and talent

What are the business drivers that drive the use of learning and assessments?

Slide 17

B

C

A

D


acknowledged.

Money, health and safety

Money, laws, health and safety

Money, laws, resources, and safety

Money, laws, time-to-market, health and safety and, access to talent and resources

What are the practical limitations of using assessments?

Slide 18

B

C

A

D

Strategy

Tactics &Plans

Job Roles

Resources

Tasks

Knowledge, Skills,

Abilities

Tactics &Plans

Job Roles

Resources

Tasks

Knowledge, Skills,

Abilities

Job Roles

Resources

Tasks

Knowledge, Skills,

Abilities

Tactics &Plans

Job Roles

Resources

Tasks

Knowledge, Skills,

Abilities

Job Roles

Resources

Tasks

Knowledge, Skills,

Abilities

Slide 19

Strategy

Tactics &Plans

Job Roles

Resources

Tasks

Knowledge, Skills,

Abilities

Tactics &Plans

Job Roles

Resources

Tasks

Knowledge, Skills,

Abilities

Job Roles

Resources

Tasks

Knowledge, Skills,

Abilities

Tactics &Plans

Job Roles

Resources

Tasks

Knowledge, Skills,

Abilities

Job Roles

Resources

Tasks

Knowledge, Skills,

Abilities

Slide 20

Knowledge, Skills and Abilities Required

How To Do It

What Will They Do

What Will They Use

Who Will Do It

What To Do

Slide 21


How To Do It

What Will They Do

What Will They Use

Who Will Do It

What To Do

Competencies

Slide 22 Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are

acknowledged.

Mental Check to Confirm a Competency

“Given What Do What”

• Time• Tools• Materials• Circumstances• Data

Given What

• Do using Competencies

Do What • Very Well• Well• Poorly

Perform

Slide 23 Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are

acknowledged.

Knowledge (a cognitive Skill)•Memory is a

cognitive skills that provides ability to recall knowledge:•“Knowledge

How”•“Knowledge

To”

Above Knowledge Skills• Cognitive skills above recalling knowledge

(Comprehension , Application, Analysis, Synthesis and Evaluation)

Observable Skills - Motor Skills i.e. physical movement/coordination- Inter-personal skills

Attitudes and Behaviors•Motivation•Ability to

achieve goals•Team fit

Ability to Do/Perform“Competency”

Competency is the “Ability To Do”(or “Ability To Perform”) and requires….

Cognitive Skills


acknowledged.

Results Oriented Ability to meet and exceed goals, make difficult decisions, act on

opportunities that create value for the company. Demonstrate ability to exceed stated job responsibilities, take informed

risks, make principled decisions, and faces high-stakes issues head on. Create a work environment where others feel it is safe to take risks.

Customer Results Consistently meets Customer Needs. Provides solutions for

external and internal customers that meet and/or exceeds customer expectations. Identifies opportunities that will benefit the customer and works in partnership with the customer to drive the business forward.

Example - Attitudes and Behaviors

Slide 24

Slide 25

Competencies Required



acknowledged.

How do we know the Knowledge, Skills and Abilities required? Job Task Analysis

Survey people who do the job

MoSCoW analysis Knowledge, Skills and Abilities that we:

Must have Should have Could have Would be nice to have

Slide 26


Assessment


acknowledged.

Slide 27

Skill Survey, Quiz, or Test,

What We Need


Knowledge

Skills

Abilities

What We Have

Knowledge

Skills

Abilities

Assessment


acknowledged.

Slide 28

What We Need


Knowledge

Abilities

What We Have

Knowledge

Skills

Abilities- =Skills

Topics ToTrain

Knowledge

Skills

Abilities


acknowledged.

Helps us survey skills

Helps us know what actions to take

Know what knowledge we have

Determine qualified candidates

Which is the best answer to the question… Why are skill surveys useful?

Slide 29

B

C

A

D


acknowledged.

Learning

Assessment Thru Learning Process

Evidence Accumulated

Windows CDPrepare For Learning

Windows CDIn/As LearningAssessments

Windows CDOf-LearningAssessments

Mentoring InstructorCollaborative & E-learning Work Place

Slide 30


acknowledged.

Learning

Assessment Thru Learning Process




Windows CDOf LearningAssessments

Windows CDJob Task Analysis Windows CDCourse Evaluations

Windows CDNeeds Analysis Mentoring InstructorCollaborative & E-learning Work Place

Windows CDSlow ForgettingCurve

Slide 31


acknowledged.

Learning

Pre-Learning Assessments



Purpose:• Create intrigue• Route to course• Inform instructors• Establish benchmarks




Slide 32


acknowledged.

Learning

In-Learning Assessments



Purpose:• Encourages focus on learning• Provide instant feedback to

instructor • Stimulate and strengthen memory• Correct misconceptions




Slide 33


acknowledged.

In a Perfect World

Slide 34


acknowledged.

Slide 35

In the Actual WorldThey don’t get it All!


acknowledged.

Learning

Post-Learning Assessments



Purpose:• Was learning environment

okay?• Knowledge transfer occurred?• Reduce Forgetting Curve• Certifications




Slide 36


acknowledged.

Learning Curve

Slide 37

Ouch! The Forgetting Curve

Time

Know

ledg

e/Sk

ills

Lear

ning

Ex

perie

nce

Lea

rnin

g E

xper

ienc

e

Lea

rnin

g E

xper

ienc

e

Que

stion

s

Que

stion

s

Purpose:• Content repetition• Memory retrieval practice• Strengthens memory

recall


acknowledged.

In the Actual WorldThey Forget It!

Slide 38


acknowledged.

Slow the Forgetting Curve

Slide 39

Time

Know

ledg

e/Sk

ills

Lear

ning

Ex

perie

nce

Lea

rnin

g E

xper

ienc

e

Lea

rnin

g E

xper

ienc

e

Que

stion

s

Que

stion

s

Que

stion

s

Que

stion

s

Purpose:• Memory retrieval practice• Strengthens memory

recall• Reduce Forgetting Curve


acknowledged.

Questions Thru the Learning Process

Slide 40

Time

Know

ledg

e/Sk

ills

Que

stion

s

Que

stion

s

Que

stion

s

Que

stion

s

Q

uesti

ons

Create Intrigue

Strengthen Memory

Reduce Forgetting


acknowledged.

30 minutes of instruction and 30 minutes of questioning

30 minutes of instruction and 30 minutes of review and study time

Both will produce similar results

Which of the following combination of instructional activities will increase learning and reduce forgetting?

Question borrowed from the work of Dr. Will Thalheimer

Slide 41

B

C

A


acknowledged.

Remember what is learned

Retrieval Practice

Develop misconceptions

Feedback

Forget what’s been learned

Repetitions, Feedback, Retrieval Practice, SpacingDon’t absorb everything

Repetitions

Attention wanders

Pre-questions; directs attention

Why Don’t They Get It?

Slide 42

Dr. Will Thalheimer research:

Learning Benefits from Questions Learning Benefit

Min Max

Pre-questions (to focus attention) 5% 40%

RepetitionsInitial

Subsequent30%15%

110%40%

Feedback 15% 50%

Retrieval Practice 30% 100%

Questions Spaced Over Time 5% 40%

100% 380%

Slide 43


acknowledged.

30 minutes of instruction and 30 minutes of questioning

30 minutes of instruction and 30 minutes of review and study time

Both will produce similar results

Which of the following combination of instructional activities will increase learning and reduce forgetting?

Question borrowed from the work of Dr. Will Thalheimer

Slide 44

B

C

A


acknowledged.

Tests and Exams That Measure

Slide 45


acknowledged.

Regulatory Governments Health and Safety Examples

Driving Tests Pilots Exams Doctor Certification

Non-Regulatory, certification Vendor Specific

Microsoft Oracle SAP

Vendor Neutral ECDL/ICDL

Tests and Exams That Measure

Slide 46

Regulatory Compliance Demonstrating compliance to government

Health/Safety/Environmental Financial Services


acknowledged.

Norm Referenced(pass/fail determined after test)

Criterion Referenced(pass/fail score determined before test)

Setting a Pass/Fail score?

Slide 47

250 people

Only 50 peoplecan pass

250 people

1st group ofpeople

65 pass

2nd group ofpeople45 pass

3rd group ofPeople

75 pass


acknowledged.

How each type of assessment measures the participant…

Norm ReferencedParticipant is compared to others in a group?• Typically used for standardized college placement tests

like the SAT, ‘competition’ tests (e.g. advancement test for limited number of openings)

• Pass/Fail can only be determined after the assessment

Criterion ReferencedParticipant is compared to a benchmark?• Typically used for certification tests, licensing exams, end

of course exams, etc.• Typically Pass/Fail score is determined before the

assessment

Slide 48


acknowledged.

Norm referenced

Criterion referenced

Neither of A nor B

Both A and B

Are Regulatory Compliance tests Norm or Criterion referenced?

B

C

A

D

Slide 49


How Do I Know It’s a Good Test

Slide 50


acknowledged.

Choices# of

Respondents

% Selecte

d

Difficulty(also p-value)

0 = Too difficult

1 = Too easy

Participant Mean

OutcomeDiscrimin

-ation

Outcome Correlatio

n

A

B

C

D

Item Statistics- Choices

Slide 51


acknowledged.

Choices# of

Respondents

% Selecte

d


0 = Too difficult

1 = Too easy

Participant Mean

OutcomeDiscrimin

-ation

Outcome Correlatio

n

A 21

B 52

C 323

D 112

Item Statistics- # of respondents

Slide 52


acknowledged.

Choices# of

Respondents

% Selecte

d


0 = Too difficult

1 = Too easy

Participant Mean

OutcomeDiscrimin

-ation

Outcome Correlatio

n

A 21 4%

B 52 10%

C 323 64%

D 112 22%

Item Statistics- % of respondents

Slide 53


acknowledged.

Choices# of

Respondents

% Selecte

d


0 = Too difficult

1 = Too easy

Participant Mean

OutcomeDiscrimin

-ation

Outcome Correlatio

n

A 21 4%

B 52 10%

C 323 64% .64

D 112 22%

Item Statistics- Difficulty (also P-value)

Slide 54


acknowledged.

Choices# of

Respondents

% Selecte

d


0 = Too difficult

1 = Too easy

Participant Mean

OutcomeDiscrimin

-ation

Outcome Correlatio

n

A 21 4% 36%

B 52 10% 47%

C 323 64% .64 66%

D 112 22% 52%

Item Statistics- Participant Mean

Slide 55


acknowledged.

Choices# of

Respondents

% Selecte

d


0 = Too difficult

1 = Too easy

Participant Mean

OutcomeDiscrimin

-ation

Outcome Correlatio

n

A 21 4% 36%

B 52 10% 47%

C 323 64% .64 66% 0.49

D 112 22% 52%

Item Statistics- Outcome Discrimination

Slide 56


acknowledged.

Choices# of

Respondents

% Selecte

d


0 = Too difficult

1 = Too easy

Participant Mean

OutcomeDiscrimin

-ation

Outcome Correlatio

n

A 21 4% 36% -0.32

B 52 10% 47% -0.22

C 323 64% .64 66% 0.49 0.24

D 112 22% 52% -0.15

Item Statistics- Outcome Correlation

Slide 57

Cronbach’s Alpha

Internal consistency test reliability: A measure of how well the assessment questions relate to one another to measure the same topic

Crucial information for legal defensibility – is the test reliably measuring what is was designed to measure?

Speaks to the quality of assessment questions: Are they of high statistical quality?

)(1 2

22

T

pT

S

SS

k

k

Slide 58


acknowledged.

Skew Mode Kurtosis Mean (arithmetic) Median Mode Standard Deviation Variance Standard Error of measurement Standard Error of the Mean

Other Measures

Slide 59


acknowledged.

Informed Judgment Poll SMEs to determine criteria for passing (no. of correct

answers, no. of attempts) Not recommended for high-stakes assessments

Borderline Group Method Candidates are interviewed prior to an exam and borderline cases are

identified After the exam, the scores of borderline candidates can be used to determine

the cut score Contrasting Group

Compare SME to others Angoff Method

Poll SMEs Consider marginal candidates and the probability of getting specific questions

right (0-100%) Make a mathematical calculation to determine cut score

Setting Cut Scores for Criterion Reference Tests

Slide 60


acknowledged.

SME A SME B SME C Total

Q1 75% 75% 75% 75.00%

Q2 70% 80% 80% 76.67%

Q3 65% 75% 70% 70.00%

Q4 60% 85% 90% 78.33%

Q5 80% 80% 85% 81.67%

Q6 80% 80% 80% 80.00%

Q7 75% 80% 75% 76.67%

Q8 65% 90% 65% 73.33%

Q….. 75% 80% 75% 76.67%

Q50 65% 85% 65% 71.67%

Totals 71% 81% 76% 76%

Angoff MethodWhat is the % chance that a borderline candidate will get question right?

Slide 61


acknowledged.

Emerging Trends and Practices


acknowledged.

Mobile Delivery of Assessments

Observational

assessments

Mobile exam rooms

Deliver to the device

in your hand

Slide 63


acknowledged.

QR = Quick Response Type of barcode Allows more data than

traditional barcode Used in marketing, advertising,

entertainment Increasing use in

e-learning/m-learning

What are QR Codes?

Slide 64

Can encode 4000+ characters of data!


acknowledged.

QR Codes are read by Apps

Slide 65

www.url.com/xxx

www.url.com/xx

x

iPhone•Redlaser•Neoreader• ScanLife

Android•Google Goggles•QR Droid• Scanlife

Windows Mobile•QR Reader•BeeTag•Quickmark

Blackberry• ScanLife•BeeTag•QR Code Scanner


acknowledged.

Books PowerPoint Signage

Enables quick launching of quizzes and course evaluations

Slide 66

Benefits• Ease of access for increased response rates• Demographic data easily captured• Better data accuracy• Easier analysis of survey results by demographic


acknowledged.

Embed in a Wiki

Slide 67

Embedded in a Wiki


acknowledged.

Embedding in Blogs, Portals, Wiki’s

For Learners For Instructors

Focus the learners’ attention Practice memory recall Identify what was learned Provide feedback to assist the

learner to: Correct misconceptions Instill confidence Point to relevant material

Easily use assessments throughout the learning process

Provide seamless user experience Centralised management of

assessment content and results

Slide 68


acknowledged.

QuestionmarkAssessments Management System


acknowledged.

Assessment Management System

Slide 70

Software-as-a-Service

For OnPremise Deployment


acknowledged.

Criterion-Referenced Test Development Sharon A. Shrock & William C. Coscarelli ISBN: 1-890289-09-4

Work-Learning Research white papers Will Thalheimer www.work-learning.com

Tests That Work Odin Westgaard ISBN 0-7879-4596-X

Performance-Based Certification: How to design a valid, defensible, cost-effective program Judith Hale, Ph.D., CPT ISBN: 0787946400

Recommended Books

Slide 71

http://www.work-learning.com/


acknowledged.

Assessments through the Learning Process

Security of Questionmark’s OnDemand Service

Alignment, Impact and Measurement with the A-model

Embedded Assessments: Building Knowledge Checks, Surveys and Other Assessments into Learning Materials

How Questionmark Aids ISO 10667 Compliance

Recommended White Papers

Slide 72

https://www.questionmark.com/whitepapers

https://www.questionmark.com/whitepapers


acknowledged.

Keep in Touch!

Slide 73

blog.questionmark.com

www.questionmark.com/newsletter

twitter.com/questionmark

linkedin.com Look for Questionmark under “Groups” and join

youtube.com/questionmarkvideo

flickr.com/questionmarkpictures

facebook.com Become a “fan” for news, updates and more

http://www.questionmark.com/newsletter


acknowledged.

Session Evaluation

Slide 74

https://www.questionmark.com/go/709015




What questions do you have?Eric Shepherd. Questionmark, CEO

[email protected] www.questionmark.com

mailto:[email protected]

http://www.questionmark.com/

Documents

Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark