The Future of Online Testing with MOOCs: An Exploratory Analysis of Current Practice

The Future of Online Testing with MOOCs: An Exploratory Analysis of Current Practice

Eamon Costello (National Institute for Digital Learning, Dublin City University)

Jane Holland (Royal College of Surgeons in Ireland)

Mark Brown (National Institute for Digital Learning, Dublin City University)

Background

• NIDL - DCU and Funded MOOC research– MOOCs and building regional capacities

SCORE2020 Project– HOME Project• MOOCs and the Media• Institutional Drivers

Possible MOOC Futures

Possible MOOC Futures

Possible MOOC Futures

Brightest MOOC Futures

Dillenbourg, P. (2015) Proposal for a Digital Education Strategy for Flanders Universities. “Thinkers in Residence” Programme from KVABKoninklijke Vlaamse Academie van België voor Wetenschappen en Kunsten. Available from: http://www.kvab.be/denkersprogramma/files/DP_BlendedLearning_No-time-to-lose.pdf

“Sooner or later, online tests will be as reliable or even more reliable than on campus exams”

Current State of Play

• The future is taken care of• But what about the present?– How mature are (x)MOOCs?– How reliable and valid are MCQ type tests in

MOOCs?

Multiple Choice Question (MCQ) Tests (Single Best Answer)

– Reliability: If we repeated this would we get the same result?

- Validity:Are we measuring what we think we are?

Don’t get “fooled by randomness”

Taleb, N. (2004). Fooled by randomness: The hidden role of chance in life and in the markets. Random House Incorporated.

Best Practice

Case, S. M., & Swanson, D. B. (2003). Constructing written test questions for the basic and clinical sciences (3rd ed.). Philadelphia, PA: National Board of Medical Examiners.

• Ambiguous or unclear information• Negative worded stem (not, incorrect, except)• Implausible distracters • Gratuitous information in stem• More than one or no correct answer• Longest option is correct• Logical cues in stem • Word repeats in stem and correct answer• Unfocused stem • True/false question • Use of all of the above • Vague terms (sometimes, frequently)• Absolute terms (never, always)• Use of none of the above• Fill-in-blank • Complex or K-type • Grammatical cues in sentence completion• Convergence cues

Best Practice

Tarrant, M., Knierim, A., Hayes, S. K., & Ware, J. (2006). The frequency of item writing flaws in multiple-choice questions used in high stakes nursing assessments. Nurse Education Today, 26(8), 662-671

• Ambiguous or unclear information• Negative worded stem (not, incorrect, except)• Implausible distracters • Gratuitous information in stem• More than one or no correct answer• Longest option is correct• Logical cues in stem • Word repeats in stem and correct answer• Unfocused stem • True/false question • Use of all of the above • Vague terms (sometimes, frequently)• Absolute terms (never, always)• Use of none of the above• Fill-in-blank • Complex or K-type • Grammatical cues in sentence completion• Convergence cues• Position of correct option

Best Practice

Tarrant, M., Knierim, A., Hayes, S. K., & Ware, J. (2006). The frequency of item writing flaws in multiple-choice questions used in high stakes nursing assessments. Nurse Education Today, 26(8), 662-671

Methodology

• Use Tarrant et. al. (2004)’s diagnostic tool to analyse MCQs in MOOC systematically

• Look for item writing flaws

The Data

• 12 Courses – from six MOOC platforms – From 12 Universities/Providers

• Total MCQs: 115

None/All of the Above

• None of the above: 1 (0.87%)• All of the above: 2 (1.74%)

The kernel is defined as:

A. The graphical user interface on top of the operating systemB. The glue between hardware and software applicationsC. the software libraries need to run the systemD. all of the above

Holsgrove, G., & Elzubeir, M. (1998). Imprecise terms in UK medical multiple choice ‐questions: what examiners think they mean. Medical Education, 32(4), 343-350.

Number of Correct Options per Question

• Greater than 1 correct: 10 (8.7%)• Average: 1.39

Check the words that are used synonymously in a JMeter test plan

A. end usersB. virtual usersC. concurrent usersD. threads

Number of Options per QuestionGreater than 4 options: 9 (8%)Average: 3.77

1 option 2 options 3 options 4 options 5 options 7 options0

20

40

60

80

0%

10%

20%

30%

40%

50%

60%

70%

80%

2%6%

17%

68%

6%2%

Correction Option is the Longest

• In 47 of the 115 MCQs the correct option is the longest (40.87%)

Position of Correct Option 105 MCQs

1st 2nd 3rd 4th 5th 0

10

20

30

0%

5%

10%

15%

20%

25%

30%

35%

21%

30%29%

19%

1%

1st 2nd 3rd 4th 0

5

10

15

20

25

15%

27%

32%

26%

Position of Correct Option in 73 Four-Option MCQs

Overall

• 17 (14.78%) all the questions contain a defined item writing flaw

• When counting only four item writing flaws• Two more item writing flaws are apparent in

characteristics that appear more often than they should be chance (small sample)

Further workAdd qualitative analysis using the Tarrant et. al. (2004) evaluation tool

• Ambiguous or unclear information• Negative worded stem (not, incorrect, except)• Implausible distracters • Gratuitous information in stem• Logical cues in stem • Word repeats in stem and correct answer• Unfocused stem • Vague terms (sometimes, frequently)• Absolute terms (never, always)• Fill-in-blank • Complex or K-type • Grammatical cues in sentence completion• Convergence cues

Implications

• Validity and Reliability of MOOC Testing• Replicating unsound pedagogies• MOOC teachers/developers need evidence-lead

teaching

Questions?

Me: eamon.costello@dcu.ie