e-Exams The story so far…transformingexams.com/files/e-exams_Dec_2015_96dpi.pdf · e-Exams The...

e-Exams

Thestorysofar…(2013-2015)

TransformingExams.com

Compiled by Mathew Hillier, Dec 2015

TransformingExamsAcrossAustraliaAustralianGovernmentOfficeforLearningandTeachingNa:onalGrantID15-4747

Acknowledgements-DiamondandGoldSponsors!SupportforthisprojecthasbeenprovidedbytheAustralianGovernmentOfficeforLearningandTeaching.TheviewsexpresseddonotnecessarilyreflecttheviewsoftheAustralianGovernmentOfficeforLearningandTeachingorpar:cipa:ngins:tu:ons.Addi:onalsupportisbeingprovidedbyMonashUniversity. D

emo&Guides

Dr Mathew Hillier; Leader OLT national project. Seed leader

Monash University (was University of Queensland)

Dr Andrew Fluck; Originator of USB e-exam concept. Seed partner.

University of Tasmania

Dr Michael Cowling, Mr Kenneth Howah Seed trial site.

Central Queensland University

Dr Kim Blackmore Australian National University Assoc. Prof. Paul Newhouse Edith Cowan University Dr Matthew Bower, Prof. Dominic Verity Macquarie University Prof. Marilyn Baird (H), Mr Scott Grant (A) Monash University Assoc. Prof. Shona Leitch RMIT University Dr Ruth Geer, Mr Bruce White University of South Australia TransformingExams.com

Civil Service Exam Under Emperor Jen Tsung (fl.1022) from a history of Chinese emperors(colour on silk): Bibliothèque Nationale, Paris, France.

Is this your exam space?

Thedissonanceofitall!RealworldofworkExams

4

World Economic Forum – How will digital change your working world. https://agenda.weforum.org/wp-content/uploads/rtr2m8vm1-628x330.jpg

Exams at Monash Caufield in 2015 (mathew.hilier[at]monash.edu)

Wearefacedwithagrowingdisconnectbetweenthewayhighstakestes*ngisconductedusingpenonpaperexamsandstudents’everydayexperiencesofstudy,workandlife.

21stcenturyemployabilityskillsWaysofthinking •  crea:vityandinnova:on

•  cri:calthinking,problemsolving•  learningtolearn,metacogni:on

Waysofworking •  communica:on•  collabora:on(teamwork)

Toolsforworking •  informa:onliteracy•  ICTliteracy•  disciplineresourcesandtools

Livingintheworld •  ci:zenship–localandglobal•  lifeandcareer•  personalandsocialresponsibility(including

culturalawarenessandcompetence)

5

http://atc21s.org/ Binkley, M., Erstad, O., Hermna, J., Raizen, S., Ripley, M., Miller-Ricci, M., & Rumble, M. (2012). Defining Twenty-First Century Skills. In Griffin, P., Care, E., & McGaw, B. Assessment and Teaching of 21st Century Skills, Dordrecht, Springer.

21Clearners

•  skilleduseoftools•  ac:velearningratherthanpassivereceivingofknowledge•  authen:clearningexperiencesratherthancontrivedtasks•  construc:onratherthaninstruc:on•  task(notprocess)oriented•  justin:melearning•  searchnotmemorise•  u:lisesocialnetworks•  doesn’tknowanswerbutknowswheretofindit•  Googlenotlibraries•  collaboratenotcompete.

6

Elliott, B (2007)

BobbyEllio]andassessment1.5to2.0

7

Tradi:onalassessment(Assessment1.0)

Computer-basedassessment(Assessment1.5)

Tool-assistedassessment(Assessment2.0andbeyond)

Elliott, B (2007) “Assessment in the age of Web 2.0” Scottish Qualifications Authority. http://wiki.cetis.ac.uk/images/d/de/Assessment_2_v2.pdf

Ye Olde style… mostly paper-based mostly classroom-based very formalised (in terms of administration) highly synchronised (in terms of time and place) highly controlled (in terms of contents and marking).

Add computer …

21C •  user-generated content, blogs, •  the power of the crowd, wikipedia •  data on an epic scale, •  architecture of participation, easy use •  network effects, •  openness. OER, Mashups

Future à Web 3.0., IoT, Ubiquitous

HypeCycleforEduca:on2015• Cloud email • Mass notification • OS learning repository • E-book readers • Tablets • Mashups • Self-publishing

8

• MOOCS • Low/mid range handsets

2014

2014 Numerous technology tools are available.

Targe:ng...

•  Supervised•  Highstakes•  Oncampus•  Largescale

(imagecredit:DrFluckUTAS)

Whatwearenotspecificallyaddressinghereisoffcampus,

onlineonly,distanceeduca*on,crossins*tu*onalstudents

–thereareextraissues(later!)and

somepossiblee-solu*onstoaddresstheseneeds.

9

Whye-Exams?Ra:onale:Concerns,drivers,possiblesolu:onsfore-Exams.Atruly'wicked','messy'problemandalongroadtogetitright!...

Presentedra:onaleat2013andSeedgrantfindingsat2014,2015conferences.Moretocome…andexploredindepthinHillier&Fluck(2013)h]p://www.ascilite.org.au/conferences/sydney13/program/papers/Hillier.pdf

ta.vu/eexam-map

e-Exams:Online,Offline,OnCampusorDistance

11

Online

(net)

•  Spaceissuesforins:tu:ons.•  Improvedexammanagement

efficiency.•  Equipment:needcomputerlabsto

caterfor2000atonceorBYO.•  Moresecure:liveITmonitoring/

controlandspacesaresupervised.•  Needsreliablenetwork.

•  Nospaceissueforins:tu:ons.•  Moreefficientexam

management.•  Studentssupplyequipment.

•  Lesssecure:ITmonitoringbutwiderspacesareunsupervised.

•  Needsreliablenetwork.

Offline

•  Spaceissuesforins:tu:ons.•  Lessefficientexammanagement.•  Equipment:needcomputerlabsto

caterfor2000atonceorBYO.•  Moresecure:ITcontrolpossible,

spacesaresupervised.•  Networkreliabilitynotanissue.

•  Nospaceissueforins:tu:ons.•  Lessefficientexammanagement.•  Studentssupplyequipment.

•  Lesssecure:nousefulmonitoring/supervision

•  Networkreliabilitynotanissue.OnCampus(controlledspaces) Distance(athome)

Therearetrade-offsforanye-examsolu:on.

APossibleFutureAnevolu:onratherthanarevolu:on.Someaspectsmayoccurquickerthanothersdependingonpar:cularimplementa:on,technicalmodelschosen,socio-cultural-policyenvironmentcondi:ons.OutlookforAustralasia.(DenmarkandNorwayalreadyhave‘internetinexams’).

12Source: Hillier & Fluck (2015) http://transformingexams.com/files/hillier_fluck_2015_exam_futures.pdf

Wherewearegoing:Post-paperexamsWeneedgreaterpedagogicalflexibilityandmoreauthen:cassessmentsintheexamroom.…alignment!

13

Simula:ons,toolsofthetrade,virtualexperiments…

'Windows'sopwareviaWINE.E.g.CAD/3Dmodeling,Celes:a.

Moodlequizwithmedia(automarked).

14

Physical hardware can be connected to the internet or we can use software simulations of labs and experiments.

SimulatedLabs

VirtualImmersiveEnvironments

TAwebinarh]p://transformingassessment.com/events_1_april_2015.php15

Sco]Grant(MonashUniversity,Australia)h]p://www.virtuallyenhancedlanguages.com

VirtualImmersiveEnvironmentsTaskBasedLearningorTaskBasedLanguageLearning(TBLL).Includescommunica:onac:vi:es,usinglanguagetocarryouttasks,languageusethatismeaningfultothelearnerandhasapurpose,andcommunica:onac:vi:esthatreflectreal-lifeac:vi:eswithauthen:cmaterials.Learnbydoing:Purchasesuppliesthencooknoodles–allinChinese.

16

MoodleandOpenSimWorkingTogether

Dataflowsasifthestudentwas

doingtheac*vityintheLMS

Asetofscripts

forMoodleand

VWthatactsas

abridge.

Student

undertakes

assessmentin

thevirtual

world

SetupQuizin

theLMS.

Resultsare

storedinthein

gradebook.

Videos:TransformingAssessmentYoutubeChannel

Undertakinganassessmentac:vityintheVWini:atesdatatransferstotheLMS.

BasicExamples-Interac:veapps

18

1.  Studentclicksalinkembeddedinthequiztolaunchaseparateapp.

2.  Undertakesataskasinstructed.

3.  Thenrespondsusingselectedresponseornumericalinputtosuit.

BasicExamples-spreadsheetinquizSpreadsheetlinkedtoquiz

19

Examples–Conversa:onSim(Monash)StudentsrespondtoaseriesstatementsviaMCQs(maybe,yes,no)withfeedbackperchoicetosimulateaconversa:one.g.Moodlelessonac:vity.

h]p://conversa:onsim.org/Nelson,R&Dawson,P(2013)Assessment-as-learning:introducingtheConversa:onSim,TAwebinar/e-AssessmentScotland,21Augh]p://transformingassessment.com/eAS_2013/events_21_aug_2013.php

Examples–ShorttextresponseStudentstypeinashortsentenceresponsewhichcanbemarkedbycomputerbasedonpa]ernmatching.AvailableinMoodlenow.Exampleques:onExamplesetngs

SallyJordan,OpenUniUK,TAwebinar,5June2013transformingassessment.com/events_5_june_2013.php

Kine:cenergywillbeatmaximumwhenatthebo]omoftheslide.

evalua:on

Examples–Confidenceques:onsConfidencebasedapproachespenaliseguessing.Studentsneedtochoosearesponseanddeclaretheirlevelofcertainty.AvailableinMoodlenow.

22

UniversityCollegeLondon

CertaintyvMarkExpected

Certaintylevelsandconsequences

TonyGardner-Medwin,UCL,TAWebinar6April2011h]p://transformingassessment.com/events_6_april_2011.php

e-ExamSystemAffordances

23

Per:nentFeatures AffordancesA'Wholecomputer'environment(OS,LMS,applica:ons…)onas:ck.

Vastlyexpandedpedagogicalscopeoverthatofabrowserwindow.

TypedstudentresponsesviaWordprocessor,constructedviaapps(humanmarked)oron-boardlearningmanagementsystemquiz(computermarked).

Catersforintroduc:ontoadvanceduses.Componentsadded/removedtosuit.Electroniccollec:onfacilitatesanaly:cs,itemresponseanalysis…

Nolivenetworkrequiredduringexam,evenforLMSques:ons.

Robust.Greatercontrol.(networkcouldbeusedforadmin)

Studentownedequipmentusedashostandlepuntouched.

Anethicalapproachtoscalability(noinvasivesopwaretoinstall)

Modular,opensourcecodebaseandcommodity'offtheshelf'components.

Leveragingpopularandsustainableprojectsforbe]erefficiency.Fully'known'(no'blackbox').Available!

OneversionworksonmostIntelbasedlaptops-Apple,'windows',Linux,thathaveaUSBport.

Onesopwareversioncanserveall.Streamlinesdevelopmentandmaintenance.

Bootable ‘live’ USB drive

Wherewearenow:PaperEquivalent(UQ)Worddoc!Ques:ontypesused:shortanswer/essay,matching,constructatable,labeladiagram/image(byfillingatable).Manualmarking.

24

Wherewearenow:PaperPlus(UTAS)Worddoc,plusapps.Ques:ondocumentwithlinkstolaunchlocalappsandresources:graphics,Scratchprogrammingtools,presenta:onslides,PDFs.Allon-boardtheUSBs:ck.‘ITinEduca:on’exams:AndrewFluck,UTAS.

25

PDF

Scratch SDK

Exam doc

e-ExamWorkflowusedinTrials

26

Postsession:retrieveresponsesandassessment

Examroomuse

ResponsesretrievedfromUSBs. Collatede-responsessenttoacademic.

CreatemasterUSB(tested)

USBsduplicatedperstudentAcademiccreatesexamlearningmaterial

Set-up:prepareexamlearningmaterials Pre-session:Studentlaptopsetup&prac:ce.

CollectUSBs(responses)

1. Studentsenterroom.

2. GivenUSB.3. Bootlaptop.4. Doexam.5. ReturnUSB.6. Leaveroom.

e-Examsystem

takesover

laptop.

Ubuntu

LiveUSB.

LibreOffice.

FirstandMostRecente-ExamsUQ

27

VETS2100 S2 2014

DENT4092 S1 2015

Used standard teaching rooms, sought rooms with tables and power sockets.

ß VETS: hand-writers sat in rows. Attempted to separate typists and hand-writers where possible. DENT: typists at the back, à hand-writers at the front.

Spaces–Triedinavariety

28

FloorPlansPurple=typistsOrange=hand-writers

VETS

DENT

BIOL OCTY

CRIM PHTY

WalkThroughofCurrente-ExamPlaxorm

2.HolddownALTthenpoweron.KeepholdingALTun:lyousee…

29

2.Poweronwhiletapping‘bootkey’(e.g.F12or…)Keeptappingun:lyousee…(orsimilar)

*Win8:firstneedtodisablesecureboot.

1. Start with the computer turned OFF. Then insert USB stick

Apple Other/Windows*

Taptaptap...Holddown

Pressandreleasepower

Pressandreleasepower

WalkThrough

3.Selectayellowicon.EFIbootor‘Windows’

Couldbeeitherone!Sojusttry.Ifyouget

Trytheotherone!30

3.SelectUSBdevice.Itmightbelabeledsomethingelseandprobablywontbefirst.

Apple Other/Windows

e-Exam

e-Exam

WalkThrough

4A.Somesystemmessagesmayappear,ifsojustwaitandsee.4B.Examsystemshouldstart.

5.Arriveate-Examsystemdesktop.

31

WalkThrough6.StudentnowtypesintheirstudentIDnumberandname.ClickStartExam.

32

WalkThrough7.Examfileopensreadytoenterexamdetailsandresponses.

33

Note:originalfilecopiedandstudentnumberprefixedtofilename.

WalkThrough8.Studenttypesresponsesintoareasindicated.

34

WalkThrough9.Remembertosave(CTRLS).WhendoneuseFile>Exit.

35

WalkThrough10.Shutdownthesystem.Whenthesystemhaspoweredoff,handintheUSBS:ck.

36

Shutdownbu]on

Ready?

37

TrialOutcomes

Resultsfrom2014-2015trialsfollow.

38

e-ExamsSeedWork2013to2015

OLTProjectleader/Presenter: DrMathewHillier,UniversityofQueenslandOLTProjectcollaborator: DrAndrewFluck,UniversityofTasmaniaOLTProjectsystemdeveloper: MarisaEmerson,UniversityofQueenslandUQcourseacademics:

Acknowledgement:SupportforthisprojecthasbeenprovidedbytheAustralianGovernmentOfficeforLearningandTeaching.TheviewsexpresseddonotnecessarilyreflecttheviewsoftheAustralianGovernmentOfficeforLearningandTeachingorpar:cipa:ngins:tu:ons.

Dr Arosha Weerakoon (Dentistry) Dr David Booth (Zoology), Elizabeth Springfield (Occupational Therapy), Katrina Williams (Physiotherapy), Prof. Malcolm Jones (Veterinary Biology), Rebekah Scotney (Veterinary Technology) and Dr Robin Fitzgerald (Criminology) Getthedemoanduserguides

h]p://transformingexams.com

UQe-ExamTrialOutcomes

40

DatacollectedfromstudentsResultsavailableuponrequest,someath]p://transformingexams.com/research.html

•  2013pre-projectonlinesurvey(UQwide:briefresultsshown)–  UQstudentssurveyedabouttheirpreconcep:onsaboute-exams.

ASCILITEpaperHillier2014h]p://transformingexams.com/files/hillier_2014_ascilite_full_paper_prepress.pdf

•  2014-5Trialspre-examshortsurvey(8courses–typistsonly).–  Conductedatthepre-examprac:cesetupsessions.–  Covered:studentpreliminaryimpressions,technicalhardwarecompa:bility.

•  2014-5Trialspost-examextendedsurvey(8courses–resultsshown)–  Conductedattheconclusionoftheexam(intheroom-allstudents).–  Covered:ra:onale,studentexamexperience,reac:ontoexamsessioncondi:ons,e-examsystemimpressions,examwri:ngstrategiesandproduc:on,generalnon-examwri:ngstrategies.Resultfor2014(sixcourses)h]p://transformingexams.com/files/Hillier_2015_ascilite_fp.pdf

•  2015Analysisoftextproduc:on(DENTonly)–  Marksvwordcount,typingvhandwri:ng(moretocome;languagedensity…)

ResearchStudyPhases

41

Phase 1 Institution wide online survey (see Hillier 2014, 2015). Phase 2, Step 1 e-Exam Trial Expression of interest

Typists Handwriters

Phase 2, Step 2 Pre-exam preparation survey

Phase 2, Step 3 Type the exam Handwrite the exam

Phase 2, Step 4 Post-exam survey Par:cipa:oninPhase1:approx.928respondents(Nov2013-Nov2014)Par:cipa:oninPhase2:Eightcourses(sixin2014,twoin2015)

Phase1SurveyDesign•  Surveyconstructedtocoverarangeofpossibleconcerns.

•  PriorresearchDermo(2009),Limetal.,(2006),MogeyandHartley(2012),Sorensen(2013),Fluck,Pullen&Harper(2009),Fluck(2011),Fluck(2013),Hillier&Fluck(2013)providedameanstoscopetheissues.

•  AsurveybyDermo(2009)providedthecore.–  Acknowledgethatwewouldbeusingitinadifferentmanner(preratherthenpost).

–  But!weneverintendedtoreplicateit,insteadweusedthisasameansforelici:ngstudentconcernsacrossarangeofissues.

–  Studentswouldlargelyberespondingspecula:velybasedontheirpreconcep*ons.(instruc:onsgiventostudentsaccordingly)

42

Phase1Ins:tu:onSurvey

Mobiledeviceownership(excludingdesktopcomputers)wasanaverageof2.3devicesperstudent(standarddevia:onof0.8).

43

94%

84%

41%

34%

0% 20% 40% 60% 80% 100%

OwnLaptop

OwnSmartPhone

OwnTablet

OwnDesktopComputer

PercentageofRespondentsThatOwnEachTypeofDevice

Phase1Theques:ons

44

Plustwoopenendedcommentques:ons

Phase1Par:cipa:on•  488#students(37%males,63%females)=1%*•  9%post-grad,remainderwereundergrads(withanevenspreadacrossyearlevels).

•  45programs,thosewithatleast10arelisted:

45

*Krejcie & Morgan (1970) state that for a population of 50,000 a random sample 381 would be sufficient to be representative in relation to opinions expressed by respondents to 95% confidence. (it wasn’t random). #analysis performed on responses received at Feb 2014. Subsequent responses up to Nov 2014 were 928.

Phase1Analysisa.  Themesdrawnfromopenendedques:onson

currentlyheld‘concerns’aboute-examsandgeneralcomments.Ourfocushere!

b.  Sta:s:cs*usedtoexplorethebodyofopinionrepresentedbyLikertscalesratherthanasasearchforasingletruth.Tendedtos:cktonon-parametrictests.

ReportedfurtherinHillier,M.(2014).TheVeryIdeaofe-Exams:Student(Pre)concep:ons.PresentedattheAustralasianSocietyforComputersinLearninginTer:aryEduca:onconference,Dunedin,NewZealand.Retrievedfromh]p://ascilite.org/conferences/dunedin2014/files/fullpapers/91-Hillier.pdf

46

Phase1ThemesPreconcep:ons.

47

0 20 40 60 80 100

Technicalreliability

Chea:ng

Matchwithdiscipline

Keyboardingprowess

ComputerliteracyComment Count

Emergent themes from Phase 1 survey

Thetechnologybeingunreliablestressesmeoutmorethanthethoughtofdoingtheexam–Lawstudent

ArealprogrammerwouldbelookinguptheAPIsfortheirlanguageevery:metheywantedtodosomething,buttheycan'tbecausethey'reforcedtoonlyusepaper-basednotestheyhaveonhand.It'sinfuria:ng-Computersciencestudent.

Asamatureagedstudent,IwouldfeelatadisadvantagedoingacomputerisedexamasIamnotascomputerliterateasmanyoftheyoungerstudents-Chemistrystudent.

It'struethat'computerisedexamsfavoursomestudentsmorethanothers'-i.e.,theonesthatareproficienttypistsovertheonesthataren't-butthesameistrueofpaper-basedexamina:ons,whichfavourthosewiththeabilitytoworkthroughstrongpainintheirwri:nghand–Artsstudent.

Phase1Findings:MatchtoDisciplineByprogram(major)

48See Hillier (2014) for more. SD D N A SA

Phase1Findings:IwouldliketoType

49

“Iwouldliketobeabletotypeanswersinanexam”Byprogram(major)

See Hillier (2014) for more. SD D N A SA

Phase1InterimConclusionStudentswere•  Cau:ouslyop:mis:c•  Justoverhalfwouldliketoseeane-examop:on.Meanagreement(3.3)“Iwouldliketobeabletotypeanswersinanexam.”

•  Werea]unetothenatureoftheirdisciplineandhowtheideaofane-exammightfit.

•  Thefearoftheunknown(?)espregardingtechnicalfailuresandreliability.

50

Phase2:UQe-ExamTrials2014-2015

51

Datacollectedfromstudents(opt-in)•  Viapre-projectonlinesurvey(UQwide–2013-2014):

Hillier,M.(2014).TheVeryIdeaofe-Exams:Student(Pre)concep:ons.PresentedattheAustralasianSocietyforComputersinLearninginTer:aryEduca:onconference,Dunedin,NewZealand.Retrievedfromh]p://transformingexams.com/files/hillier_2014_ascilite_full_paper_prepress.pdf

•  Viapre-examshortsurvey(8courses–typistsonlynext).–  Conductedatthepre-examprac:cesetupsessions.–  Covered:studentpreliminaryimpressions,technicalhardwarecompa:bility.

•  Viapost-examextendedsurvey(8courses)–  Conductedattheconclusionoftheexam(intheroom).–  Covered:ra:onale,studentexamexperience,reac:ontoexamsession

condi:ons,e-examsystemimpressions,examwri:ngstrategiesandproduc:on,generalnon-examwri:ngstrategies.

–  (six2014courses)h]p://transformingexams.com/files/Hillier_2015_ascilite_fp.pdf

•  Analysisoftextproduc:on(DENTonly)–  Marksvwordcount,typingvhandwri:ng(moretocome;languagedensity…)

Theeightcoursesinthetrials2014-2015

Condi:ons•  First‘toeinthewater’trials.•  Par:cipa:onwasop:onal.•  Midtermexamsworth15%to20%ofthecoursegrade.Note:Splitgroup=typistsandhand-writersindifferentrooms. 52

Course Minutes DescripVon

ANIM2014AnimalBiology 45 MixedshortanswerandMCQ(type'x')[splitgroup]

BIOL2014Zoology 50 Shortanswer(Mul:plechoicesec:ondonepenonOMRsheet)[splitgroup]

CRIM2014Criminology 70 Singlelongessayresponsesec:on(andaMul:plechoicesec:ondonepenonOMRsheet)

OCTY2014Occupa:onalTherapy 90 MixedshortanswerandMCQ(type'x')

PHTY2014Physiotherapy 15 Diagnosis:watchvideoandwriteintotable.Doneinsmallgroupsof16priortoOSCE.

VETS2014Veterinarytechnology 90 Theory,mostlyshortanswer.(withinternalandexternalgroups)

CRIM2015Criminology 90 Singlelongessayresponsesec:on(andaMul:plechoicesec:ondonepenonOMRsheet)[splitgroup]

DENT2015ResearchMethodsinDen:stry 60 Theory,shortanswer,onecalcula:onques:on

TrialPhaseA]ri:on

53

Numberoftypistsateachstageofthetrial(Surveyresponses)•  Notallrespondentscompletedeveryques*on.

•  Anumberofstudentselec*ngtohand-writedidnotfillintheEOIandthe

post-examsurveysoareslightlyunderrepresented.

•  Similarlynotalla\endeesatthepre-examset-upsessionreturnedasurvey

(~90%+did).

Steps of trial Yes Maybe Total typists Attrition No -

hand-write 1 Expression of Interest 241 241 420 2.1 Pre - before try 124 17 141 100 38 2.2 Pre - after try 112 19 131 10 52 4 Exam (after) 98 98 33 549 Tableupdatedtoinclude2015participants.Finaltypistsbasedonreturnedsurveys.

Pre-examFirstImpressionsSelectedpre-examsessionsurveyques:ons(typistsonly)Studentscametotesttheirlaptopandtrythesystemacoupleofweekspriortotheexam.

54

QuesVon N Mean SD

Thewri]eninstruc:onswereeasytofollow 140 4.0 1.0

Itwaseasytolearnthenecessarytechnicalsteps 137 4.0 1.0

Itwaseasytostartmycomputerusingthee-ExamUSBs:ck 140 4.1 1.2

IfeelconfidentIwillbeabletodothesestepsinarealexam 138 4.0 1.1

Thesopwarewithinthee-ExamSystemwaseasytouse 137 4.1 1.1

Inowfeelrelaxedabouttheideaofusinge-Examformyupcomingexam 138 3.8 1.1

Iwouldliketouseacomputerforexamsinthefuture*(newin2015) 32 4.1 0.9

Updatedtoincludes12015results.

Bars represent medians. Means shown for clarity. Strongly Disagree Strongly Agree

TrialPhase:Pre-examSurvey

55

Datacollectedfromstudentsatpre-examset-up/prac:cesessions(2014-2015).Beforetryinge-exam Apertryinge-exam

TrialPhase:Pre-examSurveyOpentextcomments–concernsandpraise[2014]

56

0 5 10 15 20 25

fear of technical failure/crash remember procedure/use difficulties

scrolling/touchpad use power/battery

fear of data loss drawing/diagram difficulty fear of computer damage

newness/unfamiliarity security

panic eye strain

fail the exam rules/contingency

general positive/praise good ease to use

convenient fence sitting

0 5 10 15 20 25

Goodconcept.Providesanalterna:vetowri]enexams.Easytonavigate.

Simplyadifferentformatofexam.Rememberingthisfortheexam.

Thatmycomputermayfreezeornotworkproperly.Randomtechnologicalmalfunc:on.

Fairlysurei'llusethelaptopbutjustneedtothinkaboutitali]lemore.

AmIallowedtouseamouse?Willitaffectthesystem?Scrollingopposite.

Availabilityofpower.Ba]eryrunout.

Doingsomethingwrongandpanicking

Woulditsavemyanswersproperly.

Numberoflaptopspassedandtechnicalissues[2014].

So…wehadapoolofloanlaptops.UpgradetonextversionofOSwillhelptoo.

Pre-examLaptopTes:ng

57

80%

20%

compatablemachines

incompatablemachines

0 2 4 6 8 10 12

notcompa:ble/unknown

graphicsdriverissues

manualstart

screenresolu:onissues

needclearerinstruc:ons

ba]erydrain

usererror

hardwarefault/keyboardbroken

save-asproblem(defaultsave

difficulty/complexityinset-up

slowtoboot

‘Wintel’LaptopConfigura:onTherequiredsetngadjustmentsappliedtoallowthelaptoptobootfromthee-ExamUSBs:ck(version12.04).Collatedsemester22014onward.

58

Note: Apple laptops do not require adjustment. To do: Need to collate from earlier sessions.

BrandsandOpera:ngSystemsBYOlaptopstock

59

Able to boot from e-Exam USB stick

Estimated Battery Life in Hours

Hours

Count Includess12015results–8cohorts.

Phase2:UQe-ExamTrials2014-2015

60

Datacollectedfromstudents(opt-in)•  Viapre-examprojectonlinesurvey(UQwide):

Hillier,M.(2014).TheVeryIdeaofe-Exams:Student(Pre)concep:ons.PresentedattheAustralasianSocietyforComputersinLearninginTer:aryEduca:onconference,Dunedin,NewZealand.Retrievedfromh]p://transformingexams.com/files/hillier_2014_ascilite_full_paper_prepress.pdf

•  Viapre-examshortsurvey(8courses–typistsonly).–  Conductedatthepre-examprac:cesetupsessions.–  Covered:studentpreliminaryimpressions,technicalhardwarecompa:bility.

•  Viapost-examextendedsurvey(8courses–allstudents–next)–  Conductedattheconclusionoftheexam(intheroom).–  Covered:ra:onale,studentexamexperience,reac:ontoexamsession

condi:ons,e-examsystemimpressions,examwri:ngstrategiesandproduc:on,generalnon-examwri:ngstrategies.

–  2014(sixcourses)h]p://transformingexams.com/files/Hillier_2015_ascilite_fp.pdf

•  Analysisoftextproduc:on(DENTonly)–  Marksvwordcount,typingvhandwri:ng(moretocome;languagedensity…)

Typistsandhand-writersbycourse

61

Typed15%

Handwrote85%

Combinedallcohorts

Cohort Typed HandwroteCRIM2014 25.4% 74.6%PHTY2014 18.8% 81.2%VETS2014 12.4% 87.6%ANIM2014 4.4% 95.6%OCTY2014 11.1% 88.9%BIOL2014 9.9% 90.1%CRIM2015 12.1% 87.9%DENT2015 28.8% 71.2%

Proportionoftypistsandhandwritersineachoftheeightcohorts2014-2015

0% 20% 40% 60% 80% 100%

ProporRonoftypistsandhandwritersbycohortTyped Handwrote

Genderra:os:Typistsandhand-writers

62

Typists 15%

Hand-writers 85%

All Eight Cohorts

ModeMWU 27737.5

Z -1.516Sig.(2-tailed) 0.13

Finding: Gender did not play a role in the choice to type. Includess12015results–8cohorts.

Reasonsfortypingtheexam(2014)

63

(added30October2014)

[2014]

Phase2Post-examTypist’sComments

64

Itiscleaner,ImakelotsofmistakeswhenI'mwri:nganditusuallyendsinlotsofscribbleseverywhere.

Ihaveterriblehandwri:ng.Feltbadaboutit. Icouldgetinfodownfaster

andexaminercouldreadit.

Quickertypingandtheabilitytoeditorcompletelydeletemyanswerwithoutcompromisingonspace.

Youcanwriteasmuchasyouotherwisewouldbutdon'tgetasorehandwhentyping.

Typingismorenaturalforme.IthinkbestwhenIamtyping.

[2014]

Reasonsforhandwri:ngtheexam(2014)

65

(added30October2014)

[2014]

Phase2Post-examHand-writers

66

It'seasiertohandwrite.Thoughprobablynoteasierforyoutoreadmywri:ng.

Threeyearsofpriorexamswri:ngsos:ckwithwhatyouknow.

Lazytobringlaptop.

Ifeltmorecomfortablehandwri:ngasnothingcangowrong&Iwasn'trelyingonthecomputersystemtocompletemyexam.

IthinkmoreaboutwhatI'mwri:ngwhenIhandwritebutmyhandgetssoreanditisn'tfast.

Iwasini:allyplanningtotypethisexambutdecidedagainstitduetotheunpredictabilityofmachines

I'maslowtyperandfeeldisadvantaged.

Iamworriedaboutcomputermalfunc:ons.

[2014]

Post-examImpressions

67

Boxplots: responses from typists. Bars represent medians. Means shown for clarity. N = 91

Strongly Disagree Strongly Agree

4.4

4.0

4.2

4.5

4.3

Updatedtoincludes12015results–8cohorts.

Student(typists)impressionsofusingtheexamsystem

Post-examImpressionsDidtypiststhinktheexamsuitedtheuseofcomputers?

68

Boxplots: responses from typists by cohort. Bars represent medians. Means shown for clarity. Overall mean agreement 4.2 Largely that was a ‘yes’. However two factors at play: a)  Self-selecting sample.

Typists would be positive. b)  Exam was ‘paper

equivalent’ thus not taking advantage of what was possible with IT e.g. multimedia, simulations etc

Strongly Disagree Strongly Agree

Updatedtoincludes12015results–8cohorts.

3.9

3.6

4.4

4.8

4.2

4.6

4.0

4.2

Reac:ontocondi:onsintheexamTypistshadamoreposi:veexperienceoverallbutnosignificantdifferenceswerereportedfor:merunningoutandstresslevels.

69

Mann-WhitneyU 21172.5 23645 23065 24252Z -2.539 -0.647 -1.081 -0.447

Asymp.Sig.(2-tailed) >.05 n/s n/s n/s

Typists = purple, Hand-writers = orange

Strongly Agree

Strongly Disagree

Updatedtoincludes12015results–8cohorts.

4 3.7 2.7 2.6 2.6 2.7 3.5 3.5Means

Reac:ontocondi:onsintheexamOverallexamexperiencebycohort

70

stronglydisagree(notgood)stronglyagree(be]er)

Updatedtoincludes12015results–8cohorts.

Reac:ontocondi:onsintheexamTimeavailabilitybycohort

71

stronglydisagree(notgood)stronglyagree(be]er)

Updatedtoincludes12015results–8cohorts.

Generalkeyboardandwri:ngprowessStudentreportedtypingandwri:ngskills

72

Key: Purple (top) = typists, Orange (bottom) = hand-writers Bars represent medians. Means shown for clarity. Mann-Whitney U test results shown.

Means Sig 3.28, 3.48 n/s 4.46, 4.31 n/s 3.37, 3.61 n/s 4.49, 3.88 sig >.001 4.23, 3.49 sig >.001 4.52, 3.67 sig >.001

Strongly Disagree Strongly Agree Updatedtoincludes12015results–8cohorts.

73

Was the sound of typing distracting? In each boxplot Typers (left) and Hand writers (right)

Means 3.4 2.3Both exhibited significant differences to >.01 on Mann-Whitney U test Likert Scale: 5 = Strongly Agree, 1 = Strongly Disagree Updated to include S1 2015 results.

The cohorts ANIM2014, BIOL2014 and CRIM2015 were removed from the analysis because typists and hand writers sat in different rooms. Those that could hear typing (who selected 5, 4 or 3) were included in the determination of distraction by typing sound. Cohort exams were held in different venues.

2.2 3.3

74

Was the sound of typing distracting (VetSci)? In each boxplot Typers (left) and Hand writers (right)

Means 3.4 2.47

Likert Scale: 5 = Strongly Agree, 1 = Strongly Disagree

VetSci Course: VETS both the internal and external cohorts used same room, but at different times (4 weeks apart). VETS internal: warm day, ceiling fans and construction noise. VETS external: cooler day, no fans, quiet. Environmental conditions and acoustics play a large role in the degree to which ‘typing noise’ becomes a distracting factor. Hand-writers were not all quiet either!

2.85 3.57

FutureInten:onsTypistsweremoreposi:vetowardsfuturee-Exams,asexpected,buthand-writerswerenotnega:veasawhole.

75

Mean N SDTypists 4.2 39 0.8Hand-writers 2.2 167 1.2

Mann-WhitneyU672.5Z-7.961

Sig(2tail)>.001

SD SA

Typists = purple, Hand-writers = orange Updatedtoincludes12015results(4cohorts-new question added in Semester 2, 2014). .

Post-examImpressionsHand-wri:ngintheexam

76

Boxplots: responses from hand-writers. Bars represent medians. Means and counts shown for clarity. Note: 1= Strongly Disagree, 5 = Strongly Agree

m N3.9*162.6 243.2 252.8 482.8 462.3 802.41071.8107

m N3.4 763.8 243.7 253.6 482.9 443.5 803.51093.2107

KruskalWallisTestChi-Square 61.060 19.631

df 7 7Asymp.Sig. 0.000 0.006

Updatedtoincludes12015results–8cohorts.

Are some students over estimating the neatness of their hand writing?!

* Note 20% response rate by VETS for this item. All others near 90%

TrialTechnicalIssuesIssuelog(2014):15ofthe69whotypedreported‘technicalissues’viathepost-examsurvey.1morewasiden:fiedbyobserva:on.Themajoritywereminor.

77

Issue N Notes,AddiVonalObservaVons,SuggestedSoluVonsBoot/startup 2 Inrealitymostpar:cipantsneededassistance/forgotbootkey.

Familiarity:needtoprac*ce!

EnteringID 0 Allgood.(somestudentsentered‘s’ratherthan8digitnumberbutsystemcopesfine).Usingthesopware

1 Somedidnotknowhowto'exit'gracefully(i.e.Filesave,fileexit,shutdown).Needtoprac*ce!Inves*gatean'I’mfinished'script/bu\on.

Ba]ery 0 Mostpluggedin.Powerneedstobeavailable.Savingfiles 0 Allgood.(no:cedonestudentused‘saveas’whensavewas‘greyedout’)nowfixedSopwarecrashed/computerfroze

4 1xOld2009whiteMacbook.Officesuitequittodesktop.3xSystemdriveranoutofspacecausingthesystemtocrash(nowfixed).

Touchpad/mouse

7 Sensi:vityreportedbypar:cipants.Someadjustmentsweremade.

USBwiredmicehighlyrecommended!Inves*gatedrivers.

Scrolling 15 TwofingerscrollingoppositetoOSX,keyboardshortcuts.Smallscrollbars.Sensi:vity.Familiarity:needtoprac*ce.Largerscrollbars.Inves*gateauserselectableop*onfor

touchpad/scrollbehavior(andre-mappingofkeyboardshortcuts).

Furtherdevelopmentisneededtoaddresstheseissues.Warningsremaininreadmefilesavailableonpublicdownloadsites.

78

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

I write more words when

I write faster when

I think more carefully before I start writing when

I pause to think most when

I write in a style that feels more normal when

I try not to make changes unless they are really important when

I change, move or correct words or phrases most when

I think the overall structure/argument of my responses is better when

I make more effective use of the time available when

I go back and read over my responses before submitting most when

I feel more stressed when

I am more likely to run out of time when

Overall I feel I perform better in an exam when

Using my computer Same equally Using a pen and paper

Student consideration of general exam conditions when using computer versus pen [2014]: All six cohorts. Response pairs: those who typed (line 1) & those that hand-wrote* (line 2)

* Note - Many of those that hand-wrote their exam had no prior experience of using a computer for an exam so the results presented here are largely speculative on their part. However, it is reasonable to assume that they drew on their general use of computers. Note! Updated March 2015 edition places ‘same equally’ in the middle rather than on the right.

79

0%# 20%# 40%# 60%# 80%# 100%#

I think carefully before I start writing when using my computer

I think carefully before I start writing when using pen and paper

I take notes in lectures using my computer

I take notes in lectures using pen & paper

I make quick, rough notes before writing essays/reports properly using my computer

I make quick, rough notes before writing essays/reports properly using pen and paper

I make a detailed plan before writing essays/reports properly using my computer

I make a detailed plan before writing essays/reports properly using pen and paper

I just start writing (there is no plan!) when using my computer

I just start writing (there is no plan!) when using pen and paper

I make lots of notes using pen & paper

I tend to go back and re-read and revise my writing quite a lot

I prepare most of my assignments using a computer

Yes# No#

Writing strategies under non-exam conditions – general writing habits [2014]: All six cohorts. Response pairs: Typers (line 1) and Hand writers (line 2)

ZSig.(2-tail)

-4.25 .000***

-1.75 .080*

-1.56 .120

-2.98 .003***

-3.19 .001***

-1.84 .067

-2.96 .003***

-1.20 .230

-0.08 .934

-0.12 .904

-1.40 .161

-0.52 .606

-1.48 .138

Nonparametric U & Z used to compare those who typed in the exam to those that hand wrote. Note! The September 2014 edition of this chart was incorrectly reversed against the stats.

Typingandwri:ngabili:esStudenttypingandwri:ngingeneral

80

4.5 3.8 4.2 3.5 4.5 3.9 3.3 3.5 4.4 4.3 3.3 3.4 Means

Updatedtoincludes12015results–8cohorts.

Likertsc

ale:1=stronglydisa

gree,5=stronglyagree.

Mann-WhitneyU 14703 13079.5 14514 18196.5 18969 19746.5Z -4.708 -5.677 -4.762 -1.694 -1.366 -0.676Sig.(2-tailed) >.001 >.001 >.001 n/s n/s n/s

Typers (left) and Hand writers (right)

Looks like a ‘yes’!

Did the nature of prior experience of e-exams impact on the decision to type this exam?

All participants, 6 cohorts [2014]. OfthosewithPriorexp. AllMann-WhitneyU 502Z -2.734Sig.(2-tailed) >.01

Yes No

81

82Looks like a ‘yes’!

Does the nature of prior experience of e-exams impact future intended use? [2014]

All participants, all cohorts. Hand writers, all cohorts.

All HandwritersMann-WhitneyU 22.5 21Z -3.262 -2.248Sig.(2-tailed) >.01 >.05

Phase2:UQe-ExamTrials2014-2015

83

Datacollectedfromstudents(opt-in)•  Viapre-examprojectonlinesurvey(UQwide):

Hillier,M.(2014).TheVeryIdeaofe-Exams:Student(Pre)concep:ons.PresentedattheAustralasianSocietyforComputersinLearninginTer:aryEduca:onconference,Dunedin,NewZealand.Retrievedfromh]p://transformingexams.com/files/hillier_2014_ascilite_full_paper_prepress.pdf

•  Viapre-examshortsurvey(8courses–typistsonly).–  Conductedatthepre-examprac:cesetupsessions.–  Covered:studentpreliminaryimpressions,technicalhardwarecompa:bility.

•  Viapost-examextendedsurvey(8courses–allstudents–next)–  Conductedattheconclusionoftheexam(intheroom).–  Covered:ra:onale,studentexamexperience,reac:ontoexamsession

condi:ons,e-examsystemimpressions,examwri:ngstrategiesandproduc:on,generalnon-examwri:ngstrategies.

–  2014(sixcourses)h]p://transformingexams.com/files/Hillier_2015_ascilite_fp.pdf

•  Analysisoftextproduc:on(DENTonly)–  Marksvwordcount,typingvhandwri:ng(moretocome;languagedensity…)

AnalysisofExamResponses-DENT

84

DENT Mid Semester Exam S1 2015 – •  Six short answer questions •  20% of course •  Scripts N = 68 •  19 typed •  49 handwritten

Analysis •  Production (word count) •  Marks per question and overall

Produc:onDENT

85

DENT Mid Semester Exam S1 2015 – Six short answer questions; 20% of course.

Scripts N = 68 (19 typed, 49 handwritten)

Score Mann-Whitney U 295, Z -2.333, Sig .02

Words Mann-Whitney

U 245.5 Z -3.007 Sig .003

WordCountforeachQues:onDENT

86

Comparing number of words typed and handwritten by question number. Significance per question by mode. Mann-Whitney: Q1 >.05 Q2 >.05 Q3 NS Q4 >.05 Q5 >.05 close Q6 NS

Domorewordsmeanbe]ermarks?

87

DENT 2015 exam. It depends! A higher hand written word count generally led to passing and better marks, but lesser words did not always result in poor marks. Typists did better overall. More typed words only slightly increased marks. Not claiming causation!

Moreinforma:on….Demoset-upGuide,StudentPrac:ceandUserGuideh]p://transformingexams.com

Demovideosstart-up,useandrecoveryexamples.Appleh]p://ta.vu/eexam-demo-a'Wintel'(Dell)h]p://ta.vu/eexam-demo-dContact:m.hillier[at]uq.edu.au

e-ExamProjectResources

88

ReferencesDermo,J.(2009).e-Assessmentandthestudentlearningexperience:Asurveyofstudentpercep:onsofe-assessment.Bri:sh

JournalofEduca:onalTechnology,40(2),203–214.h]p://doi.org/10.1111/j.1467-8535.2008.00915.xFluck,A.(2013).Implementa:onofon-campusdigitalexamina:onprac:ces.PresentedattheTransformingAssessmentina

DigitalEra,Melbourne,Australia.Retrievedfromh]p://www.eexams.org/stories/Documents/TA%20in%20a%20Digital%20Era%20-%20Fluck.pdf

Fluck,A.E.(2011).eExamsforTransforma:onofHigherEduca:on.PresentedattheAustralianAssocia:onforResearchinEduca:onConference,HobartTasmania.Retrievedfromh]p://www.aare.edu.au/data/publica:ons/2011/aarefinal00107.pdf

Fluck,A.,&Hillier,M.(2014).eExamsTransformingCurriculum.InNowIT’sPersonal(pp.151–158).Adelaide,Australia:ACEC.Retrievedfromh]p://acec2014.acce.edu.au/sites/2014/files/a]achments/eExams%20paperd%20-%20REV2b.docx

Fluck,A.,Pullen,D.,&Harper,C.(2009).CaseStudyofaComputerBasedExamina:onSystem.AustralasianJournalofEduca:onalTechnology,25(4),509–523.

Hillier,M(2015)"Totypeorhandwrite:student'sexperienceacrosssixe-Examtrials",ASCILITEConference,Perth,29Nov-2Dech]p://transformingexams.com/files/Hillier_2015_ascilite_fp.pdf

Hillier,M&Fluck,A(2015)"Apedagogicalendgameforexams:alook10yearsintothefutureofhighstakesassessment",ASCILITEConference,Perth,29Nov-2Dec.h]p://transformingexams.com/files/hillier_fluck_2015_exam_futures.pdf

Hillier,M.(2014).TheVeryIdeaofe-Exams:Student(Pre)concep:ons.PresentedattheAustralasianSocietyforComputersinLearninginTer:aryEduca:onconference,Dunedin,NewZealand.Retrievedfromh]p://ascilite.org/conferences/dunedin2014/files/fullpapers/91-Hillier.pdf

Hillier,M.,&Fluck,A.(2013).Arguingagainfore-examsinhighstakesexamina:ons.InH.Carter,M.Gosper,&J.Hedberg(Eds.),ElectricDreams(pp.385–396).MacquarieUniversity.Retrievedfromh]p://www.ascilite.org.au/conferences/sydney13/program/papers/Hillier.pdf

Lim,E.C.H.,Ong,B.K.C.,Wilder-Smith,E.P.V.,&Seet,R.C.S.(2006).Computer-basedversuspen-and-papertes:ng:students’percep:on.AnnalsoftheAcademyofMedicine,Singapore,35(9),599–603.

Mogey,N.,&Hartley,J.(2012).Towriteortotype?Theeffectsofhandwri:ngandword-processingonthewri]enstyleofexamina:onessays.Innova:onsinEduca:onandTeachingInterna:onal,50(1),85–93.h]p://doi.org/10.1080/14703297.2012.748334

Sorensen,E.(2013).Implementa:onandstudentpercep:onsofe-assessmentinaChemicalEngineeringmodule.EuropeanJournalofEngineeringEduca:on,38(2),172–185.h]p://doi.org/10.1080/03043797.2012.760533

220moreat:h]ps://www.zotero.org/groups/e-assessment/items/tag/e-examSeee-ExamProjectpageath]p://transformingexams.com/research.html 89

e-ExamsOnlineConferenceRecordings

Viewath]p://ta.vu/eAS20141.  Monday,8September-GMT07:00:BYODon-campuse-examsatUniversityofTasmania(UTAS).AndrewFluck,

UniversityofTasmania,Australia.2.  Wednesday,10September-GMT07:00:Bring-your-own-laptope-examforalargeclassatNUS.SeowTeckKeongand

JefferyTay,Na:onalUniversityofSingapore.3.  Thursday,11September-GMT07:00:LargescalefullyonlineBYODfinalexams:Notyourparentsmul:plechoice.Rob

Peregoodoff,UniversityofBri:shColumbia,Canada.4.  Friday,12September-GMT07:00:Finland'sna:onalmatricula:onexamsgoeselectronic.MatLa]u,Matricula:on

Examina:onBoard,Finland.5.  Tuesday,16September-GMT07:00:eOSCE-robustreal:meelectronicmarkingforclinicalexamina:ons.Sebas:an

HunkelerandDrPhilippeZimmermann,Ins:tuteofMedicalEduca:on,UniversityofBerne,Switzerland.6.  Wednesday,17September-GMT07:00:Gamifica:onofClickerswithBYOD.PaulLam,ChineseUniversityofHong

Kong.7.  Thursday,18September-GMT07:00:SafeExamBrowser:Amodularapproachtosecureandflexibleonline-exams.

DanielR.SchneiderandTobiasHalbherr,SwissfederalIns:tuteofTechnologyZurich.8.  Friday,19September-GMT07:00:TenYearsofe-ExamsatFreieUniversitatBerlin:anOverview.AlexanderSchulz&

NicolasApostolopoulos,FreeUniversityBerlin,Germany. 90

End

CitethisresourceHillier,M(2015)“e-Exams:Thestorysofar”.December.