MAKINGACASEFORONLINEEXAMSEFFICIENCY,INTEGRITYANDINSIGHT

DR.PATRICKTRAN&DR.MATHEWHILLIER

UNIVERSITYOFNEWSOUTHWALES

OutlineProposedDAStrategyatUNSW(Mathew)DAFramework(Patrick)AuthenLcityEfficiencyIntegrityInsightDApilotsandexamples(Mathew)

AholisLcdigitalassessmentstrategyisneededPedagogicalignmentacrossandwithincurriculum–formaLvetosummaLve.TechnicalArchitecture–

Acommoncorecapabilitywiththeabilitytoplugincustomassessmenttoolsusedbydisciplineareasandconnecttootherservices

UNSW:Oncampuscontexts:MoodleQuiz+SafeExamBrowserUNSW:Off-campuscontexts:MoodleQuiz+Examity(removeinvigilaLonandrecording)

AddiLonalservices–configuraLon,recording,integritychecking,e-marking,analyLcs,reporLng…Supportforacademics–Professionaldevelopment,training,development,exemplars:UNSW:DigitalAssessmentCommunityofPracLce,DigitalAssessmenttoolkit,technologytraining…Supportforstudents–includingabroaderBYODstrategy,equity,transiLonsupport.Policyandprocedure–assessment,exams,equipment…Aneyetothefuture–studentperformanceatthemacroandmicroacrossthecurriculumneedsaccesstodata(avoidisolatedblackboxes).Workingtogether–faculLes,schools,TELTadmins,learningdesigners,PVCE,ETS,examsunit,IT,studentsupport,disabilityoffice,…andmanyothers!

Proposal:AholisKcassessmentarchitectureacrossmulKplecontextsEnablesecuredigitalassessmentstobedoneacrossarangeofcontextusingcommoncoretools.

Digitalassessmentanywhere,any1me,anyhow!

DAstrategyforUNSW(possible)

On-campusinvigilatedspaces

InvigilaLonservice

SOELabPCsSEB

OtherLabPCSEB

ClassroomLaptopSEB

ExamhallLaptopSEB

HomeBYOlaptopBrowser+Examity

Moodle:providesexamcontent

Off-campusremote

SEB=safeexambrowser:securesacomputerforanexam

DAarchitectureforUNSW(core+extensible)

WinOSXSEBAppsLAN/Wifi

WinOSXSEBAppsLAN/Wifi

BootOSSEBAppsWiFiLAN

BootNetSEBAppsP.upLAN

On-campusinvigilatedspaces

SpecialistCases

InvigilaLonservice:live+record

ConfiguraLonservices

Recordingservices

ExamhallPenonPaper

ScanningcentrePapertodigital

Pen-on-PapertoDigital:encouragedigitalconversion,improveturnaroundandfeedback,retainforspecialuses.

Rapidtransfer

Rapidupload(toLMSassignment)

SOELabPC#WinOSXSEBrowserAppsLAN

OtherLabPCWinOSXSEBAppsLAN

ClassroomLaptop* ExamhallLaptop* HomeBYOLWinOSXe-InvigilaLonviabrowserandwebcam.NBNlol

PC/BYOLWinOSXSecBrwsrSpecialApp/SaaSLAN

*transiLontoBYOlaptop#surfacestudioorstandardSOE

LMSserver:quizorassignment

Integritycheckservices:textmatch+authorverificaLon

Off-campusremoteSEB=SafeExamBrowser LTI Proprietaryengine

Acommoncorethatisextensibletoincludeotherassessmenttoolsandservices.UNSW:MoodleandSafeExamBrowserorremoteproctoring.

Digitalassessmentanywhere,any1me,anyhow!

TowardsauthenLcinvigilatede-assessment(DETBYODroadmap)Start> >>> >>> >>> >>> >Future>GetReady Phase 1 Phase 2 Phase 3 Phase 4 Phase 5

InsLtuLonalapprovals,researchethics,

hardwareandinfrastructure

Paper equivalent

small scale.

Post-paper small to medium.

Medium to large scale.

Whitelisted and logged

Internet

Open but fully logged Internet

Basic doc exams to

begin!

Expanding the app and media

landscape.

Adding the power of an

LMS.

Network BYOD exam.

Network mixed mode BYOD exam.

Crawling Walking Running Jumping Flying!

6

hfp://ta.vu/e-exam-roadmap

Extensionwork:Anofflinee-learningplahormseemoleap.org

Moodleresistanttonetworkoutages+encryptedautosave.

DigitalAssessment(OnlineExams)Framework

DAAuthenLcityAuthenLcassessment*=realism,contextualisaLonandproblemaLsaLon(Villarroeletal2018).Meansofferingworthwhile,significantandmeaningfultasksforstudents.LearningObjecKves DAIdeasandToolsAbilitytoaccessandmanageinformaLon(collecLngdata,observingandinterpreLng)

Researchtasks(usingLibraryresources),Createadigitalartefact(image,audio,video,blog)

Knowledgeandconceptunderstanding(recalling,describing,relaLng)

Onlinequizzes,OralpresentaLons(viaZoom),Createonlineglossaries,Onlinediscussionforum

KnowledgeapplicaLon(problemsolving)

Onlinequizzes,Casestudies,Capstoneprojects,Groupwork,InteracKveassignments(withsimulaLon,game-based,usingsopwaretoconstructresponses)

Analysis(definingandsolvingproblems,analyzingdata,designingexperiments)

Researchproposal,Project-basedtasks,Casestudies

EvaluaLng(criLcalthinking,buildingarguments,reflecLng)

Onlinequizzes,Literaturereview,ResponsetotriggerquesLonspostedonforums,Onlinediscussioninprofessionalnetwork,Groupwork

Designing,creaLng OnlineporRolio,Createadigitalartefact(image,audio,video,blog)

DAEfficiencyThemostimportantmoLvaLngfactorforadopLngDA(OnlineExams)q Efficiency(marking,anywhere/anyLme,recordkeeping,paperless,audit)q Instantresultandfeedback(oratleastquicker)q Morecontrol(randomizaLon,overrides,examkeys,sessionrecording)q VariousquesKontypes(MCQ,calculatedquesLons,drag-drop,drawing…)

q PracKcalexams(sopwaretools-Excel,CAD,programming…)q Richresources(webpages,mulLmedia,databases)q Dataanalysis

DAEfficiency-QuizProducLonQuizMarkupLanguage(QML)PatrickTran,UNSWCanberra

Usingthewebforms

QuestionText

Choice1

Choice2

Choice3

Choice4

Grade

Grade

Grade

Grade

Whatis theslope(gradient)ofthefollowingline?

UsingQML

[[[Whatistheslope(gradient)ofthefollowingline?**pic01**]]]A.-3B.-0.8C.0.8D.1.25[[*]]

[[[...]]]...

Quiz01.txtpic01.png

OnlineQMLConverter:hWp://qml2xml.pro/

OriginalQuesKon

MoodleXML

üExternalgraphicfilesüHuman-readableformat(vsXML)üCopy&PasteüFind&ReplaceüSharing

Demo:hfps://youtu.be/dkfYsj-Hpm8

QMLFormat

hfp://patricktran.info/quiz-generator-for-moodle/

DAIntegrityv Assessmentprotocols(examrulesfromcentraloffice+onlinecontext)

v OnlineProctoring(live,recorded,automated)

v LockdownApplicaKon

Off-CampusInvigilatedExams

MoodleQuiz

InvigilationMonitoringSystem

WebResources

Lockdown

ApplicationsProvidedStart-ExamFile

xPresetConfigxExamKey

Usedefaultconfiguration

QuizTaker

Invigilator

QuizTaker

AdministratorDemo:hfps://youtu.be/K_4-IkSAljw

26216977236

62394391213556433783

1834816269124697321645964354056231822641945184117551522135611611145853807569439430407352315307271265172147132130128103352524165

0 100000 200000 300000

Resource(file)LabelURLPage

ForumFolderLTItool

QuizAssignment

TurniLntooltwoBook

TurniLntoolEcholinkChoice

HSUForumGroupselect

QuesLonnaireFeedback

H5PScorm

MediagalleryOUblog

Workshop(UNSW)Lesson

Elluminate(old)ChatWiki

AfendanceOUwiki

LightboxgalleryScheduler

WebPAGlossaryDialogueDatabase

IMScontentpackageTeambuilder

Workshop(old)Checklist

StudentquizReadinggame

CerLficateAssignment(old)

Survey

Moodletooluse DAInsights:usagepafernsSnapshottaken25Oct2019UNSWMoodleprod.Filteredtoteachingcourseswith>5enrolmentsduring2015to2019.MathewHillier&SamuelZhang(Studentaspartner,ETS)

184035

80906

29005

14771

13773

7552

7348

5150

5005

4911

4493

2260

687

639

531

449

415

208

155

28

0 40000 80000 120000 160000 200000

MulLchoiceRandom

CalculatedTruefalse

EssayNumericalDescripLon

ShortanswerMulLanswer(cloze)

StackMatch

GapselectDragdropintotext

CalculatedmulLDragdropimage

FormulasCalculatedsimpleDragdropmarkers

Poodllrecording(drawing)RandomSAmatch

QuizquesKontypes

2695

228

11 9 73

0

500

1000

1500

2000

2500

3000

Password Javascriptwindow

RequireSEB

SEBkey IPaddressrange

Moodlequizzesusingsecurityfeatures

WeloveMCQsabittoomuch?!

SophisLcatedquesLontypesneedmoresupport,APD,resources?

TheLMSisafilerepository

Approx.17%ofquizzeshaveapassword.

Aperjust2months!

ItisveryearlydaysforSEB.

MulLchoice

Random

Calculated

TruefalseEssayDescripLon

0

20000

40000

60000

80000

100000

120000

140000

160000

180000

200000

2015 2016 2017 2018 2019

QuizquesLontypes(2015-2019)

DescripLon

ShortanswerMulLanswer

(cloze)StackMatch

Gapselect

Dragdropintotext

0

1000

2000

3000

4000

5000

6000

7000

8000

2015 2016 2017 2018 2019

Dragdropintotext

CalculatedmulL

Dragdropimage

FormulasCalculatedsimple

Dragdropmarkers

Poodllrecording(drawing)

RandomSAmatch0

100

200

300

400

500

600

700

800

2015 2016 2017 2018 2019

MoodleisarichtoolsetforassessmentbutismuchunderuLlised.AholisLcsupporteffortisrequired!

DAInsightsAssessmentAnalyKcsv Psychometricquality

q Facilityindex(quesLondifficulty)q DiscriminaLonindex/coefficient

TotalScore(%) Q1 Q2 Q390 1 0 190 1 0 180 0 0 180 1 0 170 1 0 160 1 0 060 1 0 150 1 1 050 1 1 040 0 1 0

Q3Difficulty=correct/all=6/10=0.60Q3DiscriminaKon=(high.correct–low.correct)/groupsize=(5-1)/5=0.80

Moodle:Quiz>Results>StaLsLcs

DAInsights–VisualAnalyLcsMoodlequiz:GradedistribuKonandconfidenceintervals(ExportfromMoodleintoTableau)

Studentsdemographics

MoodleQuiz:QuesKonperformancebycohort

PickingDAtech:Goode-Exams–Threedimensions–Atradeoff?

Mustacknowledgethetensions,

Picktwo?….

Weneedtostriveforallthree.

Relevant and rigorous

Doable and cost effective

Valid and identity verified

AuthenLcity

Integrity Scalability

Weneedtoavoidatechnologicalandpedagogiccul-de-sacduetopoortechnologychoice!

UNSWPilots:PCLabs:SEB+Moodle(orother)

Digitalopenbookwithcontrollede-notes

Moodlequizinlockdownbrowser(SEB)

PilotBIOC11July2019220studentsacross4labs

PotenLal(approx):SOElabs1400seatsOtherlabs+500seatsTotal1900seats

SafeExamBrowsercanbeusedtosecureothertoolssuchasEd

UNSWPilots2019

Stronglyagree

Post-examstudentfeedback(Mean)

BYODUSB CampuslabCampuslab CampuslabSEBMoodleSEBMoodle SEBMoodle SEBMoodleN:107 26 216 17

1 2 3 4 5

ItypefasterthenIhandwrite

ThewrifeninstrucLonswereeasytofollow

Itwaseasytostarttheexam

OverallIfeelthesopwareusedforthisexamwaseasytouse

Ifelttheeexamsystemwasreliableagainsttechnicalfailures

IfelttheeexamsystemwassecureagainstcheaLng

Iwouldrecommendthisapproachtodoingexamstoothers

Iwouldliketouseacomputerforexamsinthefuture

2018Moansh SOLAR3010 BIOC2181 CHEM1011Stronglydisagree

In-classBYOLaptops.SEB+robustMoodleOpLonA:SEBinstalledinstudent’sOS(Windows/MacOS)–lesssecurethanLabsoropLonB.OpLonB:BootfromalternaLveoperaLngsystem(NetworkorUSB).

Example:DETe-Examprojectpilots.Robusttonetworkoutageswithofflineautosave(studentworkcanconLnueregardlessofthenetworkcondiLons).◦ Tables+chairsinstandard‘collaboraLve’layoutclassroom.◦ Powersocketsintables+WiFi(Ethernetcanbeusedtoo,butnotinthiscase).

Pilot2018

DETBYOD:Dynamic,robust,networkede-Exams(2019)

ConfiguraLonserver

Submissionserver

Moodleserver:quizorassignment

Integritycheckservices:textmatch+authorverificaLon

Examcontentserver

HostBYODorLabPCe-ExamOS(USB)SafeExamBrowser

ApplicaLonsLANorWiFi

e-ExamOS(USB)SafeExamBrowser

ApplicaLons

1.  Boothostcomputerwithe-ExamUSB.2.  Systemstartsupinaminimallockedstate.3.  Logontonetwork(viausercredenLalorauto

logon).4.  DownloadconfiguraLonse�ngsfrom

configuraLonserver.5.  ApplynewconfiguraLontosystem.6.  Downloadexamcontentandse�ngsfrom

examcontentserver(orMoodleserver).7.  Deployexamcontent,applyexamse�ng.8.  Launchexamanddotheexam.9.  Progressiveuploadofresponsedataand

monitoringdataduringtheexamtoMoodleorsubmissionserver.Includesofflinebackupduringnetworkoutages*.

10.  Examends.Systemfinalisesdatauploadthencleans-uptheUSB(wipesuserandexamdata),thenshutsdown.

e-ExamOSUSBprovidesthelock-downenvironment.*Manualdataretrievalispossibleincasesoftotalnetworkfailure.UseAdmintoolandUSBhub.

DigitalExamsforHallsPop-uplaba)  Networking:wiredLAN

thenmovetoWiFi.b)  Computers:Unilaptops

thenmovetoBYOD.Islesforpowercables(noaccess).InvertedseaLng.

DETe-Exampilotsinhalls(phase1and2):BYOD+USBboot+nonetwork.

DigitalExamsforHalls

MonashUniversity2019–Caulfieldracetrack(limitedtosub-setofLMSquizquesLons)

TowardsmoreauthenLcassessmentsinexams.•  WefirststartwithMoodlequizwithSEB.•  Thenaddotherappstothewhiteliste.gWord,excel,calculator…simulaLons.

DigitalInkingfore-ExamsatUNSWMoodlefreehanddrawinginstalled8Aug–piloted15Octremoteonlineexam

Annotateanimageorfreehanddrawing,wriLngformulaeanddiagrams.

Commodity$60USBgraphicstabletworksfine(withsomepracLce)ortouchscreendevice.

Infuture… imagescouldbetransferredtoarLficialintelligencemarkingtool?(D.Kellerman)

Off-campusInvigilatedDigitalExamsSteps:1.   Wellbeforetheexam:

•  Academicpreparesexam:SetupexaminMoodleandsetupexamrulesandscheduleininvigilaLonserviceprovidersite.

•  Studentpre-registersattheinvigilaLonserviceprovider(createaccount,setLmezone,setupID).

2.   Beforetheexam:StudentbooksexamLmewithinvigilaLonserviceprovideranddoestechnologycompaLbilitychecks.

3.   ExamKme:StudentlogintoMoodle.Clickslinktoexam.LandsatinvigilaLonprovider.Doestechnologycheck,pre-idchecks,roomscan.Invigilatordoessystemchecks.Clicksstartexam.IstakentoMoodle(orothertool).DoesexamwhilemonitoredbyinvigilaLonservice.Followexitstepswhendone(submitfile,clearcacheetc).

4.   AlerExam:ExamavailableformarkinginMoodleandrecordingreviewed/flaggedbyinvigilator.Videoviewablebyteacher..

Equipmentathome:BYOlaptopwithwebcam.StandardwebbrowserwithconnecLonstoinvigilaLonserviceproviderandexammaterialsinMoodle(orO365).OpLonal/add-on–USBgraphicstablet(digitalpen)fordigitalinkinput.InvigilaKonservice:Examityprovideshumaninvigilators.IdenLtychecks.Sessionoversightviawebcamandmachinesessionrecording(screen,webcamandaudiostream)andpostsessionreview.

Home:BYOLaptopwithwebcam.WindowsorAppleMacOSInternetconnecLon3MBSup3MBSdown

InvigilaLonServiceprovider

Webbrowser

InvigilaLonmonitoringsystemMoodleorassessmenttool

Teacher

cam

LMSServeronAWS

Off-campusinvigilatedexamsarchitecture

Pilotexam:remoteinvigilaLonwithinkingDatascience15Oct2019:60studentssimultaneous.Mixedtextandinkingdrawingandformulae.

AuthenKcproblemsolvinginexams?!

27

AuthenLcNotAuthenLcityisunderstoodasrealism,contextualisaLonandproblemaLsaLon(Villarroeletal2018)

AuthenKcQuesKonsinLMSConstructedenquiry

2.SopwareapplicaLonusedtointerrogateandconstructaresponse.

3.Respondviaform

1.Downloadfile

AuthenKcQuesKonsinLMSConstructedresponse(fileupload)

3.Respondbyfileupload

1.Opensopware

2.UsesopwareapplicaLontoconstructaresponse.

Enhancing LMS Possibility?CodeRunnerques1ontype:Structuredresponsesevaluatedagainsttestcases.Examples:programminglanguagesRstaLsLcs.Note:Requiresseparate‘Jobe’servertoevaluatethecoderesponsessubmifedbystudents.Thisissimilarto‘STACK’(math)usingscriptablequesLontype(wxMaxima)availableinUNSWMoodle.

Getintouch!DrMathewHillierAcademicLead:DigitalAssessment,OfficeofPVCE,UNSWm.hillier@unsw.edu.auhfp://mathewhillier.comUNSWDigitalAssessmenttoolkit(withcasesetcbyDACoPgroup)hfps://teaching.unsw.edu.au/digital-assessment-toolkitOtherInfo:DETnaLonale-Examprojecthfp://TransformingExams.com

DrPatrickTranUNSWCanberraLearningandTeachingGrouppatrick.tran@adfa.edu.auhfp://patricktran.info