Totypeorhandwrite:student'sexperience

acrosssixe-Examtrials

TransformingExams.com

Dr Mathew Hillier Monash University and University of Queensland &

TransformingExamsAcrossAustraliaAustralianGovernmentOfficeforLearningandTeachingNa:onalGrantID15-4747:AU$500Kover3Years+100KMonash

Acknowledgements-DiamondandGoldSponsors!500KinsupportforthisprojecthasbeenprovidedbytheAustralianGovernmentOfficeforLearningandTeaching.TheviewsexpresseddonotnecessarilyreflecttheviewsoftheAustralianGovernmentOfficeforLearningandTeachingorpar:cipa:ngins:tu:ons.100Kaddi:onalcashfromMonashUniversity.

Demo&Guides

Dr Mathew Hillier; Leader OLT national project. Seed leader

Monash University 5 Oct 2015 (was University of Queensland)

Dr Andrew Fluck; Originator of USB e-exam concept. Seed partner.

University of Tasmania

Dr Michael Cowling, Mr Kenneth Howah Seed trial site.

Central Queensland University

Dr Kim Blackmore Australian National University Assoc. Prof. Paul Newhouse Edith Cowan University Dr Matthew Bower, Prof. Dominic Verity Macquarie University Prof. Marilyn Baird (H), Mr Scott Grant (A) Monash University Assoc. Prof. Shona Leitch RMIT University Dr Ruth Geer, Mr Bruce White University of South Australia TransformingExams.com

Civil Service Exam Under Emperor Jen Tsung (fl.1022) from a history of Chinese emperors(colour on silk): Bibliothèque Nationale, Paris, France.

Is this your exam space?

Thecoreissue!…

Wearefacedwithagrowingdisconnectbetweenthewayhighstakestes*ngisconductedusingpenonpaperexamsandstudents’everydayexperiencesofstudyandlife.

4

e-Exams:Online,Offline,OnCampusorDistance

5

Online

•  Spaceissuesforins:tu:ons.•  Improvedexammanagement

efficiency.•  Equipment:computerlabsbig

enoughtocaterfor2000atonce.•  Moresecure:itissupervised.•  Needsreliablenetwork.

•  Nospaceissueforins:tu:ons.•  Moreefficientexam

management.•  Studentssupplyequipment.

•  Lesssecure:studentsathome.•  Needsreliablenetwork.

Offline

•  Spaceissuesforins:tu:ons.•  Lessefficientexammanagement.•  Equipment:needcomputerlabsto

caterfor2000atonce.•  Moresecure:itissupervised.•  Networkreliabilitynotanissue.

•  Nospaceissueforins:tu:ons.•  Lessefficientexammanagement•  Studentssupplyequipment.

•  Lesssecure:studentsathome.•  Networkreliabilitynotanissue.

OnCampus Distance

Therearetrade-offsforanye-examsolu:on.

Wherewearegoing:Post-paperexamsWeneedgreaterpedagogicalflexibilityandmoreauthen:cassessmentsintheexamroom.…alignment!

6

Simula:ons,toolsofthetrade,virtualexperiments…

'Windows'socwareviaWINE.E.g.CAD/3Dmodeling,Celes:a.

Moodlequizwithmedia(automarked).

UQTrials:PaperEquivalentSuitable format adjustments were made to cater for both paper and screen.

e-ExamTrialsWorkflow

Postsession:retrieveresponsesandassessment

Examroomuse

ResponsesretrievedfromUSBs. Collatede-responsessenttoacademic.

CreatemasterUSB(tested)

USBsduplicatedperstudentAcademiccreatesexamlearningmaterial

Set-up:prepareexamlearningmaterials Pre-session:Studentlaptopsetup&prac:ce.

CollectUSBs(responses)

1. Studentsenterroom.

2. GivenUSB.3. Bootlaptop.4. Doexam.5. ReturnUSB.6. Leaveroom.

e-Examsystemtakesoverlaptop.UbuntuLiveUSB.LibreOffice.

UQ:FirstandMostRecente-Exams

9

VETS2100 S2 2014

DENT4092 S1 2015

Used standard teaching rooms, sought rooms with tables and power sockets.

ß VETS: hand-writers sat in rows. Attempted to separate typists and hand-writers where possible. DENT: typists at the back, à hand-writers at the front.

StudyDesign–Focusisonphase2

10

Phase 1 Institution wide online survey (see Hillier 2014, 2015). Phase 2, Step 1 e-Exam Trial Expression of interest

Typists Handwriters

Phase 2, Step 2 Pre-exam preparation survey

Phase 2, Step 3 Type the exam Handwrite the exam

Phase 2, Step 4 Post-exam survey Par:cipa:oninPhase1:approx.928respondents(Nov2013-Nov2014)Par:cipa:oninPhase2:Eightcourses(sixin2014reportedinpaper,plustwoin2015~updated)

Steps of trial Yes Maybe Total typists Attrition No -

hand-write 1 Expression of Interest 241 241 420 2.1 Pre - before try 124 17 141 100 38 2.2 Pre - after try 112 19 131 10 52 4 Exam (after) 98 98 33 549 Tableupdatedtoinclude2015participants.Finaltypistsbasedonreturnedsurveys.

Typistsandhand-writersbycourse

11

Typed15%

Handwrote85%

Combinedallcohorts

Cohort Typed HandwroteCRIM2014 25.4% 74.6%PHTY2014 18.8% 81.2%VETS2014 12.4% 87.6%ANIM2014 4.4% 95.6%OCTY2014 11.1% 88.9%BIOL2014 9.9% 90.1%CRIM2015 12.1% 87.9%DENT2015 28.8% 71.2%

Proportionoftypistsandhandwritersineachoftheeightcohorts2014-2015

0% 20% 40% 60% 80% 100%

ProporRonoftypistsandhandwritersbycohortTyped Handwrote

Pre-examFirstImpressionsSelectedpre-examsessionsurveyques:ons(typistsonly)Studentscametotesttheirlaptopandtrythesystemacoupleofweekspriortotheexam.

12

QuesJon N Mean SD

Thewripeninstruc:onswereeasytofollow 140 4.0 1.0

Itwaseasytolearnthenecessarytechnicalsteps 137 4.0 1.0

Itwaseasytostartmycomputerusingthee-ExamUSBs:ck 140 4.1 1.2

IfeelconfidentIwillbeabletodothesestepsinarealexam 138 4.0 1.1

Thesocwarewithinthee-ExamSystemwaseasytouse 137 4.1 1.1

Inowfeelrelaxedabouttheideaofusinge-Examformyupcomingexam 138 3.8 1.1

Iwouldliketouseacomputerforexamsinthefuture*(newin2015) 32 4.1 0.9

Updatedtoincludes12015results–8cohorts.

Bars represent medians. Means shown for clarity. Strongly Disagree Strongly Agree

Post-examImpressionsSelectedpost-examsessionsurveyques:ons(likert5=stronglyagree) Updatedtoincludes12015results–8cohorts.

13

Ques:on Typists Hand-writersN Mean SD N Mean SD

Ityped(orhand-wrote)thisexam 98 - - 522 - -Ifeltthee-examsystemwaseasytouse 91 4.4 0.7 - - -Ifeltthee-examsystemwasreliableagainsttechnicalfailures 91 4.0 1.0 - - -Ifeltthee-examsystemwassecureagainstchea:ng 91 4.2 0.9 - - -IlikedthefactIcouldusemyowncomputer 79 4.5 0.8 - - -Iwouldrecommendthee-examsystemtoothers 90 4.3 0.9 - - -Overallmyexperienceofthisexamwasposi:ve 98 4.0 1.0 511 3.7 1.1Iranoutof:me 97 2.7 1.4 508 2.6 1.5IfeltmorestressedinthisexamthanInormallydoinotherexams 97 2.6 1.3 510 2.7 1.3Iwentbackandreadovermyresponsesbeforesubmitng 98 3.5 1.5 509 3.5 1.4Iwouldliketouseacomputerforexamsinthefuture 39 4.2 0.8 167 2.2 1.2Ifeltthispar:cularexamsuitedtheuseofcomputers 92 4.3 0.9 - - -Ithinkmyhandwri:ngwasneatandlegible - - - 513 3.4 1.2Iexperienceddiscomfort(sore/:red/cramp)inmywri:nghand - - - 453 2.4 1.3ItypefasterthanIhandwrite 94 4.5 0.9 439 3.8 1.4Itypeaccurately 93 4.2 0.9 440 3.5 1.1WhenImakeerrors,Iamabletoquicklycorrectthemaspartoftyping 94 4.5 0.8 438 3.9 1.1Iocenrelyonspellchecktodetecterrors 93 3.3 1.3 439 3.5 1.3IworkmoreefficientlywhenItypeonafamiliarkeyboard 94 4.4 0.9 439 4.3 0.9Myhand-wri:ngisnormallyneatandlegible 94 3.3 1.3 439 3.4 1.1

Post-examImpressions

14

Boxplots: responses from typists. Bars represent medians. Means shown for clarity. N = 91

Strongly Disagree Strongly Agree

4.4

4.0

4.2

4.5

4.3

Updatedtoincludes12015results–8cohorts.

Student(typists)impressionsofusingtheexamsystem

Post-examImpressionsDidtypiststhinktheexamsuitedtheuseofcomputers?

15

Boxplots: responses from typists by cohort. Bars represent medians. Means shown for clarity. Overall mean agreement 4.2 Largely that was a ‘yes’. However two factors at play: a)  Self-selecting sample.

Typists would be positive. b)  Exam was ‘paper

equivalent’ thus not taking advantage of what was possible with IT e.g. multimedia, simulations etc

Strongly Disagree Strongly Agree

Updatedtoincludes12015results–8cohorts.

3.9

3.6

4.4

4.8

4.2

4.6

4.0

4.2

Post-examImpressionsHand-wri:ngintheexam

16

Boxplots: responses from hand-writers. Bars represent medians. Means and counts shown for clarity. Note: 1= Strongly Disagree, 5 = Strongly Agree

m N3.9*162.6 243.2 252.8 482.8 462.3 802.41071.8107

m N3.4 763.8 243.7 253.6 482.9 443.5 803.51093.2107

KruskalWallisTestChi-Square 61.060 19.631

df 7 7Asymp.Sig. 0.000 0.006

Updatedtoincludes12015results–8cohorts.

Are some students over estimating the neatness of their hand writing?!

* Note 20% response rate by VETS for this item. All others near 90%

Typingandwri:ngabili:esStudenttypingandwri:ngingeneral

17

4.5 3.8 4.2 3.5 4.5 3.9 3.3 3.5 4.4 4.3 3.3 3.4 Means

Updatedtoincludes12015results–8cohorts.

Likertsc

ale:1=stronglydisa

gree,5=stronglyagree.

Mann-WhitneyU 14703 13079.5 14514 18196.5 18969 19746.5Z -4.708 -5.677 -4.762 -1.694 -1.366 -0.676Sig.(2-tailed) >.001 >.001 >.001 n/s n/s n/s

Typers (left) and Hand writers (right)

Reac:ontocondi:onsintheexamTypistshadamoreposi:veexperienceoverallbutnosignificantdifferenceswerereportedfor:merunningoutandstresslevels.

18

Mann-WhitneyU 21172.5 23645 23065 24252Z -2.539 -0.647 -1.081 -0.447

Asymp.Sig.(2-tailed) >.05 n/s n/s n/s

Typists = purple, Hand-writers = orange

Strongly Agree

Strongly Disagree

Updatedtoincludes12015results–8cohorts.

4 3.7 2.7 2.6 2.6 2.7 3.5 3.5Means

Reac:ontocondi:onsintheexamOverallexamexperiencebycohort

19

stronglydisagree(notgood)stronglyagree(beper)

Updatedtoincludes12015results–8cohorts.

Reac:ontocondi:onsintheexamTimeavailabilitybycohort

20

stronglydisagree(notgood)stronglyagree(beper)

Updatedtoincludes12015results–8cohorts.

FutureInten:onsTypistsweremoreposi:vetowardsfuturee-Exams,asexpected,buthand-writerswerenotnega:veasawhole.

21

Mean N SDTypists 4.2 39 0.8Hand-writers 2.2 167 1.2

Mann-WhitneyU672.5Z-7.961

Sig(2tail)>.001

SD SA

Typists = purple, Hand-writers = orange Updatedtoincludes12015results(4cohorts-new question added in Semester 2, 2014). .

Moreinforma:on….Demoset-upGuide,StudentPrac:ceandUserGuidehpp://transformingexams.com

Demovideosstart-up,useandrecoveryexamples.Applehpp://ta.vu/eexam-demo-a'Wintel'(Dell)hpp://ta.vu/eexam-demo-dContact:mathew.hillier[at]monash.edu

e-ExamProjectResources

22

End

CitethisresourceHillier,M(2015)“Totypeorhandwrite:student'sexperienceacrosssixe-Examtrials”Presenta:onatASCILITEConference,Perth,Australia,30Nov-2Dec.FeedbackSurvey

hpp://ta.vu/eexamsurv

References•  Alfreosson,F.(2014).Bring-your-own-deviceExamSystemforCampuses.Presentedatthe28thNORDUnet

Conference,UppsalaUniversity,Sweden.Retrievedfromhpps://events.nordu.net/display/NORDU2014/Bring-your-own-device+Exam+System+for+Campuses

•  Binkley,M.,Erstad,O.,Hermna,J.,Raizen,S.,Ripley,M.,Miller-Ricci,M.,&Rumble,M.(2012).DefiningTwenty-FirstCenturySkills.InGriffin,P.,Care,E.,&McGaw,B.AssessmentandTeachingof21stCenturySkills,Dordrecht,Springer.hpp://link.springer.com/chapter/10.1007%2F978-94-007-2324-5_2

•  Dahlstrom,E.,&Bichsel,J.(2014).ECARStudyofUndergraduateStudentsandInforma:onTechnology2014.EDUCAUSECenterforAppliedResearch.Retrievedfromhpp://net.educause.edu/ir/library/pdf/ss14/ERS1406.pdf

•  Dahlstrom,E.,&diFilipo,S.(2013).TheConsumeriza:onofTechnologyandtheBring-Your-Own-Everything(BYOE)EraofHigherEduca:on(ResearchReport).Louisville,CO,USA:EDUCAUSECenterforAppliedResearch.Retrievedfromhpp://www.educause.edu/library/resources/byod-and-consumeriza:on-it-higher-educa:on-research-2013

•  Dermo,J.(2009).E-Assessmentandthestudentlearningexperience:Asurveyofstudentpercep:onsofe-assessment.Bri:shJournalofEduca:onalTechnology,40(2),203–214.hpp://doi.org/10.1111/j.1467-8535.2008.00915.x

•  Fluck,A.(2015).Repor:ngProgressWithPost-paperDigitalExams.Educa:onTechnologySolu:onsMagazine.12May.Retrievedfromhpp://educa:ontechnologysolu:ons.com.au/2015/05/12/repor:ng-progress-with-post-paper-digital-exams/

•  Fluck,A.,&Hillier,M.(2014).eExamsTransformingCurriculum.InNowIT’sPersonal(pp.151–158).Adelaide,Australia:ACEC.Retrievedfromhpp://acec2014.acce.edu.au/sites/2014/files/apachments/eExams%20paperd%20-%20REV2b.docx

•  Frankl,G.,Schartner,P.,&Zebedin,G.(2011).The‘SecureExamEnvironment’forOnlineTes:ng(Vol.2011,pp.1201–1211).PresentedattheWorldConferenceonE-LearninginCorporate,Government,Healthcare,andHigherEduca:on,Honolulu,Hawaii,USA.Retrievedfromhpp://www.editlib.org/p/38879/

•  Frankl,G.,Schartner,P.,&Zebedin,G.(2012).Secureonlineexamsusingstudents’devices(pp.1–7).PresentedattheIEEEGlobalEngineeringEduca:onConference(EDUCON),Marrakech.hpp://doi.org/10.1109/EDUCON.2012.6201111

24

References•  Gibbs,G.(1999),UsingAssessmentStrategicallytoChangetheWayStudentsLearn,inS.BrownS.&GlasnerA.(eds),

AssessmentMapersinHigherEduca:on,SocietyforResearchintoHigherEduca:onandOpenUniversityPress,Buckingham,UK

•  Hillier,M.(2014).TheVeryIdeaofe-Exams:Student(Pre)concep:ons.AustralasianSocietyforComputersinLearninginTer:aryEduca:onconference,Dunedin,NewZealand.Retrievedfromhpp://ascilite.org/conferences/dunedin2014/files/fullpapers/91-Hillier.pdf

•  Hillier,M.(2015).e-Examswithstudentowneddevices:Studentvoices.PresentedattheInterna:onalMobileLearningFes:valConference(pp.582-608),HongKong.22-23May.Retrievedfromhpp://transformingexams.com/files/Hillier_IMLF2015_full_paper_formatng_fixed.pdf

•  Hillier,M.&Fluck,A.(2013).Arguingagainfore-examsinhighstakesexamina:ons.InH.Carter,M.Gosper,&J.Hedberg(Eds.),ElectricDreams(pp.385–396).MacquarieUniversity.Retrievedfromhpp://www.ascilite.org.au/conferences/sydney13/program/papers/Hillier.pdf

•  Jamieson,S.(2004).Likertscales:howto(ab)usethem.MedicalEduca:on,38(12),1217–1218.DOI:10.1111/j.1365-2929.2004.02012.x

•  Keong,S.T.,&Tay,J.(2014,September).Bring-your-own-laptope-examforalargeclassatNUS.PresentedattheeAssessmentScotland2014OnlineConference,Dundee,Scotland,UK&Brisbane,Australia.Retrievedfromhpp://transformingassessment.com/eAS_2014/events_10_september_2014.php

•  Kruskal,W.H.,&Wallis,W.A.(1952).UseofRanksinOne-CriterionVarianceAnalysis.JournaloftheAmericanSta:s:calAssocia:on,47(260),583–621.DOI:10.1080/01621459.1952.10483441

•  Lapu,M.(2014).Digitalisa:onoftheFinnishMatricula:onExamina:on-geographyonthefirstwavein2016.InvitedtalkpresentedattheOpenSourceGeospa:alResearchandEduca:onSymposium,Otaniemi,Espoo,Finland.10-13June.Retrievedfromhpp://2014.ogrs-community.org/2014_papers/Lapu_OGRS2014.pdf

•  Mann,H.B.,&Whitney,D.R.(1947).OnaTestofWhetheroneofTwoRandomVariablesisStochas:callyLargerthantheOther.TheAnnalsofMathema:calSta:s:cs,18(1),50–60.DOI:10.1214/aoms/1177730491

25

References•  Melve,I.(2014).DigitalAssessments,onCampusandNetworks.Presentedatthe28thNORDUnetConference,Uppsala

University,Sweden.Retrievedfromhpps://events.nordu.net/display/NORDU2014/Digital+Assessments%2C+on+Campus+and+Networks

•  Mogey,N.,&Fluck,A.(2014).Factorsinfluencingstudentpreferencewhencomparinghandwri:ngandtypingforessaystyleexamina:ons:Essayexamsoncomputer.Bri:shJournalofEduca:onalTechnology.hpp://doi.org/10.1111/bjet.12171

•  Nielsen,K.G.(2014).DigitalAssessmentwithStudents’OwnDevice:ChallengesandSolu:ons.Presentedatthe28thNORDUnetConference,UppsalaUniversity,Sweden.Retrievedfromhpps://events.nordu.net/display/NORDU2014/Digital+Assessment+with+Students%27+Own+Device%3A+Challenges+and+Solu:ons+-+2

•  Peregoodoff,R.(2014).LargeScale-FullyOnlineBYODFinalExams:NotYourParentsMul:pleChoice.PresentedattheeAssessmentScotlandandTransformingAssessmentjointonlineconference.11September.Retrievedfromhpp://transformingassessment.com/eAS_2014/events_11_september_2014.php

•  Ramsden,P.(1992),LearningtoTeachinHigherEduca:on,Routledge,NewYork.•  Ripley,M.(2007).E-assessment:anupdateonresearch,policyandprac:ce.UK:FutureLab.Retrievedfrom

hpp://archive.futurelab.org.uk/resources/documents/lit_reviews/Assessment_Review_update.pdf•  Schulz,A.,&Apostolopoulos,N.(2014).TenYearsofe-ExamsatFreieUniversitatBerlin:anOverview.Presentedatthe

eAssessmentScotlandandTransformingAssessmentjointonlineconference.Retrievedfromhpp://transformingassessment.com/eAS_2014/events_19_september_2014.php

•  Terzis,V.,&Economides,A.A.(2011).Theacceptanceanduseofcomputerbasedassessment.Computers&Educa:on,56(4),1032–1044.hpp://doi.org/10.1016/j.compedu.2010.11.017

•  TransformingExams(2014).'e-ExamSystem'project,hpp://transformingexams.com

220moreat:hpps://www.zotero.org/groups/e-assessment/items/tag/e-exam26