Transcript

TheEffectofDigitizingandGamifyingQuizzinginClassroomsAlfIngeWang,MengZhu,andRuneSætreNorwegianUniversityofScienceandTechnology,Trondheim,[email protected]@[email protected]:Theuseofgame-basedlearningintheclassroomhasbecomemorecommoninrecentyears.Manygame-basedlearningtoolsandplatformsarebasedonaquizconceptwherethestudentscanscorepointsiftheycanchoosethecorrectansweramongmultipleanswers.Thearticledescribesanexperimentwherethegame-basedstudentresponsesystemKahoot!wascomparedtoatraditionalnon-gamifiedstudentresponsesystem, aswell as the usage of paper forms for formative assessment. The goal of the experimentwas toinvestigate whether gamified formative assessments improve the students’ engagement, motivation,enjoyment, concentration, and learning. In the experiment, the three different formative assessmenttools/methods were used to review and summarize the same topic in three parallel lectures in an ITintroductorycourse.The firstmethodwastohavethestudentscompleteapaperquiz,andthenreviewtheresults afterwards using hand raising. The second method was to use the non-gamified student responsesystemClickerwherethestudentsgavetheirresponsetoaquizthroughpolling.Thethirdmethodwastousethe game-based student response system Kahoot!. All three lectures were taught in the exact same way,teachingthesamesyllabusandusingthesameteacher.Theonlydifferencewasthemethodusetosummarizethelecture.Atotalof384studentsparticipatedintheexperiment,where127subjectsdidthepaperquiz,175used thenon-gamified student response system, and82 students using the gamified approach. The genderdistributionwas48% female studentsand52%male students.Pre-andapost-testwereused toassess thelearningoutcomeofthelectures,andaquestionnairewasusedtogetdataonthestudents’engagementandmotivation. The results show significant improvement in motivation, engagement, enjoyment, andconcentrationforthegamifiedapproach,butwedidnotfindsignificantlearningimprovement.Keywords: Game-based learning, Student response system, Student engagement, Evaluation, Formativeassessment,Kahoot!

1. IntroductionPrototypesof student responsesystems (SRSs)havebeenaroundsince thesixties (Judson2002),and thesesystemsstartedtobeusedinbiologyandchemistryteachingintheearlyseventies(BesslerandNisbet1971,Casanova1971).ThefirstgenerationofSRSswasbasedonspecialhardwarethatallowedthestudentstogivetheir answers using clickers, key-pads, handsets or zappers (Caldwell 2007). Amajor disadvantagewith thisfirstgenerationofsystemswasthattheyrequiredinvestmentinhardwaredevicesandinfrastructureaswellas administration and maintenance of the hardware and software. The Bring Your Own Device wave hasopened up for a new generation of SRSs, where students can use their own devices to respond. After theintroduction of smart phones and tablets, easy access to wireless network access and support for HTML5,manynewSRSsandsimilar toolshavepopulated themarket: forexampleSocrative (CocaandSlisko2013),Quizlet (Gruenstein,McGrawet al. 2009), Poll Everywhere (Sellar 2011), iClicker (Lucas2009), and LearningCatalytics(Schell,Lukoffetal.2013).TheuseofHTML5web-technologymakesitpossibletousethesesystemswithout installing any applications, and opens up for a range of newways of interacting in the classroom.Kahoot! isagame-basedSRS (GSRS) thatwas introduced to thepublic in the fall2013.Themaindifferencebetween a GSRS and a SRS is that the game-based version focuses more on engaging andmotivating thestudents through attractive graphical user-interfaces and audio, as well by gamifying the whole studentresponseexperience.ThegamificationisdonebytemporarilytransformingtheclassroomintoagameshowasshownonTV,where the teacherplays the roleofagameshowhostand thestudentsare thecompetitors.Well-designedvideogamesaresaidtobelearningmachines(Gee2003),andtheyhavethepotentialtogettheplayerssomotivatedandengagedthattheyarenotawarethatlearningisactuallyhappening.InK-12,gameshave been found to be beneficial for academic achievement, motivation and classroom dynamics (Rosas,Nussbaumet al. 2003). Games have also been found to have a similar effect in higher education (Sharples2000).Previousresearchindicatesthatgamescanbemadeanintegratedpartoftraditionalclassroomlecturestoimprovelearning,motivationandengagement(CarverJr,Howardetal.1999,Carnevale2005,Wang,Øfsdaletal.2007,Wang,Øfsdaletal.2008,Wu,Wangetal.2011).

ThisarticlepresentsanexperimentwheretheGSRSKahoot!wascomparedtoaSRS,aswellastotheusageofpaper forms for formative assessment. Section 2 presentsmaterial andmethods including related work, adescription of the formative assessment tools used in the experiment, the data sources used, the researchcontext and participants of the experiment, the experiment procedures, and the data analysis. Section 3presentstheresultsfromtheexperiment.Section4concludesthearticle.

2. RelatedWorkTherehavebeenmanyexperimentsandstudiesconductedonSRSs,andaliteraturestudyfrom2007reportsthat such systems have been found to have a positive effect on student exam-performance, and that theycreateamorepositiveandactiveatmosphereinclassrooms(Caldwell2007).Morespecifically,studentsusingSRSsweretwiceas likely toworkonaproblempresentedduringclass (Cutts,Kennedyetal.2004),studentattendanceroseto80-90%(BurnsteinandLederman2001),andabout88%ofthestudentseither“frequently”or “always” enjoyed using the SRS in class (Caldwell 2007) Further, Caldwell’s survey summarized somecommonusesofclickerquestionsfoundintheliterature:toincreaseormanageinteraction,toassessstudentpreparationandensureaccountability,tofindoutmoreaboutstudents,forformativeassessment,forquizzesortests,todopracticeproblems,toguidethinkingrevieworteach,toconductexperiments,tomakelecturesfun, to differentiate instruction, and to prompt discussion. Another study summarizes similar findings ofbenefitsofSRSsinthethreeareasClassroomenvironment,Learning,andAssessment(KayandLeSage2009).SRSs were found to improve attendance, provide more focused students, provide anonymous studentparticipation, improved student engagement, increase learning performance, improved teaching, andgenerally improve interaction between teacher and students. The benefits listed in these surveys are allbenefitswehaveexperiencedfromusinggame-basedSRSinclassroomsaswell.There are several studies on the effects of educational games related to learning outcome and increasedmotivation,andwewillpresentsomeofthesestudieshere.Onestudyexploredtheimpactofusingagamenamed Supercharged! on pre-service teachers’ understanding of electromagnetic concepts compared tostudentswhoconductedamoretraditionalinquiryorientedinvestigationofthesameconcept(AndersonandBarnett 2011). The effectiveness was investigated through an experiment that used both qualitative andquantitative data that included pre- and post-scores, student notebooks, video recordings of laboratoryactivities and observations. The results of this study showed that the group using the video gamesoutperformed the group that did not use the video game in terms of learning outcomes (statisticallysignificant). In another study, the video game relative asteroids was used to teach physics (Carr andBossomaier 2011). A pre- and post-test with 8 questions was used tomeasure the learning outcome. Theresultsshowedimprovementofthetestscoresfornewlearners.Thestudentsalsofoundthatlearningphysicsthrough a game was motivating and engaging, and it was effective at improving their comprehension ofphysics. Ina chemistry class,anexperimentwasconducted tocompare students’achievementandattitudefromtraditionalvs.game-basedteachingmethods(Tüysüz2009).Thestatisticallysignificantconclusionofthestudywasthatgame-basedlearningincreasedthestudents’achievementinchemistrycomparedtotraditionallearningmethods. The study also showed that game-based learning increased the students’ interest in thecourse: they enjoyed the coursemore, and weremore focused and engaged on the subject being taught.Similarresultsonimprovedlearningoutcomewerealsofoundforusingaweb-basedadventuregametoteachneuroscience (Miller, Schweingruber et al. 2002), in an experiment comparing teaching computer memoryknowledge with a game vs. a non-game application (Papastergiou 2009), and for using a mobile game toengagestudentsinarithmeticpractices(Liao,Chenetal.2011).Inthecomputermemorystudy,thestudentsthatusedagamefoundthislearningapproachsignificantlymoreappealingandeducationalfruitfulthanthestudents with the non-game application (Papastergiou 2009). The same students also found their learningapproachmore engaging,more effective,more active and relaxed compared to the students that used thenon-gameapproach.Therearealsostudiesthatshowthat introducinggames intotheclassroomnotalwaysproducepositiveresultsandcanresultincomplainingstudentsandlackofmotivation(Squire2005).Kahoot! represents a new generation of student-response systems that has a main focus on studentmotivationandengagementthroughgamification.ThetoolisaresultoftheresearchprojectLectureQuizthatstarted in2006 (Wang,Øfsdaletal.2007),where results fromexperimentationofearlyprototypes showedpositive results in terms of increased engagement, motivation and perceived learning (Wang, Øfsdal et al.2008,Wu,Wangetal.2011).Educationalgamescomparedtomainstreamentertainmentgamesareknowntosuffer from running on very few platforms (usually Windows PCs), too simplistic, being single player and

offline,offering lowproductionvalue,andaretypicallymoretargetedtowardsparents, teachersandformallearning curriculum thanbeing fun for the students (Kirriemuir andMcFarlane2004). This is especially truewheneducationalgamestrytocopyexistinggameconceptsandaddsomelearningontopofit.Kahoot!wasnotdesignedtocopyanyexistinggame,butrathertofindagameconceptthatcouldfitaclassroomsettingandthatcouldbealignmentwithTomMalone’stheoryofintrinsicallymotivatinginstructions(Malone1980).Malone’s theory lists three categories that make things fun to learn: Challenge (goals with uncertainoutcomes),Fantasy (captivate through intrinsic or extrinsic fantasy), andCuriosity (sensor curiosity throughgraphics and audio, and cognitive curiosity). As the game should be used in the classroom, it was alsoimportanttoincorporateasocialgameplay.Theresultwastodevelopagameconceptwherethefantasy isthat the classroom temporarily is changed to a game show where the teacher is the game host and thestudentsare thecompetitors.Thechallenge is toanswerquestionsandcompeteagainstotherplayers,andthecuriosityisprovidedthroughinspiringgraphicsandaudio,aswellassolvingacognitivepuzzle.Thelackofvariety ingameplay iscompensatedbythecompetitivenatureofplayingagainstawholeclassofstudents.Reports from happy teachers and students all over theworld give an indication that the conceptworks asintended.Learninggamesarecommonlyusedtoreviewfactsusingmultiple-choicequestionssimilartowhatis done in Kahoot!. However, such games can also be used to teach skills, judgment, behaviors, theories,reasoning,process,procedures,creativity, language,systems,observation,andcommunicationusingvariousapproaches(Prensky2005).

3. MaterialandMethodThissectionpresentsthethreeassessmenttoolsused,thedatasources,theresearchcontextandparticipants,researchprocedures,andthemethodfordataanalysis.

3.1 ResearchQuestionsandResearchApproachTheresearchgoaloftheexperimentpresentedinthisarticlewastoinvestigatehowdigitizingandgamifyingquizzing in the classroom affects the students’ motivation, enjoyment, engagement, concentration andlearning.Specifically, thisexperiment investigates the impactof choicebetween runningaquizusingpen&paper,usingastudentresponsesystem(SRS)andusingagame-basedSRSaffectsthestudents’perceptionofthe quiz. The researchmethod used is based on the Goal, QuestionMetrics (GQM) approach (Basili 1992)wherewefirstdefinearesearchgoal(conceptuallevel),thendefineasetofresearchquestions(operationallevel),andfinallydescribeasetofmetricstoanswerthedefinedresearchquestions(quantitativelevel).

3.1.1 ResearchGoalandResearchQuestions

TheresearchgoalofthisstudywasdefinedasthefollowingusingtheGQLtemplate:Thepurposeofthisstudywastoevaluatetheeffectofchoiceofquiztoolforreviewtaughtmaterialfromthepointofviewofastudentinthecontextofalecture.Thefollowingresearchquestions(RQs)weredefinedbydecomposingtheresearchgoal:

• RQ1:Howdoesthechoiceofquiztoolaffectthestudents’motivation?• RQ2:Howdoesthechoiceofquiztoolaffectthestudents’enjoyment?• RQ3:Howdoesthechoiceofquiztoolaffectthestudents’engagement?• RQ4:Howdoesthechoiceofquiztoolaffectthestudents’concentration?• RQ5:Howdoesthechoiceofquiztoolaffectthestudents’learningoutcome?

3.2 ThreeMethodsforRunningQuizzesinaClassroom

In theexperimentpresented, threedifferentapproacheswereusedtorunaquizasapartofa lecture.Thequizwasusedtoreviewwhatbeingtaughtinthelectureandconsistedof12multiple-choicequestions.Thequizmethodsusedwerepaper forms, a simple student response systemnamedClicker, and a game-basedstudentresponsesystemnamedKahoot!.Figure1showspictures fromthethree lecturesofstudentsdoingthequiz(left:paperquiz,middle:Clicker,right:Kahoot!).Thethreequizmethodswillnowbedescribedmoreindetail.

Figure1Picturesfromlecturesusingthreedifferentquizmethods

3.2.1 PaperFormandHandRaising

Thepaperquizisananalogue,well-knownandprovenapproachforrunningquizzesinaclassroom.Beforealecture, the teacherpreparespaper formswithmultiple-choicequestionswherestudentscan tickofoneoftwotofouranswersusingapenorapencil.Duringthelecture,theteacherhandsoutthepaperforms,andthe students answers aswell as they can. The normal procedure for such quizzes is that the teacher thencollectsthepaperforms,andthestudentsgetfeedbackinthefollowinglectureonhowtheyhaveperformed.Inourexperiment,wechangedthereviewparttobemorecompatiblewiththedigitalcounter-parts.Afterthestudents had completed their forms, the teacher reviewed their answers by going through the questionsaskingstudentstoraisetheirarmsforthealternativetheyhadanswered.Inthisway,theteachergottoknowhowthestudentshadansweredandgavehimthechancetogivefeedbacktotheclass.Thepaperformsusedconsistedofaquizoftwelvequestions,allwithfouralternativeanswers.

3.2.2 TheClickerStudentResponseSystem

Clickerisasimplestudentresponsesystem(SRS)allowingstudentstogivetheirresponsestoquestionsbeingaskedusingaweb-browseronanydigitaldevice.ThequestionsandanswersaretypicallyshownusinganothertoollikeKeynote,PreziorPowerPoint,andClickerisusedtocollect“votes”fromthestudents.Figure2showsanoverviewoftheClickerSRSandthestepsneededtogetresponsesfromstudents.First,theteacherneedsto name a classroom which will be the ID used for students to connect (Figure 2a). Second, the teacherchooseshowthestudentscanresponsetoaquestion(Figure2b).Third,theteacherhastoshowthequestionsandanswersonthescreenusingPowerPointorsimilartoolaswellashestartsthevotingprocess(Figure2c).In this step, the teacher canmonitor howmany students that have given their answers. Forth, the teacherstopsthevoting,andadistributionofhowthestudentshavevotedisshown(Figure2d).Thedistributionofanswersisalsoshownonthestudentclient.TheClickerSRSdoesnotdirectlygivetheteacherorstudentsanyfeedback on correctness of answers. It is up to the teacher to comment on correctness of the students’responsesbasedonthegivendistributionofvotes.

Figure2OverviewoftheClickerStudentResponseSystem

3.2.3 TheKahoot!Game-basedStudentResponseSystem

Kahoot! is a game-based student response system (GSRS) launched by the teacher in a web-browser on alaptop connected to a large screen. Unlike the Clicker SRS, Kahoot! provides a tool for creating quizzes

includingaddingpicturesandYouTubevideostothequestions.Kahoot!alsomakesitpossibletopublishandshare yourownquizzes, andedit quizzesmadebyothers.Anotherdifference is thewayKahoot! is played.Studentswilllogintothesystemusingagamepin(anumber)andanickname.Thegoalforthestudentsistoanswerthecorrectanswerasfastaspossibletogetasmanypointsaspossible.Figure3showshowKahoot!isplayed.Aquestion is shownon the large screenalongwith fouror less alternatives indifferent colorswithassociated graphical symbols. The students give their answers by choosing the color and symbol she or hebelievescorrespondstothecorrectanswer.

Figure3PlayingKahoot!Betweeneveryquestion,adistributionofhowthestudentsansweredisshownbeforeascoreboardofthefivebestplayers.Thestudentsget individualfeedbackontheirquestions intermsofcorrectness,thenumberofpoints,theranking,howfarthestudentisbehindthestudentrankedabove,andthecorrectanswerifwronganswerisgiven.AttheendofaKahoot!session,thewinner’snicknameandpointswillbeshownonthelargescreen.Duringthequiz,Kahoot!usesaplayfulgraphicaluserinterfaceaswellasmusicandsoundstogiveitaplayful and competitive atmosphere similar to a game show on TV. The students are also asked to givefeedbackonthequiztheyhaveplayedthroughgivingscoresonwhetherthequizwasfun,educational,canberecommendedtoothers,andhowyougenerallyfeelaboutthequiz.Finally,Kahoot!providesthefunctionalityfortheteachertodownloadtheresultsfromthequizinanExcelspreadsheet.

3.3 DataSourcesTheinstrumentsforcollectingdatainourexperimentincludedadomainknowledgetestandaquestionnaireon students’ engagement and motivation. The domain knowledge test consisted of seven multiple-choicequestionsdevelopedbyadomainexpert,andthisquestionnairewasusedbothasapre-testandapost-testtomeasuretheknowledgebeforeandafterthelecture.Themotivationquestionnairewasdevelopedtomeasurethemotivationandtheengagementofthestudents.Thequestionnairewasadaptedfromthecoursemotivationsurvey(CMS)(Kebritchi,Hirumietal.2010)toourresearch context, and integrated with relevant questions in the Motivated Strategies for LearningQuestionnaire (MSQL) (Pintrich 1991) and (Lepper, Corpus et al. 2005). The questionnaire used a five-pointLikertscalefromstronglydisagreetostronglyagree.

3.4 ResearchContextandParticipantsTheexperimentwasperformed in the IT introductory course (TDT4105) atNorwegianUniversity of ScienceandTechnology(NTNU).Thereweretworeasonsforchoosingthisparticularcoursefordoingtheexperiment.First, the IT introductory course is a large coursewithmany students,meaning that itwouldbepossible tocollectdatafrommanysubjects.Second,duetothesizeofthiscourse,thesamelecturehastobetaughtinthree parallels. Thismeans that the same teacherwill teach exactly the same lecture for three parallels ofstudents.TheIT introductorycourseisamandatorycourseforall firstyearstudentsattheuniversity,giving

thatthegroupsofstudents intheexperimentshouldbefairlyuniform.TheexperimentwasconductedoverthreedaysattheendofSeptember2013,andthetopicofthelecturewasonbasiccomputerknowledge.384students participated in the experiment where the distribution of the subjects that completed thequestionnaireswas127subjectsforpaperquiz(58%femalevs.42%male),175subjectsforclickerquiz(37%malevs.63%male),and82subjectsforKahoot!(54%femalevs.46%male).

3.5 ProceduresThelectureintheexperimentwasconductedaccordingtoFigure4.First,theteacherintroducedthelectureby presenting the agenda and the current topic, before the students carried out a paper pre-test on thelecture’stopic.Second,theteachertaughtthetopicbasiccomputingusingPower-pointslides.Third,aquizonthetopicwascarriedoutinthreevariationsforthethreeparallels(Paper,Clicker,andKahoot!).Forth,attheendofthelecture,thestudentshadtofill inamotivationquestionnaireaswellasdoingthepaperpost-test(sameasthepre-test).

Figure4ExperimentProcedures

3.6 DataAnalysisTheanswers fromthepre-andpost-testswereevaluatedtoascore from0to7points,whereeachcorrectanswer contributed one point to the total score. The learning outcome was computed by comparing thedifference between the post- and the pre-test scores. The Mann-Whitney test was used to compare thedifferenceinthelearningoutcomesfromthedifferentquizmethods(Paper,ClickerandKahoot!).TheMann-Whitney test isanonparametric test for the significanceof thedifferencebetween thedistributionsof twoindependentsamplesofdifferencesizes.TheKruskal-Wallistestwasrunonthedatafromthemotivationquestionnairetoinvestigatethedifferencesbetween the responses from the three groups Paper, Clicker, and Kahoot!. The Kruskal-Wallis test is anonparametric test for the significance of the differences among the distributions of in our case threeindependentsamplesthathaddifferencesizes.

4. ResultsThissectionpresents theresults fromthecontrolledexperiment. In theanalysiswe lookedatdifferences instudents’motivation, enjoyment, engagement, and concentration in regards to the used quizmethod. Thissectionalsoreportsondifferencesinthelearningoutcome.

4.1 RQ1:EffectonMotivation

Table1showsthedescriptivestatisticsandtheresultsfromtheKruskal-Wallistestforstatementsrelatedtomotivation.Theresultsshowthatthere isastatisticallysignificantdifference inthestudents’motivationfordoingthequiz (statement1).TherewasnotmuchdifferencebetweenthePaperandtheClickerquizwherehalfof thestudentshad internalmotivationfordoingthequiz.ForKahoot!almost fouroutof fivestudentshadaninternalmotivation,andonly5%didthequizonlybecausetheteachertoldthemto.However,therewas no significant difference related to the motivation of the lecture as a whole (statement 2), nor the

motivationforlearningmoreaboutthetopic(statement3).Thereisatendencythatthestudentsweremoremotivatedforalectureusinggame-basedquizzes(statement2),butnotstatisticallysignificant.Table1ResultsonMotivationStatement Group Disagree Neutral Agree H P1. I did the quiz only because theteachertoldmeto

Paper 50% 25% 25%14.2 0.0008Clicker 56% 29% 15%

Kahoot! 77% 18% 5%2.Idonotthinkthislecturewasworthmytimeandeffort

Paper 56% 29% 15%2.79 0.2478Clicker 51% 32% 17%

Kahoot! 63% 24% 12%3. I enjoyed the quiz so much that Iwanttoknowmoreaboutthetopic

Paper 37% 43% 20%1.4 0.4966Clicker 34% 39% 27%

Kahoot! 35% 49% 16%

4.2 RQ2:EffectonEnjoyment

Table2showsthedescriptivestatisticsandtheresultsfromtheKruskal-Wallistestforstatementsrelatedtoenjoyment.Thetableshowsthatthereisastatisticallysignificantdifferenceregardingtheengagementofthequizitself(statement5)wherethegame-basedapproachwasclearlyperceivedasmoreengagingcomparedtothe paper quiz and the Clicker student response system. No statistically significant differences were foundrelated to completing the quiz (statements 4 and 6), although there is a tendency that the game-basedapproach to a larger degree gave students a satisfaction on completion compared to the two otherapproaches.Table2ResultsonEnjoymentStatement Group Disagree Neutral Agree H P4. To complete the quiz gave mesatisfaction

Paper 35% 33% 32%3.1 0.2122Clicker 28% 41% 31%

Kahoot! 22% 38% 40%5. The quiz was boring and notengaging

Paper 60% 34% 6%9.59 0.0083Clicker 63% 26% 10%

Kahoot! 84% 12% 4%6. It gaveme satisfaction to completethequizinasatisfactoryway

Paper 18% 41% 41%1.65 0.4382Clicker 16% 40% 44%

Kahoot! 15% 34% 51%

4.3 RQ3:EffectonEngagement

Table3showsthedescriptivestatisticsandtheresultsfromtheKruskal-Wallistestforstatementsrelatedtoengagement.Thetableshowsthattherewasastatisticallysignificantdifferenceinhowthestudentsperceivedthe engagement of the quiz for both statements. For Kahoot! over half of the students expressed that themost satisfactory in the lecture was to do well on the quiz. There was even a larger difference for thestatementonwhetherthequizcauseincreasepulsewhereabouthalfofthestudentsdoingthegame-basedquizclaimedtohaveincreasedpulse,comparedtoaround10%fortheothertwoapproaches.Table3ResultsonEngagementStatement Group Disagree Neutral Agree H P7.Todowellonthequizwasthemostsatisfactoryinthelecture

Paper 38% 40% 22%15.91 0.0004Clicker 35% 37% 27%

Kahoot! 22% 26% 52%8. I felt increased pulse when Iansweredquestionsinthequiz

Paper 65% 22% 13%31.78 <0.0001Clicker 66% 26% 8%

Kahoot! 34% 20% 46%

4.4 RQ4:EffectonConcentrationTable4showsthedescriptivestatisticsandtheresultsfromtheKruskal-Wallistestforstatementsrelatedtoconcentration. The table shows that the statistically significant difference for students doing a game-basedquiztoalargedegreewishedtodobetteronthequizthantheirfellowstudents(statement11).Almost70%ofthestudentsdoingtheKahoot!quizhadahigherwishtodowellonthequiz,comparedtoaround40%forthe twoother approaches.On the statementdirectly related to concentrationondoing thequiz, there is atendencythatstudentsdoingtheClickerquizconcentratedmorethanstudentsdoingthepaperquiz,andthestudentsdoingtheKahoot!quizconcentratedmorethanthestudentsdoingtheClickerquiz.Interestinglyonthequestionregardingworkingindependently(statement10),thereisatendencythatthestudentsusingthestudentresponsesystemstoalargerdegreewantedtoworkontheirowncomparedtothosedoingthepaperquiz.Table4ResultsonConcentrationStatement Group Disagree Neutral Agree H P9. I concentrated on the quiz to getcorrectanswer

Paper 8% 31% 61%4.59 0.1008Clicker 7% 22% 71%

Kahoot! 6% 16% 78%10. I wanted to answer quiz withouthelpfromothers

Paper 20% 20% 60%5.19 0.0746Clicker 10% 17% 73%

Kahoot! 9% 20% 72%11. I wished to do better on the quizthanmostotherstudentsintheclass

Paper 24% 39% 38%20.66 <0.0001Clicker 16% 41% 43%

Kahoot! 4% 28% 68%

4.5 RQ5:LearningOutcome

WewereableonlytocomparethelearningoutcomefromthetwoparallellectureswherePaperandKahoot!wereusedastheresultsfromthepost-testintheClickerquizlecturewereincompleteduetolackoftimeattheendofthelecture.ResultsfromtheMann-WhineytestalongwiththedescriptivestatisticsareshowninTable6.TheMin,Max,MeanandMedianshowsimprovementfrompre-testtopost-testinnumbercorrectanswersinthetest.ThereisatendencyforahighermeanvalueforthelecturewithKahoot!quizcomparedtothelecturewiththePaperquiz,butthedifferenceisnotstatisticallysignificant.Table5LearningOutcomefromPaperQuizvs.Kahoot!quiz

Treatment N Min Max Mean Median UA Z PPaper 127 0 7 3.669 4

5655.5 -1.05 0.1469Kahoot! 82 1 6 3.817 4

5. ConclusionIn this article,wehavepresentedanexperiment to investigatehow theusageofquizzes in review-lecturesaffectsmotivation (RQ1),enjoyment (RQ2),engagement (RQ3),motivation (RQ4),andthe learningoutcome(RQ5). Our experiment revealed that students using a game-based student response system compared topaper forms and a simple non-game-based student response system were more engaged, motivated andconcentrated, and enjoyed it more. The statistically significant differences were only related to activity ofdoing the quiz itself, and not the lecture in general. Regarding the learning outcome, the results from theexperimentdidnot showany statistically significantdifferencesbetween thequizmethods (onlypaperandKahoot! was tested). In future studies, wewill investigatemore thoroughly whether the learning outcomevariesbythemethodorbythequiz-toolusedforalongerperiodoftime.

ReferencesAnderson,J.andM.Barnett(2011)."Usingvideogamestosupportpre-serviceelementaryteacherslearningofbasicphysicsprinciples."JournalofScienceEducationandTechnology20(4):347-362. Basili,V.R. (1992).Softwaremodelingandmeasurement: theGoal/Question/Metricparadigm,UniversityofMarylandforAdvancedComputerStudies. Bessler,W.C.andJ.J.Nisbet(1971)."Theuseofanelectronicresponsesysteminteachingbiology."ScienceEducation55(3):275-284. Burnstein,R.A.andL.M.Lederman(2001)."Usingwirelesskeypads in lectureclasses."ThePhysicsTeacher39(1):8-11. Caldwell, J. E. (2007). "Clickers in the large classroom: Current research and best-practice tips." CBE-LifeSciencesEducation6(1):9-20. Carnevale, D. (2005). "Run a class like a game show:‘Clickers’ keep students involved." Chronicle of HigherEducation51(42):B3. Carr, D. and T. Bossomaier (2011). "Relativity in a rock field: A study of physics learning with a computergame."AustralasianJournalofEducationalTechnology27(6):1042-1067. CarverJr,C.A.,etal.(1999)."Enhancingstudentlearningthroughhypermediacoursewareandincorporationofstudentlearningstyles."Education,IEEETransactionson42(1):33-38. Casanova,J.(1971)."Aninstructionalexperimentinorganicchemistry.Theuseofastudentresponsesystem."JournalofChemicalEducation48(7):453. Coca,D.M.andJ.Slisko (2013)."SoftwareSocrativeandSmartphonesasToolsFor ImplementationofBasicProcesses of Active Physics Learning in Classroom: An Initial Feasibility Study With Prospective Teachers."EuropeanJournalofPhysicsEducation4(2). Cutts,Q.I.,etal.(2004).MaximisingDialogueinLecturesusingGroupResponseSystems.CATE. Gee,J.P. (2003)."Whatvideogameshavetoteachusabout learningand literacy."Comput.Entertain.1(1):20-20. Gruenstein, A., et al. (2009). A self-transcribing speech corpus: collecting continuous speechwith an onlineeducationalgame.SLaTEWorkshop. Judson, E. (2002). "Learning from past and present: Electronic response systems in college lecture halls."JournalofComputersinMathematicsandScienceTeaching21(2):167-181. Kay,R.H.andA.LeSage(2009)."Examiningthebenefitsandchallengesofusingaudienceresponsesystems:Areviewoftheliterature."Computers&Education53(3):819-827. Kebritchi, M., et al. (2010). "The effects of modern mathematics computer games on mathematicsachievementandclassmotivation."Computers&Education55(2):427-443. Kirriemuir,J.andA.McFarlane(2004)."LiteratureReviewinGamesandLearning." Lepper,M.R.,etal.(2005)."Intrinsicandextrinsicmotivationalorientationsintheclassroom:Agedifferencesandacademiccorrelates."Journalofeducationalpsychology97(2):184. Liao, C. C., et al. (2011). "My-Mini-Pet: a handheld pet-nurturing game to engage students in arithmeticpractices."JournalofComputerAssistedLearning27(1):76-89.

Lucas,A. (2009). "Usingpeer instruction and i-clickers to enhance studentparticipation in calculus." Primus19(3):219-231. Malone, T. W. (1980). What Makes Things Fun to Learn? Heuristics for designing Instructional ComputerGames. The 3rd ACM SIGSMALL symposium and the first SIGPC symposium on Small systems. Palo Alto,California,UnitedStates,ACMPress. Miller,L.,etal.(2002)."Teachingneurosciencethroughwebadventures:adolescentsreconstructthehistoryandscienceofopioids."TheNeuroscientist8(1):16-21. Papastergiou,M.(2009)."DigitalGame-BasedLearninginhighschoolComputerScienceeducation:Impactoneducationaleffectivenessandstudentmotivation."Computers&Education52(1):1-12. Pintrich,P.R.(1991)."AmanualfortheuseoftheMotivatedStrategiesforLearningQuestionnaire(MSLQ)." Prensky, M. (2005). "Computer games and learning: Digital game-based learning." Handbook of computergamestudies18:97-122. Rosas, R., et al. (2003). "BeyondNintendo: design and assessment of educational video games for first andsecondgradestudents."ComputerEducation40(1):71-94. Schell, J., et al. (2013). "Catalyzing learner engagement using cutting-edge classroom response systems inhighereducation."Cutting-edgeTechnologiesinHigherEducation6:233-261. Sellar,M.(2011)."Polleverywhere."TheCharlestonAdvisor12(3):57-60. Sharples,M.(2000)."Thedesignofpersonalmobiletechnologiesforlifelonglearning."Comput.Educ.34(3-4):177-193. Squire, K. (2005). "Changing the game:What happens when video games enter the classroom." Innovate:Journalofonlineeducation1(6). Tüysüz, C. (2009). "Effect of the computer based game on pre-service teachers’ achievement, attitudes,metacognitionandmotivationinchemistry."SciResEssays4(8):780-790. Wang,A.I.,etal.(2007).LectureQuiz-AMobileGameConceptforLectures.IASTEDInternationalConferenceonSoftwareEngineeringandApplication(SEA2007).Cambridge,MA,USA,ActaPress:6. Wang,A.I.,etal.(2008).AnEvaluationofaMobileGameConceptforLectures.Proceedingsofthe200821stConferenceonSoftwareEngineeringEducationandTraining-Volume00,IEEEComputerSociety. Wu,B.,etal.(2011).ImprovementofaLectureGameConcept-ImplementingLectureQuiz2.0.Proceedingsofthe3rdInternationalConferenceonComputerSupportedEducation.