Interactive engagement strategies for large classes

Interactive Engagement Strategies For large and diverse classes (a choose-your-adventure workshop)

14-15 JUNE 2017 LIVERPOOL JMU TEACHING & LEARNING CONFERENCE “VISIONS FOR LEARNING”

Choose your own adventure

Lectures as 2-way conversations Interaction (in / out of class)

Assessments (student-generated) Assessments (exams)

Lectures

Context - class time (specifically lectures)

CC BY-NC 2.0 https://flic.kr/p/f3ynHx!

Derek Bruff: Class time reconsidered!http://prezi.com/donq036eunko/class-time-reconsidered/!

https://www.brookes.ac.uk/services/ocsld/resources/20reasons.html!

Sourcear)cle:Poh,M.Z.,Swenson,N.C.,Picard,R.W.,"AWearableSensorforUnobtrusive,Long-termAssessmentofElectrodermalAc)vity,"IEEETransac)onsonBiomedicalEngineering,vol.57,no.5,pp.1243-1252,May2010.doi:10.1109/TBME.2009.2038487Ac)vity:JaredStang,UBCPhysics

Weekly rhythm for our 1A class!

Poh,M.Z.,Swenson,N.C.,Picard,R.W.,"AWearableSensorforUnobtrusive,Long-termAssessmentofElectrodermalAc)vity,"IEEETransac)onsonBiomedicalEngineering,vol.57,no.5,pp.1243-1252,May2010.doi:10.1109/TBME.2009.2038487

Weekly rhythm for our 1A class!

Learning gains on PI !

1. Pre class material!

© Jorge Royan / http://www.royan.com.ar / CC-BY-SA-3.0, via Wikimedia Commons!

1. Pre class material - DIY media!

1. Pre class material - DIY media!

Learning gains on PI !

Reproduced from Eric Mazur (search “Confessions of a converted lecturer” on YouTube)

1!

2!

3!

A ball initially at rest is!thrown upwards, comes !back down & is caught!!Which of the following is a!plausible graph of the!acceleration of the ball !with time?!

Source activity: Simon Lancaster (UEA, UK) Ross Galloway (Edinburgh, UK)

A large truck collides head on with a small compact car. Which of the following statements is true? 1. The forces on the car is greater 2. The force on the truck is greater 3. The force on the car and truck are

equal 4. Can’t specify without knowing mass

and speed of vehicles

R R Hake !American Journal of Physics: Volume 66, Issue 1, Pages 64-74!http://dx.doi.org/10.1119/1.18809!

Freeman et al PNAS www.pnas.org/cgi/doi/10.1073/pnas.1319030111

Wieman commentary PNAS !http://www.pnas.org/cgi/doi/10.1073/pnas.1407304111

“This meta-analysis makes a powerful case that any college or university that is teaching its STEM courses by traditional lectures is providing an inferior education to its students”

1.  Real (and perceived) workloads!

2.  Dealing with resistance !

3.  Suitability of teaching spaces!

4.  About ‘covering content’!

5.  Loss of (total) control !

!

… and the upsides too. !

Interaction / communication

FOSTERING INTERACTIONa case study

Simon Bates Senior Advisor, Teaching and Learning

Academic Director, CTLT

Professor of Teaching, Department of Physics and Astronomy

OVERVIEW  Landscape of LT tools and applications

  My course context

  Tools matter!

  Implementation

  Features

ContentAuthoring ContentDelivery Publisher Simula4ons

AdobeCrea)veSuite Livescribe ArtsFileShare Kaltura(*) UBCBlogs/WordPress(Content)(*) Cengage(PIC)(*) GoogleCardboard

Ar)culateStudio MediaSite(*) CengageiLrn LearnDash(WordPress)(*) Macmillan(PIC)(*) Layar

Audacity Microso]Excel Connect(ContentManagement)(*)

LibraryOnlineCourseReserves(LOCR)(*)

UBCiTunes(*) McGraw-Hill(PIC)(*) MannequinSimula)ons

BbCollaborateVoiceAuthoring(*) Microso]PowerPoint DocumentCameras Lyryx UBCWiki(*) Pearson(PIC)(*) Mo)onControl

Camtasia(*) OnebuconStudio(*) DropBoxModernDigitalImageDatabase(MDID)

UBCYouTube(*) SaplingLearning(PIC)(*) NeoReality

Collabora)veLearning(*)Annota)onSystem(CLAS)

Panopto Drupal(*) Moodle(*) Vimeo Wiley(PIC)(*) PhetSimula)ons

ExplainEverything Dspace Omeka Vitalsource Praxis

FinalCutPro Prezi edX(*) Owncloud WebAssign VCER

Entrada Perusall Wikipedia

HotPotatoes RespondusStudymate(*) Evernote Piazza(*) Wis)a

Jing Snagit ExamPrepDatabase(*) Plone Workspace(*) PorLolios

Keynote TimelineJS Github Podcasts Zimbra ChalkandWire

Lectora GoogleDocs Reflector Zoomify Connect(ePorLolio)(*)

Lightboard(*) VideoScribe(*) HTML5FlashCards SourceTree Pep

iBooks SugarSync

Assessment ResponseSystem

Adap)veCompara)veJudgement(ACJ)(*) MechanicalTA Aplia

AutoMul)pleChoice(AMC) Moodle(*) iClicker(*)

CalibratedPeerReview(CPR)(*) OpenBadgesUBC(*) Kahoot!

CengageiLrn PearsonMyTest LearningCataly)cs

Crowdmark PeerScholar PollEverywhere

Entrada PeerWise(*) REEFpolling(*)

ExamPrepDB(*) Remark TopHat

ExamSo] RespondusQuiz(*)

FormBuilder(*) Scantron

GradeGrinder Turni)n(*)

iPeer(*) WebAssign

LearnDash(WordPress)(*) Webwork(*)

Analy4cs CourseEvalua4on CourseAdmin Other

ArtsDatamart BlackboardOutcomesAssessment(*) CengageiLrn 3Dprin)ng

BlackboardOutcomesAssessment(*) CoursEval(*) Connect(GradeCenter)(*) GoogleEarth

Connect(PerformanceCenter)(*) SEoTDatamart(*) Doodle iStudiezPro

GoogleAnaly)cs TeachEval(*) Entrada SCORM(*)

IBMSPSS GoogleCalendar TinCanAPI(ExperienceAPI)

JMP LearnDash(WordPress)(*)

Microso]Excel Moodle(*)

SEoTDatamart(*) One45

Stata Turni)n(*)

Tableau UBCBlogs/WordPress(*)

WebAssign

LEARNING TECHNOLOGY TOOLS - FUNCTIONAL MAP

CONTENT ASSESSMENT

COURSE MANAGEMENT & EVALUATIONINTERACTIONS

Discussion SurveyTool SocialMedia VC

Connect(Discussions)(*) Connect(EnterpriseSurveys)(*) Facebook AdobeConnect

Piazza(*) Connect(Surveys)(*) Figure1 BbCollaborateWebConf.(*)

PulsePress(*) FluidSurveys(*) Google+ BlueJeans(*)

Slack GravityForms(WordPress) LinkedIn GoogleHangouts

UBCBlogs/WordPress(Discussion)(*) LimeSurvey Twicer Lifesize

Qualtrics Skype

SurveyMonkey VC(MedIT)

Vovici WebEx

Bold = Integrated Tool(*) = Supported by LT Hub

CONTEXT  First year intro Physics courses

  P101 / P117

  Non-majors

  Flipped class design

  Heterogeneous cohort

TOOLS MATTER

  Connect discussions vs Piazza

TOOLS MATTER

  Connect discussions vs Piazza

IMPLEMENTATION

  “No email” policy

  Give reasons to engage

  Support TA: to monitor and respond

A BUNCH OF NICE FEATURES…..

Assessment: student Generated material

Selected results & analysis

Engagement - how do students use the system?

Benefits - what is the impact on learning?

Question quality - how good is what students produce?

Relevant publications:

Scaffolding student engagement via online peer learning - European Journal of Physics 35 (4), 045002 (2014)

Student-Generated Content: Enhancing learning through sharing multiple-choice questions. International Journal of Science Education, 1-15 (2014).

Assessing the quality of a student-generated question repository - Phys Rev ST PER (2014) 10, 020105

Student-generated assessment - Education in Chemistry (2013) 13 1

Typical implementation

Minimum participation requirements for each of two assessment exercises (PW1,

PW2) Write 1 Answer 5 Rate / comment 3

5% course credit

Physics 101, Energy & WavesWinter Semester: 3 sections, ~800 students

Not so typical implementation

Writing original questions is a demanding activity

Extensive scaffolding exercises

Revisited in subsequent tutorials

Engagement with PeerWise

Number Multiplier Number Multiplier

Questions 1105 [1.7] 998 [1.6]

Answers 11393 [17.2] 11807 [18.7]

Comments 4901 [7.4] 5509 [8.7]

PW 1 PW 2

Engagement with PeerWise

Engagement with PeerWise

Engagement with PeerWise

Engagement with PeerWise

Generally, students did

•  Participate beyond minimum requirements •  Engage in community learning, correcting errors •  Create problems, not exercises •  Provide positive feedback

Generally, students did not •  Contribute trivial or irrelevant questions •  Obviously plagiarize •  Participate much beyond assessment periods •  Leave it to the last minute (sort of….)

58

Correlation with learning

59

Correlation with learning

60

Correlation with learning

61

62

Question/Explanation Quality

Bloom’s Taxonomy of levels in the cognitive domain

Score Level Description

1 Remember Factual knowledge, trivial plugging in of numbers

2 Understand Basic understanding of content

3 Apply Implement, calculate / determine. Typically one-stage problem

4 Analyze Typical multi-step problem; requires identification of strategy

Evaluate Compare &assess various option possibilities; often conceptual

Synthesize Ideas and topics from disparate course sections combined. Significantly challenging problem.

Score Level Description

1 Remember Factual knowledge, trivial plugging in of numbers

2 Understand Basic understanding of content

3 Apply Implement, calculate / determine. Typically one-stage problem

4 Analyze Typical multi-step problem; requires identification of strategy

Evaluate Compare & assess various option possibilities; often conceptual

Synthesize Ideas and topics from disparate course sections combined. Significantly challenging problem.

63

Textp>0.05, NS

64

Question/Explanation Quality

Score Level Description

0 Missing No explanation provided or explanation incoherent/irrelevant

1 Inadequate Wrong reasoning and/or answer; trivial or flippant

2 Minimal Correct answer but with insufficient explanation/justification/ Some aspects may be unclear/incorrect/confused.

3 Good Clear and detailed exposition of correct method & answer.

4 ExcellentThorough description of relevant physics and solution strategy. Plausibility of all answers considered. Beyond normal expectation for a correct solution

65

66

Results (UoE 2010-11)

2 successive years of the same course (N=150, 350)!

‘High quality’ questions: 78%, 79%!

Over 90% (most likely) correct, and 3/5 of those wrong were !

identified by students. !

69% (2010) and 55% (2011) rated 3 or 4 for explanations!

Only 2% (2010) and 4% (2011) rated 1/ 6 for taxonomic level. !

67

Bottomley & Denny Biochem and Mol Biol Educ. 39(5) 352-361 (2011)

107 Year 2 biochem students 56 / 35 / 9 % of questions in lowest 3 levels.

Momsen et al CBE-Life Sci Educ 9, 436-440 (2010)

“9,713 assessment items submitted by 50 instructors in the United States reported that 93% of the questions asked on examinations in introductory biology courses were at the lowest two levels of the revised Bloom’s taxonomy”

Comparison with literature

68

69

Assessment: exams

IAD Course Organisers Forum – Edinburgh, Oct 2015

Two-stage exams!assessments of, for and as learning !

Simon.Bates@ubc.ca@simonpbatesbit.ly/batestalks

Disclaimers!•  Not ours; not new

•  Similar to elements of well-established pedagogy e.g. TBL

•  But….. Significant due to ease, effectiveness and take-up

Overview•  Define two-stage exams (UBC style)

•  Discuss advantages and disadvantages of two-stage exams

•  Take your questions on logistics of administering them

•  Describe some research done with two-stage exams at UBC

Two-stageExamsThe basics: Summative assessments During the exam (midterms / final / in-class test)

– Students complete then hand in individual exam – Get into groups of 4 to work on a group exam (for about ½

the time of the individual portion)

What they look (and sound!) like http://www.cwsei.ubc.ca/resources/SEI_video.html

Two-stageexams•  The Group Exam is identical to the individual exam

(+/- one or two “difficult” questions)

•  Students work on the group exam collaboratively Consensus

•  The group only gets one exam sheet!

Two-stageexamsatUBCAt UBC: now well over 100 courses

Physics, Chemistry, Biology, Math, Statistics, Earth, Ocean, and Atmospheric Sciences, Computer Science, Forestry, Pharmacy, Psychology, and Land and Food Systems

Many class formats / assessment types

900 student final exams (that was me J ) 450 student 1st year lectures 20 person laboratories 20 student 4th year seminars 5 student graduate classes

Advantages•  Immediate feedback •  Models classroom behaviour •  Engaged students •  Teaching!

AdvantagesYour list: • Nearly instant feedback / reflection • More collaboration •  Engagement •  Responsibility • Different perspectives • Happier students learn better •  Develops group work skills •  Lower achieving students get extra explanation •  Reduces anxiety

Advantages•  Higher achieving students benefit from explaining and

reinforcing knowledge •  ALL students participate! •  Quieter students get a chance to contribute even in large

classes •  Students like it and believe it helps their learning

SomeStudentCommentsPhys250andEOSC114

“Some problems, its a good way to find out what you did wrong on the individual exam

almost immediately.”

“Great idea! The group exams give you a chance to go over your answers to the exam

while you still care about the questions.”

SomeStudentCommentsPhys250andEOSC114

“Discussion over tricky questions facilitate learning immediately and the answer/

concept is stuck in your brain FOREVER!”

“You actually learn what you got wrong right away from a student

perspective”

DisadvantagesYour list Compressing the grades Group composition Loud students No agreement

PossibleDisadvantages•  “Only certain subjects” •  Social loafing •  Dominant group members •  Sidetracked by process •  Assigning marks •  A & D •  Student inexperience with groups! •  Time! Shorter exams •  Getting “convinced” of wrong answers •  Incompatible with curving as standard

SomeStudentCommentsPhys250andEOSC114

“… That said, there was the issue of excessive discussion in the group exam. That is, there were

several times where a part of a question was contentious within our group and the ensuing debate, frequently ended only by calling over

<instructor>, often took up so much time that doing the last few questions was hurried and messy.”

FeedbackfromEOSC114“Group Exams are…”

Posi)ve‘because…’comments

Discussion 48 Learn why you were wrong 37 New perspectives 29 Better grades 21 Instant feedback 16 Review 10 Build confidence 8 Understand questions better 6 Learn techniques from others 6 Other 27 Total 208

Nega)ve‘because…’comments

Coming to consensus 21 Time consuming 13 Unbalanced knowledge in group 6 Convinced of wrong answer 3 Realize did poorly individually 3 Worth too much 2 Other 8 Total 56

HowGroupsChoose(EOSC111)

Researchevidenceforeffec)veness

Gilley, B.H. and Clarkston, B. (2014) Collaborative Testing: Evidence of Learning in a Controlled In-Class Study of Undergraduate Students Journal of College Science Teaching Vol. 43 No. 3

Resources•  Journal articles:

– Gilley & Clarkston (2014) Journal of College Science Teaching Vol. 43 No. 3

– Rieger & Heiner (2014) Journal of College Science Teaching Vol. 43 No. 4

– Wieman, Rieger & Heiner (2014) Phys. Teach. Vol 52

•  For more information and videos visit: http://blogs.ubc.ca/eoassei/two-stage-exams/ • Misc articles:

http://www.macleans.ca/education/multiple-choice-multiple-students/ http://blogs.ubc.ca/catherinerawn/2014/07/22/two-stage-exam-introduction-and-resources/ https://teachingcommons.stanford.edu/teaching-talk/turn-exam-learning-experience-two-stage-exams http://www.artsrn.ualberta.ca/mengel/huco5002014/?p=356

Resources

www.cwsei.ubc.ca!

http://blog.peerinstruction.net/ !

http://flippedlab.learning.ubc.ca/!

http://diy.open.ubc.ca/!