1. Understanding In-Video Dropouts and Interaction Peaks in Online Lecture Videos Juho Kim (MIT CSAIL) Philip J. Guo (MIT CSAIL, U of Rochester) Daniel T. Seaton (MIT Office of Digital Learning) Piotr Mitros (edX) Krzysztof Z. Gajos (Harvard EECS) Robert C. Miller (MIT CSAIL)2014.03.04 Learning at Scale

2. Video Lectures in MOOCs 3. Classrooms: rich, natural interaction dataMaria Fleischmann / Worldbank on Flickr | CC by-nc-nd Love Krittaya | public domainarmgov on Flickr | CC by-nc-sa unknown author | from pc4all.co.kr 4. liquidnight on Flickr | CC by-nc-sa 5. How do learners use videos?Data-Driven Approach: Analyze learners interaction with the video player 6. Why does data matter? detailed understanding of video usage design implications for Instructors Video editors Platform designers new video interfaces and formatsImproved video learning experience 7. How do learners use videos? Watch sequentially Pause Re-watch Skip / Skim 8. Collective Interaction Traces Student #7888 Student #7887 ...... Student #4 Student #3 Student #2 Student #1 video time 9. Collective Interaction Traces into Interaction Patterns interaction eventsvideo timesecond-by-second activity tracking 10. ~40M video interaction events from 4 edX coursesLearners 127,839Mean Video Processed Videos Length Events 862 7:46 39.3MCourses: Computer science, Statistics, Chemistry 11. Analyzing Clickstream Events: play / pause In-video time and absolute time Learner ID: first-time or re-watching Clickstream interaction logPer-learner watching segmentslearner xxxlearner xxx0:00 play 0: 34 pause 0:57 play 1:47 pauseSegment 1 - 0:00-0:34 Segment 2 - 0:57-1:47Per-second stats for views, re-watches, plays, & pauses 12. Collective Interaction Patterns 1. In-video dropout viewershipvideo time2. Interaction peaks interaction eventsvideo time 13. 1. In-video dropout viewershipvideo time2. Interaction peaks interaction eventsvideo time 14. In-video Dropout % watching sessions that end before the video finishes.viewershipvideo time 15. 36%: dropouts withinrate few seconds 55%: overall dropout first viewership36% 19%55%video timeWhy? Auto-play playing unwanted videos Misleading video titles / interfaces 16. Longer videos lead to more dropouts. 17. Re-watching sessions lead to more dropouts than first-time sessions. 18. Tutorial videos lead to more dropouts than lecture videos. Lecture introduction to concepts continuous flowTutorial supplementary examples step-by-step demos 19. 1. In-video dropout viewershipvideo time2. Interaction peaks interaction eventsvideo time 20. Interaction Peaks Temporal peaks in the number of interaction events, where a significant number of learners show similar interaction patternsinteraction eventsvideo time 21. Analytic Workflow Bin data into per-second segmentsApply a kernel smootherDetect peaks 22. Re-watching sessions show stronger and more peaks than first-time sessions. 23. Tutorial videos show stronger and more peaks than lecture videos. LecturesTutorials 24. What causes an interaction peak? 25. Observation: Visual transitions in the video often coincide with a peak. 26. lecture videonumber of re-watching sessions 27. Analytic Workflow Step 1. Visual Analysis second-by-second pixel differences between adjacent frameshead slidehead slidehead slide 28. Analytic Workflow Step 2. Peak Categorization Manually inspected 80 videos Interaction peak Visual transition 29. Five Explanations for an Interaction Peak Type 1. Beginning of new material Type 2. Returning to content Type 3. Tutorial step Type 4. Replaying a segment Type 5. Non-visual explanation 30. Type 1. Beginning of new materialinteraction 31. Type 2. Returning to contentinteraction 32. Type 3. Tutorial stepinteraction 33. Type 4. Replaying a segmentinteraction 34. Type 5. Non-visual explanationinteraction 35. 61% of interaction peaks involved a visual transition. Peak CategoryFrequencyType 1. Beginning of new material25%Type 2. Returning to content23%Type 3. Tutorial step7%Type 4. Replaying a segment6%Type 5. Non-visual explanation39%61% 36. Can interaction data improve the video learning experience? 37. Lessons for instructors, video editors, and platform designers 1. Make shorter videos. 2. Add informative titles and easy navigation. 3. Avoid abrupt visual transitions. 38. Lessons for instructors, video editors, and platform designers 4. Make interaction peaks more accessible.5. Enable one-click access for tutorial steps. 39. Next Steps: More Data Streams What would transcript / text add to the analysis? How about acoustic data? 40. Next Steps: Scalability Reliably & automatically detect peak types? How much data is needed until we see patterns? viewership5 learners video timeviewership5,000 learners video time 41. For instructors & editors: Video Analytics 42. For learners: Data-Driven Video UI 43. Contributions A first MOOC-scale in-video dropout rate analysis A first MOOC-scale in-video interaction peak analysis Categorization of learner activities responsible for an interaction peak Data-driven design implications for video authoring, editing, and interface design 44. Understanding In-Video Dropouts and Interaction Peaks in Online Lecture VideosJuho Kim MIT CSAIL juhokim@mit.edu juhokim.com 45. Backup slides 46. Domain educational videosTheory interactive learningHow-to videos MOOC videosRole of interactivity Learner control Subgoal labelingMethod scalable data collection to realize theory Crowdsourcing Learnersourcing 47. Vision in learnersourcing Feedback loop between Learners: natural, pedagogically useful activities System: improve interaction using learner data Visualize and analyze large-scale video learning activities Use data to inform learning platform design 48. How can we design online video learning platforms that are as effective as in-person classrooms?