1
How online educational resources provide novel affordances for conducting practical interventions
and doing psychology experiments
Joseph Jay Williams, Stanford UniversityLytics Lab (lytics.stanford.edu), Graduate School of Education
Office of the Vice Provost of Online Learning (vpol.stanford.edu)[email protected] www.josephjaywilliams.com
Originally from Trinidad – apologies for thick accent!
www.josephjaywilliams.com/experiments-online-education
2
Overview
• MOOCs VS Digital Online Resources• Experiments: Link Real World & Laboratory• Motivational Messages Experiment• Simultaneous Basic & Applied Research• Ed-Surge Meetup to link Research & Practice
(Wed Feb 19 @ 7)• Current Directions
Teaching Learning StrategiesLearning CoachMOOC Experiment
• MOOClets Research & Practice Group Meets Tuesdays 11-12 PST (tiny.cc/lyticshangout, tiny.cc/mooclet)
Online Education
Moving from Physical to Digital World
Experiments: Link Real World & Laboratory
• Randomized assignment• Experimental Control• Rich data
• Real-world environment
• Authentic activities• Practical Challenges
6
Experiment
• Practical goal on Khan Academy• Extend motivation research
7
Motivate students to practice math
• Online (Math) Exercise
1. Number of Problems Practiced
2. Improve Accuracy
Motivational Message: Keep up the good work!
8
Add motivational messages
Practice-as-usual
Remember, the more you practice the smarter you become!
Growth Mindset Message
9
Which messages to add?
• "This might be a tough problem, but we know you can do it!”
• Remember, the more you practice the smarter you become!
• Some of these problems are hard. Just do your best.”
• Encourage “Growth Mindset” beliefs about intelligence (Dweck, 2008)
• Fixed Mindset• Growth Mindset
10
Embedded in vivo Experiment
• Growth Mindset Message
• "Remember, the more you practice the smarter you become.”,
• "Mistakes help you learn. Think hard to learn from them.”
• Practice-as-usual
Jascha Sohl-Dickstein
11
Results: More motivated?
• Growth Mindset Message > Practice-as-Usual
• reduction in dropout rate• increase in Number of problems practiced • increase in Number Correct
• Accuracy: Problems correct/Problems attempted
• Accuracy improvement
12
Interpreting results
• Obvious?• Skepticism: Anything’s statistically
signficant in a MOOC
13
Practice-as-usualGrowth Mindset Message
Positive Message
Different motivational messages
Some of these problems are hard. Do your best!Remember, the more you practice the smarter you become!
14
Does any positive message work?
• Growth Mindset Message
• "Remember, the more you practice the smarter you become.”,
• "Mistakes help you learn. Think hard to learn from them.”
• Positive Message• "Some of these problems are
hard. Just do your best."• "This might be a tough problem,
but we know you can do it.”
• Practice-as-usual
15
Effects of Positive Messages?
• Positive Messages not better than Practice-as-Usual!
• Growth Mindset better than Positive• reduction in dropout rate• increase in Number of problems practiced
• increase in Number Correct
• Accuracy improvement
16
Next: Situationally appropriate messages
• Message blast, can be ignored• Providing motivational message Just-
in-Time: after incorrect answer• Laboratory experiments• www.assistments.org Neil Heffernan
17
Simultaneous Applied & Basic Research• Start with “Product”• Constraints
Randomization (Technical, use Qualtrics)Behavioral/Learning MeasuresWilliams & Williams (2013). Using Interventions to improve Online Learning. NIPS Data Driven Education workshop. tiny.cc/experimentsonlinelearning
• Consider SynthesisWilliams, J.J. (2013) Improving Learning in MOOCs by Applying Cognitive Science. AIED MOOCshop & NIPS Data-Driven Workshop. tiny.cc/cognitivescienceinmoocs
Reading List: www.josephjaywilliams.com/education
Improving outcome of psychosocial treatments by enhancing memory and learning.(in press) Perspectives in Psychological Science.
• High Investment but High Return
Multidisciplinary – Relevant Communities
• Association for Psychological Science 2013. Online Learning and Psychological Science: Opportunities to integrate research and practice. Williams, Saxberg, Means, Mitros.
• Cognitive Science Society 2013. Online Education: A Unique Opportunity for Cognitive Scientists to Integrate Research and Practice. Williams, Renkl, Koedinger, Stamper.
• AERA 2014– American Education Research Association. How online resources can facilitate interdisciplinary collaboration. Williams, Thille, Trumbore, Stigler, Siemens.
• ICLS (under review) – International Conference for the Learning Sciences. Online & Blended Education: Opportunities and Obstacles for the Learning Sciences. Williams, Linn, Slotta, Rose, Dillenbourg.
• CHI 2014 – Computer Human Interaction. Learning Innovations at Scale. Williams, Kizilcec, Russell, Klemmer.
• E-learning guild’s DevLearn• Educational Data Mining• Artificial Intelligence in Education
• Computer Supported Collaborative Learning• Professional & Organizational Development Network• American Society for Training & Development
19
Laboratory Paradigms & Online Products
• A matching problem• www.edsurge.com,
www.edudemic.com• [email protected] • Williams, J.J., Teachman, B.A., Richland, L., Brady, S.T, &
Aleahmad, T. (2014). Leveraging the Internet to do Laboratory Research in the Real World. Symposium to be conducted at the annual convention of the Association for Psychological Science. San Francisco, CA.
20
Example: Explanation & Learning
• Williams & Lombrozo (2010, Cognitive Science; 2013, Cognitive Psychology; w Rehder, JEP: General)
Right
WrongDrent
Glorp
21
EdSurge Meetup Wed Feb 19 7 pm CERAS
• “Improving Ed-Tech using Research on Learning & Advancing Research on Learning using Ed-Tech”
• Ed-Tech Startup Survey• Researcher Survey• Featured Company & Researcher• Panelist & Question suggestions?
22
Completely New Horizons for Experimentation
• “Bridging Studies”• www.josephjaywilliams.com
/experiments-online-education • Williams, J. J. (2013). Enhancing Educational
Research & Practice using Experiments on Online Educational Resources. Talk at Pittsburgh Science of Learning Center LearnLab Summer School, Pittsburgh, PA. tiny.cc/onlineexperiments
Next: Teaching Learning Strategies • Limited teaching time: Specific content
or general strategies?• Online Advantage:
Easier to run RCTs & collect longitudinal data Reinforce new educational habits & behaviors
• Teach metacognitive self-questioning strategies (Palinscar & Brown, 1984, Cognition and instruction; McNamara, 2004, Discourse Processes; Williams & Lombrozo, 2010, Cognitive Science)
23
Drop-down text about strategies
Clickable link.
[Click here to learn about the “What? Why? How?” strategy]
25
Learning Coach
• Tiny.cc/kalearningcoach
26
MOOC experiment
• Motivational VideoControlEnergetic Thank you & EncouragementGrowth Mindset (by Instructor)Growth Mindset (outside Instructor)“Belonging”
• Questions for reflection after videoNone VS Just Think VS Write response
27
MOOClets Group – Develop Digital Modules
• E.g. MOOClet: Math exercise, 5 minute interactive video, 30 minute lesson, Study Strategy coach, Quiz/Assessment
• Instructors can use for blended classroom education (High School, Community College, University)
• Continuously in use and being improved• Easy to build, prototype, test & iterate• Researchers directly implement & collect data• Small scale & online access facilitates multidisciplinary
collaboration & research-practitioner development (see AERA 2014 invited panel)
• Illustrate/Operationalize instructional design principles• MOOClet Reading & Discussion Group meets Tue 11-12
PST @ Barnum (Stanford) or tiny.cc/lyticshangout• Multidisciplinary group of researchers & practitioners (tiny.cc/
mooclet)
28
Review
• MOOCs VS Digital Online Resources• Experiments: Link Real World & Laboratory• Motivational Messages Experiment• Simultaneous Basic & Applied Research• Ed-Surge Meetup to link Research & Practice (Wed
Feb 19 @ 7)• Current Directions
Teaching Learning StrategiesLearning CoachMOOC Experiment
• MOOClets Research & Practice Group Meets Tuesdays 11-12 PST (tiny.cc/lyticshangout, tiny.cc/mooclet)
29
Acknowledgements
• Jascha Sohl-Dickstein• Jace Kohlmeier & Khan Academy• PERTS• Sam Maldonado• Lytics Lab (lytics.stanford.edu)• VPOL (Vice Provost of Online
Learning, online.stanford.edu)