1

How online educational resources provide novel affordances for conducting practical interventions

and doing psychology experiments

Joseph Jay Williams, Stanford UniversityLytics Lab (lytics.stanford.edu), Graduate School of Education

Office of the Vice Provost of Online Learning (vpol.stanford.edu)josephjaywilliams@stanford.edu www.josephjaywilliams.com

Originally from Trinidad – apologies for thick accent!

www.josephjaywilliams.com/experiments-online-education

2

Overview

• MOOCs VS Digital Online Resources• Experiments: Link Real World & Laboratory• Motivational Messages Experiment• Simultaneous Basic & Applied Research• Ed-Surge Meetup to link Research & Practice

(Wed Feb 19 @ 7)• Current Directions

Teaching Learning StrategiesLearning CoachMOOC Experiment

• MOOClets Research & Practice Group Meets Tuesdays 11-12 PST (tiny.cc/lyticshangout, tiny.cc/mooclet)

Online Education

Moving from Physical to Digital World

Experiments: Link Real World & Laboratory

• Randomized assignment• Experimental Control• Rich data

• Real-world environment

• Authentic activities• Practical Challenges

6

Experiment

• Practical goal on Khan Academy• Extend motivation research

7

Motivate students to practice math

• Online (Math) Exercise

1. Number of Problems Practiced

2. Improve Accuracy

Motivational Message: Keep up the good work!

8

Add motivational messages

Practice-as-usual

Remember, the more you practice the smarter you become!

Growth Mindset Message

9

Which messages to add?

• "This might be a tough problem, but we know you can do it!”

• Remember, the more you practice the smarter you become!

• Some of these problems are hard. Just do your best.”

• Encourage “Growth Mindset” beliefs about intelligence (Dweck, 2008)

• Fixed Mindset• Growth Mindset

10

Embedded in vivo Experiment

• Growth Mindset Message

• "Remember, the more you practice the smarter you become.”,

• "Mistakes help you learn. Think hard to learn from them.”

• Practice-as-usual

Jascha Sohl-Dickstein

11

Results: More motivated?

• Growth Mindset Message > Practice-as-Usual

• reduction in dropout rate• increase in Number of problems practiced • increase in Number Correct

• Accuracy: Problems correct/Problems attempted

• Accuracy improvement

12

Interpreting results

• Obvious?• Skepticism: Anything’s statistically

signficant in a MOOC

13

Practice-as-usualGrowth Mindset Message

Positive Message

Different motivational messages

Some of these problems are hard. Do your best!Remember, the more you practice the smarter you become!

14

Does any positive message work?

• Growth Mindset Message

• "Remember, the more you practice the smarter you become.”,

• "Mistakes help you learn. Think hard to learn from them.”

• Positive Message• "Some of these problems are

hard. Just do your best."• "This might be a tough problem,

but we know you can do it.”

• Practice-as-usual

15

Effects of Positive Messages?

• Positive Messages not better than Practice-as-Usual!

• Growth Mindset better than Positive• reduction in dropout rate• increase in Number of problems practiced

• increase in Number Correct

• Accuracy improvement

16

Next: Situationally appropriate messages

• Message blast, can be ignored• Providing motivational message Just-

in-Time: after incorrect answer• Laboratory experiments• www.assistments.org Neil Heffernan

17

Simultaneous Applied & Basic Research• Start with “Product”• Constraints

Randomization (Technical, use Qualtrics)Behavioral/Learning MeasuresWilliams & Williams (2013). Using Interventions to improve Online Learning. NIPS Data Driven Education workshop. tiny.cc/experimentsonlinelearning

• Consider SynthesisWilliams, J.J. (2013) Improving Learning in MOOCs by Applying Cognitive Science. AIED MOOCshop & NIPS Data-Driven Workshop. tiny.cc/cognitivescienceinmoocs

Reading List: www.josephjaywilliams.com/education

Improving outcome of psychosocial treatments by enhancing memory and learning.(in press) Perspectives in Psychological Science.

• High Investment but High Return

Multidisciplinary – Relevant Communities

• Association for Psychological Science 2013. Online Learning and Psychological Science: Opportunities to integrate research and practice. Williams, Saxberg, Means, Mitros.

• Cognitive Science Society 2013. Online Education: A Unique Opportunity for Cognitive Scientists to Integrate Research and Practice. Williams, Renkl, Koedinger, Stamper.

• AERA 2014– American Education Research Association. How online resources can facilitate interdisciplinary collaboration. Williams, Thille, Trumbore, Stigler, Siemens.

• ICLS (under review) – International Conference for the Learning Sciences. Online & Blended Education: Opportunities and Obstacles for the Learning Sciences. Williams, Linn, Slotta, Rose, Dillenbourg.

• CHI 2014 – Computer Human Interaction. Learning Innovations at Scale. Williams, Kizilcec, Russell, Klemmer.

• E-learning guild’s DevLearn• Educational Data Mining• Artificial Intelligence in Education

• Computer Supported Collaborative Learning• Professional & Organizational Development Network• American Society for Training & Development

19

Laboratory Paradigms & Online Products

• A matching problem• www.edsurge.com,

www.edudemic.com• josephjaywilliams@stanford.edu • Williams, J.J., Teachman, B.A., Richland, L., Brady, S.T, &

Aleahmad, T. (2014). Leveraging the Internet to do Laboratory Research in the Real World. Symposium to be conducted at the annual convention of the Association for Psychological Science. San Francisco, CA.

20

Example: Explanation & Learning

• Williams & Lombrozo (2010, Cognitive Science; 2013, Cognitive Psychology; w Rehder, JEP: General)

Right

WrongDrent

Glorp

21

EdSurge Meetup Wed Feb 19 7 pm CERAS

• “Improving Ed-Tech using Research on Learning & Advancing Research on Learning using Ed-Tech”

• Ed-Tech Startup Survey• Researcher Survey• Featured Company & Researcher• Panelist & Question suggestions?

22

Completely New Horizons for Experimentation

• “Bridging Studies”• www.josephjaywilliams.com

/experiments-online-education • Williams, J. J. (2013). Enhancing Educational

Research & Practice using Experiments on Online Educational Resources. Talk at Pittsburgh Science of Learning Center LearnLab Summer School, Pittsburgh, PA. tiny.cc/onlineexperiments

Next: Teaching Learning Strategies • Limited teaching time: Specific content

or general strategies?• Online Advantage:

Easier to run RCTs & collect longitudinal data Reinforce new educational habits & behaviors

• Teach metacognitive self-questioning strategies (Palinscar & Brown, 1984, Cognition and instruction; McNamara, 2004, Discourse Processes; Williams & Lombrozo, 2010, Cognitive Science)

23

Drop-down text about strategies

Clickable link.

[Click here to learn about the “What? Why? How?” strategy]

25

Learning Coach

• Tiny.cc/kalearningcoach

26

MOOC experiment

• Motivational VideoControlEnergetic Thank you & EncouragementGrowth Mindset (by Instructor)Growth Mindset (outside Instructor)“Belonging”

• Questions for reflection after videoNone VS Just Think VS Write response

27

MOOClets Group – Develop Digital Modules

• E.g. MOOClet: Math exercise, 5 minute interactive video, 30 minute lesson, Study Strategy coach, Quiz/Assessment

• Instructors can use for blended classroom education (High School, Community College, University)

• Continuously in use and being improved• Easy to build, prototype, test & iterate• Researchers directly implement & collect data• Small scale & online access facilitates multidisciplinary

collaboration & research-practitioner development (see AERA 2014 invited panel)

• Illustrate/Operationalize instructional design principles• MOOClet Reading & Discussion Group meets Tue 11-12

PST @ Barnum (Stanford) or tiny.cc/lyticshangout• Multidisciplinary group of researchers & practitioners (tiny.cc/

mooclet)

28

Review

• MOOCs VS Digital Online Resources• Experiments: Link Real World & Laboratory• Motivational Messages Experiment• Simultaneous Basic & Applied Research• Ed-Surge Meetup to link Research & Practice (Wed

Feb 19 @ 7)• Current Directions

Teaching Learning StrategiesLearning CoachMOOC Experiment

• MOOClets Research & Practice Group Meets Tuesdays 11-12 PST (tiny.cc/lyticshangout, tiny.cc/mooclet)

29

Acknowledgements

• Jascha Sohl-Dickstein• Jace Kohlmeier & Khan Academy• PERTS• Sam Maldonado• Lytics Lab (lytics.stanford.edu)• VPOL (Vice Provost of Online

Learning, online.stanford.edu)