RELATE :

1

REsearch in Learning, Assessing, and Tutoring Electronically

RELATE.mit.edu

Postdocs: Phil Dukes

Sofia MoroteRasil Warnakulasooriya

PI: Dave Pritchard$: MIT, NSF, DEP

RELATE:

2

an expert systembased on educational expertise, not AI

The most advanced tutorial and assessment system in the worldMade by Effective Educational Technologies, a Pritchard company.

Also known as

and CyberTutor.MIT in publications

by EET

3

Outline

• Objective

• Pedagogy that Works

• Feedback – Closed Loop Education

• Research from MIT

• Revolution in Assessment

• Closing Thoughts

4

Digital Education Future?!Broadcast Radio

Passive

Class

Uniform Style

Next Edition

Teacher

Author

High Stakes Tests

Two-way Radio

Interactive

Student

Stylized (e.g. Audio)

Next Day

Coach

Authors/Researchers

Embedded Assessment

Perspective

5

Why Homework??

Teachers’ Priorities:

1. Lectures

2. Exams

3. Notes and Demonstrations

4. Homework

Students spend most time and learn most from 1. Homework

6

TWO WAY LEARNING

Books, lectures, most WWW education

Students, Teachers, Authors, ResearchersLearn from each other

DATA, EXPERIMENT, ANALYSIS, CONCLUSIONS

7

SOCRATIC LEARNING

Authors, Researchers

Student System

Teacher

8

Outline

Objective

• Pedagogy that Works

• Feedback – Closed Loop Education

• Research from MIT

• Revolution in Assessment

• Closing Thoughts

9

PedagogyDesign Philosophy of myCyberTutor

Emulate the interaction between a human tutor and a student.

Results:an effective interactive learning toolyou can author, deliver and improve contentan expert program embodying your expertise

The tutor informs the teacher.

The process informs the author.

10

Pedagogy

Mastery LearningThe amount learned should be constantand time allowed

to vary

OthersSocratic pedagogy and learning styles

(to be implemented)

ConstructivistsAllow students to

construct knowledge in their own way

Student

Student-Centered Instruction

11

Pedagogy

Pedagogical Principles• Actively engage the student • Adapt problem to less skillful students with hints• Prompt feedback addresses wrong answers• Mastery Learning >90% get solution• Declarative and procedural knowledge are both

important hints and subproblems• Solidify and extend the solution followups• Free response answers reduce guessing

12

13

14

15

16

17

Gain on the MIT Final Exam

December 2000 to May 2001

-0.100

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

Group Problems WrittenHomework

ClassParticipation

CyberTutor

P-value 0.69 0.69 0.35 0.010

Results from MIT

18P-value 0.854 0.807 0.198 0.087 0.015

-0.2

-0.1

0

0.1

0.2

0.3

0.4

0.5

0.6

Small TutorialSessions

PIVOTMultimedia

WrittenHomework

Group Problems CyberTutor

FCI gain=0.41 for course

Gain on Force Concept Inventory

data C. Ogilvie 2000

Results from MIT

19

Outline

Objective

Pedagogy that Works

• Feedback – Closed Loop Education

• Research from MIT

• Revolution in Assessment

• Closing Thoughts

20

Two Way Learning - Feedback

• Typical student returns to server 10 times during course of each problem (cf. Web Assign ~4 times per assignment)

• Students achieve the correct answer 90% of the time (cf. ~60% first time right)

• Students comment on ~3% of all problems– More if problem has flaws

myCyberTutor interactions:

TWO WAY LEARNING

21

Student Comments

Feedback – Closed Loop Education

22

Wrong Answers

Feedback – Closed Loop Education

23

Feedback to AuthorImproves Problems

• < 90% correct: Need more hints

• Wrong Answers: Respond to common ones

• Comments: Revise wording, remove confusion, revise program

• Time: Is this problem worthwhile?

Feedback – Closed Loop Education

24

Feedback Enables Revisionsthat Improve Problems

91.6 %

83.5 %

PercentAnsweringCorrectly

6.0%

12.3 %

Percent Requesting Solutions

0.89

1.51

AverageWrong Answers/part

0.83

0.76

Average hints/part

1.5Spring 2002

1.5Spring 2001

Median Minutes/part

Room for even more improvement!!

Fall

2003

93.4%

25

Outline

Objective

Pedagogy that Works

Feedback – Closed Loop Education

• Research from MIT

• Revolution in Assessment

• Closing Thoughts

26

Pedagogy

RELATE:REsearch in Learning, Assessing, and

Tutoring ElectronicallyRELATE.mit.edu

Postdocs: Phil Dukes

Sofia MoroteRasil Warnakulasooriya

27

Data with resolution

Capability for Split Class Assignments

Your Data PackageLog of your students’ interactionsAnonymous student # vs name & ID for your classDatabase of your class’ assignments

Plus - General Data PackagePerformance Data - your class vs. standardSML skeleton each problem (subparts, hints, etc.)Format key

Attractions for ResearchersObjective

28

Inductive vs. Deductive Instruction

Inductive: Students learn by doing a problem from the hints, from the subproblemsby figuring it out from feedback

Transfer learning to tutorial questions??

Deductive: Students learn from a tutorial from the learning goal & text from the hintsfrom the self-assessment questions

Transfer learning to related problem??

Tutorials

• Tutorial problems in Mastering Physics are

carefully planned and sequenced

instruction with SAQ’s

• They are used as instructional material to

impart principles in deductive learning.

30

Problems

• Problems require a student to apply an already familiar concept, formula, or procedure

• Socratic help is available, including explanation of the concept, a formula that is needed, etc.

• Related Problems cover same topic as adjacent tutorial

Pedagogy

31

p=0.01*

p=0.06**

p=0.03*

Deductive: Related Problem Difficulty Reduced by

working Tutorial First

After working tutorial

Tor

que

New

ton

3rd

Law

Har

mon

ic O

s.

Results from MIT

32

Tor

que

New

ton

3rd

Law

Har

mon

ic O

s.Inductive:

Tutorial Difficulty Reduced by working Related Problem First?

Results from MIT

33

Conclusion: Deductive Works

• Interactive tutorials significantly increase performance on

subsequent related problems (~25% less difficult)

• Students don’t learn inductively from a multi-part example

• We recommend using online tutorials in the old fashioned

way - preparation for subsequent deductive exercises

Results from MIT

34

0.000

0.002

0.004

0.006

0.008

0.010

0.012

0.014

0.016

0.018

0.020

Imp

rove

me

nt

per

un

it t

ime

Tutorial-first

Problem-first

Torque Newton III SHM

Twice as much learning per unit time spent on the tutorial compared with time spent on the preparatory

problem

35

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Cross-section for asteroid impact

fraction of students who request hints

Colliding cars

Unprepared group Prepared group by solving a related problem

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Newton III

fraction of students who request hints

Finding Torque

Unprepared group Prepared group by solving a tutorial problem

p < 0. 1

p < 0. 005p < 0. 05

p < 0. 01

Prior tutorial reduces the hints requested on the related problem by ~19%

(based on 5 problems)

Prior related problem reduces the hints requested on the related problem by ~12%

(based on 6 problems)

36

Time to Completion

The real time environment allows us to study how long it takes students to work problems, whether good students do problems quicker or slower, etc.

We have discovered that there areThree groups of students in time:

37

When Students Finish: Three distinct groupsQuick solvers < 2.5 minutes

Real-time solvers 2.5 min – 2.2 hours

Interrupted solvers > 2.2 hours

2 4 6 8 10 12 14-0.05

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

7s 20s 55s 2.5m 6.7m 18m 50m 2.2h 6h 17h 2d 5d

Rate of completion

Ln(t)

Colliding cars

Quick solvers

Real-time solvers

Interrupted solvers

38

2 4 6 8 10 12 14

0.0

0.1

0.2

0.3

0.4

7s 20s 55s 2.5m 7m 18m 50m 2.2h 6h 17h 2d 5d

Rate of completion

Ln(t)

Finding torque Flywheel kinematics Parallel-axis theorem

Median time ~ 7min

2 4 6 8 10 12 14-0.05

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

Median time ~ 11min

7s 20s 55s 2.5m 7m 18m 50m 2.2h 6h 17h 2d 5d

Rate of completion

Ln(t)

Colliding cars Shooting a block up an incline

2 4 6 8 10 12 14-0.05

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

Median time ~ 18-30min

7s 20s 55s 2.5m 7m 18m 50m 2.2h 6h 17h 2d 5d

Rate of completion

Ln(t)

The parallel-axis theorem A person standing on a leaning ladder Collision at an angle

2 4 6 8 10 12 14

0.00

0.05

0.10

0.15

0.20

0.25

0.30

Median time ~ 30min

7s 20s 55s 2.5m 7m 18m 50m 2.2h 6h 17h 2d 5d

Rate of completion

Ln(t)

Cross-section for asteroid impact Post-collision orbit

39

2 4 6 8 10 12 14

0.00

0.05

0.10

0.15

0.20

0.25

Rate of completion

Ln(t)

total no hints, no wrong ans. no hints, at least one wrong at least one hint, one wrong

7s 20s 55s 2.5m 6.7m 18m 50m 2.2h 6h 17h 2d 5d

Real-time solvers make errors and ask for hints

40

2 4 6 8 10 12 14

0.00

0.05

0.10

0.15

0.20

0.25

Rate of completion

Ln(t)

total no hints, no wrong answers no hints, at least one wrong at least one hint, one wrong

Collision at an angle:Prepared group

7s 20s 55s 2.5m 6.7m 18m 50m 2.2h 6h 17h 2d 5d

2 4 6 8 10 12 14

0.00

0.05

0.10

0.15

0.20

0.25 total no hints, no wrong ans. no hints, at least one wrong at least one hint, one wrong

7s 20s 55s 2.5m 6.7m 18m 50m 2.2h 6h 17h 2d 5d

Rate of completion

Ln(t)

Collision at an angle: Unprepared group

Note that:1. The quick solvers do not make mistakes or ask for hints2. The real-time solvers make mistakes and ask for hints3. The interrupted solvers make mistakes and ask for hints4. Fewer real-time solvers in the prepared group ask for hints

41

Fraction Finished curves with hints & feedback

2 4 6 8 10 12 14

0.0

0.2

0.4

0.6

0.8

1.0

fraction

Ln(t)

Colliding cars

7s 20s 55s 2.5m 6.7m 18m 50m 2.2h 6h 17h 2d 5d

2 4 6 8 10 12 14

0.0

0.2

0.4

0.6

0.8

7s 20s 55s 2.5m 6.7m 18m 50m 2.2h 6h 17h 2d 5d

fraction

Ln(t)

Collision at an angle

For 14 problems: fraction of real-time solvers = 65 4%

64% 44%

42

Time to completion curves without hints & feedback

2 4 6 8 10 12 14

0.0

0.2

0.4

0.6

0.8

1.0

7s 20s 55s 2.5m 6.7m 18m 50m 2.2h 6h 17h 2d 5d

fraction

Ln(t)

End-of-chapter 10.46

For 3 typical homework problems:

fraction of real-time solvers = 29 3%

2 4 6 8 10 12 14

0.0

0.2

0.4

0.6

0.8

1.0

7s 20s 55s 2.5m 6.7m 18m 50m 2.2h 6h 17h 2d 5d

fraction

Ln(t)

End-of-chapter 10.38

35%

2 4 6 8 10 12 14

0.0

0.2

0.4

0.6

0.8

1.0

7s 20s 55s 2.5m 6.7m 18m 50m 2.2h 6h 17h 2d 5d

fraction

Ln(t)

End-of-chapter 10.40

28%

43

Outline

Objective

Pedagogy that Works

Feedback – Closed Loop Education

Research from MIT

• Revolution in Assessment

• Closing Thoughts

44

Imagine that a rich ship-owner has hired Socrates to tutor his children. At the end of the month he desires to assess the amount they have learned. Would you advise him to:

a) Administer a standardized hour-long test to the children?

 b)  Ask Socrates how much they have learned?

Low-Error Embedded Assessment

Future - Assessment

This Assessment gives ~6 times as reliable an assessment per unit of student time as a good final exam!

MyCyberTutor Assessment has ~100 times less variance due to error than a good final exam!

45

Embedded Assessment

myCyberTutor vs. Final ExamSocraticTutor

R2 = 0.986

0

0.1

0.2

0.3

0.4

0.5

0 0.1 0.2 0.3 0.4 0.5

Score (Even Problems)

Score (Odd Problems)

Final Exam

R2 = 0.4053

0

0.1

0.2

0.3

0.4

0.5

0 0.1 0.2 0.3 0.4 0.5

Score (Even Problems)

Score (Odd Problems)

•…is 6 times more reliable per unit time•…has ~100 times less error variance

myCyberTutor

Future - Assessment

46

Assessment: Detailed Skill Profile

Future - Assessment

47

R2 = 0.51

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

SocraticTutor Prediction

MIT Final Score

Implication: MyCyberTutor can replace tests

Predicting Final Exam Score

Future - Assessment

48

myCyberTutor Assessment Implies:

1) More Accurate

2) Fine Grained Assessment on Subtopics

3) Immediate Remediation

-Select Next Problem    

4) JITT Guide for Teacher

5) Learning vs. Avoiding Lost Points

6) Predict Test Scores

-Eliminate Tests

7) Incredible Tool for Education Research

8) Replace High Stakes Tests

Assessment

49

Outline

Objective

Pedagogy that Works

Feedback – Closed Loop Education

Research from MIT

Revolution in Assessment

• Closing Thoughts

50

What you can gain

»Write interactive problems»Educational Research on them»Educational Research in general

What we can accomplish together»Partnerships in each - ideally all!!

51

You can write truly interactive material

Improve your problems from student data

Author in just your area of research expertise

Author tutorials, example problems, and problems involving applets

New education research tools improve your problems and exercises

Attractions for Problem Writers

52

New research tool with resolution of student’s capabilities and difficulties

Improve instructional material via Feedback

Experimentally Compare material and pedagogy

Develop texts, tutorials, traditional exercises, and more sophisticated problem sequences

Attractions for Curriculum Developers

53

Credits and ThanksEffective Educational Technologies, Inc.*

Alex Pritchard - sole programmer for 4 yearsAdam Morton - chief programmerDavid Kokorowski - content developmentAndrea Pritchard - president & treasurer & HR

Postdocs and UndergradsGabe Rockefeller - now at U. ArizonaPhil Dukes - now at UTBrownsvilleSofia Morote - now at Dennison College David Kokorowski - now at Effective Education TechRasil Warnakulasooriya - presentSupport:MIT - esp. Physics Dept. - support with TA’sNSF*DEP’s family has controlling financial interest

54

END

55

Student Opinion“How does the amount you learn per unit time with CyberTutor compare with time (including

checking solutions) spent on written homework?”

MIT Semesters

Much Less

Same

Much More

Results from MIT

56

Student Opinion“Would you recommend myCyberTutor for

8.01 next year?”Ratio of Yes to No

MIT Semesters

Results from MIT

57

Feedback – Closed Loop Education

58

Author training and manual

Pedagogy instructional material

Review first problems

Instruction on using feedback data to improve problems, guidelines for wrong answers

Help for Authors

59

85% Relilability is Unfair

1/4 of those students who failed had a passing true scoreAnother 1/4 should have failed, but passed

Future - Assessment

60

Attractions for Teachers

Tutoring available 24/7Better than written Homework

- not lower quality substitute

Shift grading effort into instruction

Teachers have instant access to detailed student performance data, facilitating Just-In-Time Teaching (JiTT)

Problem Library with Metadata on difficulty, time, student rating, topics involved

Objective

61

PedagogyWritten Homework

• No help for student when stuck

• Feedback to student takes 1 week

• Labor to grade• No feedback to

teacher• Copying is easy

Assign Do homework

Hand in

Grade

Learn from hw result?

62

Pedagogy

• Provides immediate feedback and help to the student

• Does the grading • Offers immediate

feedback to the teacher

• Supplies powerful insight into the student’s thinking

Assign

Learn whiledoing CyberTutor

homework

Detailed knowledgebase

myCyberTutor is a Web-Based Homework Tutorial System

Tutorials

• Tutorial problems in myCybertutor are carefully planned and sequenced instruction with SAQ’s

• They are used as instructional material to impart principles in deductive learning.

Pedagogy

64

Outline

ObjectivePedagogy

• Demo

• Results from MIT

• Feedback – Closed Loop Education

• Future - Assessment

• Closing Thoughts

65

Assessment/Testing Error

Any assessment has testing error - did you study problem that was on test?- careless mistakes?- lucky guess?

How to determine reliability (reproducibility)? - compare two equivalent tests

- split single test into two equivalent tests

If test is reliable (error-free), split grades will correlate

Future - Assessment

66Reliability 0.85 error =0.41

observed

Final Exam 2001-2002Future - Assessment