1 Andes Tutoring: Freedom, Support, and Accelerated Learning as Students Solve Complex Physics Problems Kurt VanLehn & Brett van de Sande School of Computing,

1

Andes Tutoring: Freedom, Support,

and Accelerated Learning as Students

Solve Complex Physics Problems

Kurt VanLehn & Brett van de SandeSchool of Computing, Informatics and Decision Engineering

Arizona State University

USN

2

Andes development team Developers

– Brett van de Sande

Instructors/designers– Bob Shelby– Don Treacy

Experimenters– Scotty Craig– Bob Hausmann– Sandy Katz– Tim Nokes– Michael Ringenberg

Instructors4-year colleges

Jim Culbertson – Arizona State University, AZJohn Fontinella – US Naval Academy, MDDavid Guerra – St. Anselm College, NHTroy Hacker – US Air Force Academy, COAndrew Heckler – Ohio State University, OHGerd Kortemeyer – Michigan State UniversityTed McClanahan – US Naval Academy, MDMary Wintersgill – US Naval Academy, MD

2-year collegesTom Wilbur – Ann Arundel Com. College, MD

High SchoolsSophia Gersham – Watchung Hills Reg. HS, NJPaul Perkins – Belleview Christian School, WADavid Richardson – Packer Academy, NY

3

Outline

Goals of the Andes project Andes – the core Andes – the surroundings Andes – technology Evaluation Why does Andes succeed?

Next

4

Why physics?

Required for the other sciences Major source of attrition Well-studied in cognitive science Other courses are similar

5

A typical physics courseRepeat: Read chapter Attend lectures Solve problems Do lab Review for exam Take chapter exam

Take finals: Standardized & conceptual

Tutoring needed

6

Why focus on problem solving?

Students spend most of their time on problem solving

Frustrating; May causes attrition Bad habits can develop

– “Symbol pushing” instead of deep understanding

Done right, it can elicit deep, conceptual understanding

Professional human tutors focus on it

7

What do professional tutors do?

Pick a good problem– Neither trivially easy nor impossibly difficult– Problem targets a weakness of the student

Help the student solve it– Let student try each step– Give immediate feedback if step is wrong– Give hints sparingly

Reflective debriefing– What were main principles?– What did you learn?

Andes could, but doesn’t

Andes does

Under development

8

How to evaluate Andes?Repeat: Read chapter Attend lectures Solve problems Do lab Review for exam Take chapter exam

1. Paper & human graders

2. Paper & grading service

3. Andes

4. Professionalhuman tutor

Take finals: Standardized & Conceptual

Com

paris

on c

ondi

tions

Just these 2 as of todayJust these 2 as of today

9

Main goals: Andes should be a “workbook” where:

Instructors select & assign Andes problems– Need lots of problems, covering all textbooks

Students solve problems on Andes, not paper – getting immediate feedback and hints

Students learn more, compared to paper

10

More goals

Instructors agree with all of Andes’ advice Instructors no longer need to grade

homework Students prefer Andes to paper Andes is as effective as human tutors

11

Outline


Next

12

A typical physics problem

University introductory physics courses High school physics courses

A 2000 kg car at the top of a 20º inclined driveway 20 m long slips its parking brake and rolls down. If we ignore friction and drag, what is the magnitude of the velocity of the car when it hits the garage door?

13

Andes user interface

Read a physics problem

Type in equations

Draw vectors

Type in answer

14

Andes feedback and hints“What should I do next?”

Green means correctRed means incorrect

“What’s wrong with that?”

Dialogue & hints

15

Frequently asked questions (slide 1 of 3) Why is Andes sometimes called an intelligent

tutoring system?– Tutors usually replace classes; Andes doesn’t.– Andes should be called a “step-based homework

helper”

Is Andes the same as web-based grading services such as WebAssign, Mastering Physics & LON-CAPA?– They give feedback & hints on answers only– Andes gives feedback & hints on steps leading to

answers

16

Most existing tutoring systems and web-based grading services accept only the answer

W = 25

Answer

What is the value ofangle w?

45°

w°

x°

30°

40°

z°y°

u°

17

With step-based systems (e.g., Andes), students enter steps leading up to the answer

Step

Step

Step

Answer

Step

40+30+x=180

x=110

x=z

w+45+z=180

w+45+110=180

w=180-155

w=25

Step

Step

Step


45°

w°

x°

30°

40°

z°y°

u°

18

Step-based homework helpers usually give immediate feedback on each step

w = 40

OPPS!

OK

45°

w°

x°

30°

40°

z°y°

u°


19

Hints start general…

w = 40

Lines that look parallel often are not.

OK

45°

w°

x°

30°

40°

z°y°

u°


20

Hints become more specific.

w = 40

Try summing the angles of the triangle that include angle w. (see pg. 212)

OK

45°

w°

x°

30°

40°

z°y°

u°


21

Usually, the last hint tells the student exactly what to enter.

w = 40

You should apply the triangle sum rule by entering 45+z+w=180.

OK

45°

w°

x°

30°

40°

z°y°

u°


22

Frequently asked questions(slide 2 of 3)

Does Andes require students to do all the steps?– No. But skipping conceptual steps lowers the score.

Does it require doing steps in a particular order?– No, except that variables must be defined before being

used in equations– The only way to define a vector component is to draw

the vector

Are there correct solutions that Andes doesn’t accept.– No (we hope).

23

Frequently asked questions(slide 3 of 3)

Why does Andes focus on problem solving? Problem solving doesn’t improve conceptual understanding [sic]. Instruction really should focus on __________ instead of problem-solving. – Andes requires more conceptual steps than paper.– Reflective debriefing is being added– Perhaps instructors should let Andes to handle

problem-solving so they can focus on __________.

24

Outline


Next

25

Two methods for accessing Andes problems

Stand-alone– Downloads as a standard Windows application

Open-Learning Initiative (Web-based LMS)

26

Standalone opening screen is a menu of physics topics (apprx. chapters)

27

After clicking on “Translational dynamics,” get menu of problems

28

Two methods for accessing Andes problems

Stand-alone– Downloads as a standard Windows application

Open-Learning Initiative (Web-based LMS)– Free & open usage

» OLI keeps no record of student’s usage

– Authenticated member of a registered class» OLI keeps a grade book for the instructor

Next

29

OLI top level screen

Andes

30

OLI top level of Andes is a menu of topics (apprx. Chapters)

31

After clicking on “Translational Dynamics” module, get menu of problems

32

OLI grade book Looks/acts like a spreadsheet

– One row per student– One column per problem– Cell has student’s score on the problem

Can export to Excel or database Clicking a cell displays student’s solution in

instructor’s Andes.– Can check student’s work– Good for office hours

33

Outline


Next

The major components of an ITS

P(mastery) for each knowledge compo.

Problem to be solved

All correct steps in all orders

Response pattern for each student step

Expert

Assessor

Helper

Use

r in

terf

ace

The expert’s computation Expert can be authors,

students or an expert system

Solve the problem in all acceptable ways

Record steps taken Record knowledge

components used at each step




Response patterns for each student step

Expert

Assessor

Helper

Andes uses an expert system

Knowledge base organized by “problem solving methods” – One per major physics principle– Often called as “principle schemas”– Approximately 100 principles covered in 1-year course

Can solve more that 500 physics problems For each, finds all acceptable solutions

– Merges them into a solution graph

Pre-generates solution graphs & saves on disk– Allows regression testing (compare new to old)– Allows instructors to inspect

36

Problem solving method for Newton’s second lawTo apply Newton’s second law to <body> at <time> along <axis>: Draw a free-body diagram for <body> at <time> including <axis>

– For each force on <body> at <time>, draw it.– Draw the acceleration of <body> at <time>

Write the F=m*a in terms of components along <axis> For each vector (i.e., acceleration and each force), write a projection equation

for the vector along <axis> For each minor principle of the form <force magnitude> = <expression> where

the force is one of the ones on <body> at <time>, write the minor principle’s equation.

37

38

Problem & its solution graph

Draw coord. axes @ 20 deg

Define given quantitiesd=20 m; m= 200kg

Apply conservation of energyvf^2 = 2*g*sin(20 deg)

Apply translational kinematicsvi^2 = 2*a*d

Apply Newton’s second lawa= -g*cos(200 deg)

Solve equations for final velocityvf= 11.59 m/s

Enter answer: 11.59 m/s

Draw body for car

1. Draw free-body diagram2. Apply F=m*a along x-axis3. Project vectors onto x-axis4. Apply weight law W=m*g

•Draw weight force•Draw normal force•Draw acceleration

4 popular designsfor the Expert

Hand-author all possible solutions per problem– AutoTutor, CTAT

Rule-based AI problem solver + problem all possible solutions– Andes, Cognitive tutors

Hand-author one solution & use constraints to generalize to all possible solutions– Constraint-based tutors e.g., SQL Tutor

Given log files from students, induce shortest/best paths– iLisp, Barnes tutor



Expert

The helper’s computation

When the student enters a step, match it to a correct step

Give feedback & hints as necessary

Record response pattern

P(mastery) for each knowledge compo




Expert

Assessor

Helper

How to tell if student’s equation is correct? Color by numbers algorithm Given student’s equation “Fw_x+m*g*sin(20 deg)” Substitute values from solution point

– Fw_x 6704 N– g 9.8 m/s^2– m 2000 kg

Check arithmetic.– If checks, equation is correct

See paper for dealing with variables that don’t have values in the solution point

41

When the student needs a hint, how to choose next step?

Which solution branch is student probably following?

First PSM along path that is not finished? First step inside the PSM that is not yet

done?

But how to tell which equation-writing steps have been done already?

42

Matching equations via the indy check algorithm

Student enters “Fw_x+m*g*sin(20 deg)”– Row S below

Which equations in solution were combined?– Rows A, B, C, D– Because S’s gradient is a linear combination of their gradients

43

Function f f/m f/g f/Fw f/Fw_x gradient

A Fw_x –Fw*cos(250º) 0 0 –cos250º 1 (0, 0, 0.342, 1)

B Fw – mc*g –g –mc 1 0 (–9.8 ,–2000, 1,0)

C mc – 2000 1 0 0 0 (1, 0, 0, 0)

D g – 9.8 0 1 0 0 (0, 1, 0, 0)

S Fw_x+mc*g*sin(20º) g*sin20º mc*sin20º 0 1 (3.352, 684, 0, 1)

3 main design issuesfor the Helper

Matching student’s step to correct steps– Natural language: Use LSA, keywords, Atlas…– Math expressions: Substitute numbers for variables– Physical actions: Fuzzy, Bayesian

Handling pedagogically important student errors Managing the student-tutor dialogue

– Immediate feedback + hint sequences

– Delayed feedback + student or tutor controlled debriefing

– Adaptive, especially decision theoretic & fading



Helper

The assessor’s computation Given

– Response patterns for each step taken by the student

– Old P(mastery) for each knowledge component

Calculate– New P(mastery)





Expert

Assessor

Helper

Currently, Andes doesn’t do such assessment

Where’s the scale-up bottleneck?

Number of problems

Cod

e si

ze

Expert

Helper

Assessor

Userinterface

Other scaling up issues= same as other reforms

Coordination with curriculum & standards Teacher buy-in and training Support Etc…

Implementation: Summary

Expert– Expert system,

not humans Helper

– Matching student equations is challenging

Assessor– Dynamic

Bayesian networksP(mastery) for each knowledge component




Expert

Assessor

Helper

49

Outline


Next

50

Prior work with answer-only homework helpers

Compared to ordinary paper-based homework– Modest effect size: 0.42 (Kulik et al., 1983)

Compared to paper-based homework that iscollected & graded– No benefits (Pascarella, 2002; Dufresne, Mestre & Rath, 2002)

Interpretation – Motivating students to do their homework provides

benefits– But answer-only feedback & hints provides no

additional benefits

51

Prior work with step-based homework helpers

Lisp Tutor (Corbett, 2001) and many others– Same homework problems & text– Experimenter’s exams only– Not a whole semester (only 5 lessons)

Cognitive Tutors (Koedinger et al; Carnegie Learning)

– Whole year of high-school algebra, geometry– Both experimenter’s exams & standard exams– Curriculum confounded with tutoring system

52

Evaluation of Andes at the US Naval Academy

Fall semesters 2000, 2001, 2002 & 2003 Only the homework modality was varied:

Andes vs. paper-based– Same textbook

– Similar lectures, labs, recitations

– Similar homework problems

– Same exams

Students were motivated to do paper-based homework– Either collected and graded

– Or 1 homework problem on each quiz

53

Exams Midterm exam

– 1 hour, 4 problems– Scored on derivation & answer

» Drawings (30%)» Variable definitions (20%)» Equations (40%)» Answers (10%)

Final exam– 3 hours, 50 problems– Multiple choice

54

Subjects; non-random assignment

Prior competence equal, all 4 years– Grade-point averages equal – Distribution of majors equal

2000 2001 2002 2003

N Andes 140 129 93 93

N control 135 44 53 44

55

Midterm exam results(All differences reliable, p < .01)

50

55

60

65

70

75

2000 2001 2002 2003

ControlAndes

Effect size: 0.61

56

Final exam: Methodological details

Andes coverage of the course increased to 70% by 2003, so used only that year’s final exam

Non-random sample– 89 Andes students (3 sections)– 823 non-Andes students (rest of course)– GPAs, Majors reliably different

Regressed out incoming GPA, Major

57

Final exam results

-1

-0.5

0

0.5

1

1.5

2

2.5

Control Andes

Difference is reliable (p = 0.028)

Effect size = 0.25

58

Effect sizes for subscores of midterm exam

-1

-0.5

0

0.5

1

1.5

Drawings Variables Equations Answers

The more conceptual the subscore, the greater the benefit

59

Andesy = 0.9473x - 2.4138

R2 = 0.2882

Controlsy = 0.7956x - 2.5202

R2 = 0.2048

-3.0000

-2.0000

-1.0000

0.0000

1.0000

2.0000

3.0000

1 1.5 2 2.5 3 3.5 4

GPA

Z-s

core

on

exam

ANDES

CONTROLS

Linear (ANDES)

Linear (CONTROLS)

Benefits same regardless of GPA

60

Benefits varied by major on final exam but not on midterm exam

Midterm exam results

-0.5

-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

Engineers Scientists Others

Control Andes

Final exam results

-1

-0.5

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

Engineers Scientists Others Overall

Non-Andes Andes

61

Interpretation of results Engineering & science majors knew

red path & preferred it for answer-only– Andes didn’t affect their final exam scores

Other majors did not have red path, so they used the blue path on answer-only– Andes increased their final exam scores

Everyone used blue path on midterms– Andes increased everyone’s midterm exam

scores– Biggest benefit for diagrams & variables– Smaller on equations; none on answer

Problem

Diagram & variables

Equations

Answer

Andes

Andes

Prior math & physics

62

Summary of results Main result: Replacing graded paper homework with

Andes provides benefits– Midterm exam effect size: 0.61 – Final exam effect size: 0.25

Andes helps students learn conceptual skills– Effect sizes on conceptual subscores: 1.21 & 0.69– Effect sizes on calculational subscores: 0.11 & -0.08

Engineering & Science majors appear to have a non-conceptual method for solving problems– Competes with the conceptual method taught by Andes– They use it on the (answer-only) final exam– This dilutes the benefit of Andes on final exam

63

Andes’ effect sizes are typical of other step-based homework helpers

0

0.2

0.4

0.6

0.8

1

1.2

1.4

Experimenter'sexam:

Conceptualparts

Experimenter'sexam: Lessconceptual

parts

Standard exam

Lisp tutorCognitive tutorAndes

64

Take-home message on homework helpers

Baseline:– Paper-based with light or no human grading

Better: – Answer-based e.g., WebAssign

Best: – Step-based e.g., Andes–

65

Outline


Next

66

Why these results? Hypotheses:

Baseline:– Paper-based with light or no human grading– Students often do not do their homework

Better: – Answer-based e.g., WebAssign – Students do homework, but with non-optimal methods

Best: – Step-based e.g., Andes– Students do homework with optimal method

67

3 methods for doing homework

1. Get answers from friends

2. Copy & edit another problem’s solution

3. Generate each step oneself

68

1. Getting answers from friends

No learning, so should be discouraged Even if the numbers in a problem are

randomly generated, students circulate spreadsheets that calculate answers

Andes requires that students “show their work”– Andes can analyze time per step, too.– So Andes can make this kind of cheating very

difficult

69

2. Copy and edit another problem’s solution

Students learn general solution schemas– E.g., for pulley problems, a = g(m1-m2)/(m1+m2)– Good for learning to solve algebra word problems– High math students tend to use it (cf. USNA)– But should learn physics principles, not problems

Andes mildly discourages– Must close current problem in order to open an old one– Example solutions are videos, not paper

Andes should implement “fading”

70

3. Generate each step oneself

If stuck, get hints (from Andes), or refer to textbook or videos of examples– Should focus on learning the principle, not just

getting unstuck– Hint sequences start vague, become specific

Causes learning of principles, not problems

71

Hypothesized distributions of solution methods explains results

0%

20%

40%

60%

80%

100%

Light or no grading

Answer-based

Step-based e.g., Andes

Fre

qu

en

ce

of

so

luti

on

me

tho

d

Type of homework helper

3. Each step oneself

2. Copy & edit solutions

1. Copy answers

No solution

72

Take-home message, again

Baseline:– Paper-based with light or no human grading– Students often do not do their homework

Better: – Answer-based e.g., WebAssign – Students do homework, but with non-optimal methods

Best: – Step-based e.g., Andes– Students do homework with optimal method

73

Outline

Goals of the Andes project Andes – the core Andes – the surroundings Evaluation Why does Andes succeed? Future work

Next

74

Future work

Andes3– Client/server (no downloading necessary)– Client runs in any browser & any OS– User interface ≈ PowerPoint (for familiarity)– Compatible with WebAssign & other services

Help others to author Andes problems Support conceptual problem solving

75

Quantitative problem solving is necessary but not sufficient for deep understanding

Necessary– Avoids inert knowledge

Not sufficient– Question:

» If a massive truck and a lightweight car have a head-on collision, on which vehicle is the impact force greater?

– Students who got A’s say: The car– Correct answer: Equal (Newton’s third law)

Many concept inventories (exams) exist

76

Does Andes improve scores on concept inventories?

Learning principles instead of problems should help

But not enough– USNA: FCI not significantly different

77

Why so little transfer from quantitative to conceptual problem solving?

Conceptual prob. solving knowledge

Quantitative prob. solving knowledge

Traditional assumption lots of transfer


Cognitive modeling little transfer


78

Solution: Add conceptual problem solving practice

Question: In the problem you just solved, what would be different if the car had no mass?



W=m*gIf no mass, then no weight

79

Katz & Connelly’s experimental addition to Andes After students have finished an Andes problem,

use a natural language dialogue system to discuss the problem’s solution

System asks lots of conceptual questions

80

Students first solve an Andes problem

81

Andes then asks about principles, alternative situations…

Andes: Which major principle did you need to solve the problem? Student: Newton's 2nd law Andes: Yes. The main difference between this problem and most of the

previous Newtons 2nd law problems is the fact that the direction of acceleration is __________. (fill in the blank)

Student: centripetal Andes: Correct. All else being the same, how would the velocity at the

lowest point of the swing change if the length of the rope were doubled to 4.0 m?

Student: it would also double Andes: Well, it would increase, but it would not double. Recall that the

formula for centripetal acceleration involves the square of the velocity. …

82

Results of Katz/Connelly expt. Successes

Treatment students significantly outperformed control students on a mostly qualitative post-test

Instructors would like reflective dialogue to become a permanent feature of Andes

Limitations Currently covers only 8 weeks of Andes No significant effect on FCI at the end of the

semester Next steps

Revise content to target FCI Cover whole semester

83

Thanks for your attention!

At www.andestutor.org you can…– Download the stand-alone version of Andes– Try the OLI version of Andes– Download papers on Andes– View videos of Andes being used

http://www.andes/

Questions?

84




Response pattern for each student step

Expert

Assessor

Helper

Use

r in

terf

ace

Andesy = 0.9473x - 2.4138

R2 = 0.2882

Controlsy = 0.7956x - 2.5202

R2 = 0.2048

-3.0000

-2.0000

-1.0000

0.0000

1.0000

2.0000

3.0000

1 1.5 2 2.5 3 3.5 4

GPA

Z-s

core

on

exam

ANDES

CONTROLS

Linear (ANDES)

Linear (CONTROLS)

Documents

1 Andes Tutoring: Freedom, Support, and Accelerated Learning as Students Solve Complex Physics Problems Kurt VanLehn & Brett van de Sande School of Computing,