1

The Pépite project

Élisabeth Delozanne, Paris Universitas, UPMCD. Prévit, B. Grugeon, F. Chenevotot

Automatic Multi-criteria Assessment of Open-Ended Questions: a case study in

school algebra

ITS’2008

Cognitive modeling authoring tool Problem

Multi-step reasoning, multiple equivalent reasonings Our approach

1.An expert teacher (or a researcher) defines diagnosis exercises

2.A cognitive engineer implements templates that generalize these particular diagnosis exercises

3.A teacher clones these diagnosis exercises by filling template forms

4.A domain specific application • generates the clone and a set of plausible

correct and incorrect anticipated solutions• matches the student’s reasoning with

anticipated solutions2

OutlineAn introductory example

Pépite : a specific diagnosis toolPépiGen : a system to clone Pépite

Author’s and Student’s points of view Automatic Diagnosis

How does it work ? Pépinière *

•Formal processing of expression treesConclusion

3* in French : tree nursery

Blandine

Validity Incorrect V3

Use of letters Incorrect L3

Translation Step-by-step with incorrect chains T4

Algebraic Expressions writing

Incorrect use of parentheses with memory of meaning

EA31

Justification By algebra using incorrect rules J3

Aliou

Validity Incorrect V3

Use of letters No L5

Translation Step-by-step T2

Algebraic writing No EA?

Justification By example J2

Definitions Diagnosis exercise

An exercise (statement and user interface) an analysis grid to assess every plausible solution

anticipated by experts Clone

A similar exercise • has the same kind of statement and user

interface• gives the same kind of information on students’

competence an analysis grid

• to assess every plausible solution • automatically generated by the system 6

PépiGen A system to clone the Pépite diagnosis tool An author (a teacher)

Chooses an exercise to be cloned Enters the statement of the clone

PépiGen generates The student’s interface Each plausible solution (correct or incorrect) and

its assessment on several dimensions

7

The Author’s interface

8

The Student’s interface

9

The Automatic Diagnostic

10

Outline An introductory example

Pépite : a specific diagnosis tool PépiGen : a system to clone Pépite

How does it work ? Pépinière Expanding the tree of plausible steps of

correct and incorrect algebraic transformations Walking through the tree to anticipate different

solutions and their assessment Diagnosing the student’s reasoning

Conclusion

11

Plausible steps

(x+6)*3-3x

-2x+18

18

3x+18-3x

x*3+6*3-3x

x+6*3-3x

3x+18-3x

18x

21x-3x

R1

R3

R3

R2

R4

R3

R3

21x-3x

18x

R5

Correct rules

R1 : (A+B)C AC+BC

R3 : AB+AC A(B+C)

R2: (A+B)C A+BC

R4: AB+C B(A+C)

R5: A+B*C (A+B)*C

Incorrect rules

18

R3

R4

V1,EA1 V3,EA42 V3,EA31

V3,EA31EA42

V3,EA32

Analysis grid generation PépiGen

1. sends the algebraic expression to Pépinière that returns a tree of plausible steps• Validity and Algebraic Expression Writing

2. completes the plausible solutions set with• Non optimal algebraic

3. completes each solution assessment on the 5 dimensions• V, EA, L, T, J

4. saves each algebraic solution and its assessment• XML file : solution analysis grid

Note : arithmetic reasonings are analyzed by the

diagnosis system 13

Analysis grid (extract)<UnexpectedCorrectSolutions> (…) <Comment>Algebraic proof ; the student interprets

the statement as an equation</Comment> <Solution>

<Assessment>V2,EA1,L1,T1,J1</Asssessment> <Expression>(x+6)*3-3*x = 18</Expression><Expression>x*3+6*3-3*x = 18</Expression><Rule>C,3</Rule><Expression>x*3+18-3*x = 18</Expression> <Expression>18 = 18</Expression>

</Solution> </UnexpectedCorrectSolution>

14

Automatic diagnosis

XM L

Diagnosis system

loaded

ExpressionsTree

processor

Pépinière

Equivalent expression

tree ?True/False

save

XM L

Analysis grid

XM L

Student’s reasonin

g

loaded

Student’s reasoning+ assessment

Diagnosis algorithm Numerical or algebraic approach? Loop on each expression of the student’s reasonning

Build the expression tree (ST) Loop on each Plausible solution in the analysis grid

• Build the expression tree (PT)• If numerical approach

- substitute the numerical value in PT• If ST PT

- keep :PT, the rule and the comment and stop At the end

walk through PT to set up the final assessment save the final assessment, the comment and the

applied rules16

Results and tests On going work A demonstration prototype implements a complex

exercise cloning Authoring clones Solving Diagnosing

Preliminary Tests assessment of a corpus of 141 students’ solutions

• Multi-step reasoning• Multiple equivalent reasonings

3 teachers tested it in the lab

17

Discussion Diagnosis

compared with model tracing≈Tree of plausible steps: correct and incorrect

rules≠Emphasis : whole reasoning/step-by-step≠Several student’s types of reasoning derived

from a single solution branch≠Multidimensional assessment

Authoring Filling template forms

- Limited to specified exercisesAutomatic multidimensional diagnosis validated

by expertsNo programming, no modeling for teachers 18

Automatic Multi-criteria AssessmentOur proposal

Teachers clone a diagnosis tool previously designed by experts

The cloning process relies on• A preliminary educational study in the

domain• An implementation of templates of

diagnosis exercises• A specific application to analyze

reasonings that are not pre-formatedDemo: Friday afternoonhttp://pepite.univ-lemans.fr 19