Upload
vidor
View
39
Download
0
Tags:
Embed Size (px)
DESCRIPTION
The Pépite project. Evaluating the performance of a diagnostic system in School Algebra. Naima El- Kechaï , Élisabeth Delozanne , D. Prévit, B. Grugeon, F. Chenevotot. Outline. Why a cognitive diagnosis ? The Pépite project The diagnostic system A simple example PépiDiag Evaluation - PowerPoint PPT Presentation
Citation preview
1
The Pépite project
Naima El-Kechaï, Élisabeth Delozanne, D. Prévit, B. Grugeon, F. Chenevotot
Evaluating the performance of a diagnostic system
in School Algebra
Outline Why a cognitive diagnosis ?
The Pépite project The diagnostic system
A simple example PépiDiag
Evaluation Conclusion
2
The Pépite project Objective
To design and implement a web-based application To support math teachers To manage the students’ cognitive diversity
Domain Algebra class 9th-10th grade (15-16 years old)
3
A teaching scenario Who ?
Mary is a Math teacher in 9th grade Context
To bring all her class to the same level Steps
1. Students have the online diagnostic test2. Pépite
a) analyzes the students’ answers b) displays a cognitive profile of each student and of
the whole classc) suggests 6 groups of students and a session of
exercises adapted to their cognitive profiles3. Mary can adjust the groups and the exercises4. Students log in and do the exercises in their assigned
session 4
A cognitive profile of a classGroupe A+
Engage in algebraic thinking
Use algebraic calculation appropriately
Groupe B+ Begin to use algebra to
solve problems Use sometimes mal-rules
Groupe C- Stuck in arithmetical
thinking Algebra makes no sense
5
A B+ student’s cognitive profile
6
Level Personal featuresAlgebraic calculation Techniques
Sucess rate on asked questions: 4/12Interpreting algebraic expressionsSuccess rate on asked questions: 11/23
Strong pointsPartial mastery of algebraic rulesSome good interpretations of algebraic expressions
Weak pointsLow mastery of algebraic calculation
Usage of algebra Mathematical modelingSuccess rate on asked questions: 5/9
Strong pointsGood mastery of algebra usage on some problems
Weak pointsJustification by school authority
Algebraic translation Translating situations to algebraSuccess rate on asked questions: 12/24
Strong pointsSome good translations of mathematical relations
Weak pointsSome translations without reformulation
More
More
More
More
More
More
A differentiated session
7
Pépite : Cognitive Diagnosis Process
8
A simple example
9
Answer assessment (local diagnosis)
10
Assessment Dimensions Values CodesValidity Incorrect V3Use of Letters Incorrect L3Translation Incorrect T332Algebraic Expressions writing
turning a product into a sum a*a-> 2a
EA41
Justification By algebra using incorrect rules J3Type of answers :
Recognition of sub-figures but turning a product into a sum
Exercise coding prescription file
Student’s answers Answers coding+ Student’s profile
PépiDiag general architecture
11
Student’s interface
PépiDiag
Teacher’s interfaceStudent
Database
Student’s answers
Exercise Database
Solves exercises
A coding prescription file (partial)Type Label Code Patterns of
equivalent expressions
1 Correct expression of the area as a product Length × Width
V1, T1,J1 (a + 3)×(b + a)
2 Correct expression of the area by adding the areas of the different rectangles
V2, T1, J1 ab+a²+3b+3a; (a×(a+b))+(3×(a+b));a(a+3)+b(a+3)
3.1 Recognition of sub-figures with parentheses errors (e.g. a+3 a+b)
V3, EA3, T3, J3 (a+3)×b+a; a+3×(b+a); a+3×b+a; a+3×a+b; a+b×a+3
3.2 Recognition of sub-figures with gathering transformations
V3, EA42, T3, J3 (a+3)(ab); (b+a)×(3a); a+3 × ab ; ab×a+3; 3a×a+b ; a+b×3a
3.3 Recognition of sub-figures but turning a product into a sum a²->2a
V3, EA41, T3, J3 5a+ab+3b; ab+2a+3b+3a
…9 No interpretation Vx 12
PépiDiag
• Local Diagnosis
• Exercise coding
prescription file
• Coding of student’s answers
• Computer• Algebra
System
• Student’ s
answers file
13
• Global Diagnosis
• Student’s profile
Outline Why a cognitive diagnosis ?
The Pépite project The diagnostic system
A simple examplePépiDiag
Evaluation Conclusion
14
Evaluation Quality requirements
1. No correct answer badly coded2. Better no code than a wrong one3. Minimal number of answers left unanalyzed
Method Comparison between experts & system
assessment 3 experts assessed 360 answers of each exercise
(one by one) Experts worked separately and then agreed on
some corrections to minor errors
15
Unanalyzed answers
16
Empty answers
Students’ answers
94/360 266/360
Data collected (on the exercise used as an example)
PépiDiag analyzes • More answers than the teacher• Less than the experienced researcher
Answers coded by PépiDiag
Answers Unanalyzed by PépiDiag
Answers Unanalyzed by 3 experts
218/266 (82%)
48/266 (18%)
24/266 (9%)
Correct answers
No correct answer is analyzed incorrectly Unanalyzed answers
5/13 : Using X instead of * 8/13 : Mixing algebra and natural language :
• « (a+b) times (3+a) » Solutions
To prevent students from typing letters other than those relating to the exercise statement
17
Correct answers
Correct automatic coding
Unanalyzed answers by PépiDiag
Unanalyzed answersBy experts
149/266 136/149 13/149 0(56%) (91%) (9 %)
Incorrect coding b+ axa +3 (parenthesis errors) b + a² + 3 (gathering the figure items) Both are incorrect (V3) distinct for human experts
and the same for the CAS Unanalyzed : incompleteness + interface
Incorrect answers
18
Incorrect answers
Correct automatic coding
Incorrect automaticcoding
Unanalyzed answers by PépiDiag
Unanalyzed answersBy experts
Unanalyzed by PépiDiagAnalyzed by experts
117/266 (44%)
80/82 2/82 35/117(30 %)
24/117(21 %)
11/117(9%)
Conclusion Our approach to asses students’ open answers
Human experts• typify anticipated patterns of correct and
incorrect solutions • code each type on several dimensions
PépiDiag, using a CAS,• matches the student’s solution with a pattern
PépiDiag is reliable 100 % : closed answers 80 % : answers with one algebraic expression 70 % : answers with a multi-step reasoning [ITS
2008] PépiDiag saves teachers’ time and tedious effort
19
Current work http://www.sesamath.net/ Diagnosis
To evaluate the building of the student’s profile from the answer assessment
Comparing• Heuristic method (implemented)• Statistical building (being implemented)• Neural Networks (being implemented)
A differentiation support tool for teachers To validate with teachers the different
learning paths adapted to each (group of profile) To build a tool to automatically
• retrieve indexed exercises from a data base • suggest a session adapted to a student’s profile 20