Embed Size (px)
Citation preview
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
A Rasch Analysis of Math Testsusing M.In.E.R.Va. Platform
Grazia Messineo - Salvatore Vassallo
ICETC2014
Messineo - Vassallo Rasch Analysis in Minerva Platform 1/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
General frameThe Web Platform
The problem
Students must achieve some automatism in eachdiscipline to succeedMany students have difficulties in Mathematics(see Ocse-Pisa reports)Traditional ways (courses) to help students aredifficult to implement, also because of their costs
Messineo - Vassallo Rasch Analysis in Minerva Platform 2/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
General frameThe Web Platform
Goals
To help students to recover their deficiencies in adisciplineTo provide teachers a tool to assign common orpersonalized tasks to studentsTo provide teachers a tool to automatically verifystudents’ achievements
Messineo - Vassallo Rasch Analysis in Minerva Platform 3/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
General frameThe Web Platform
The structure: contents
Theoretical schemes to strengthen or createknowledgeGuided exercises to illustrate the correct way forsolving the problemsConsolidation and self-assessment exercisesPerformance assessment exercises
Messineo - Vassallo Rasch Analysis in Minerva Platform 4/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
General frameThe Web Platform
The structure: feedback
Assignment of a scoreAutomatic answers correctionHint to get a solutionReferences to theoretical contentSolution (complete or in partial form)
Messineo - Vassallo Rasch Analysis in Minerva Platform 5/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
General frameThe Web Platform
How it works
The software consists of some LATEX files: Thepackage minerva, the master files, and the files ofthe exercisesEach file of exercises contains one or morevariants of a basic exerciseThe output is a pdf file with interactive contentmanaged by Javascript
Messineo - Vassallo Rasch Analysis in Minerva Platform 6/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
General frameThe Web Platform
Advantages
Possibility of being used offlineWide range of exercises:
Multiple choice testsOpen questions (with powerful automaticcomputation)Multiple answers questions (with different marksfor any question)
Messineo - Vassallo Rasch Analysis in Minerva Platform 7/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
General frameThe Web Platform
Advantages
Use without installing any additional applicationExcellent display of mathematical textIndependence from operating systemAbility to use the same tool in different situationsand disciplines
Messineo - Vassallo Rasch Analysis in Minerva Platform 8/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
General frameThe Web Platform
The generation of a test
A variant is randomly chosenVariants are randomly scrambledAnswers are shuffledValues are assigned to random parametersA test is produced (PDF file)
Messineo - Vassallo Rasch Analysis in Minerva Platform 9/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
General frameThe Web Platform
The url: http://progettominerva.unicatt.it
Messineo - Vassallo Rasch Analysis in Minerva Platform 10/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
General frameThe Web Platform
Students’ interface - 1
Messineo - Vassallo Rasch Analysis in Minerva Platform 11/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
General frameThe Web Platform
Students’ interface - 2
Messineo - Vassallo Rasch Analysis in Minerva Platform 12/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
General frameThe Web Platform
Teachers’ interface - 1
Messineo - Vassallo Rasch Analysis in Minerva Platform 13/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
General frameThe Web Platform
Results
Messineo - Vassallo Rasch Analysis in Minerva Platform 14/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
Initial dataIndividual AnalysisDifferential Analysis
Secondary Schools
Trial has been going on since school year 2011-12Results before and after the use of the tool
Messineo - Vassallo Rasch Analysis in Minerva Platform 15/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
Initial dataIndividual AnalysisDifferential Analysis
University
Trial started in February 2014Students can perform exercises and studytheoretical material onlineA group gives a part of the final exam online:figure shows positive marks.
Messineo - Vassallo Rasch Analysis in Minerva Platform 16/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
Initial dataIndividual AnalysisDifferential Analysis
Rasch model
Mathematical model to compute the probability𝑝𝑖𝑗 that the subject 𝑖 with ability 𝜃𝑖 answerscorrectly to question 𝑗 with difficulty 𝛽𝑗Rasch measure based on the difference 𝜃𝑖 − 𝛽𝑗Data collected in a raw score matrix: 𝑛 rows(subjects) and 𝑚 columns (items)Conversion of values of measures of ability anddifficulty into a common scale (logit)
Messineo - Vassallo Rasch Analysis in Minerva Platform 17/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
Initial dataIndividual AnalysisDifferential Analysis
The analysis performed in 2014
Data collected from a test of MathematicsTwo sets of students: 20 of Catholic Universityand 16 of a Secondary SchoolQuestionnaire with 16 multiple choice questions,8 on derivatives and 8 on domains
Messineo - Vassallo Rasch Analysis in Minerva Platform 18/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
Initial dataIndividual AnalysisDifferential Analysis
Summary Statistics
Messineo - Vassallo Rasch Analysis in Minerva Platform 19/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
Initial dataIndividual AnalysisDifferential Analysis
Summary Statistics interpretation
For students: separation index lower than 2 andreliability index lower than 0.8
→ more items maybe needed for a better separation of ability levels(actually we can divide the sample into 3 levels)For items: separation index lower 3 and reliabilityindex lower than 0.9 → the students’ sample isnot large enough to confirm the item difficultyscale.
Messineo - Vassallo Rasch Analysis in Minerva Platform 20/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
Initial dataIndividual AnalysisDifferential Analysis
Summary Statistics interpretation
For students: separation index lower than 2 andreliability index lower than 0.8→ more items maybe needed for a better separation of ability levels(actually we can divide the sample into 3 levels)
For items: separation index lower 3 and reliabilityindex lower than 0.9 → the students’ sample isnot large enough to confirm the item difficultyscale.
Messineo - Vassallo Rasch Analysis in Minerva Platform 20/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
Initial dataIndividual AnalysisDifferential Analysis
Summary Statistics interpretation
For students: separation index lower than 2 andreliability index lower than 0.8→ more items maybe needed for a better separation of ability levels(actually we can divide the sample into 3 levels)For items: separation index lower 3 and reliabilityindex lower than 0.9
→ the students’ sample isnot large enough to confirm the item difficultyscale.
Messineo - Vassallo Rasch Analysis in Minerva Platform 20/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
Initial dataIndividual AnalysisDifferential Analysis
Summary Statistics interpretation
For students: separation index lower than 2 andreliability index lower than 0.8→ more items maybe needed for a better separation of ability levels(actually we can divide the sample into 3 levels)For items: separation index lower 3 and reliabilityindex lower than 0.9 → the students’ sample isnot large enough to confirm the item difficultyscale.
Messineo - Vassallo Rasch Analysis in Minerva Platform 20/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
Initial dataIndividual AnalysisDifferential Analysis
Person-Item Distribution Map
Messineo - Vassallo Rasch Analysis in Minerva Platform 21/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
Initial dataIndividual AnalysisDifferential Analysis
Person-Item Distribution Map
Messineo - Vassallo Rasch Analysis in Minerva Platform 21/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
Initial dataIndividual AnalysisDifferential Analysis
Person-Item Distribution Map
Messineo - Vassallo Rasch Analysis in Minerva Platform 21/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
Initial dataIndividual AnalysisDifferential Analysis
Person-Item Distribution Map
Messineo - Vassallo Rasch Analysis in Minerva Platform 21/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
Initial dataIndividual AnalysisDifferential Analysis
Person-Item Distribution Map interpretation
students’ ability is on the left, from highest (top)to lowest (bottom);items are on the right, ordered by difficulty, frommost difficult to easiest;both are measured on the same scale;average of difficulty of items is set equal to 0;students’ mean value is 0.12 → students’performance is a little above the expectedperformance.
Messineo - Vassallo Rasch Analysis in Minerva Platform 22/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
Initial dataIndividual AnalysisDifferential Analysis
Guttman Scalogram
Messineo - Vassallo Rasch Analysis in Minerva Platform 23/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
Initial dataIndividual AnalysisDifferential Analysis
Guttman Scalogram
Messineo - Vassallo Rasch Analysis in Minerva Platform 23/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
Initial dataIndividual AnalysisDifferential Analysis
Guttman Scalogram
Messineo - Vassallo Rasch Analysis in Minerva Platform 23/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
Initial dataIndividual AnalysisDifferential Analysis
Guttman Scalogram
Messineo - Vassallo Rasch Analysis in Minerva Platform 23/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
Initial dataIndividual AnalysisDifferential Analysis
Guttman Scalogram interpretation
𝐴 → unexpected correct answer@ → unexpected wrong answerhelps identify “lucky guesses” and other patternsof response
Messineo - Vassallo Rasch Analysis in Minerva Platform 24/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
Initial dataIndividual AnalysisDifferential Analysis
Item Category/Option/Distractor Frequencies
Messineo - Vassallo Rasch Analysis in Minerva Platform 25/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
Initial dataIndividual AnalysisDifferential Analysis
Scatterplot
Messineo - Vassallo Rasch Analysis in Minerva Platform 26/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
Initial dataIndividual AnalysisDifferential Analysis
Scatterplot interpretation
Comparison of performance in the two groupsoutliers are the items outside the two bands thatrepresent the limits of the confidence interval(95%)DOM20 is significatively more difficult forUniversity studentsDER15 is significatively more difficult forSecondary School students
Messineo - Vassallo Rasch Analysis in Minerva Platform 27/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
Initial dataIndividual AnalysisDifferential Analysis
THANK YOU
Messineo - Vassallo Rasch Analysis in Minerva Platform 28/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
Initial dataIndividual AnalysisDifferential Analysis
Messineo - Vassallo Rasch Analysis in Minerva Platform 29/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
Initial dataIndividual AnalysisDifferential Analysis
Messineo - Vassallo Rasch Analysis in Minerva Platform 30/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
Initial dataIndividual AnalysisDifferential Analysis
Messineo - Vassallo Rasch Analysis in Minerva Platform 30/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
Initial dataIndividual AnalysisDifferential Analysis
Messineo - Vassallo Rasch Analysis in Minerva Platform 30/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
Initial dataIndividual AnalysisDifferential Analysis
Messineo - Vassallo Rasch Analysis in Minerva Platform 30/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
Initial dataIndividual AnalysisDifferential Analysis
Messineo - Vassallo Rasch Analysis in Minerva Platform 31/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
Initial dataIndividual AnalysisDifferential Analysis
Messineo - Vassallo Rasch Analysis in Minerva Platform 32/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
Initial dataIndividual AnalysisDifferential Analysis
Messineo - Vassallo Rasch Analysis in Minerva Platform 32/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
Initial dataIndividual AnalysisDifferential Analysis
Messineo - Vassallo Rasch Analysis in Minerva Platform 33/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
DIF Plot for students
Messineo - Vassallo Rasch Analysis in Minerva Platform 34/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
DIF Plot - interpretation
Plot shows the average difference for each groupbetween the observed and expected responseDER15, DER03, DOM20 show significativedifferences in the two groups
Messineo - Vassallo Rasch Analysis in Minerva Platform 35/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
Scatterplot Female-Male
Messineo - Vassallo Rasch Analysis in Minerva Platform 36/37
The M.In.E.R.Va ProjectStatistical Analysis and Rasch Model
Other Tables
DIF Plot for items
Messineo - Vassallo Rasch Analysis in Minerva Platform 37/37
