Upload
others
View
0
Download
0
Embed Size (px)
Citation preview
Paper Architecture Paper Architecture and and Exam Scoring ApplicationExam Scoring Application
Narcis LozanoNarcis LozanoHidetoHideto OdaOda
Masaki NakagawaMasaki NakagawaTokyo University of Agriculture and Tokyo University of Agriculture and
TechnologyTechnology
Table of contentsTable of contents
Pen interfacesPen interfacesPaper architecturePaper architectureExam scoring applicationExam scoring applicationConclusion Conclusion Future workFuture work
Pen InterfacesPen Interfaces
Devices:Devices:–– Graphic tabletsGraphic tablets–– MonitorsMonitors–– Tablet PCsTablet PCs–– PDAPDA
Characteristics:Characteristics:–– Constantly connected to Constantly connected to
the computerthe computer–– Instant visual feedbackInstant visual feedback–– High interactivityHigh interactivity
Pen InterfacesPen Interfaces
CharacteristicsCharacteristics–– Use paperUse paper–– Online data stored in memoryOnline data stored in memory–– Delayed data processDelayed data process–– Low interactivityLow interactivity
DevicesDevices–– UltrasoundsUltrasounds–– InfraredInfrared–– Patterned PaperPatterned Paper
Need for an unified architecture for Need for an unified architecture for pen and paper devicespen and paper devices
Several brands and technologies.Several brands and technologies.No standard neither de facto standard for pen No standard neither de facto standard for pen devices that write on paper.devices that write on paper.Several versions must be developed for each Several versions must be developed for each pen device that must be supported by the pen device that must be supported by the system.system.
Application
Application’
Application’’
Paper ArchitecturePaper ArchitectureUnified software architecture.Unified software architecture.Following Following purepure lazy recognition (do not show lazy recognition (do not show recognition until needed). recognition until needed). Allows using several types of pen devices with Allows using several types of pen devices with the same application.the same application.
ApplicationInkML PaperInterface
XML
Template
Paper Interface InputPaper Interface Input
Server
Client
Client
Client
InkMLFile
<Ink><Trace>
<Point X = 1 Y = 1><Point X = 31 Y = 11>
</Trace><Trace>
<Point X = 14 Y = 61>…
The users fulfils the printed document using the device of their choice. When they connect the device to the computer, the data, which is in a proprietary format, is converted into InkML using a client application. This InkML file is then sent to the server.
TemplatesTemplatesInput by writing in a Input by writing in a printed documentprinted documentAreas:Areas:–– QuestionsQuestions–– CheckboxesCheckboxes–– Input with recognitionInput with recognition–– Raw inputRaw input–– Virtual keyboardVirtual keyboard
Contextual information Contextual information neededneeded–– Template filesTemplate files
Template creationTemplate creation
ORIGINAL DOCUMENT EDITOR
TEMPLATEFILE
SERVER
IdentifierSegmentation
SharedIdentifierTemplate
Identifier230
Segmentationand
data process
Data process in ServerData process in Server
Template230
Template Folder
XMLResult
File
<XML><seqField>
<content>Student Id</content><content>05646211</content>
</seqField><question>
<usrAnswer>1196</usrAnswer><result>CORRECT</ result >
</question1>。。。
InkMLOnlineData
OutputOutput
SERVER
XMLResult
File
<XML><seqField>
<content>Student id</content><content>05646211</content>
</seqField><question>
<usrAnswer>1196</usrAnswer><result>CORRECT</ result >
</question1>。。。
XML Viewer
Web Page
Application
Pen Devices in EducationPen Devices in EducationDevices:Devices:–– Electronic whiteboardElectronic whiteboard–– Tablet PCsTablet PCs–– PDAsPDAs
Applications:Applications:–– Use on lecturesUse on lectures–– Video lecturesVideo lectures–– Collaborative learningCollaborative learning–– Learning by gameLearning by game--like competitionlike competition–– Etc.Etc.
Pen & Paper Interface ApplicationPen & Paper Interface Application
Systems designSystems design–– Students fulfill exams using a pen & paper Students fulfill exams using a pen & paper
devicedevice–– Teachers score the answers using the exam Teachers score the answers using the exam
scoring applicationscoring application–– Results and statistics are calculated and sent to Results and statistics are calculated and sent to
the studentsthe studentsExam scoring applicationExam scoring application–– Not fully automatic scoring but hybrid approach Not fully automatic scoring but hybrid approach
Problems withProblems withautomatic evaluationautomatic evaluation
Segmentation and Segmentation and recognition errorsrecognition errorsQuestions with Questions with several possible several possible answers:answers:–– Free text questionsFree text questions–– Even short questions,Even short questions,
There are synonyms.There are synonyms.
Hybrid evaluationHybrid evaluation
Boolean questions and questions with one Boolean questions and questions with one answer:answer:–– Automatic scoring.Automatic scoring.–– Handwriting recognition when necessary.Handwriting recognition when necessary.Rest of the questions:Rest of the questions:–– TeacherTeacher’’s scoring assisted by computers.s scoring assisted by computers.–– Handwriting recognition is not performed.Handwriting recognition is not performed.
Application FlowApplication Flow
PaperArchitecture
ServerScoringResult
Computer AssistedScoring Application
Xml ResultFiles
Computer Assisted Exam Scoring InterfaceComputer Assisted Exam Scoring Interface
Video of the system runningVideo of the system running
Answer reorganizationAnswer reorganization
Question 1Question 2Question 3Question 4
Question 1Question 1Question 1Question 1
Question 2 Question 3 Question 4Question 2Question 2Question 2
Question 3Question 3Question 3
Question 4Question 4Question 4
Video of marking by student or by answerVideo of marking by student or by answer
Video of correctionVideo of correction
Preliminary EvaluationPreliminary Evaluation
Exam scoring of 15 exam answer sheets.Exam scoring of 15 exam answer sheets.Exam consisting of vocabulary and fillExam consisting of vocabulary and fill--inin--thethe--gaps questions.gaps questions.Processed by the Paper InterfaceProcessed by the Paper InterfaceWe asked 7 people to score the answer We asked 7 people to score the answer sheets using the scoring application.sheets using the scoring application.
AdvantagesAdvantages
Answer reorganization Answer reorganization seems efficient for seems efficient for teachersteachers
Relatively low costRelatively low costGeneration of detailed Generation of detailed statisticsstatisticsPaper versions of the Paper versions of the original exam available.original exam available.
Table 1. Experimental result.
40.001220No. 7
52.171123No. 6
41.671424No. 5
50.001224No. 4
45.451222No. 3
18.18911No. 2
45.831324No. 1
Time reduction ratio (%)
Comp. assistedmarking time (min.)
Manual marking time (min.)
Partici-pant
Preliminary EvaluationPreliminary Evaluation
ProblemsProblems–– Missing strokesMissing strokes–– Incorrect segmentation of the answers caused Incorrect segmentation of the answers caused
by displacement of the document.by displacement of the document.
ConclusionConclusion
Paper architecturePaper architecture–– Unified platform for pen and paper devicesUnified platform for pen and paper devicesExam scoring applicationExam scoring application–– Automatic scoring for selection type questions Automatic scoring for selection type questions –– TeacherTeacher’’s scoring for free text questions.s scoring for free text questions.–– Question reorganizationQuestion reorganization–– OneOne--click question evaluation, statistics click question evaluation, statistics
export and automatic eexport and automatic e--mailing of exam mailing of exam results to the students were also implemented. results to the students were also implemented.
Future workFuture work
Remaining issues:Remaining issues:–– Students and teachers benefits. Added effect Students and teachers benefits. Added effect
v.sv.s. mental burden, human error rates, etc.. mental burden, human error rates, etc.–– Unexploited information from digital ink. Unexploited information from digital ink.
StudentsStudents’’ answering speed or fluency of answering speed or fluency of writing might signal something about the writing might signal something about the student confidence.student confidence.
Improvement of the segmentation.Improvement of the segmentation.Evaluation in real situations for a larger Evaluation in real situations for a larger number of students.number of students.
Paper Architecture Paper Architecture and and Exam Scoring ApplicationExam Scoring Application
Narcis LozanoNarcis LozanoHidetoHideto OdaOda
Masaki NakagawaMasaki NakagawaTokyo University of Agriculture and Tokyo University of Agriculture and
TechnologyTechnology