Evaluation of the Evaluation of the ‘Electronic Feedback’ ‘Electronic Feedback’ Marking Assistant and Marking Assistant and

Analysis of a Novel Analysis of a Novel Collusion Detection FacilityCollusion Detection Facility

Phil DentonPhil DentonFaculty Learning Development ManagerFaculty Learning Development Manager

Liverpool JMULiverpool JMU

Typical Class at Liverpool JMUTypical Class at Liverpool JMU(Scouse Stereotype #34)(Scouse Stereotype #34)

Computer Assisted Computer Assisted Assessment (CAA)Assessment (CAA)

““CAA is essentially automatedCAA is essentially automated

assessment through assessment through

objective testing.”objective testing.”

UK University WebsiteUK University Website

Types of CAATypes of CAA Objective TestingObjective Testing Data Retrieval ExercisesData Retrieval Exercises Electronic SubmissionElectronic Submission Free Text AnalysisFree Text Analysis Marking AssistantsMarking Assistants

- May be used to assist in the marking of May be used to assist in the marking of work that would normally be done work that would normally be done exclusively by handexclusively by hand

- Includes packages that can generate and Includes packages that can generate and email feedback, after appropriate data email feedback, after appropriate data have been inputtedhave been inputted

Marking in the UKMarking in the UK Vast majority is still done by hand Vast majority is still done by hand

CAA preferred for formative/diagnostic testing CAA preferred for formative/diagnostic testing

Summative ‘high stakes’ assessments do not normally benefit Summative ‘high stakes’ assessments do not normally benefit from electronic assistance from electronic assistance

There is an “... absence of any front-running software in this There is an “... absence of any front-running software in this regard, compared to the high profile of Question Mark.” (D. P. regard, compared to the high profile of Question Mark.” (D. P. Stephens 2001)Stephens 2001)

Most of the existing report-writing assistants are designed for the Most of the existing report-writing assistants are designed for the US High School marketUS High School market

Electronic Feedback: Designed for UK academics by a UK Electronic Feedback: Designed for UK academics by a UK academicacademic

Electronic FeedbackElectronic Feedback MS Excel/Word based Marking AssistantMS Excel/Word based Marking Assistant Functionalised with VBA programmingFunctionalised with VBA programming Consists of 3 files:Consists of 3 files:

- Feedback.xlsFeedback.xls- Guide.xlsGuide.xls- Fb.docFb.doc

Version 8 (2001/2): 22 UsersVersion 8 (2001/2): 22 Users Version 9 (2002/3): 44 UsersVersion 9 (2002/3): 44 Users Version 10 (Summer 2003): ??Version 10 (Summer 2003): ??

DemonstrationDemonstration

Electronic Feedback 10Electronic Feedback 10- Work in progressWork in progress

Electronic Feedback 9Electronic Feedback 9- Collusion detectionCollusion detection

- Feedback generationFeedback generation

Evaluation: Feedback FacilityEvaluation: Feedback Facility

Emailed questionnaire returned by Emailed questionnaire returned by 22 users of Version 822 users of Version 8

It was agreed that the software It was agreed that the software allows users to return:allows users to return:- more feedback (2.1)more feedback (2.1)- of higher quality (1.6)of higher quality (1.6)- in a shorter space of time (2.1)in a shorter space of time (2.1)

(Averaged Likert Scale responses)(Averaged Likert Scale responses)

Evaluation: Collusion DetectionEvaluation: Collusion Detection Tested over 11 JMU assessmentsTested over 11 JMU assessments IdenticalIdentical feedback detected in 5 feedback detected in 5

assessments, 19 instances in total.assessments, 19 instances in total.- 17 cases were coincidental17 cases were coincidental- 1 case was suspicious 1 case was suspicious - 1 case was plagiarism1 case was plagiarism

One ‘failed’ test repeated using One ‘failed’ test repeated using customisable detection criteriacustomisable detection criteria 5 comments in common, 5 comments in common, 1 different 1 different

- 9 cases found, 1 was plagiarism9 cases found, 1 was plagiarism

ConclusionsConclusions Electronic Feedback is ...Electronic Feedback is ...

- returned more rapidly,returned more rapidly,

- more extensive,more extensive,

- of higher quality.of higher quality.

V10 has these advantages and ...V10 has these advantages and ...- Facility for multiple markers,Facility for multiple markers,

- Normal & Criterion styles available Normal & Criterion styles available simultaneously,simultaneously,

- Reduced Memory problems.Reduced Memory problems.

The Future?The Future?

Development of a wholly Development of a wholly web-based version of web-based version of Electronic Feedback, Electronic Feedback,

integratable with existing integratable with existing Institutional databasesInstitutional databases

The EndThe End

To arrange a training session at To arrange a training session at your institution, or receive a your institution, or receive a

copy of V10, when ready, please copy of V10, when ready, please email the authoremail the author

p.denton@livjm.ac.ukp.denton@livjm.ac.uk