Dr Peter Vitartas, Prof Damminda Alahakoon, Dr Sarah Midford, Ms Nilupulee Nathawitharana, Dr Kok-Leong Ong, Prof Gillian Sullivan-Mort

Toward an automated student feedback system for text-based assignments     

2

Agenda

• Background

• Automated writing evaluation tools

• Next generation rubrics project

• Results

• Discussion

3

Background

4

Automated writing evaluation

e-rater®

Revision Assistant

5

Types of error comments provided by Criterion ®

6

Automated writing evaluation

e-rater®

Revision Assistant

7

Pearson’s Intelligent Essay Assessor

8

IEA’s feature list used in Write-to-learn

9

Automated writing evaluation

e-rater®

Revision Assistant

10

Vantage Learning

• My access provides:

• writing aids,

• word processing capabilities and

• teacher analytics

• IntelliMetric analyses “400 semantic-, syntactic-, and discourse-level features to form a composite sense of meaning”

11

Automated writing evaluation

e-rater®

Revision Assistant

12

LightSide and Revision Assistant

• LightSide is used as the turnitin scoring engine

• Revision Assistant – gives ‘signal checks’ as formative feedback– Needs to be trained

13

Automated writing evaluation

e-rater®

Revision Assistant

WriteLab

14

• Provides suggestions about:– Clarity– Logic– Cohesion– Grammar

15

Next Generation Rubrics

Push towards larger subject cohorts:– 30% of La Trobe subjects = 100+ students– 41 subjects = 500+ students– BA Interdisciplinary Cores: ‘Rethinking Our Humanity’ (HUS1FAS) and ‘Ideas that Shook the World’ (HUS1TEN) = 1000+ students

Consequences:– Increased pastoral care, assessment feedback and marking– Less time to interact with individual students– Slower turn around on assessment feedback– Difficult to monitor gaps in knowledge at a cohort level– Difficult to assess critical thinking and written skills frequently and meaningfully– Lower student satisfaction

Good pedagogical practice demands quick turn-around times on student assessment feedback, but staff resources are limited and ever-increasing student diversity requires more pastoral care

16

Our solution

Online feedback tool for text based assessments

Students get: immediate feedback on their written work designed to highlight areas that could be improved before submission for marking

Staff get: a dashboard summarising student performance, knowledge gaps and patterns at a cohort level

Benefits—

• Students receive more guidance on their written assessments than staff could otherwise provide

• Student feel less self conscious about the quality of their work because online submission is anonymous

• Staff can look at submission data to see where student skills and knowledge gaps lie and target their teaching more effectively to student needs

• Staff can use data to better design their assessments

17

Student interface

18

Student interface

19

Staff interface

20

Staff interface

21

A framework for evaluating assignments

Assignment statistics

Readability

Research

Critical thinking

Discipline theory

22

Pilot study subjects

Undergraduate Subject:

HUS1TEN – Ideas that Shook the World

1st year BA Interdisciplinary Core Subject

Assessment: Essay

Postgraduate Subject:

MKT5MMA – Marketing Management

MA level Business Subject

Assessment: Report

23

Assignment statistics

Calculates average and individual:– word count– paragraph count– page count– spelling and grammar error count

Assignment statistics

24

Readability

• Calculates Flesch reading ease and Flesch-Kincaid grade level to measure the readability

• Investigates the average readability of the assignments across marks categories

Marks Category Average Flesch Reading Ease

0-49 44.07000046

50-59 41.39999989

60-69 38.35882308

70-79 33.25000004

80-89 29.19999997

90-100 19.5

HUS1TEN Readability statistics

Readability

25

Research

• Counts correctly formatted in-text references (citations) written following La Trobe Harvard style

• Counts all in-text references (citations) disregarding format• Counts number of distinct authors in in-text references (citations)

• Counts number of references in reference list

Research

26

Critical thinking

• Extracts frequencies for both stemmed/non stemmed words using Porter’s stemming algorithm (Porter, 1980)

• Captures phrases with frequencies using TerMine web tool (Frantzi, Ananiadou, & Mima, 2000)

• Identifies Critical Thinking words and phrases and calculates the occurrence of relevant words and phrases across marks categories

Critical thinking

27

Discipline theory

• Extracts frequencies for both stemmed/non stemmed words using Porter’s stemming algorithm (Porter, 1980)

• Captures phrases with frequencies using TerMine web tool (Frantzi, Ananiadou, & Mima, 2000)

Discipline theory

28

Most frequent stemmed words – HUS1TEN

Word Frequency

relationship 1788

love 1558

peopl 648

idea 638

societi 590

half 564

marriag 513

find 507

individu 503

sexual 409

Essay Question:The idea that in love you are finding “your other half” has a long history. But is the idea that we are all one half of a whole still relevant in the current climate of changing gender norms, open relationships and soaring divorce rates? Please discuss the extent to which we are simply one half of a whole with reference to one or more of the following:• Gender ideals• Polyamory• Platonic love• Individualization• The ‘pure relationship’ (Giddens)

29

Most frequent phrases

Phrase Frequency

pure relationship 197

romantic love 125

soul mate 76

many people 46

confluent love 39

platonic love 35

monogamous relationship 22

current climate 21

gay marriage 21

polyamorous relationship 20

Essay Question:The idea that in love you are finding “your other half” has a long history. But is the idea that we are all one half of a whole still relevant in the current climate of changing gender norms, open relationships and soaring divorce rates? Please discuss the extent to which we are simply one half of a whole with reference to one or more of the following:• Gender ideals• Polyamory• Platonic love• Individualization• The ‘pure relationship’ (Giddens)

30

Human marked Grade 0-19(F/<50%)

20-23(D/50-59 %)

24-27(C/60-69%)

28-31(B/70-79%)

32-40(A/80+%)

All Assignments

# of Assignments 3 12 18 20 7 60Critical Thinking Average # Critical Thinking terms

16.7 17.5 15.7 17.7 18.7 16.3

Top 3 critical thinking terms (by frequency)

AnalyseWorldFact

WorldAnalyseData

AnalyseProblemWorld

AnalyseDataWorld

DataAnalyseProblem

AnalyseDataWorld

Citations Av Citations 9.7 17.7 12 13.2 30.7 15.2Av Distinct Authors 5.3 8.25 7.2 6.8 13.6 7.9Word Statistics Av Word Count 2151 2603 2697 2758 3284 2739Av Paragraph Count 45 66 66 68 110 70Av Spelling Error 8.3 26.8 27.5 26.5 35 26.9Av Grammar Error 1.7 6.8 10.0 7.7 4.9 7.6Readability Flesch Reading Ease 35.3 39.3 39.3 38.7 33.4 37.8Flesch-Kincaid Grade Level 14.5 12.8 12.7 12.9 14.1 13.0

31

Initial findings

• Greater understanding of student performance in a large class– Distribution of mistakes– Terms and concepts not being considered– Use of references– Extent of critical thinking terms being used in assignments

• Greater understanding of tutor performance– Variability in marking weightings– Distribution of grades– Consistency in marking assignments

32

Continuing work

• Refinement of dimensions for the assessment classifier– Testing of alternative word extraction methods (LDA, semantic analysis and NLP)– Development of ontologies for discipline areas

• Improved staff user interface being built– User-friendly settings for assignment parameters– Easy to use and relevant reporting outputs

• Improved student interface being built– User-friendly assignment submission– Assignment feedback that is easy to read, understand and put into action

• Further testing of assignments

33

Thank you!

Dr Peter Vitartasp.vitartas@latrobe.edu.au

Acknowledgement: La Trobe Learning and Teaching Digital Learning Strategy Innovative Research GrantSpecial Thanks: James Heath, Aleksandra Michalewicz and Tanvir Ahmed