OpenEssayist: A supply and

demand learning analytics tool

for drafting academic essays

Institute of Educational

TechnologyDenise Whitelock, Alison Twiner, John

Richardson, Debora Field, Stephen Pulman

Providing meaningful automatic feedback

• Can we provide Advice

for Action to assist

essay writing?

• How can we capture

progress with a

visualisation?

SAFeSEA: Supportive Automated Feedback for

Short Essay Answers

An automated tool

supporting

online writing and

assessment

of essays providing

accurate targeted

feedback

SAFeSEA

Professor Denise Whitelock

Professor John Richardson

Professor Stephen Pulman

About SAFeSEA

• No tutor support for drafts of first

assignment

• Reduce dropout rate with automatic

feedback?

• Effect of summaristion

• What are the beneficial factors?

• Correlate measures of learner activity

and essay improvement

• http://www.open.ac.uk/iet/main/resea

rch-innovation/research-

projects/supportive-automated-

feedback-short-essay-answers

Talk Back

• Checking

understanding by ‘talk

back’

• Summaries in

OpenEssayist

• Key words = key

ideas

• http://www.open.ac.uk

/researchprojects/safe

sea

Key word extraction What we mean by 'key word'

• Not author-assigned key words

• Words that "describe the contents of a document" (Collins ED)

Process: Before graph making comes NLP pre-processing(Python-base NLP toolkit)

• Filter out unwanted parts of speech

o cardinals, modals, adverbs, symbols,...

• Filter out stop words (meaning-poor words)

o Large corpus analysis shows function words the most frequent

o Stop words prone to becoming key words owing to strong relationship between word

frequency and word 'key-ness'

Essay

V1 top

10 key

words

Key word graphs

All remaining meaning-rich words of the essay become

nodes in a graph (network)

Adjacent words represented by links (edges) between

nodes

An algorithm traverses the graph to derive the key

words

• A key word is one that co-occurs (within a window of

N words) with lots of words that co-occur with lots of

words that co-occur...

No external trained model or reference source involved

Different from word frequency and from collocation

count

• Algorithm captures a word's connectedness to the

entire text

Key sentence extraction Roughly similar process to key words: Same 33 x 1500-word essays

Some NLP text pre-processing

Each sentence becomes a node in a graph

Every sentence is compared to every other sentence to derive a

similarity score for each pair Currently using Cosine Similarity (vectors based on co-occurrence)

Links (edges) connect pairs of nodes that have a similarity score > 0 Similarity score = Strength of connection (edge weight)

TextRank algorithm (Mihalcea & Tarau 2004) calculates 'global

importance' score for each sentence Uses graph structure (what links to what) and edge weights

Like PageRank but with edge weights in the mix

Result is list of all essay's sentences returned in order of global

importance

ESSAY

GRAPHICS

ANALYSIS

OpenEssayist: What it tells you

The system’s focus is to present summaries of your own work in different ways, to encourage you to reflect constructively on what you have written.

In other words Open Essayist tells you from its analysis what are the most important or key points in your essay. You can then think about whether that was what you intended to emphasis in your essay. If not then you can make the appropriate changes.

A very important aspect of the OpenEssayist system is that it will not tell you what to write, or how to rewrite sections of your essay, or even what is correct or incorrect in your essay.

Sample key phrases dispersion plot

Most Used Features

Total access

Key words 115

Keyword cloud 103

Key sentences 99

Keyword dispersion plot 96

Grades and use of OpenEssayist with H817

• Used by MAODE students

• Positive correlations

1. Grades for Essay 1 and number of drafts (r=+0.41)

2. Number of site visits and number of drafts (r=+0.65)

3. Number of visits and grade for Essay 2 was significant one

tailed test (r=+0.5)

4. Mean grade for overall module for students in cohort who used

OpenEssayist (64.2) and students in previous cohort (53.7)

(p=0.4)

Student comments

The summary is an extension of

the key words and phrases and

will make me think about

whether this is really what I

wanted to say in the essay.

I can see the benefit because it is

talking

about the structure. It will help you

understand where you need to work

in, the different sections, what you

are missing maybe you need to fill

in a bit more or not.

New feature Rainbow Diagrams

Pretend essay: 10 identical paragraphs

Pretend essay: 50 identical sentences

Stanford University Boothe Prize essay

OU essay awarded high grade

OU essay awarded low grade

Good vs. bad?

Good:

densely connected

Red nodes (conclusion) central

Close links between violet (intro) and red notes

Bad:

Not densely connected

Red nodes (conclusion) not central

Few links between violet (intro) and red nodes

Rainbow diagrams related to mark awarded

Multivariate analysis of variance on marks awarded to 45

students

Submitted two essays

Rainbow diagrams produced from these essays and rated as

high, medium or low attainment

Covariate showed a significant relationship with the marks

F(1, 43) = 5.92, p = .01 using a directional test

Essays rated as high would be expected to receive 8.56

percentage points more than essays rated as medium

17.2 percentage points higher than essays rated from rainbow

diagrams as low

How about feedback first?

Hints before writing?

R.C.T.

2 essays

F(1,41) = 3.23 p = 0.04 for

hints

http://www.eden-

online.org/publications/best

-research-papers.html

Creating teaching and learning dialogues:

towards guided learning supported by

technology

Learning to judge

Providing reassurance

Providing a variety of

signposted routes to

achieve learning goals

ReferencesWhitelock, D., Twiner, A., Richardson, J.T.E., Field, D. & Pulman, S. (2015). OpenEssayist: A supply and demand

learning analytics tool for drafting academic essays. The 5th International Learning Analytics and Knowledge (LAK)

Conference, Poughkeepsie, New York, USA. 16-20 March 2015. ISBN 978-1-4503-3417-4

Whitelock, D., Twiner, A., Richardson, J.T.E., Field, D. & Pulman, S. (2014). OpenEssayist: Real-life testing of an

automated feedback system for draft essay writing. Paper presented at the #design4learning conference, 27 November

2014 at The Open University http://design4learning.org.uk/

Alden, B., Van Labeke, N., Field, D., Pulman, S., Richardson, J.T.E. & Whitelock, D. (2014) Using student

experience as a model for designing an automatic feedback system for short essays. International Journal of e-

Assessment, 4(1), article no. 68

Field, D, Richardson, J.T.E., Pulman, S., Van Labeke, N. & Whitelock, D. (2014) An exploration of the features of

graded student essays using domain- independent natural language techniques. International Journal of e-Assessment,

4(1), article no. 69

Whitelock, D., Twiner, A.; Richardson, J.; Field, D.; Pulman, S. (2014). Feedback on Academic Essay Writing Through

Pre-Emptive Hints – Moving Towards ‘Advice for Action’ . Challenges for Research into Open & Distance Learning:

Doing Things Better – Doing Better Things Proceedings of the European Distance and E-Learning Network 2014

Research Workshop Oxford, 27-28 October, 2014. ISBN 978-615-5511-00-4

in service of The Open University