Online Marking of Essays Trends and Developments

Patris van BoxelVrije Universiteit Amsterdam

30 November 2012

2

Content

• Role of (written) feedback in learning • Challenges in marking of essays• Trends and developments– Growth in marking and feedback tools – Feedback types (annotations, rubrics, voice)– Learning analytics

• Pilots and evaluation in Dutch SCALA project• Conclusions

3

The role of feedback

• Powerful impact on study behaviour & performance

• Characteristics of good feedback– Timely, personalised, information-rich– ‘Feedforward’ function: bridging the gap

between current and desired (end) result(The Power of Feedback, Hattie &

Timperley, 2007)

4

Giving feedback on essays

Typical teacher challenges:• Logistical workflow– Collection, printing, distribution, storage

• Content– May involve a lot of repetitive commentary– Inter-marker inconsistency

• Contextual demands– Large cohorts– Often need for fast turnaround

5

Receiving feedback on essays

• Is feedback– legible?– understandable?– timely? – actionable?– accessible?

Illegible demand for legibility OUR item expressing suspicion that the UK Royal Mail is deliberately smudging the printing on its "write the postcode clearly" messages reminded Howard Greenwood that he once received a piece of homework marked by his physics teacher bearing an unreadable scrawl in the usual red ink.

Howard took his homework book up to the teacher and asked him what his comment said. The teacher looked at the page and exploded, sending Howard back to his desk with grave threats of what would happen if he were to be so impertinent in future. Howard was completely bewildered until, after much deciphering, he and his friends worked out that the scrawl said, "You must write more legibly".

SOURCE: NEW SCIENTIST, March 2012

6

Tools for essay marking

• Generic software solutions• Dedicated marking tools– Annotation tools (e.g. Markin)– Automated feedback (e.g. ETS)– Electronic assignment management (e.g.

GradeMark/Turnitin)

• Integration with learning management systems(e.g. Blackboard-GradeMark or Moodle-Lightwork)

7

Marking with GradeMark in Bb at VU

• Turnitin software suite

• Advantages of integration with LMS• Lowers barrier for

teachers to use• Less system admin

overhead• Integrates with pre-

and post-marking workflows

• Shared marking

GradeMark in Blackboard

9

Feedback types

• Annotations• Rubrics• Voice grading

10

Annotations

• Contextual• Can be stored • Reused• Categorised• Shared amongst

teachers• Basis for learning

analytics

11

Rubrics

• Matrix with multiple assessment criteria• Described in terms of achievement levels• May also be used for grading• Advantages– Provides insight in current and required

performance– Offers growth perspective– Higher-level feedback over lifespan of course and /

or curriculum

Rubrics• Growing number of tools for (digital) rubric development (e.g.

Waypoint Outcomes) • Basic rubric functions emerging in LMS environments• Self-made solutions e.g. in Google docs

13

Voice grading

• Personalised feel• Localised – inline - feedback• Emphasis, clarification, repetition

Examples• Voice comments in GradeMark• Audio feedback via iAnnotate (demo)

14

Grading analytics

• Marking tools’ ability to provide data – Feedback type and

quantity– Individual / group

level• Diagnostic evidence

of recurring problems• More information:

EBEAM project (JISC, UK)

C. Ellis (2011), Univ of Huddersfield

15

‘Green grading’Imagine an instructor has 15 students each turning in two-page journalentries printed single-sided. Let’s say the instructor will spend about eight minutes grading each journal entry with corrections and comments regardless if it is digital or on paper.

→Traditional grading: It would take 1,800 watt-hours (1.8 kWh) of energy to produce 30 sheets of paper to print the assignments.*

→Paperless grading: It would take 316 watt-hours (0.316 kWh) to run an average computer continuously for 2 hours to grade electronically.**

That’s a savings of 1,484 watt-hours (1.484 kWh) by grading on acomputer as opposed to using paper, the equivalent of saving 1.4 lbs (.635 kg) of coal and reducing emissions by 3.9 lbs (1.769 kg) of carbon dioxide.

SCALA Investigating the potential of digital marking • ‘Scaffolding Assessment for Learning’

(2012-2014)• National funding (SURF) to use (digital) methods for

efficient provision of written feedback • Project aims– Reduction in feedback workload

• Storage and re-use of feedback • Beter assignment management procedures

– Improvement in feedback quality and timeliness

17

SCALA project plan

• 30 pilots across four institutions• Variation of marking tools & feedback types• Variation in disciplines, study years, cohort sizes• Deliverables– Good practices– Teacher guidelines on digital marking– Generic feedback components e.g. rubrics for

assessment of writing skills

18

Conclusions

Marking tools • make anytime, anywhere marking increasing

possible• offer potential for efficiency and quality gains• stimulate more consistent approaches

towards marking and use of assessment criteria

• are able to provide timely, rich, and personal feedback

19p.van.boxel@vu.nl