Reliant on Readers, Stuck with Scribes or Independent with ICT?

Can technology replace human

support in examinations?

9th European e-Accessibility Forum Cité des sciences et de l'industrie - Universcience – Paris

8 June 2015

CALL Scotland

• Communication, Access, Literacy and Learning

• Funded primarily by Scottish Government

• National Assistive and Communication Technology

• Service, research and development

www.CALLScotland.org.uk

Scottish School Assessments

• Formal examinations sat by most students in Scotland.

• Traditional paper format.

• 675,361 examinations sat by 132,588 candidates in 2014.

• Administered by the Scottish Qualifications Authority (SQA).

“Assessment arrangements allow candidates who are

disabled and/or who have been identified as having additional support needs appropriate arrangements to access the assessment without compromising its integrity.”

Introduction to Assessment Arrangements for Schools and Colleges SQA January 2010, revised July 2014

SQA Assessment Arrangements

Number of Candidates requesting AA 14,214

% of total candidates 10.7%

Number of requests for AA 43,344

% of total examination entries 6.42%

SQA Assessment Arrangements 2005

Total number of requests 41,454

Assessment Arrangement Number of requests which included the specific arrangement

Percentage of requests which included the specific arrangement

Extra Time 33,146 79.96%

Reader 15,740 37.97%

Scribe 14,505 34.99%

PA Referral 3,658 8.82%

Use of ICT 2,388 5.76%

Transcription with correction 1,178 2.84%

Coloured Paper 1,108 2.67%

Calculator 893 2.15%

Transcription without correction 742 1.79%

Enlarged Print 737 1.78%

Question Paper signed to candidate 92 0.22%

Candidate Signs Responses 38 0.09%

Use of tape recorder for responses 26 0.06%

Braille 19 0.05%

• Visually identical to hard copy papers

• Read questions with text-to-speech

• Use on-screen drawing tools

• Type and choose answers on screen

Digital Question Papers

• Zoom in and out • Papers (without

answer boxes) can be re-flowed

• Set page and text colours

• Navigate with keyboard/mouse

• Read with text-to-speech

Visual impairment / dyslexia

• Type into answer boxes

• Draw and annotate

• Spellchecker

• Word prediction

• On-screen keyboards

• Switch access

• Speech recognition

Writing and answering

“many of the students commented that the synthetic voice was of poor quality and was difficult to understand” “The most common suggestion for improvement to the PDF examinations was to improve the quality of the synthetic voice.”

Nisbet, P.D., Aitken, S., Shearer, N. (2004) Trial of External Papers in Accessible PDF for Candidates with Additional Support Needs.

http://www.adapteddigitalexams.org.uk/Downloads/Reports/

Text-to-speech voice quality

2008: ‘Heather’ licensed for Scottish schools

2011: ‘Stuart’ developed and licensed

> €2.7 million cost saving compared to schools buying high quality TTS Schools get the voices from: http://www.thescottishvoice.org.uk

..so we licensed high quality Scottish voices from CereProc in Edinburgh

www.CereProc.co.uk

Low cost -> wider access

All schools and students have access at zero cost to: • Adobe Reader • Text reader

software • High quality

voices • Past papers

from SQA’s web site

“Simplify, then add lightness” Colin Chapman, Lotus

Can technology replace human support?

0

5 000

10 000

15 000

20 000

25 000

2008 2009 2010 2011 2012 2013 2014

Reader

Scribe

ICT or DigitalPaper

Number of AA requests

Assessment arrangements 2014

Extra Time 34,494 Transcription with correction 863

Separate Accommodation 24,277 Transcription without correction 421

Reader 9,399 Calculator 341

Scribe 7,105 Modified Content (e.g. text description of images)

308

ICT / word processor

6,786

Referral of script to the Principal Assessor

192

Digital Question Papers 3,540 Adapted Certificate (e.g. in Large Print / Braille)

68

Rest Period 2,232 Question Paper signed to candidate

32

Coloured Paper 1,793 Taped transcription-Live Presentation

26

Prompter / Practical Helper 1,783 Braille 18

Enlarged or Adapted Print Question Papers

1,064 Candidate Signs Responses 11

Problems and limitations

1. “Integrating the question papers and answer booklets into one.”

2. “Pronunciation of some of the longer and geographical terms was poor.”

3. “takes up a lot of paper/ink when printing”

4. “Two machines crashed during Higher English exam.”

5. “give symbols for Science/Maths in order to do more straight on to the digital paper”

Accommodation and costs

• 9,399 reader requests; 7,105 scribe requests

• ~ 18,000 individual exams • ~ 27,000 hours • Say average €28/hour for

reader/scribe?

= €756,000 • Say €14/hour for invigilator?

= €378,000 TOTAL = €1,134,000 in 2014

(Although it was €2.26 m in 2013)

• “The pupils' experience of using the digital exams was a very positive one. Pupils adapted to the technology with ease and each found their own preferred way of using it - some pupils used the digital reader only for text that they struggled with in the paper copy whereas others did not look at the paper exam and worked exclusively on the computer. It was this flexibility of use that impressed me.”

• “I think the digital exam system is an excellent option for pupils with ASN and gives them more independence.”

• Increases candidates’ feelings of independence and control over their speed of progress. Helps with confidence to face next step, e.g. college, work.

Independence

Conclusions – PDF Digital papers are:

• Accessible for many candidates who require Assessment Arrangements, but they are not a universal solution for all candidates and all assessments.

• Low cost to produce and use.

• Reliable, familiar and secure.

• Accessible on different platforms.

• A pragmatic solution to the problem of providing digital versions of assessments.

• A good solution for paper-based assessment systems.

A viable alternative to human readers and scribes.