Evaluating electronic textbooks: a methodology

Ruth Wilson and Monica Landoni

ECDL 2001

5th September 2001

Outline

• The Visual Book

• The WEB Book

• EBONI’s methodology for evaluating electronic textbooks

The Visual Book

• Studied the application of the paper book metaphor to the design and production of electronic books

• Found that users were able to rely on their experience with paper books to interact with the electronic book.

The WEB Book

• Focused on the impact of appearance on the usability of textbooks on the Web.

• Comparison of two versions of a textbook chapter:– Original version: very plain, one “page” per

chapter, basic use of hyperlinks.– Revised, “scannable” version: chapter divided

between 10 “pages”, heavier use of hyperlinks, extra headings, indented lists, coloured keywords.

• Revised version was 92% more usable.

EBONI: aims

EBONI aims to:• Evaluate the different approaches to the design of

electronic textbooks

• Identify and report on the individual requirements of students and academics in learning and teaching using electronic textbooks

• Compile and promote a set of best practice guidelines for the publication of electronic textbooks

Experiments:psychology Web books

• Peter Kaiser. The Joy of Visual Perception. http://www.yorku.ca/eye/thejoy.htm

• Eric Chudler. Neuroscience for Kids. http://faculty.washington.edu/chudler/neurok.html

• C. Robin Timmons and Leonard W. Hamilton. Drugs, Brains and Behaviour. http://www.rci.rutgers.edu/~lwh/drugs/

Experiments:Hypertext in Context

• Cliff McKnight, Andrew Dillon and John Richardson. Hypertext in Context.

–Print

–Original electronic version

–Revised electronic version

Experiments:commercial formats

• Rosa Ainley (ed.). New Frontiers of Space…

–MobiPocket Reader–Adobe Acrobat eBook Reader–Microsoft Reader–Paperback

Experiments:e-encyclopaedias

• Britannica

• Columbia

• Encarta

Experiments:portable ebooks

• HP Jornada

• Franklin eBookMan

• PalmPilot

• Rocket eBook

• Softbook

Ebook evaluation model

• Selection of material• Selection of participants• Selection of tasks

–Scavenger hunt, memory tasks, high cognitive skill tasks

• Selection of evaluation techniques–Questionnaires, observation, think-aloud,

interviews

Selection of material (1)

Electronic books offer a diverse array of material for evaluation:

• Hardware devices, e.g. REBs, eBookMan, goReader

• Ebook reader software, e.g. Microsoft Reader, Adobe Acrobat Ebook Reader

• “Web books”

Selection of material (2)

• EBONI aims to identify design techniques which maximise information intake by users, in order to provide guidance to creators of digitised educational material.

• Texts can be compared according to three parameters:– Format

– Content

– Medium

Selection of actors

Four main roles can be distinguished:

• Participant

• Evaluator

• Task developer

• Task assessor

Selection of tasks

• Tasks are a way of bringing together participants and the test material in a structured manner. Three types are suggested:– “Scavenger Hunts”, or retrieval tasks

–Memory tasks

–High cognitive skill tasks

Selection of evaluation techniques

Qualitative feedback can be obtained via:

• Subjective satisfaction questionnaires

• Behaviour observation

• Think-alouds

• Interviews

Technique complexity

Task complexity

Interviews

Think-aloud

Covert observation

Questionnaires

Scavenger hunt

Memory tasks

High cognitive skill tasks

WEB Book

Study X

Guidelines

• Sophisticated experiments can be “mapped” to simple experiments.

• Results of all evaluations will be comparable at some level.

• Results will feed into a set of best practice guidelines for producing electronic textbooks.

Contact details

• Ruth Wilson: ruth.m.wilson@strath.ac.uk

• EBONI Web site: http://eboni.cdlr.strath.ac.uk/

• To join the EBONI mailing list, follow the instructions at: http://www.jiscmail.ac.uk/lists/open-eboni.html