Usability Evaluation of the Course Management Features of Sakai

Jonathan Howarth {jhowarth@vt.edu}Rex Hartson {hartson@vt.edu}

Aaron Zeckoski {aaronz@vt.edu}

Virginia Tech

Sakai Conference, Atlanta, 2006

2 | | | | | | |

Overview Introduction Usability Setup Positive Feedback Performance and Satisfaction Usability Problems Next Steps

3 | | | | | | |

Introduction - Scholar is Virginia Tech’s implementation

of Sakai Using Scholar in a series of dissertation studies Usability evaluation in exchange for input from

Scholar developers Summary report

{http://issues.sakaiproject.org/confluence/download/attachments/3115

0/ScholarUsabEvalReportSummary.pdf?version=1} Full report

{http://issues.sakaiproject.org/confluence/download/attachments/3115

0/ScholarUsabEvalReportFull.pdf?version=1}

4 | | | | | | |

Usability Examples (1) Examples from the real world

5 | | | | | | |

Usability Examples (2) Examples from the digital world

6 | | | | | | |

Usability Examples (3) Some additional contrived examples

World Usability Day posters provided by the Usability Professional’s Association {http://www.worldusabilityday.org/tools/world-usability-day-posters}

7 | | | | | | |

Usability Examples (4) Usability in the news

8 | | | | | | |

Usability Definition (1) Usability is a characteristic of an interactive

system that indicates ease of use, usefulness

Usability includes Effectiveness Efficiency Satisfaction

9 | | | | | | |

Usability Definition (2) It is NOT "dummy proofing” “Doing usability” is NOT (just) usability testing Usability is NOT “user friendliness” Misconceptions:

Peanut butter theory Priest in a parachute theory

10 | | | | | | |

Usability Motivation Why should you care

For the user, the UI is the system Usability can make or break your product Reduces training and help-desk costs Increases productivity Stand out among competition

An example Shutterfly increased print orders by 200% by

redesigning their website {http://

www.jaredresearch.com/why_it_worked.html}

11 | | | | | | |

Setup (1) Comparison with other Sakai usability evaluations

{http://bugs.sakaiproject.org/confluence/display/UI/Usage+Reports} Scholar version 2.2.x Course management features of Scholar Goals

Satisfaction Learnability Performance

12 | | | | | | |

Setup (2) Walkthrough Usability testing

5 participants with different backgrounds Sessions lasted 2 hours 17 tasks 2 questionnaires Gratuity

13 | | | | | | |

Positive Feedback Variety of tools Help system Mechanisms for supporting collaboration

14 | | | | | | |

Performance and Satisfaction (1) Performance

Time on task Long amounts of time required for initial tasks

Example – 10 minutes average to add lecture and lab events to the schedule

Much variance among participants Improvement for repeated tasks

Assists As a group, the participants were only able to complete 2 initial

tasks Task of adding an assignment resulted in 14 assists

15 | | | | | | |

Performance and Satisfaction (2) Satisfaction

Scores on questionnaire Average around a 5 (scale of 1 to 10, 1 = poor, 10 = excellent) No change in scores between 1st and 2nd questionnaire

Summary Need to decrease time on task Results suggest that participants were able to learn

quickly Performance/perceived performance did not have a

substantial effect on satisfaction

16 | | | | | | |

Usability Problems (1)

Group P1 P2 P3 P4 P5 Total

Terminology 9 11 15 4 9 48

Consistency 6 9 10 5 7 37

Sections and groups 2 10 6 5 4 27

Conformance with standards 2 8 5 1 5 21

Assignments and gradebook assignments 5 1 6 3 4 19

Workspaces and worksites 2 4 4 5 1 16

Distribution of functionality 2 3 4 5 1 15

Miscellaneous 11 12 12 9 8 52

Usability problems – verbal feedback, hesitations, deviation from correct sequence of actions

17 | | | | | | |

Usability Problems (2) Terminology

Roster {video – explaining what a roster is}, Resources {video – what are resources?}

18 | | | | | | |

Usability Problems (3) Consistency

Lists, confirmation, feedback, . . .

19 | | | | | | |

Usability Problems (4) Sections and Groups

Sections are tools (appear in the menu bar) and are exclusive

Groups are not tools (appear in site info) and are not exclusive

Conformance with standards Partial saving of page state {video – trying to get to the

main Site Info page} Functioning of the browser’s back button

20 | | | | | | |

Usability Problems (5) Assignments and gradebook assignments {video

– working in both tools to correct a point value}

21 | | | | | | |

Usability Problems (6) Workspaces and Worksites {video – trying to

explain the difference}

22 | | | | | | |

Usability Problems (7) Distribution of functionality

Adding and removing students {video – separation of add and remove causes problems on repeated task}

Editing assignments {video – ability to access assignment from multiple locations}, etc

Example – RSmart.com

23 | | | | | | |

Usability Problems (8) Miscellaneous problems

Reoccurring schedule entries, system-centered permissions, clean Word text {video – difficulties pasting text}

24 | | | | | | |

Next Steps Usability evaluation must be iterative

It’s not possible to get it right the first time (in any complex design domain)

Need:

Ready, Fire, Aim!

25 | | | | | | |

Questions

?