Usability Testing Report (ENGLISH Only)

8/14/2019 Usability Testing Report (ENGLISH Only)

http://slidepdf.com/reader/full/usability-testing-report-english-only 1/7

Sree Anirudh J BhandaramPortfolio of a Usability, User Experience & Web development Professional

UsabilityWeb DevOthersPhotography

You are here: Home » Usability » Usability Testing

Usability Testing

Prepared for:Staci Autovino Kemp

Prepared by:Kate Cushman WaldersSanket Anil NaikSharmishsta DhuriSree Anirudh J Bhandaram

May 15, 2009

1 Executive SummaryDataMentor is the friendly front door to assessment data and instruction with solutions for theclassroom level teacher. It allows teachers, administrators, curriculum specialists, and other professionals interested in improvement through the use of data to analyze district assessmentscoring results and select from an array of valuable options to target student growth andimprovement based on individual or district needs. It allows the user to analyze district assessmentscoring data, identify performance gaps between individual school districts and the local BOCESaggregate result, and explore professional development resources.

The goal of the usability test was to validate the usability of the most common tasks of the website.Participants agreed to test the site. Four tests were conducted in the usability test lab and three

others were conducted at the participant’s location of choice. All the participants except one useda laptop computer running windows vista. A screen capturing software, Webinaria, was used torecord mouse clicks, pages visited and the participant’s comments.

Not all participants were able to complete all the tasks successfully. For the task which requiredthe participant to navigate to a trend summary chart, 85% of the participants successfullycompleted the task. For the task which required the participant to find a lesson plan, only 42%successfully completed the task! The task that required participant to navigate to the lesson plansprovided the most difficulty for participants, with an average task time of 427 seconds, and acompletion rate of 42%.

lity Testing Sree Anirudh J Bhandaram http://www.bionecron.com/portfolio/usability/usability-tes

2010/03/1



The chart below shows the average task completion time. It’s pretty clear that task 4 was the mostdifficult task.

2 Method

2.1 Participants

The main users of the DataMentor website are teachers and administrators. Many participants hadroles that were not limited to teaching. Of the seven participants, four were males and three werefemales. We had three novice users and four expert users. The participants included oneadministrator, four teachers and two data analysts.

2.1.1 Team Roles

Before conducting the usability test, we identified roles for the team members based on the dutiesand processes for the test. We came up with four team roles:

Test Administrator Test Moderator Note taker Observer

Then, we assigned roles to all of us members as following:

Team Member Team Role

Kate Walders Administrator

Sanket Naik Moderator, Note taker

Sharmishtha Moderator, Note taker

Anirudh Bhandaram Note taker, Observer

After assigning roles, we conducted a pilot test to familiarize ourselves with these roles. Althoughthe team roles were specified, the note takers occasionally played different roles during the tests.

As the test administrator, Kate recruited all the participants. She was the team’s point of contactwith Staci and the participants. During the test, Kate greeted the participants, read off theorientation script, obtained written consent and gave the pre-test to the participants. She alsoconducted the tests by handing the tasks and reading the tasks aloud to the participant. She alsotook notes on participant’s comments in addition to the note takers. Anirudh helped setup the datacapture software during each session in addition to being the observer. Anirudh also took notesduring one usability test. Sanket and Sharmishtha were test moderators and note takers; theywere in the usability testing room with the participant and recorded participant’s comments. For two tests, Sanket and Anirudh couldn’t attend because of scheduled classes, Kate andSharmishtha conducted two tests on their own, assuming all the roles.

After the tests were completed, Sanket analyzed the data, Kate and Sharmishtha wereresponsible for the website recommendations and Anirudh created the usability test report.


2010/03/1



2.2 Protocols

The test was designed to collect data relevant to the tasks, which enabled us to make suggestionsfor the website. The test also helped validate our heuristic evaluation of the website. Theparticipants were encouraged to think aloud which assisted us in obtaining comments andsuggestions from the participant which otherwise would have been difficult to get. Our main focuswas the path followed by the participant to complete a task. This gave us insight into the design of

the website and the frustrations experienced by a participant while completing the tasks. We alsocollected the task times, number of times the help system was accessed, and the frequency of the“back” button use.

The primary roles for the testing were ‘administrator’ and ‘note taker’. The administrator wasresponsible for direct communication with the participants. As the administrator, Kate read thetasks, answered questions, oriented the participants and conducted the post-task and post testinterviews. The note takers were responsible for recording the various types of data that came outof the test via observation or the communication between the participant and the administrator.

Users were encouraged to follow the think aloud protocol as they performed the tasks. Our

administrator kept her input to a minimum until after the tasks were completed. She discussed thetasks with the participants in more detail while she was filling out the post task and post testquestionnaires.

2.3 Procedure

The usability test lab was reserved for four of the participants who consented to test in the lab.The participants were informed about the test well in advance. One of the participants tested thewebsite at her residence. The other two tested in an empty classroom at a suburban middleschool.

During the test, Kate greeted the participants, gave introductory instructions, requested that theyfill out a pretest form and sign a consent form. They were informed that their voice and their actions on the screen would be captured. The participants were encouraged to “think aloud” and were told to inform us when they think that they completed the task. If they asked for assistancebecause they were having trouble, they were redirected but were never told the solution to thetask (unless they gave up). They were informed that we were testing the website, not them. Weattempted to make them feel as comfortable as possible.

When the participant completed each task, they were asked to answer a set of generic questionsabout the task. Participants were encouraged to discuss the task and give any comments they hadon the website or the task.

2.4 Equipment

All of the participants, except one, tested on Sharmishtha’s laptop computer which had WindowsVista installed on it. A mouse was provided to make interaction with the website easier. Thewebsite was accessed through Mozilla Firefox in its default configuration. The website required aflash plug-in which was preinstalled. Sounds were turned on to get feedback from the user aboutthe sounds on the website.


2010/03/1



2.5 Testing

Four usability tests were conducted in the usability test lab in the Thomas Golisano College of Computing and Information Sciences at Rochester Institute of Technology. The usability testinglab was designed for occupancy by 4 people at once. Although the testing lab had two computers,we decided to use our own laptop because we had users that were not going to test in the lab.Because environmental factors did not play a significant role in the test, we decided to ignore the

presence of more than two people in the room. All the tests were conducted with the same administrator, but the remaining roles were played outby different team members in the tests.

3 TasksOur task scenarios were designed to mimic the most common tasks a user would perform on thewebsite. Based on these task scenarios, we came up with a set of research questions that wewanted to explore during the test. The following are the research question used to design thetasks:

How easily and successfully do teachers and administrators find subject resources?How quickly can a teachers or administrator find their grade and subject’s trend summarycharts?What are the major usability flaws that cause the most frustration for teachers andadministrators?How easily do teachers and administrators understand what is clickable?

You can find the tasks used in the test in the appendix.

4 Test ResultsDuring the test, the note takers observed and noted all the comments that the participants made.The data collected included task times, number of times help was referenced and number of pages traversed.

The quantitative data was categorized according to the tasks. The qualitative data collected can beseen in the appendix.

4.1 Task 1 – View chart listOn average, a participant spent 4 minutes performing this task. The bulk of the time was spent onanalyzing the charts. This task was the easiest for the participants and all users successfullycompleted this task.

ParticipantTime(insec)

Pagestraversed

Backbuttonused

Time onhelptutorial

Avg.time oneachhelp use


2010/03/1



1 94 4 0 0 02 411 12 5 72 363 368 11 4 104 724 96 4 0 0 05 440 7 0 90 906 113 N/A N/A N/A N/A7 117 4 0 0 0

4.2 Task 2 – View trend summary and related performanceindicators

All of the users were able to complete this task. On an average, a participant spent 6 minutes onthis task. The average pages traversed was 15. One user traversed 51 pages to complete this taskthough the ideal number was 5 pages. The location of trend summary at the bottom of the pagewasn’t visible.

ParticipantTimePagestraversed

Back buttonused

Time onhelp

Avg. timeoneachhelp use

1 60 5 0 0 02 629 10 6 0 03 192 11 5 10 104 199 6 1 0 05 706 51 11 0 06 119 N/A N/A N/A N/A7 269 10 4 0 0

4.3 Task 3 – Distinguish resources to locate differentassessment views

One user failed this task. The average user took 7 minutes to complete the task. Some of theusers tried to locate the January 2007 ELA Regents through the chart list which is where theyexpected but did not find it.

TimePagestraversed

Back buttonused

Total time onhelp tutorial

Avg. time on eachhelp use

860 38 11 0 0398 16 8 32 32Failed8 0 0 074 4 0 0 058 3 0 0 0386 N/A N/A N/A N/A582 23 6 134 67

4.4 Task 4 – Find the resource to locate lesson plans

This task was considered to be the most difficult task of the four. 4 out of 7 users failed this task.


2010/03/1



On average, 30 pages were traversed performing this task.

TimePagestraversed

Back buttonused

Total Time onhelp tutorial

Avg. time on eachhelp use

150 21 14 0 0failed 67 17 152 38failed 23 10 60 60failed 23 6 38 38failed 29 6 18 18204 N/A N/A N/A N/A205 13 3 0 0

4.5 Conclusions

On average, all tasks went beyond 5 minutes. The maximum of time was spent on searching for the data. An average user traversed 17 web pages for each task because they couldn’t find thedata where they expected it. The help system was referred to only for 19 seconds on average,where a user spent time reading the manuals but finally gave up. Few users viewed the help video

for more than several seconds. The back button was used an average of 5 times because theusers didn’t find appropriate navigation links. Many of our users went to the Accelerate U site bymistake as the DataMentor website didn’t give the user proper notification that he/she was leavingthe site.

5 Recommendations

5.1 Task 1 – View chart list

Legends should be visible throughout the chart list so that a user can refer to it anytime theywant to.Make chart sorting buttons more obvious.

5.2 Task 2 – View trend summary and related performanceindicators

Trend Summary chart option should be listed as sub-menu for chart list

5.3 Task 3 – Distinguish resources to locate differentassessment views

List all similar kind of data at one place. For example, a user could not get to the January2007 ELA regions from the chart list.Font size of different assessment views (teachers/students) should be larger.Organize test questions in assessment such that each question is on its own page andprovide a navigation system that lets the user choose a question and/or that lets a user go tothe previous or next question


2010/03/1



5.4 Task 4 – Find the resource to locate lesson plans

The lesson plan is frequently used by target users. Almost all users had problem searchingthis option. The lesson plan should be part of resources navigation.

5.5 Overall

The data should be reorganized to provide categorical view for user to make user goals moreeasily obtainable.Rename some of the information types to be more descriptive – like new features could benew assessment updates.Define resources.Make more information choices readily apparent.Move features that are not primary data (like new features) .Search is the user’s lifeline when navigation fails. Design of search should be presented as asimple box, since that’s what users are looking for.Dealing with PDF files while browsing through website is not a preferred option for any user;

most PDFs do not follow standard browser conventions. Layouts of PDFs are often optimizedfor a sheet of paper, which rarely matches the size of the user’s browser window and it isdifficult to scroll such document.PDF is great for printing and for distributing manuals. Reserve it for this purpose and convertany information that needs to be browsed or read on the screen into real web pages.

A wall of text is deadly for an interactive experience. When user is looking into help, theactual text should be in the following format:

subheadsbulleted listshighlighted keywordsshort paragraphs

CSS style sheets unfortunately give websites the power to disable a Web browser’s “changefont size” button and specify a fixed font size. In DataMentor the font is fixed and small insize, which significantly reduces readability for most users.The website should provide a way to give a user preferences and let them resize text asneeded. It is important to consider font size for DataMentor website as some of the targetaudience age group is more than age 40.Need to provide a hierarchical view of the website to allow users to browse through itseamlessly without the use of browser back button.

Elsewhere

FacebookFlickr Deli.icio.usLinkedinTwitter

© Sree Anirudh J Bhandaram. Powered by WordPress , using the Manifest theme


Documents

Usability Testing Report (ENGLISH Only)