Usability Studies: Testing Your Website · According to Jakob Nielsen in the article “Web Usability: ... have a similar purpose to the one you are designing. ... Usability Studies:

Usability Studies: Testing Your Website

Introduction

The web is no longer “new and exciting.” Having a website does not automatically makea company innovative or creative. Web users and surfers are expecting more from thewebsites they visit. They are no longer impressed by animated gifs and interesting icons.Users come to a site for a reason. They have a goal when they visit. They are seekinginformation and they want to be able to find it quickly and easily. They want a site that iseasy to use. A new word that has entered the vocabulary within the past few years isusability. Usability can be defined as “the extent to which a product can be used byspecified users to achieve specified goals with effectiveness, efficiency, and satisfactionin a specified context of use” (Bevan, 1997). How do you know if your site is usable?The best way to find out is to conduct a usability study.

Significance of Topic

A usable website can be the difference between gaining or losing a new customer. If auser visits a site for a specific reason but can’t figure out the navigation structure or thelayout, he may leave in frustration, never to return. If a user visits a site and effortlesslyfinds the information he needs, most likely you have made a customer for life (or at leastuntil the next redesign). Usability is no longer a “nice-to-have” on a website; it is now arequirement. For this reason, companies and designers have realized that usabilitystudies (also called usability testing) are an essential part of the web design process.Usability studies enable designers to learn from representative users if the site “works”the way it is expected to. Usability testing is the best way to uncover problems in a site’sdesign.

Usability studies can be involved and detailed, with many respondents participating, ormay follow a more stripped-down model, with only a handful of participants. They canbe conducted on the live site or on paper prototypes. It is important to be aware of thedifferent characteristics of usability testing, the different types of testing that can beconducted, and how to conduct a typical usability study.

Discussion of topic

UsabilityBefore discussing usability studies, the idea of “usability” should be understood. Whendiscussing websites, usability can be defined as the aspects of a system that make itpleasing to use, easy to learn, and easy to use and navigate. Usability can make thedifference between “performing a task accurately or not, completing a job or not, andenjoying the process or being frustrated” (Usability Introduction, 1998). In other words,to be usable, a website’s layout should be intuitive; it should require as few steps as


2

possible to retrieve information; steps to retrieving information should be easy toremember; and the site should leave the users with a positive feeling about the site andthe organization (Bernard, 2000).

Usability is important in keeping users at the website. There is no training in how to usea website. A user visits the website and begins looking for information. If a website isnot effective within the first few seconds of arrival, the user is lost (Rhodes, interviewwith Jakob Nielsen, 1999). One of the core usability concepts is the ability of the user tonavigate to find what they want. “Without usability you are dead in the water in thisregard” (Rhodes, interview with Jakob Nielsen, 1999).

Jakob Nielsen is one of the chief experts in web usability. He has written a book on webusability entitled Usability Engineering and writes a biweekly column on web usability.In his article “What is ‘Usability’?” (1998) he gives the five characteristics of usability.The first characteristic is ease of learning. Can a user who has never seen the websitebefore learn it well enough to accomplish basic tasks? The second characteristic isefficiency of use. How fast can an experienced user accomplish tasks once he or she haslearned to use the system? Memorability is the third characteristic. If a user haspreviously visited a website, can he or she remember enough to use it more effectivelythe next time? The fourth characteristic is error frequency and severity. How often dousers make errors while on the website and how serious are these errors? The finalcharacteristic is subjective satisfaction. How much does the user enjoy using thewebsite?

According to Jakob Nielsen in the article “Web Usability: Why and How: How to Get aUsable Website” (1998), the best approach to creating a usable website is to follow somebasic steps during the design process.

1. If a website has already been created and will be redesigned to be more usable, don’tthrow out the old site immediately. It is a good idea to run a small test on the olddesign to see if there are some good parts that should remain.

2. Test sites that have a similar purpose to the one you are designing. Interestingsolutions to a current problem may be found on other websites.

3. Conduct a field study to learn how users work in their own environment. Real workpractice is always different than what a manager says it is.

4. Create some simple prototypes of a variety of designs on paper and test with a fewusers.

5. From these tests, choose the best design and create a web prototype (that is notcompletely built).

6. Test as many times as possible.7. Prepare an almost-finished user interface that contains the important pages and the

navigation elements. Test again.8. Once the design is launched, start planning the next redesign.


3

It is obvious that many of the steps to creating a usable website involves user testing.Once designers know exactly what makes a usable website they can begin the creationprocess. Before the site goes live, usability studies should be conducted. The nextsection will discuss different types of usability studies, and then focus on the mostcommon type of usability study, which uses a “lab test” environment.

Usability StudiesIn their most common form, usability studies take a website and put real users in front ofit with real-world situations (“Usability,” 2000). It is important to always keep in mindthat the ultimate goal is to create a great website. The best tool for determining if awebsite is great is to watch users use it. If they can use it then the goal has been attained.If not, fix the problem and test again (Rhodes, interview with Jared Spool, 1999).

Usability studies may have variations on this method. One variation is the ExpertAssessment method (Whitehand, 1997). With this method, one or two usability expertstest a site throughout the development process. This is a quick and inexpensive way toconduct a usability study and provides quick feedback. However, in using experts it maybe difficult to judge how much trouble certain problems will cause the average user(Whitehand, 1997). Another variation is Participatory Evaluations, or experts with users.Several usability experts pair up with a representative user and work together on a set ofpre-defined tasks centered around the website (Whitehand, 1997). This method enrichesthe experts’ understanding of how users use the website, but the users’ way of workingmay be affected by having an expert as a partner.

Some studies do not use actual sites at all. In some studies, a paper prototype is placed ina three-ring binder with dividers and labeled tabs that represent destinations ofhyperlinks. “The person using your prototype will view one ‘page’ at a time, and skip tothe next ‘link’ by checking the labeled tab on the divider” (“Usability: Testing Sites withProspective Users,” 1996). In fact, some usability experts recommend testing this way.“This allows testing at a stage where we know enough about what the application issupposed to do to be able to make meaningful design recommendations but where wehave not invested too many resources” (Wilt, 1997).

The most common type of usability study, however, is mentioned above: “take a websiteand put real users in front of it with real-world situations.” This method uses a “lab test”environment, where participants follow a set of pre-defined tasks while being observed(Whitehand, 1997).

Planning the StudyA usability test with five users will usually uncover 80% of usability problems (Nielsen,“Cost of User Testing a Website,” 1998). In his article “Why You Only Need to TestWith Five Users,” Nielsen states that when a single user is tested, he finds about a thirdof all there is to learn about how usable the website is. With the second user, there issome overlap in what is found because he does some of the same things as the first user.


4

He will also find some things the first user missed, but not as much as the first user did.The third user will do many of the things User 1 or User 2 did, as well as some of thethings they both did. He will offer some new insight, but less than the first and secondusers. As each new user is added, the study facilitator will learn less and less because thesame thing is seen again and again. After the fifth user, the facilitator is wasting her timeas she observes the same findings repeatedly.

Once the number of users has been determined, the next step is to decide whom to test.Define the target audience and then find people fitting that profile. Find people in thetarget audience who haven’t visited the site before and invite them to participate in theusability study (Instone, 1997). The study will yield the best results when you test a goodbreadth of users.

The next item to be determined is the goal of the usability study. Common goals includechecking if users can collect information or perform certain tasks (Gordon, 2000). It isalso important to keep in mind the goal of the website, as this will shape the tasksprovided for participants. Why do people come to the site? Is “that” easy to do on thesite (Instone, 1997)?

The final step in the planning phase is deciding what to ask participants. The questionsare influenced by the goals of the study that were previously defined (Gordon, 2000).Have users perform tasks that will be common to site visitors. Ask users what they thinkof the site: What do they like or dislike, and why (Kuniavsky, 1998)?

The facilitator may also want to test the vocabulary and icons used on the site. Does thevocabulary used for buttons and links lead the users where they want to go? Are theicons meaningful? To test vocabulary, ask participants what information they think theywould get by clicking on a certain word, or ask participants which word they would pickif they were looking for information about a specific topic (Redish, 1997). To test icons,ask participants to identify the meaning of pictures that are used as icons, or ask them tomatch the pictures with brief descriptions (Redish, 1997). For these types of questions,the facilitator may want to use paper prototypes and test this information beforedesigning the site, since this may save valuable web authoring time.

At this stage, it is important to keep a few characteristics of usability testing in mind. Ausability test is context-specific (Boling, 1996). It is designed to provide data about aspecific audience and a specific set of tasks on a specific website. The team conductingthe study should resist the temptation to “generalize findings across projects” (Boling,1996). Usability tests are data driven. Decisions on the design of the site should bedetermined from observation and participant responses, not from speculation. Finally,the usability test should be descriptive, not prescriptive (Boling, 1996). Instead ofdetermining design principles, problems should be identified and described. Thereshould be no expectation that the discovery of a design flaw will automatically yield asolution (Boling, 1996).


5

Conducting the StudyWhen a participant enters the testing room, several things should be said to her to ensurethe best results. Explain the purpose of the study if it wasn’t explained when theparticipant was enlisted (Gordon, 2000). Probably the most important thing to say is thatthe site is being tested, not the participant (Gordon, 2000). This generally relieves someof the pressure the participant may be feeling. She should also be told to be honest in herresponses—don’t worry about hurting the facilitator’s feelings (Gordon, 2000). Veryimportant to the success of the study is asking the participant to think out loud as shemoves through the site, to verbalize what she is doing and why. Finally, ask that she notdo anything differently than she would if she were at home or work. “Respondents oftentry to hard to complete tasks to please the moderator. For example, make sure they don’tspend more time reading instructions than they normally would” (Gordon, 2000).

Not required of a usability study, but important if possible, is to videotape the sessions.This allows the interested parties to go back over them later to see what participants didduring the study (Outing, 1997). The videotape is important because it means thefacilitator does not have to sit behind participants and look over their shoulder to seewhat they are doing. Instead, they can sit at a table facing the participant and look at themonitor (showing their mouse clicks) and their facial expressions at the same time.

The facilitator’s job may be made easier by having a script to work from. The scriptshould provide specific prompts for note-taking about user activities (Kantner, 1997). Inaddition, it is helpful to have a printout of the web pages that participants will beaccessing. The facilitator can take notes about where participants visited and in whatorder. Following this method provides the facilitator with information about the order ofweb pages visited and “which pages received the most time from the user” (Kantner,1997). This information is invaluable in understanding the extent of the usabilityproblems on the site.

After giving an overview of the usability study, the facilitator hands the participant a listof “tasks” to accomplish. Once the participants begin answering the questions, thefacilitator assumes a passive role. If a participant asks for help, the facilitator should notanswer. In fact, it is best not to interfere with the participant in any way (Outing, 1997).If she continues to struggle for several minutes and continues to look in the wrongdirection, tell her the correct answer and ask her to move on to the next question.

When a particular task causes problems for several users, the facilitator should removethat task from the list. The task should be noted as being a problem for later study, butthere is no need to waste time and have every participant try something that is clearlyhanging up the majority of participants (Gordon, 2000).

This is a time to get valuable feedback from the participant. When respondents identify aproblem, the facilitator should make sure to document it (Gordon, 2000).


6

Besides evaluating the participants’ performance during the test, the facilitator shouldnotice body language and facial expressions. Nonverbal communication is often morerevealing than the spoken word (Gordon, 2000). Observation is the most important partof conducting a usability study. “Keep a record of what the subjects do; then examinethese records to pinpoint problems in the design. Figure out why the problems occurred,and revise the design” (“Usability: Testing Sites with Prospective Users,” 1996).

If a video camera is not available to tape the study, then the facilitator may want to havean assistant. This person can help take notes and observe, since it is hard to facilitate andgather feedback at the same time (Gordon, 2000).

Larry Wood encourages testers to go beyond a task-based usability study and ask WHY.“Asking why had a more profound effect on the outcome that I had anticipated” (Wood,2000). He was asked to conduct a usability study for the Brigham Young Universitylibrary’s website. When he asked participants who had completed a task correctly whythey chose the option they did, he learned that most of them had guessed. Therefore, theusability of the site was actually lower than the results indicated (Wood, 2000).

After the Study: Interpreting the ResultsAfter the final participant, the facilitator should immediately summarize her notes. Thenext step is to analyze the results and determine what needs to be changed or redesignedon the website. The most effective thing to do is look for trends (Kuniavsky, 1998). Ifone person struggled with an item but the remainder of the participants had no troublewith it, it is probably best to leave that item alone. If several people missed the samething than some changes need to be made. “Focus on the shared issues that were raised”(Fleming, 1998). Ultimately, the facilitator will need to make some judgment calls.“Look for the underlying patterns in what people say; don’t try to follow their desiresverbatim” (Kuniavsky, 1998).

According to Greg Wilt, Manager of Usablity Engineering at Bell Atlantic, some of themost valuable data that comes out of usability studies is time spent to complete searchtasks (1997). Too much time spent on tasks indicates a failure on the part of designers topromote “a clear visualization of the information space” (Wilt, 1997). In other words, theinformation is not organized in a structure that is easy for participants to decipher. Whenthis is the case, users view their search tasks as “digging instead of browsing”—movingdown into material instead of across it (Wilt, 1997).

Use what was learned to improve the website (Instone, 1997). Participants may havesuggested more descriptive labels, or the majority may have ended up getting lost whiletrying to complete a task. Redesign around these observations.

At a later date, another usability study should be scheduled to evaluate the success of theredesign.


7

References

Bernard, M. (January 31, 2000). Constructing User-Centered Websites: The Early DesignPhases of Small to Medium Sites. Usability News, 2. Retrieved October 19, 2000 fromthe World Wide Web:http://wsupsy.psy.twsu.edu/surl/usabilitynews/2W/webdesign.htm.An article from a newsletter called Usability News. This newsletter is put out by theSoftware Usability Research Laboratory at Wichita State University. The article isabout how usability can be incorporated into a website before getting to the productionphase.

Bevan, Nigel (February 19, 1997). Position paper for Conference on Human Factors inComputing Systems, March 22-27, 1997. Retrieved October 20, 2000 from theWorld Wide Web: http://www.acm.org/sigchi/webhci/chi97testing/bevan.htm.This article is written by the Head of Usability Services at the National PhysicalLaboratory in the UK. He details usability goals for websites.

Boling, Elizabeth (October 5, 1996). Usability Testing for Web Sites. Position paper forLearning for the Global Community: Seventh Annual Hypermedia ’95 Conference.Retrieved October 20, 2000 from the World Wide Web:http://www.indiana.edu/~iirg/ARTICLES/usability/usability.main.html.This is a paper from a workshop from several years ago. The author answersquestions about why designers should test and what to look for when testing.

Fleming, Jennifer. (June 15, 1998). User Testing: How to find out what users want.Ahref.com. Retrieved October 20, 2000 from the World Wide Web:http://www.ahref.com/guides/design/199806/0615jefprintable.html.This article answers some basic questions about when it is appropriate to test andwhen it isn’t appropriate. It goes through the steps of setting up a test, choosing afacilitator, finding participants, and analyzing the results of the study.

Gordon, Seth. (February 15, 2000). User Testing. CNET Builder.com. Retrieved October ,2000 from the World Wide Web:http://www.builder.com/Graphics/Evaluation/index.html.This is one of the better articles on usability testing. Very simple andstraightforward. He breaks the study down into 4 steps and explains each onethoroughly.

Instone, Keith. (April 25, 1997). User Test Your Web Site. Webreview.com. RetrievedOctober 20, 2000 from the World Wide Web:http://www.webreview.com/97/04/25/usability/index.html.The author is a usability consultant. This article is the first in a series he is writingon usability testing. He is trying to raise awareness of it.


8

References, cont’d.

Kantner, Laurie (February 19, 1997). Position paper for Conference on Human Factors inComputing Systems, March 22-27, 1997. Retrieved October 20, 2000 from theWorld Wide Web: www.acm.org/sigchi/webhci/chi97testing/kantner.htm.The author is a usability consultant. The article describes two different ways to testwebsites for usability and the effectiveness of each.

Kuniavsky, Mike. (April 8, 1998). Why User Testing is Good. Webmonkey. RetrievedOctober 24, 2000 from the World Wide Web:http://hotwired.lycos.com/webmonkey/98/14/index3a.html?tw=design.The author is the Interface Designer for Hotwired. In this article, he breaks usertesting into four steps and describes each one.

Nielsen, Jakob. (May 3, 1998). Cost of User Testing a Website. UseIt.com Alertbox.Retrieved October 20, 2000 from the World Wide Web:http://www.useit.com/alertbox/980503.html.This article discusses common usability problems based on studies at the TechnicalUniversity of Denmark. It also talks about the costs involved in running a usabilitystudy, both for new testers and experienced testers.

Nielsen, Jakob. (September 15, 1998). Web Usability: Why and How: How To Get aUsable Website. ZDNet Developer. Retrieved October 21, 2000 from the WorldWide Web:http://www.zdnet.com/devhead/stories/articles/0,4413,2137433,00.html.This article lists practical ways to get a usable website. A section on user testingvery simply lists the steps for conducting a usability test and implores all designersto conduct these tests on their websites.

Nielsen, Jakob. (September 29, 1998). What is ‘Usability?’. ZDNet Developer. RetrievedOctober 21, 2000 from the World Wide Web:http://www.zdnet.com/devhead/stories/articles/0,4413,2137671,00.html.Based upon the above article, Jakob Nielsen digs deeper into the issue of usability.He lists five characteristics for a usable website.

Nielsen, Jakob. (March 19, 2000). Why You Only Need to Test With Five Users.UseIt.com Alertbox. Retrieved October 21, 2000 from the World Wide Web:http://www.useit.com/alertbox/20000319.html.This article explains the why using more than 5 users in a usability study is a wasteof time for the facilitator. It also explains the mathematical model that was used toreach this number.


9


Outing, Steve. (July 2-4, 1997). Reader Usability Wisdom From a Web Guru. Editor &Publisher Interactive: The Media Info Source. Retrieved October 19, 2000 from theWorld Wide Web:http://www.mediainfo.com/ephome/news/newshtm/stop/st070297.htm.This website is an interactive newspaper. The article discusses “do it yourself”usability studies and the future of online news.

Redish, Janice (February 19, 1997). Usability Testing of World Wide Web Sites. ). Positionpaper for Conference on Human Factors in Computing Systems, March 22-27,1997. Retrieved October 20, 2000 from the World Wide Web:http://www.acm.org/sigchi/webhci/chi97testing/redish.htm.The author teaches workshops on usability studies and task analysis. Her articlediscusses known usability problems, special characteristics of the web, and thebarriers to conducting usability tests.

Rhodes, John S. (August 8, 1999). [Interview with Jakob Nielsen]. WebWord.com.Retrieved October 20, 2000 from the World Wide Web:http://webword.com/interviews/nielsen.html.This interview discusses the history of usability and core usability concepts. Thisarticle also goes into factors that make a website excellent and the research that stillneeds to be done on usability.

Rhodes, John S. (April 12, 1999). [Interview with Jared Spool]. WebWord.com. RetrievedOctober 20, 2000 from the World Wide Web:http://webword.com/interviews/spool.html.In this interview, Jared Spool, founding principal of a company called UserInterface Engineering talks about the importance of user data taken from usabilitystudies. He discusses the usability issues that have shocked him the most and thesteps to take when usability problems are apparent.

Usability. (April 14, 2000). Usability by Design. Retrieved October 21, 2000 from theWorld Wide Web: http://www.usability.uk.com/This is the website for a company that usability and design solutions. Their siteprovides extensive information on what usability means, what it is, and presentscase studies.

Usability Introduction (1998). Usability First. Retrieved October 23, 2000 from the WorldWide Web: http://www.usabilityfirst.com/intro/newcomers.html.Usability First is an online guide to usability resources. It provides information onmaking websites and software easier to use.


10


Usability: Testing Sites with Prospective Users. (October 14, 1996). Indiana University.Retrieved October 13, 2000 from the World Wide Web:http://www.iuinfo.indiana.edu/policy/usability.html.This article focuses on using paper prototypes in usability. It then goes throughhow to conduct a usability study with prototypes.

Whitehand, Richard (February 19, 1997). Usability Testing of World Wide Web Sites.Position paper for Conference on Human Factors in Computing Systems, March 22-27, 1997. Retrieved October 20 from the World Wide Web:http://www.acm.org/sigchi/sebhci/chi97testing/whitehan.htm.The author is a trained ergonomist who consults on web usability evaluation anddesign support activities. He presents several of the different models he has used intesting and explains the pros and cons of each method.

Wilt, Gregory A. (February 19, 1997). ). Position paper for Conference on Human Factorsin Computing Systems, March 22-27, 1997. Retrieved October 20, 2000 from theWorld Wide Web: http://www.acm.org/sigchi/webhci/chi97testing/wilt.htm.The author leads the Usability Engineering program at Bell Atlantic. This articlefocuses mainly on usability in regards to web-based training.

Wood, Larry (May, 2000). Why Ask Why in a Usability Evaluation? The UPA Voice,volume 2, issue 2. Retrieved October 20 from the World Wide Web:http://www.upassoc.org/voice/vol2no2/technique.htm.This article is from the publication for the Usability Professionals’ Association. Inthe article, the author discusses a case study on a usability study he did at BrighamYoung University. He discusses the results he got and how, when he askedparticipants why they made the choices they did, he got very meaningful feedback.


11

Related Links

1. Usable Web. http://www.usableweb.com. This page divides up usability into severalcategories, and then lists multiple sites for each catetory. Their tag line is “1122 linksabout web usability.”

2. Jakob Nielsen’s (usability guru) website. http://www.useit.com/If you will be designing a website or a usability study, this is the place to start. JakobNielsen is the undisputed leader in the usability field.

3. http://www.webpagesthatsuck.com/. Examples of poorly designed websites can befound here. Very valid comments about what makes a website confusing.

4. http://www.coolhomepages.com/. Examples of well-designed web pages can be foundhere.

*This paper is written by Adrienne Schmerbeck for the course EDC385G InteractiveMultimedia Design and Production at the University of Texas - Austin

Documents

Usability Studies: Testing Your Website · According to Jakob Nielsen in the article “Web Usability: ... have a similar purpose to the one you are designing. ... Usability Studies: