[IEEE IPCC 2005. Proceedings. International Professional Communication Conference, 2005. - Limerick, Ireland (July 7, 2005)] IPCC 2005. Proceedings. International Professional Communication

2005 IEEE International Professional Communication Conference Proceedings

0-7803-9028-8/05/$20.00 © 2005 IEEE.

Conducting Remote, Internet-based Experiments on Web Design

Elisabeth Cuddihy University of [email protected]

Carolyn Wei University of [email protected]

Alexandra L. Bartell University of [email protected]

Jennifer Barrick University of [email protected]

Brandon Maust University of [email protected]

Seth S. Leopold, MD University of [email protected]

Jan H. Spyridakis, PhD University of [email protected]

Abstract

This paper presents an Internet-based study of the effect of intra-article navigation affordances on Web browsing behavior. Navigation was assessed by randomly assigning participants to one of four experimental conditions: the existing intra-article navigation bar, one of two redesigned intra-article navigation bars, or an intra-article navigation bar combined with a global navigation bar. Data about participants’ browsing behavior were logged by using our experimental design toolkit to instrument the Web site’s pages with PHP scripts [1]. Demographics, perceptions, and comprehension were measured through a post-browsing survey. The study is ongoing and results will be presented at the conference. We will also report on the further development of our toolkit that has made conducting Internet-based research possible.

Keywords: Web navigation, Internet research, comprehension, browsing behavior

Introduction

The Web has become a primary resource for many people seeking information about health, educational programs, and government services. Many Web sites that seek to convey information on these topics have been carefully crafted with rubrics that are intended to enhance Web site usability. However, a majority of design guidelines

are based on best practices, small-scale lab studies, and designers’ instincts rather than large-scale empirical research conducted in a natural context.[2] We argue that the use of large-scale, rigorous empirical studies of real users “in the wild” to test design guidelines would help validate their effectiveness and applicability as well as enhance the certainty with which designers can apply them.

Our research group has been conducting remote, Internet-based experiments to test the effectiveness of various Web design recommendations, assessing such issues as heading frequency [3], local navigational link wording [4], hyperlink wording [5], and credibility features.[6] Our experiments have been conducted through the Internet (e.g., accessible from any computer in the world and at any time) on target populations and at times with populations of students drawn from university classes. Participants connect to the study wherever and whenever they like while the research team uses the experimental server to log user behaviors and record survey answers.

We contend that there are many benefits to conducting Web design research remotely through the Internet. First, it is possible to sample a large number of naturally occurring users’ interactions with a naturally occurring Web site. Second, asking users to complete a Web study in their own time and space allows researchers to study realistic behavior.

554


This paper reports on our continued work in Internet-based research on Web design, specifically focusing on our most recent study of navigational features of a health Web site. We will also present at the conference the latest features of our toolkit that we have used to simplify the Internet experimental process for researchers with little programming experience.

Usability and Health Web Sites

In 2004, one-fourth of Americans searched for information about prescription drugs online, and it is estimated that by 2005, 88.5 million Americans will seek health information on the Internet.[7], [8] The usability of online heath information is critical, yet it has not been extensively examined in the e-health field.[9-15]

Some recent empirical studies have examined navigation and usability on health-related Web sites through Internet-based testing. Mobrand and Spyridakis [4] conducted an Internet-based study that tested how various types of local navigational cues affected user comprehension, perceptions, and browsing behavior of a Web site on osteoarthritis. Similarly, Schultz and Spyridakis [3] conducted an Internet-based study of the effect of heading frequency on users’ comprehension and perceptions of Web sites about osteoarthritis or rheumatoid arthritis.

The study reported in this paper furthers this kind of Internet-based research by investigating how intra-article navigation affects readers’ browsing behavior, perceptions, and comprehension of lengthy articles about orthopaedic surgery on an informational Web site. It is hoped that the findings will provide insight about how those who seek health information navigate and process information on medical Web sites, as well as augment the growing research on empirically-based Web design guidelines.

Navigation and Comprehension of Lengthy Web-based Articles

The presentation of lengthy articles as multiple Web pages presents a number of challenges that are less problematic or non-existent when such information is presented in print.

One difficulty is that readers cannot assess lengthy Web articles at a glance without scrolling or clicking through multiple pages. In print, a reader can see an entire page at once, gauge organizational structure by scanning headings, or see the physical page count. Web-based articles, however, often obscure such important structural information. Web pages may be of indeterminate length, forcing users to scroll without a good sense of the page length.

Further, while headings in Web-based articles are employed as they are in print to help readers understand content and structure, readers cannot scan all the headings on a long Web page at once. Similarly, the Web parallel of a reader quickly thumbing through a printed book or article to get a sense of the lay of the land requires a time investment of searching, clicking links, and waiting for pages to load.

Web design guidelines suggest that long Web pages should have a table of contents to help users understand what a page contains [16], a claim supported by other researchers.[17], [18], [19], [20] It is logical to assert that a more complex case—a multi-page Web article—would also benefit from a table of contents in order to make up for the lack of cues that would otherwise be available in print.

Articles that are presented as Web pages are typically presented within a site’s design template. A Web site may have one or more templates that organize global Web site elements such as site-wide search boxes, top-level navigation, utility links (e.g., “Home,” “Site Map,” “Contact Us”), and branding logos; such templates may also provide a space for content that will vary from page to page. Articles, including their internal navigation features, are treated as a kind of content that must fit within the content area of the template.

However, when an article is broken into multiple pages and when an article may require navigation within a page, the question is raised as to how to best display internal navigation or a table of contents that will help readers comprehend the article’s structure and contents, and access its various internal pages and sections. An example of a Web site that offers internal navigation within

555


articles is Wired Webmonkey1, which provides a numbered table of contents on the left of each article page, aligned with the top paragraph. This table of contents explicitly signals that it provides article-specific internal navigation by using the word “Pages” and numbering the links.

Web sites need to clearly distinguish different navigational elements, such as global and intra-article navigation. Moreover, users quickly learn to anticipate the location of global Web elements, particularly when such elements are near the top of pages, and they will move their mouse toward important global links before moving their eyes.[16], [21], [22]

In contrast, if visually similar designs and locations are used for global Web site navigation and internal article navigation, it is likely that users will confuse the two or fail to notice that intra-article navigation has been made available. Because the Web supports many different structures serving different purposes, users can become disoriented or experience cognitive overload [23], and a menu that does not provide adequate structural cues may make users feel like they are “lost in space.”[23], [24]

University of Washington Orthopaedic Surgery and Sports Medicine Web Site

The study reported here uses an article drawn from the Web site2 of the Department of Orthopaedic Surgery and Sports Medicine at the University of Washington (UW). The site delivers information about the department as well as articles about orthopaedic conditions, surgery, and medical treatments. These articles are typically eight Web pages in length, and a single page is usually much longer than one screen. The site’s article template displays a Table of Contents in a left-hand navigation bar that replaces the global navigation menu that appears on other content pages. The layout and design of this intra-article navigation bar (right side of Figure 1) are nearly identical to the global navigation bar that exists outside the article pages (left side of Figure 1). The intra-article navigation bar lists the articles’ first-level headings (e.g., Summary, Review of the Condition, Considering Surgery, Preparing for Surgery) and

1 http://webmonkey.wired.com 2 http://www.orthop.uwmedicine.org

links to separate Web pages structured with additional subheadings.

Users navigate between pages of an article by clicking on the intra-article navigation menu or by clicking the “Next Page” or “Previous Page” links at the top and bottom of each content page.

Figure 1. The image on the left shows the Web site’s global navigation bar and the image on the right shows the Web site’s intra-article navigation bar. Both use nearly identical visual and structural styles.

From within an article, beyond utility links to the Home Page of the Department of Orthopaedics and to the Site Map, the only global reference providing links to the rest of the site is a short breadcrumb trail to the Home page or a menu of other patient information articles (Figure 2).

Figure 2. Breadcrumbs provide the only form of global navigation within an article.

Hypotheses

We predict that users will have greater comprehension of lengthy, multi-page articles on highly technical topics if the Web site explicitly displays article structure and differentiates that display from the global navigation structure. Specifically, we predict that multi-page articles with visually distinct tables of contents that

556


provide a heading-level outline will lead to higher comprehension than multi-page articles where the intra-article and global navigation schemes are visually conflated.

On the UW Orthopaedic Web site, we predict that users will overlook the existing intra-article navigation menu thinking it is the global navigation menu. As a result, we hypothesize that users are likely to assume that the only way to navigate between pages of an article is to click “Next Page” or “Previous Page” links, and they will read the articles linearly without understanding the article’s overall structure. They are more likely to miss important information and perhaps abort reading before reaching the end of an article, a browsing behavior that will negatively affect their comprehension.

Methods

To test our hypotheses, we remotely administered an Internet-based study investigating the effect of four intra-article navigation conditions.

ParticipantsParticipants were drawn from naturally occurring readers of orthopaedic information. A banner was placed in multiple locations on the Web site to direct users to the study, which was hosted on a different server. Fliers advertising the study were also sent to orthopaedic medical offices throughout Washington state.

MaterialsThe study Web site consisted of an article drawn from the UW Orthopaedic Web site that contained eight pages discussing a new type of total knee replacement surgery. To adapt the site for use in this study, we removed the Web site search tool, and links to other parts of the site and to video clips. This article’s eight long pages comprised more than 8,500 words and numerous illustrations of the surgical procedure. This article was chosen for this study because it is one of the highest rated of the 80 articles on the Web site concerning medical conditions and surgical procedures.

Using PHP scripts, four versions of the Web site were generated such that each version incorporated a different visual representation of the intra-article navigation bar. Three conditions were based on the intra-article navigation bar being placed to the left of the page content and the fourth condition was

placed to the right of the content: (1) the existing intra-article navigation bar on the left; (2) an intra-article navigation bar on the left that was visually similar to the existing bar but labeled at the top with ‘‘Article Table of Contents”; (3) an intra-article navigation bar on the left that was visually distinct from the original global navigation and labeled at the top “Article Table of Contents”; and (4) an intra-article navigation bar labeled “Article Table of Contents” placed to the right of the content area, with the existing global navigation bar placed to the left of the main content area. Figure 3 displays these four conditions.

The study Web site was prefaced with introductory pages providing background information about the study, an informed consent statement, and study instructions. At the conclusion of browsing, participants took an online questionnaire that collected general demographic information and assessed their perceptions and comprehension of the Web site.

Our toolkit for conducting Web-based experiments also recorded latent information about participants’ clicking behaviors. The article pages were instrumented with PHP scripts so that participants’ navigational behavior, such as which links they clicked, could be accurately recorded. As the participants browsed the article pages, the specific hyperlinks or buttons that they clicked were logged, time-stamped, and associated with each participant’s session. In order to make sure that all interactions with the Web site were recorded by the Web-server, each Web page was marked with a browser request to turn off browser caching. This forced page requests to always go to the Web-server when participants used their browser’s back button so we could identify when the back button rather than a hyperlink was used to reach a page. The research team could determine the specific links used to navigate between pages, the approximate time spent on each page, and the order in which they visited pages.

ProcedureParticipants entered the study by first visiting the UW Department of Orthopedics Web site and then clicking on a banner advertising the study. When visitors opened the initial page of the experimental Web site, they were randomly assigned to one of the four conditions. Each visitor read the introductory material and if they chose to

557


Figure 3a. The existing intra-article navigation is displayed to the left of the main content area. This is the same location and appears in the same style as the global navigation menu on higher-level pages (see Figure 1, left).

Figure 3b. A slightly revised intra-article navigation menu that is labeled “Article Table of Contents” but is otherwise visually similar to the existing intra-article navigation scheme.

Figure 3c. A substantially revised intra-article navigation bar, that is clearly labeled “Article Table of Contents,” numbers the eight pages in the article, and clearly marks the major headings within each page. This navigation menu is visually distinct from the site’s global navigation menu (see Figure 1, left).

Figure 3d. The article is displayed with the global navigation menu on the left, as found on other (non-article) pages, and the substantially revised intra-article navigation menu (same as Figure 3c) is displayed on the right, inside the article content pane.

558


participate, they were then instructed to browse the test Web site for about 10-15 minutes and then click the “Proceed to Survey” button.

After users browsed the site and clicked the “Proceed to Survey” button or any link that would take them outside of the 8-page article, the participants were sent to a Web page where they were asked if they did indeed mean to leave or finish the study. This Web page provided them with the option to return to the article, go to the survey, or leave the study completely. The survey presented the participants with a set of factual and inferential questions about the article’s contents. Survey results were logged with a participant ID such that they could be matched up with the participant’s browsing behavior during analysis.

Our toolkit for running Web-based experiments includes a set of data post-processing scripts (written in Perl) that allows us to parse our custom Web logs and disqualify participants who do not adhere to the study procedure.[1]

Results and Discussion

Data from the study are being collected and analyzed in SPSS 11.5 and will be presented at the conference.

Conclusions and Future Research

The findings in this study will further our understanding of how users navigate through lengthy, multi-page Web site articles. Such findings will also help further the development of empirically-based guidelines for the design of navigational affordances. This study is one of an ongoing series of large-scale, rigorous empirical studies that investigate the behavior, comprehension, and perceptions of the actual users of a Web site.

Besides these contributions to the development of Web design guidelines, this study allowed us to take the first steps in deploying our toolkit on a large, working Web site, extending its functionality and applicability. In its previous implementation, the toolkit was applied on a prototype Web site. Although in this study the toolkit and experimental design pages were administered on a separate server from the official Web site, in future studies we will more closely integrate the toolkit with the native Web site. Discussion of the ongoing

development of our experimental toolkit will be presented at the conference.

More research is needed to provide further empirical evidence for Web design guidelines, especially large-scale, naturalistic, Internet-based studies. In addition, more investigation is needed concerning the methodological issues of Internet-based studies of Web design as well as the requirements of software tools that support such experimentation.

References

[1] J.Barrick, B. Maust, J.H. Spyridakis, M. Eliot, C. Wei, M. Evans., and K. Mobrand, “A tool for supporting web-based empirical research: Providing a basis for web design guidelines,” in Proc. of IEEE International Professional Communication Conference, 2004, pp. 189-194.

[2] J.H. Spyridakis, C. Wei, J. Barrick, E Cuddihy, and B. Maust, “Internet-based research: Providing a foundation for web design guidelines,” Under review.

[3] L. Schultz and J. H. Spyridakis, “The effect of heading frequency on comprehension of online information: A study of two populations,” Technical Communication, vol. 51, no. 4, pp. 504-518, 2004.

[4] K. A. Mobrand, and J. H. Spyridakis, “A Web-based study of user performance with enhanced local navigational clues,” in Proc. of IEEE International Professional Communication Conference, 2002, pp. 500–508.

[5] M. Evans, C. Wei, M. Eliot, J. Barrick, B. Maust, and J. H. Spyridakis, “The influence of link wording on browsing behavior and comprehension,” in Proc. of International Conference on Technical Communication, 2002, pp. 313-317.

[6] K.S. Freeman, and J.H. Spyridakis, “An examination of factors that affect the credibility of health information on the internet,” TechnicalCommunication, vol. 51, no. 2, pp. 239-263, 2004.

[7] M. Pastore. (2000). The mess known as online healthcare. ClickZ Stats Demographics. [Online]. Available:

559


http://www.clickz.com/stats/sectors/demographics/article.php/5971_379231.

[8] Pew Internet and American Life. (2004, October 10). Prescription Drugs Online. [Online]. Available:http://www.pewinternet.org/pdfs/PIP_Prescription_Drugs_Online.pdf

[9] M. D. Fetters, N. V. Ivankova, M. T. Ruffin, J.W. Creswell, and D. Power, “Developing a web site in primary care,” Family Medicine, vol. 36, no. 9, pp. 651-659, 2004.

[10] K. Seidler-Patterson, and M.J. Patterson, “Tapping into the usability dimension: test your product before it goes on-line,” Quirk’s Marketing Research Review, July-Aug, pp.64-8, 2001.

[11] A.T. McCray, E. Dorfman, A. Ripple A, N.C. Ide, M. Jha, D.G. Katz, R.F. Loane, and T. Tse, “Usability issues in developing a Web-based consumer health site,” in Proc of AMIA Symp.2000, pp. 556-560.

[12] P.L. Elkin, B. Sorensen, D. De Palo, G. Poland, K.R, Bailkey, D.L. Wood, and N.F. LaRusso, “Optimization of a research Web environment for academic internal medicine faculty,” Journal of the American Medical Informatics Association, vol. 9, no. 5, pp. 472-480, 2002.

[13] D.M. Fuller, and P.G. Hinegardner, “Ensuring quality Web site redesign: the University of Maryland’s experience,” Bulletin of the Medical Library Association, vol. 89, no. 4, ppg 339-345, 2001.

[14] D.R. Croft, and Peterson, “An evaluation of the quality and contents of asthma education on the World Wide Web,” Chest, vol. 121, no. 4, pp 1301-1307, 2002.

[15] G. Eysenbach, and C.Köhler, “How do consumers search for and appraise health information on the World Wide Web? Qualitative study using focus groups, usability tests, and in-depth interviews,” BMJ,2002. [Online]. Available: http://bmj.bmjjournals.com/misc/terms.shtml.

[16] S. Koyani, R. Bailey, and J. Nall,. Research-based Web Design and Usability Guidelines.Bethesda, MD: National Cancer Institute, 2004.

[17] M. Bieber, “Enhancing information comprehension through hypertext,” in Intelligent Hypertext: Advanced Techniques for the World Wide Web. C. Nicholas and J. Mayfields, Eds. Berlin: Springer-Verlag, 1997, pp. 1-11.

[18] D. Farkas, and J. Farkas, “Guidelines for designing web navigation,” TechnicalCommunication, vol. 47, no.3, pp. 341-358, 2000.

[19] S. Haas, and E. Grams, “A link taxonomy for web pages,” in Proc. 61st Annual Meeting of the American Society for Information Science. 1998, pp. 485-495.

[20] J. Spyridakis, “Guidelines for authoring comprehensible web pages and evaluating their success,” Technical Communication, vol. 47, no. 3, pp. 59-382, 2000.

[21] M. Bernard. (2001). Developing schemas for the location of common web objects. Usability News. [Online]. 3(1), Available: http://psychology.wichita.edu/surl/usabilitynews/3W/web_object.htm.

[22] M. Bernard. (2002). Examining user expectations for the location of common e-commerce web objects. Usability News. 4.1. [Online]. Available: http://psychology.wichita.edu/surl/usabilitynews/41/web_object-ecom.htm.

[23] B.M. Yu, and S-Z Roh, “The effects of menu design on information-seeking performance and user’s attitude on the world wide web,” Journal of the American Society for Information Science and Technology, vol. 53, no. 11, pp. 923-933, 2002.

[24] A. Dieberger, “Supporting social navigation on the World-Wide Web,” International Journal of Human-Computer Studies, vol. 46, pp. 805-825, 1997.

About the Authors

Elisabeth Cuddihy is a doctoral student in the Department of Technical Communication at the University of Washington. Her primary area of interest is in the development of methodologies and supporting tools that aid user interface designers during the design process. She is also interested in information design, information visualization, and education. Elisabeth holds a

560


Master’s degree in Computer Science from the State University of New York at Buffalo and has worked in the software industry designing scientific software applications.

Carolyn Wei is a doctoral student in the Department of Technical Communication at the University of Washington. Her research focuses on how diverse people interact with information and communication technologies such as the Internet and mobile devices. In particular, she is interested in how people start using these technologies and adapt them for their own cultures and personal needs. Previous projects have studied interactions within virtual communities such as distributed work groups, instant messaging buddy groups, and blog networks.

Alexandra (“Sandy”) Bartell recently received her Masters of Science in Technical Communication at the University of Washington where she is continuing her studies as a PhD student. Her primary areas of interest are information design and usability applied to electronic media, particularly in the area of e-government.

Jen Barrick is a Masters student in the Technical Japanese Program at the University of Washington, a sub-department of the Department of Technical Communication. She is interested in cross-cultural and computer-mediated communication and has a background in Web programming and computer technical support.

Brandon Maust is majoring in English and Biochemistry at the University of Washington, and planning on pursuing a career in computational biology. His programming background is in system administration and Web-based applications.

Seth S. Leopold, MD, is an associate professor in the Department of Orthopaedics and Sports Medicine at the University of Washington. His clinical specialty is hip and knee replacement surgery, and minimally-invasive joint replacement. His areas of research interest outside the surgical domain include identification and measurement of the influence of unusual sources of bias in the clinical literature, the relationship between confidence and competence in psychomotor skills education, and use of the Internet to promote patient self-education on health-related topics.

Jan H. Spyridakis is a Professor in the Department of Technical Communication at the University of Washington where she teaches courses on style in writing, research methodology, and international and advanced technical communication. Her research focuses on document and screen design variables that affect comprehension and usability, cross-cultural audiences, and the refinement of research methods. She has a special interest in Internet-based research. She has published numerous articles and received many publication and teaching awards; she is a member of IEEE and also a Fellow of the Society for Technical Communication.

561

Documents

[IEEE IPCC 2005. Proceedings. International Professional Communication Conference, 2005. - Limerick, Ireland (July 7, 2005)] IPCC 2005. Proceedings. International Professional Communication