NIDDK WEB SITE USABILITY ANALYSIS
August 20, 2005
Prepared By:
Bilen Aynu, Rob Fay, & Nneka Okereke INFM 702
User Interaction with Information Systems University of Maryland
College Park, MD 20742
Prepared For:
Roberta Albert Computer Technology Branch
OD/NIDDK/NIH/DHHS Two Democracy Plaza, Room 940
6707 Democracy Boulevard Bethesda, MD 20892-5462
NIDDK Web Site Usability Analysis
Page 2
Executive Summary
Objectives of the Assessment
The NIH National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK)
web team is in the beginning stages of redesigning the Institute’s www.niddk.nih.gov domain.
The NIDDK’s primary motivation for the redesign is to both make the web site more informative
and usable for the customer, but also to introduce upgraded technologies so that the site will be
updated in a more timely fashion using a content management system (CMS). The current
NIDDK site uses static web pages to inform customers.
The purpose of this assessment is to provide objective feedback to the web team and
other interested stakeholders regarding the usability of the current site and to make
recommendations for improvements. Our team conducted an evaluation that includes current
research, a competitive analysis, trusted heuristic ratings, and detailed personas.
Importance of the Assessment
The INFM 702 team believes that a usability analysis will provide some rich feedback to
the NIDDK web team and to the Institute’s stakeholders. Although the NIDDK has collected
some web survey information that polled both the public and the NIDDK staff regarding the
current site, we believe that our evaluation could not happen at a better time. Since proper
information architecture activities and usability studies must take place before a content
management system is implemented, we feel that our evaluation and subsequent
recommendations fill a very vital need.
Key Findings and Recommendations
Although only seven percent of visitors to the NIDDK site focus on the section geared for
researchers, the team feels that the current NIDDK site focuses primarily on these stakeholders
NIDDK Web Site Usability Analysis
Page 3
and focuses less on the information needs of its general public stakeholders. The team has found
that the site is very text-intensive and does not seem to support readability at a sixth grade level.
Fortunately, the preponderance of text makes for quick page load times and makes for easy
viewing on mobile devices. However, all reports are presented in PDF format and do not
provide alternate viewing options such as viewing them as a web page.
The INFM 702 team first recommends that the NIDDK accept and use the personas
included in this deliverable. The NIDDK design team needs to have each stakeholder in mind
when approaching the redesign. In addition, we make a bold recommendation: The NIDDK
needs to include in its mission a responsibility for informing the general public. We’re confident
this is already generally accepted, but the team argues that expanding its mission will allow the
NIDDK to better respond to the increasing information needs of its customers.
Practically, the INFM 702 team recommends that the NIDDK a) provide a consistent
interface across NIDDK pages, including the health information and intramural sites, b) better
organize the text elements using easily understood bite-sized words, c) make better use of white
space, and d) break up text with images, graphics, and media that might more easily
communicate important information to all consumers of the site. Additionally, we recommend
that the NIDDK web team refer to our competitive analysis, using the NCI (www.cancer.gov)
and NSF (www.nsf.gov) sites as best of breed models.
Finally, since the Institute is moving towards implementing a content management
system, we feel that now is the perfect time to a) introduce content ownership so that content
owner names and dates posted are provided with available content and to b) develop rich
metadata elements to aid the consumer in “findability” – through better search (taxonomies, best
bets, thesauri) and navigation (labeling, layouts, topics, etc.).
NIDDK Web Site Usability Analysis
Page 4
Table of Contents
Executive Summary ...............................................................................................................2 Objectives of the Assessment..........................................................................................2 Importance of the Assessment.........................................................................................2 Key Findings and Recommendations..............................................................................2
Table of Contents...................................................................................................................4 Introduction............................................................................................................................5
Background......................................................................................................................5 Importance and Context of the General Topic ................................................................6 Overview/Current Knowledge of the Topic ....................................................................7 Assessment Objectives ....................................................................................................9 Definition of Key Terms .................................................................................................10 Thesis/Point of View .......................................................................................................10
Evaluation of the Usability of the NIDDK Web Site ............................................................11
Description of the Research Issue ...................................................................................11 Method of Data Collection Approach for Conducting the Evaluation............................13 Underlying Assumptions .................................................................................................15 Analysis Techniques Used and Justification ...................................................................15 Development and Pre-testing of Data Collection Instruments ........................................17 Data Collection, Analysis, and Findings .........................................................................18
Heuristic Evaluations..................................................................................................18 Nielsen’s 10 Usability Heuristics .........................................................................20 Shneiderman’s 8 Golden Rules of Interface Design ............................................20 Guidelines for Usability Based on the 4 Learning Styles ....................................20
Persona Development .................................................................................................21 Competitive Analysis..................................................................................................28
Summary/Conclusion.............................................................................................................32
Findings and Conclusions................................................................................................32 Identified Limitations of the Study..................................................................................33 Recommendations ...........................................................................................................33 Future Research Directions .............................................................................................35
References..............................................................................................................................36 Appendices.............................................................................................................................38
Appendix 1: Nielsen’s 10 Usability Heuristics ...............................................................38 Appendix 2: Shneiderman’s 8 Golden Rules of Interface Design ..................................40 Appendix 3: Guidelines for Usability Based on the 4 Learning Styles...........................41 Appendix 4: Competitive Analysis .................................................................................42 Appendix 5: NIDDK Web Site Main Pages....................................................................44
NIDDK Web Site Usability Analysis
Page 5
Introduction
Background
For over one hundred years, the National Institutes of Health (NIH) has played an
important role in improving the health of the nation. There are currently 27 institutes and centers
providing leadership and financial support to researchers throughout the nation and the world.
With over 17,000 employees, the NIH funds over 28 billion annually in research. Of the
Congressional funding it receives, the NIH uses approximately 80% to support publicly funded
research (extramural research) and 10% is used to support NIH employee-conducted research
(intramural research).
The mission of the National Institute of Diabetes and Digestive and Kidney Diseases
(NIDDK) is to “conduct and support research on many of the most serious diseases affecting
public health” (http://www.niddk.nih.gov/welcome/mission.htm). Much of the Institute’s
research focuses on medical topics including kidney disease, diabetes, digestive diseases, and
nutrition and obesity. In addition, according to its web site, the NIDDK
…supports much of the clinical research on the diseases of internal medicine and
related subspecialty fields as well as many basic science disciplines. The NIDDK
supports basic and clinical research through investigator-initiated grants, program
project and center grants, and career development and training awards. The
Institute also supports research and development projects and large-scale clinical
trials through contracts. (http://www.niddk.nih.gov/welcome/mission.htm)
NIDDK Web Site Usability Analysis
Page 6
Importance and Context of the General Topic
The NIDDK is currently planning a redesign of its www.niddk.nih.gov domain (see
Appendix for NIDDK main web page images). The primary motivator for the redesign is to both
make the web site more informative and usable for the consumer, but also to introduce upgraded
technologies so that the site will be updated in a more timely fashion using a content
management system (CMS). The current NIDDK site uses static web pages to inform
consumers.
The reason our team decided to conduct a usability evaluation of the NIDDK site was
because one of the INFM 702 team members was approached by the NIDDK webmaster to act as
an information architect resource for the redesign project. As a result, this team member
approached the webmaster to see if the team could conduct a usability analysis while the
redesign was in its initial stages. Our team hopes that this analysis can provide some substantive
feedback to the NIDDK redesign team as they choose a content management tool and prepare a
new design. Due to time restrictions, our team’s evaluation focused on the site’s overall
usability characteristics – we primarily evaluated the high level pages of the site.
According to the NIDDK web site redesign committee, initially, the project will focus on
the “Research Funding Opportunities” portion of the site, the NIDDK Home page, and the “look
and feel” of the entire web site. The web site “look and feel” redesign will apply to the entire
niddk.nih.gov domain regardless of whether or not the content has been revised. A great deal of
focus will be placed on the design phase of this project to guarantee that the new site is easy to
navigate, and information is easy to read and find. In addition, a major requirement for the
selected “Interactive Publishing” tool is that it be user friendly and intuitive to use. This will
reduce the learning curve for contributors and limit the number of helpdesk calls.
NIDDK Web Site Usability Analysis
Page 7
Overview/Current Knowledge of the Topic
There are no clear guidelines for evaluating the usability of health information web sites.
The increasing number and variety of user types who are using the web to access health
information make the challenge more complex. In her Washington Post article, Agnvall (2005)
stated that consumers in greater numbers are turning to the Internet for health information. The
article looked at research released in May 2005 by the Pew Internet & American Life Project
found that about 95 million American adults use the Internet to find health information. Pew
research also previously found that nearly a fifth of Web seekers say they have gone online to
diagnose or treat a medical condition without consulting a doctor. These users don’t like to be
bogged down in detail, which is one reason why the average user was attracted to commercial
sites such as MayoClinic.com, because it presented information in a manageable way. On the
other hand, non-commercial sites such as the National Institutes of Health, presented an
overwhelming amount of information. Since the information may not be presented in a
manageable way, consumers may feel too bogged down in detail.
Some research readily exists that evaluates the effectiveness and usability of health
information sites. For example, Consumer WebWatch issued a report in 2002 stating that
consumers did not evaluate the credibility of Web sites using rigorous criteria when assessing a
Web site's credibility. Instead, consumers focused on superficial criteria, such as attractiveness
of site design (Forster, 2002).
Furthermore, Sillence, Briggs, and Fishwick (2005) go on to indicate that medical
information sites are often “heavily jargon-laden” and difficult to read (p. 1026). The results of
their study found that customers were heavily influenced by the look and feel of a site (83%) and
NIDDK Web Site Usability Analysis
Page 8
by the degree of personalization of a site. The authors provided the following three
recommendations:
• Guideline 1 – make the purpose of the site clear;
• Guideline 2 – allow personalization, tailored experience; and
• Guideline 3 – include markers of social identity (Sillence, Briggs, and
Fishwick, 2005, p. 1026).
Siegel and Wood (2003) offer insight into the National Library of Medicine’s (NLM)
web strategy, including how this strategy has radically changed over the last 7 years. Originally,
the NLM focused on “meeting the information needs of health practitioners, researchers, and
librarians” (2003, p. 227). More recently, perhaps influenced in part by e-government
legislation, the National Library of Medicine expanded its mission to include focusing on the
information needs of the general public, including “patients, families and friends of patients,
students, and the general health consumer” (2003, p.227). Consequently, it became much more
important for the Institute to critically evaluate how they make information available to all its
stakeholders via the Internet, so they created and recommend a multidimensional approach to
web evaluation that includes usability testing, gathering user feedback, collecting web usage
data, collecting web performance data, and initiating outreach projects to gain qualitative
feedback on the usability of a site.
The usability field overall is maturing in the area of federal health web sites. The
National Cancer Institute developed the "Research-Based Web Design and Usability Guidelines"
to provide guidance to other federal agencies (Barr, 2003). The 187-page guideline covers issues
such as accessibility, home page design, site navigation, writing, graphics and content
organization. Each guideline comes with a score indicating the "strength of evidence" behind the
NIDDK Web Site Usability Analysis
Page 9
recommendation and a score for "relative importance" to help Internet developers make
judgments about competing priorities (Barr, 2003).
Assessment Objectives
The INFM 702 team believes that a usability analysis will provide some rich feedback to
the NIDDK web team and to the Institute’s stakeholders. One piece of information the INFM
702 team received early on was a NIDDK web usage report that calculated the average monthly
page visits. Results indicated that eighty percent of customers visit the NIDDK site to find
health information. On the other hand, only seven percent of customers visit the site to find
research funding information. These contrasting statistics seem very much at odds with the
NIDDK mission to both conduct and support research. Therefore, we feel that the results of our
assessment might encourage the NIDDK to rethink its current mission (conduct and support
research) to include a web site specific mission that focuses on all of its consumers of
information.
Secondly, we feel that providing a third party objective analysis will be extremely
beneficial since two of the members of the INFM 702 team have no affiliation with the NIDDK,
do not have a deep understanding of what the Institute does, and can provide feedback from the
general public’s perspective.
In addition to the benefit of providing objective third party feedback, the team will
provide research-based findings that can be used to increase the usability of the redesigned site.
Since a majority of the redesign participants are scientists and Ph.D.s (in a medical or scientific
field), our team feels that providing research findings will lend a certain level of credibility to
our findings and recommendations.
NIDDK Web Site Usability Analysis
Page 10
Definitions of Key Terms
• B2C – Business-to-Consumer services including information or financial data exchange through an information system.
• CMS – Content Management System • Competitive Analysis – The process of looking at competitive or similar organizations to
see how the usability of their web sites compares with the NIDDK web site. • Extramural Research – This refers to the non-NIH research community that receives
federal funds (grants, contracts) to conduct health-related research or provide related services.
• FAQ – Frequently Asked Questions • HCI – Human-Computer Interaction; the study of how humans interact with computers,
primarily studying the effects of how easily a person can use a given information system. • ICGI - Interagency Committee on Government Information; established as a result of the
E-Government Act of 2002. • Information Architecture – The process or method of organizing information to make it
easier for people to find and use. • Institute – One of the 27 NIH entities that supports health-related research according to
its specific science or support-related mission. • Intramural Research – This refers to the NIH employees that work for specific Institutes
to conduct research related to the mission of their Institute. • NICHD – National Institute of Child Health and Human Development • NIDDK – The National Institute of Diabetes and Digestive and Kidney Diseases • NIH – The National Institutes of Health (NIH), a part of the U.S. Department of Health
and Human Services, is the primary Federal agency for conducting and supporting medical research.
• NLM – National Library of Medicine; one of the NIH institutes • NSF – National Science Foundation; not part of NIH but provides federal grant funding
for scientific research • OMB – Office of Management and Budget • RFA – Request for Application; NIH mechanism for asking for submissions for specific
research funding initiatives. • Stakeholder – Any individual or group that has a vested interest in the NIDDK web site.
This includes the general public (taxpayer), Congress, the research community, NIDDK and NIH leadership and oversight, and NIDDK employees and affiliated contractors.
• Usability – A qualitative and/or quantitative measurement to describe how well a consumer can find and use information from an information system, and their likelihood of revisiting the system to fulfill their future information needs.
Thesis/Point of View
The INFM 702 team understands that the mission of the NIDDK is primarily to conduct
and support medical research as it relates to issues including diabetes, kidney diseases, digestive
NIDDK Web Site Usability Analysis
Page 11
diseases, and nutrition. With this in mind, it seems that the current web site is usable for some
stakeholders but not for others.
Our team believes that in order to make a web site more useful to all stakeholders, it is
vital to conduct an evaluation that provides the NIDDK web redesign team and related
stakeholders with current usability research, a competitive analysis, trusted heuristic ratings, and
detailed personas. Clearly, the redesigned web site must reflect, not only the mission of the
Institute, but it must provide customer satisfaction to all stakeholders, not only specific
audiences.
Evaluation of Usability of the NIDDK Web Site
Description of the Research Issue
The primary mission of the NIH (and the NIDDK) is to provide funding for medical
research. Approximately 80% of the NIH’s Congressionally allocated funds are used to finance
extramural research and related services, whereas approximately 10% supports intramural
research. Although the Institute’s mission emphasizes research funding, its web presence needs
to embrace the larger mission of informing every stakeholder.
Traditional web usability studies generally focus on commercial B2C sites. The primary
reason for this is because commercial entities have a strong interest in making sure they can
maximize their profits by providing customers with a very intuitive and pleasing experience
using their information system. Not-for-profit sites, however, often receive less research
attention with regard to usability. One explanation for this lack of research may be because a
non-profit’s goods and services are not offered to the consumer based on a business model of
maximizing profit. In addition, these organizations often do not have substantial human or
financial capital resources to ensure that their web presence has high usability. Recent
NIDDK Web Site Usability Analysis
Page 12
legislation, however, has recognized the need for the general public to be able to easily access
government information via the Internet.
For example, on December 17, 1999, President Clinton released "Electronic
Government," a memorandum for the heads of executive departments and agencies. The
directive of this memorandum was for executive departments and agencies, in conjunction with
the private sector as appropriate, to help citizens gain one-stop access to existing government
information and services, and to provide better, more efficient, government services and
increased government accountability. The President felt that:
• The public should be able to access government information by the type of service or
information that people may be seeking; and
• Providing information at an agency level might be burdensome since the public would
need to know which agency provides the service he or she needs.
In addition, the E-Government Act of 2002 established the Interagency Committee on
Government Information (ICGI). As a result, the ICGI formed a Content Management
Workgroup to make recommendations on issues related to federal public web sites (ICGA,
2004). The ICGA recommendations include:
1. Citizens must be able to identify official government web sites and trust that those web
sites will provide current and accurate government information;
2. Federal public web sites must be written and organized from the audiences’ point of
view;
3. Federal public web sites must be designed and written to ensure they are easy to access
and use;
4. To promote seamless government, federal organizations must work to simplify and unify
NIDDK Web Site Usability Analysis
Page 13
information across the government;
5. Federal organizations must establish priorities and a schedule for posting content on their
public web sites;
6. Organizations must continue to comply with existing federal laws, regulations, and
policies; and
7. Developing web content policies and requirements for federal public web sites is an
ongoing process, requiring structure (ICGA, 2004, p. 1)
As a result of this and other government initiatives, government web sites in particular
are being held more accountable for the information they present, including how easily
information can be accessed by the general public.
Method and Data Collection Approach for Conducting the Evaluation
In addition to reviewing relevant literature, the INFM 702 team used the following three
processes to conduct the evaluation of the NIDDK web site.
1. Heuristic Evaluations
2. Persona Development
3. Competitive Analysis
The team conducted three heuristic evaluations that sought to quantify subjective
evaluations using specific criteria. Our first two evaluations were based on established heuristics
created by Jakob Nielsen (Nielsen, 2000) and Ben Shneiderman (Shneiderman & Plaisant, 2004).
We found limitations in the first two standards and decided to create our own third heuristic
based on the Four Learning Styles (Jester, 2000). All heuristic criteria were rated by all three
members of the team using a 1 (poor) to 5 (excellent) scale.
NIDDK Web Site Usability Analysis
Page 14
Second, we created detailed personas based on the stakeholders that we felt the site was
visited by. We decided to create personas that reflected the web page traffic data provided by the
NIDDK:
Table 1.0 – NIDDK Unique Web Page Visits Per Month
We quickly noticed that out of the 1.5 million average monthly page visits, approximately eighty
percent of customers seek out health information. By contrast, only seven percent seek out
research funding information. In addition, even fewer people visit the other sections of the site,
including the intramural research pages, the reports, and the clinical trials information pages.
The final approach to our methodology included conducting a competitive analysis of
similar web sites. We felt that by seeing the good, the bad, and the ugly of similar sites, we
might therefore be able to provide some rich information to the NIDDK regarding best practices
available to them. We decided to briefly review three sites using the Nielsen heuristic: the
National Cancer Institute (NCI), the National Institute of Child Health and Human Development
(NICHD), and the National Science Foundation (NSF). The first two sites are sister NIH
NIDDK Web Site Usability Analysis
Page 15
Institutes and the latter is a peer site that provides a similar function of conducting and
supporting research related to its science focus.
Underlying Assumptions
Our team stepped into this project with the assumption that we could provide feedback to
the NIDDK, unhindered by any limitations. Two of the team members approached this
evaluation knowing little about the National Institutes of Health (NIH) or its mission, so we felt
that we could therefore be able to provide a relatively objective evaluation, specifically
representing the views of the general public stakeholder.
The NIDDK wisely set up an oversight committee to oversee its redesign effort. This
committee established both a redesign subgroup and a content subgroup with the expressed
intent to focus on issues of technology selections for the new site and to focus on how to
organize, present and manage content. Although some thought has been given to the information
architecture of the site, our team feels that the committee members may be less informed
regarding standards for web design, usability, and user experience.
Analysis Techniques Used and Justification
Heuristic Evaluations
The team adapted the underlying criteria of all of the heuristics by adding specific
descriptors to them. The purpose of adding detailed and relevant descriptions for the heuristics
was so that we could provide a richer NIDDK evaluation. Nielsen (1996) recommends that
multiple persons review the same site for usability. Therefore, all three team members evaluated
the NIDDK web site using the three heuristics.
NIDDK Web Site Usability Analysis
Page 16
The first heuristic the group used was Jakob Nielsen’s 10 Heuristics of Usability
(Nielsen, 2000). Jakob Nielsen is an acknowledged expert in information system usability,
supported by his extensive collection of research on the issue. His ten heuristics provide a
common standard for measuring the usability of the NIDDK web site. Second, the team decided
to use Ben Shneiderman’s heuristic entitled “The 8 Golden Rules of Interface Design” for our
evaluation (Shneiderman & Plaisant, 2004). Since Dr. Shneiderman is considered an expert in
Human Computer Interaction (HCI) research, the group felt that his rules of thumb would
provide us with additional criteria with which to evaluate the NIDDK site.
Our team determined that there were some limitations to the previous two heuristics.
Both focus on information access and usability, but neither specifically focuses on the learning
styles of individuals. Universal standards of usability may not translate to all persons if these
people learn differently. Therefore, in order to address this deficiency, the group decided to
adapt the 4 Learning Styles (Jester, 2000) into a new usability heuristic. This third heuristic will
be described in greater detail in the next section.
Personas
In order to develop consistent and relevant personas, the group agreed upon certain
criteria to create attributes and characteristics that need to be included while developing the
personas. Each group member created at least two personas, with a total of seven personas.
Competitive Analysis
Another way to make sure that the redesigned site is as good as it can be is to look at
other related sites to find best practices. The team picked three related peer web sites to conduct
a competitive analysis. Each group member did an evaluation of the usability of the web site
compared to the NIDDK and submitted findings. Ratings were based on Nielsen’s heuristic.
NIDDK Web Site Usability Analysis
Page 17
Development and Pre-testing of Data Collection Instruments
A heuristic evaluation is the most general of the usability inspection methods and is also
the easiest to learn and apply. Unfortunately, heuristic evaluation itself does not provide a
systematic way to generate fixes to the usability problems or provide a way to assess the
probable quality of any redesigns. However, because heuristic evaluations aim at explaining
each observed usability problem with reference to established usability principles, it will often be
fairly easy to generate a revised design according to the guidelines provided by the violated
principle for good interactive systems. Also, many usability problems have fairly obvious fixes
as soon as they have been identified (Nielsen, 2000).
In addition to reviewing NIDDK-provided survey results of NIDDK employees and the
general public’s opinion of the current site, the team applied three usability heuristics to the
NIDDK web site. The main purpose of conducting the web site usability assessment using the
three heuristics is to provide an objective third party evaluations and recommendations to the
NIDDK redesign committee. The three heuristics are:
1. Jakob Nielsen’s 10 Heuristics for Web site evaluation
2. Ben Shneiderman’s 8 Golden Rules for Interface Design
3. Guidelines for Usability Based on the 4 Learning Styles
The first two heuristics the team used were already developed. The team developed a
third heuristic using the 4 learning styles (Jester, 2000). The reason the first two heuristics were
not sufficient is because the two focus on information access and usability but do not consider
learning styles of customers. Since the NIDDK web site users have varying levels of expertise in
the topic on which information is sought; they also vary in their learning styles. The task of
searching for information has very different requirements to those who are looking for health
NIDDK Web Site Usability Analysis
Page 18
information and for those who are looking for research grants (from an individual's perspective).
Just as the team did for the previous two heuristics, our adaptation of the 4 Learning Styles
meant that we added our own descriptions for each criterion. Please see the Appendix for
specific descriptions. Jester (2000) defines the 4 Learning Styles as:
• Visual/Verbal Learning Style – This person learns best when information is
presented visually and in a written language format. This learner likes to write down information
that is seen, often in summary fashion, as a way to aid retention.
• Visual/Nonverbal Learning Style – This person learns best when information is
presented visually and in a picture or design format. This person likes to see images, charts,
graphs and diagrams to better visualize the information that is being represented. This learner
may draw pictures or color code test as a way to aid retention.
• Tactile/Kinesthetic – This person learns best when physically engaged in a "hands
on" activity. This person is an experiential learner who often uses flashcards and “experiences”
information using alternate media choices (visual, audio) to aid retention.
• Auditory/Verbal Learner – This person learns best when information is presented
auditory in an oral language format. This person benefits from speaking to and listening to
others and audio files or using a screen reader would greatly aid this learner’s retention of the
presented information.
Data Collection, Analysis & Findings
Heuristic Evaluations
The main data collection instrument used was a heuristic evaluation data collection table.
This table included 2 – 5 descriptions of the evaluation criterion for each heuristic used (see
Appendix to see specific evaluation criteria). The web sites were evaluated in terms of each
NIDDK Web Site Usability Analysis
Page 19
description using a range from 1 to 5. The rating used was 1 (poor); 2 (needs improvement); 3
(good); 4 (better than average); 5 (excellent). Table 2 provides a representative sample of one
criteria selected, along with specific descriptors of each. Each member of the team evaluated the
site using these descriptors.
Table 2.0 - Heuristic Evaluation Data Collection Table
The data collection table also had columns for subjective comments and recommendations for
criteria that scored below three. Completed data collection tables are found in the appendix. Our
team performed a quantitative data analysis by averaging the three individual ratings for each
description of each heuristic and then averaging the overall rating for the heuristic. In addition,
we performed a qualitative data analysis by compiling comments for criteria that scored below
three (these comments can be found in detailed tables, located in the Appendix). Below are the
average ratings for the three heuristics.
NIDDK Web Site Usability Analysis
Page 20
Table 3.0 - Nielsen’s 10 Usability Heuristics
The overall average rating for Nielsen’s 10 Heuristics for the NIDDK web site was 3.30
using a rating scale of 1 (poor) to 5 (excellent).
Table 4.0 - Shneiderman’s Eight Golden Rules of Interface Design
The overall average rating for Shneiderman’s Eight Golden Rules for the NIDDK web
site was 3.95 using a rating scale of 1 (poor) to 5 (excellent).
Table 5.0 - Guidelines for Usability Based on the Four Learning Styles
NIDDK Web Site Usability Analysis
Page 21
The overall average rating for Guidelines for Usability Based on the Four Learning Styles
for the NIDDK web site was 2.33 using a rating scale of 1 (poor) to 5 (excellent).
Persona Development
In addition to providing both qualitative and quantitative heuristic evaluations, our group
created personas that would reflect that customers served by the NIDDK web site. It is widely
regarded that if representative personas can be created and used, developers are more likely to
create a usable site for all customers. Seven personas consisting of names, photos, and brief bios
were created so that the NIDDK could redesign its site with specific customers in mind. The
personas represents behavior patterns that will help the NIDDK redesign committee understand
the goals, motivations, and behaviors of the people who will use the NIDDK web site and how
the interface will fit into meeting those needs. The purpose of this exercise was to help explore
the NIDDK web site’s target customers. One current concern is that the primary members of the
web redesign working groups might focus on specific audiences such as users who are looking
for research funding rather than considering the many types of people who might visit the site for
information. However, our team feels that if these personas are considered, the redesign team
might successfully make the site usable for all customers. Below is a listing of the seven created
personas:
Dr. Steven Arias
Dr. Arias is in desperate need of a grant through RFA. He is 50 years
old and is worried about losing his job. He has already had to fire his
technician. Dr. Arias has tried for RFA’s in the past, but has been unscored
NIDDK Web Site Usability Analysis
Page 22
three times. He needs to speak to someone in his field, and find other initiatives he can apply
for. Before applying for funding again. Dr. Arias hopes he can get in touch with one of the
NIDDK program staff to get some ideas on how he might improve his application.
Not knowing much about Internet search, he used google.com and stumbled onto the
NIDDK web site. On the site, he first looked at the FAQs to check for existing solutions. Then
he viewed the site map and clicked the big headings for clues. Finally, he used the contact form
to submit his inquiry for assistance.
Dr. Arias has a x486 pc running Windows NT. Since he is completing his sabbatical, he
is working from a remote location and only has dial up access. He has not spent much time
“surfing” the web. He has little patience, due to his desperate situation. He has to be able to get
answers to his questions quickly.
Dr. Arias lives in Atlanta, Georgia with his wife of 30 years Susan. He loves to play golf
in his spare time.
Dr. Helmut Heisenberg
Dr. Heisenberg is in his 60’s, with significant experience in cancer
research, has spent the last 20 years at NCI. Dr. Heisenberg is well written,
having published four papers in the last year. He is a member of the National
Academy of Sciences.
During his recent research, Dr. Heisenberg made an interesting finding that he feels may
apply to diabetes research. He has no idea what areas in diabetes are “hot” topics now, so he
needs guidance.
Dr. Heisenberg would like to hire three new post-docs to work with him in this new area.
NIDDK Web Site Usability Analysis
Page 23
The doctor normally has his team do his Internet research for him, so his web skills are not
strong. But, he has the ambition and desire to find what he is looking for. He uses a Macintosh
running OS X at his office on the NIH Campus in Bethesda, MD.
Dr. Heisenberg lives in Bethesda, Maryland with his wife Marilyn. Both have had what
could be considered by all means a good marriage. And in their 35 years of marriage, they have
traveled to over 30 countries throughout the world.
Because of his longstanding career with the NIH, Dr. Heisenberg knows that the NIDDK
web site must exist, but he is not particularly skilled in Internet searching. However, his
enthusiasm about his new discovery would not allow him wait for his team to do the search for
him. He spends part of his time looking at the NIDDK web site from his office, using a high-
speed connection on his Macintosh.
He wants to browse the NIDDK web site to look for new research findings in Diabetes.
On the NIDDK web site, he is first drawn to the following headings and functions: Diabetes,
What’s New, Highlights, and the search function. He is primarily interested in seeing the
recency of information found. He would desire graphic representations of certain things.
Although Dr. Heisenberg is still very strong and agile, his vision has started to fail and so he
would most certainly prefer the texts in bold formats with well-contrasted colors. And because
of his limitations in the use of the internet, he will be very frustrated if he has to click several
times deeper in other to get specific information.
NIDDK Web Site Usability Analysis
Page 24
Seth Warren
Mr. Warren works in the construction business out in a rural setting
about 50 miles west of Chicago, IL. He is 37 years old, married to his high
school sweetheart, Lisa, and has two children (Daniel, 10, Becky, 8). He
stumbled across the NIDDK site after searching for “nih diabetes obesity” on the Yahoo! Search
page. He is particularly interested to learn more about diabetes and its affects on the elderly
since his mother now lives with the family due to complications with diabetes.
Mr. Warren doesn’t currently have a computer at home, although his children would like
to get a new computer. Instead, he must travel to the nearby county library to access a Pentium
II PC with Windows 98 with a high-speed connection. He spends very little time on the Internet,
generally limiting it to checking email once a week or so.
Dr. Elizabeth Marquette
Dr. Marquette is 42 years old and has been with the NIDDK as a
program staffer for 6 years. She has been with the Institute for 10 years,
previously working in the Review section. Dr. Marquette is a single mother
of two, commuting to her Bethesda office from Frederick, Maryland.
She spends a great deal of time on the NIDDK web site, but needs to
find information very quickly. Her day is fast-paced to say the least.
During her research on the NIDDK web site, she needs to be able to find out who is the
Urology PD, for example. She can never remember the P01 cap, so this information needs to be
NIDDK Web Site Usability Analysis
Page 25
readily available. Also, she knows that there is something new having to do with animal sharing,
but has no idea where this information is located.
Because Dr. Marquette is often on the go either attending meetings and conferences or
tending to her parental responsibilities, she tends to spend some of her available time responding
to emails and doing some web research using her Blackberry and mobile phone. In addition,
because of occasional mobile interference and the very small screen size, she wants to be able to
locate needed information quickly. Dr. Marquette is quite web-savvy, with a background in
designing a previous version of the NIDDK web site.
Jenny Spinova
Jenny is a fifth grader with two younger brothers (Jaime -
6, Elias – 4). Her parents were killed on 9/11, so she and her
brothers now live with her grandmother (mom’s side) –
Hermosa, who is 67 years old. They live in Hoboken, New
Jersey.
Jenny is a very bright student. She is currently enrolled in a magnet school where she
excels in science and math and she enjoys spending time in the computer lab. She primarily
speaks Spanish at home with her grandmother, but she is fluent in both Spanish and English.
Since it is national nutrition month, Jenny’s teacher has asked that her students write a
report on steps that both old and young can take to remain healthy. Since Jenny’s grandfather
(dad’s side) works for the NIDDK, he suggested she look at the site to find some good
information. Unfortunately, Jenny’s grandfather does not use the site too much, but he knows
NIDDK Web Site Usability Analysis
Page 26
she must be able to find some information. He is at least able to give her the address of the home
page.
Fortunately, Jenny is relatively skilled at navigating the Internet, so she finds the site
while at school, using a pc with Windows XP and a high-speed connection. She does not know
how to use the search function, but instead, tends to browse.
Unfortunately, the home page has lots of text with little graphics, so she becomes
somewhat intimidated by all the links. She decides to click through a couple of links, but finds
that some of the words and abbreviations are confusing. She eventually finds some pages in
Spanish, but they appear to primarily provide information on food recipes. She eventually gives
up her search.
Sally Smith
Sally Smith, originally from the DC metro area, is a 25 year old
graduate student at the University of Pennsylvania, studying the effects of
diabetes on kidney function.
Since she will be graduating in May, Sally needs to find information on pre-doctoral
scholarships and where to find the pre-doctoral scholarship application forms. She logs on to the
National Institutes of Health web site (http://www.nih.gov) and then looks under the grants
section (http://grants1.nih.gov/grants). She then finds the link for the NIDDK web site.
Sally is overwhelmed by this process and would really like to talk to someone in person.
Up to this point, she doesn’t have a mentor to guide her. Sally’s goal is to make sure she has all
the right information to complete the scholarship application accurately. After Sally downloads
NIDDK Web Site Usability Analysis
Page 27
the scholarship application, she will be looking for contact information so she can reach some
one in person.
Sally’s use of the Internet is second nature; she is never more comfortable than with a
mouse in her hand. She has a high-speed cable connection and uses a Mac.
Joann Bentley, Ph.D.
Dr. Joann Bentley is a 32 year old female that has just been hired as
an Assistant Professor of Biology at University of Maryland, College Park.
She found out about the NIDDK web site through her colleagues and she
needs to find information on appropriate grants, and really doesn’t know
where to begin. “What is NIDDK looking for?” is the question that runs through her mind
constantly as she searches on the web site.
Dr. Bentley needs to be directed to information on career awards, including eligibility,
DK-specific information, application forms and deadlines. She needs basic research grant
information, but doesn’t know that is what it’s called. She also needs to find a current list of
funding initiatives. Dr. Bentley is confused because she is not familiar with the terminology that
she is finding on the NIDDK web site. Quite frankly, she is unfamiliar with the whole federal
grants process, and is more uninformed about the NIH grants process.
Dr. Bentley feels confident with her web navigation abilities; she uses the computer in
her office that is a high-speed Windows machine to do this research. This is the first time she
has needed to find these kinds of information, and may be more comfortable if there was a
NIDDK Web Site Usability Analysis
Page 28
person she could talk to directly. She has no idea how to contact someone who can guide her in
the right direction.
Competitive Analysis
In addition to providing the three heuristic evaluations and creating seven relevant
personas, the team felt it would be useful to conduct a competitive analysis. Federal agencies
can often become insulated, not knowing what other agencies may be doing or keeping up with
current best practices in web development. Therefore, our team looked at three sister sites to see
if there were best practices that might be followed.
Our competitive analysis essentially looked at the high-level usability issues with each
site, rating each using the Nielsen heuristic applied earlier to the NIDDK site. Our team
provided usability scores (as shown on Chart 1 below) for three federal government web sites
and contrasted these evaluations with the previously rated NIDDK evaluation (see the Appendix
for more detailed quantitative score attributes). The National Institute of Health and Child
Development (NIHCD) and the National Cancer institute (NCI) are other NIH Institutes and the
National Science Foundation (NSF) is a peer agency that also conducts and supports scientific
research.
Chart 1.0 - Competitive Analysis
NIDDK Web Site Usability Analysis
Page 29
Due to the limitations in time, the competitive analysis evaluations were not very
detailed. Only a first level analysis which is meant to draw a quick comparison to that of the
NIDDK web site was done. Therefore only a quick overview of the comparison is presented
below:
The National Institute of Health and Child Development (NIHCD)
- Unlike the NIDDK and its inconsistent navigation (specifically between the main pages,
the intramural site, and the health information sites), the user can identify visibly where
he or she is on each page.
- The styles, fonts and headings were standard from page to page.
NIDDK Web Site Usability Analysis
Page 30
- The logo, contact information, and links to the home page are in the same location across
each page.
- The site map is well defined and detailed information is provided.
The NIHCD web site probably included the least amount of information on its site, but its
simplicity was probably its greatest strength. The site was rated slightly higher than NIDDK,
primarily because it used consistent navigation items throughout the site. The site also reflected
Nielsen’s standard in terms of aesthetic and minimalist design. Overall, this site was rated
slightly above average, on par with the current NIDDK site.
The National Cancer institute (NCI)
- Unlike the NIDDK site the NCI site had standardized task sequences
NIDDK Web Site Usability Analysis
Page 31
- The NCI site ensured that embedded links are descriptive. The NIDDK site had no
descriptions on any of its links on the homepage.
- The NCI site used unique and descriptive headings
- The NCI site made good use of thumbnail images to preview larger images.
Unfortunately, the NIDDK site homepage did not have any images to break up text other
than its logo.
Besides being an award winning web site for usability, the NCI site rated extremely high,
in fact higher than all other compared sites, using Nielsen’s 10 heuristics. The NCI web site also
certainly made better use of powerful images and pictures to aid users’ recognition and
understanding of the NCI’s mission. It was also perceived that the NCI site probably employed
more detailed personas by the simple designs, pictures of families etc. Unfortunately, the NCI
site, like all the rest compared, did not pass the W3C validation test.
The National Science Foundation (NSF)
NIDDK Web Site Usability Analysis
Page 32
- The NSF site has a search function that allows users to search by various content types
such as a) news, b) discoveries, c) funding, d) publications, and e) awards. Every other
site that the team evaluated did not employ this method of search.
- The NSF site has consistent navigation, as well as look and feel throughout the site.
- The site uses breadcrumbs very well so users know where they are at all times.
- The site allows users to search for information by subject area, by NSF division and by
customer type. This site was very flexible in its approach to providing multiple ways for
users to locate information.
Overall, the NSF site uses easy to understand terminology and provides a great mix of
text, graphics, and media for different learning types. This site took great pains to make sure that
information appealed to various learning styles, was accessible by various stakeholder types, and
was easy to navigate. This site also had a substantial help section. We would consider this to be
a model site for the NIDDK redesign team to consider.
Summary/ Conclusion
Findings and Conclusions
The overall findings of our usability evaluation indicate that the current site was rated
better than average in terms of its usability. However, our team feels that the site could be
redesigned for even greater usability. One of our key findings suggests that the NIDDK site may
have been unintentionally too focused on its research funding audience, even though these
people only constituted approximately seven percent of the site’s visitors.
Therefore, our team feels that the redesign must focus on each stakeholder equally,
considering the learning styles and information needs of each customer.
NIDDK Web Site Usability Analysis
Page 33
Identified Limitations of the Study
The primary limitation of this project was that our team only had six weeks to conduct
this usability study. As a result, only three persons conducted the analysis without having the
time to conduct a formal usability study (usability lab, videotape, etc.). In addition, personas
were created without the convenience of being able to interview persons to identify persona
types. The seven personas were created as a result of both NIDDK staff feedback and the team’s
own ideas of the Institute’s target customers.
Another limitation resulting from the short time frame was that the INFM 702 team only
had time to review high-level pages. There simply was little time to look into deeper levels of
the site.
Finally, the team was provided with results of a public web survey of the NIDDK site.
This survey provided an overall customer satisfaction score that was very high – even higher
when compared to customer satisfaction with NIH sites as a whole as well as federal government
sites as a whole. Time again limited us because we simply did not have time to determine why
our usability results seemed to conflict with the public web survey results (public responded with
83% satisfaction with NIDDK site – better than NIH or federal sites in general).
Recommendations
As a result of this usability assessment, our team has a few recommendations to make the
NIDDK site better. We feel that these recommendations could not happen at a better time since
the NIDDK is in the beginning stages of its site redesign process and good information
architecture must precede the implementation of a content management system.
Our first recommendation simply asks the NIDDK to consider using some of the results
of our usability study. For instance, we used initial feedback from NIDDK staff and created
NIDDK Web Site Usability Analysis
Page 34
detailed personas, representative of the various stakeholders that might visit the NIDDK site.
We ask that these personas be referred to frequently when attending both content and redesign
subcommittee meetings. It is only by this constant reminder that the NIDDK redesign staff
makes sure that the redesigned site meets its usability objectives for its intended audience.
Our team purposefully limited the number of NIDDK staff personas and did not consider
the results of a survey of NIDDK employees, simply because the ICGI recommends that Internet
sites are for the public, not for employees (Interagency Commission on Government Information,
2004). Instead, the NIDDK should consider making its Intranet presence much more robust to
meet the needs of its employees.
In general, the INFM 702 team recommends the following Information Architecture
components:
• Create a consistent interface across all pages (including the health information and
intramural sites);
• Organize text groupings using easily understandable bite-sized words;
• Make better use of white space;
• Break up text with useful images (see NCI and NSF sites for best practices), graphics,
and media that might more easily communicate to persons with different learning styles;
• Focus on search and “findability” by spending time creating metadata to aid in “best
bets,” classification, search, and retrieval;
• Change the “Quick Links for Investigators” drop-down so that it instead provides quick
links for various stakeholder types, minimally including a) New Investigators, b) Current
Awardees, c) Press/Media, and the d) General Public.
NIDDK Web Site Usability Analysis
Page 35
• In addition to a site map, allow users to find in multiple ways, including by browsing
subject categories (A-Z) and possibly including a definitions/acronym/abbreviations
page; and
• Apply content ownership procedures by posting the date content was posted to the site.
In addition, include the content owner name and contact details so customers can ask
questions directly.
Lastly, we recommend that the NIDDK consider revising or expanding its mission so that its
focus moves from simply conducting and supporting research to also meeting the needs of all of
its Internet patrons equally by providing information that is easily accessible and understandable.
Future Research Directions
Finally, our team feels that there is a lot of usability research that has been largely
unexplored. For example, although much research has focused on the usability of commercial
systems, only within the last half dozen or so years have research efforts focused on the usability
of non-profit sites, including government Internet sites.
Although Internet sites are used to inform customers and should offer organization,
simplicity, and “findability,” sites are offering richer interactive experiences than ever before.
Our team argues that usability is related to personal learning styles. Therefore, future studies
should focus on this relatedness.
Furthermore, usability cannot be universally measured, in part because it can be
considered both an art and a science. Therefore, we recommend that future studies focus on
cultural and international differences with respect to usability as Internet sites focus more on
their expanding multicultural and international customer base.
NIDDK Web Site Usability Analysis
Page 36
References
Agnvall, E. (June 21, 2005). Sites for sore eyes (legs, etc.); Ads, lack of disclosure, missing data
compromise even the best consumer health web sites, new ratings show [FINAL Edition].
Washington, DC: The Washington Post, p. F.01.
Barr, S. (October 28, 2003). HHS web guidelines created with designers, users in mind.
Washington, DC: The Washington Post, p. B.02.
Forster, S. (Oct 29, 2002). Consumers using wrong criteria to evaluate medical web sites. The
Wall Street Journal Online. New York: Wall Street Journal (Eastern edition), p. D.3.
Interagency Committee on Government Information. (June, 2004). Recommended policies and
guidelines for federal public web sites: Final report of the Interagency Committee on
Government Information. Office of Management and Budget.
Jester, C. (2000). The four learning styles. Pleasant Hill, CA: Diablo Valley College. Accessed
August 2005 at: http://www.metamath.com/lsweb/dvclearn.htm.
National Institute of Diabetes and Digestive and Kidney Diseases. Bethesda, MD. Accessed
August, 2005 at: http://www.niddk.nih.gov.
Nielsen, J. (2000). Designing Web Usability. Indianapolis: New Riders.
Nielsen, J. (1994). Heuristic evaluation. In Nielsen, J., and Mack, R.L. (Eds.), Usability
Inspection Methods. New York: John Wiley.
Shneiderman, B. and Plaisant, C. (2004). Designing the User Interface, 4th ed. New York:
Addison Wesley.
Siegel, E. R. and Wood, F. B. (2003). The National Library of Medicine’s strategy for assessing
the impacts of health information web sites. Information Services & Use, 23, 227-234.
NIDDK Web Site Usability Analysis
Page 37
Sillence, E., Briggs, P., and Fishwick, L. (May 2005). Guidelines for developing trust in health
web sites. WWW 2005, 1026-1027.
NIDDK Web Site Usability Analysis
Page 38
Appendix 1: Nielsen’s 10 Usability Heuristics (NIDDK)
NIDDK Web Site Usability Analysis
Page 39
Appendix 1: Nielsen’s 10 Usability Heuristics (NIDDK) – Continued
NIDDK Web Site Usability Analysis
Page 40
Appendix 2: Shneiderman’s Eight Golden Rules of Interface Design (NIDDK)
NIDDK Web Site Usability Analysis
Page 41
Appendix 3: Guidelines for Usability Based on the Four Learning Styles (NIDDK)
NIDDK Web Site Usability Analysis
Page 42
Appendix 4: Competitive Analysis Using Nielsen’s 10 Usability Heuristics
NIDDK Web Site Usability Analysis
Page 43
Appendix 4: Competitive Analysis Using Nielsen’s 10 Usability Heuristics - Continued
NIDDK Web Site Usability Analysis
Page 44
Appendix 5: NIDDK Web Site Main Pages
NIDDK Home Page
NIDDK Web Site Usability Analysis
Page 45
Appendix 5: NIDDK Web Site Main Pages
Welcome Page
NIDDK Web Site Usability Analysis
Page 46
Appendix 5: NIDDK Web Site Main Pages
Health Information Page
NIDDK Web Site Usability Analysis
Page 47
Appendix 5: NIDDK Web Site Main Pages
Research Funding Opportunities Page
NIDDK Web Site Usability Analysis
Page 48
Appendix 5: NIDDK Web Site Main Pages
Clinical Research Page
NIDDK Web Site Usability Analysis
Page 49
Appendix 5: NIDDK Web Site Main Pages
NIDDK Laboratories Page
NIDDK Web Site Usability Analysis
Page 50
Appendix 5: NIDDK Web Site Main Pages
Reports, Testimony, and Plans Page