7

Usability Testing of the Virtual Data Center

Jessica Hovater, M. Krot, D. L. KiskisUniversity of Michigan

1301 Beal Ave.Ann Arbor, MI 48109-2122

{jhovater, krot, dlk}@umich.edu

Ilona Holland and Micah AltmanHarvard University

G-4 Littauer Center, North YardCambridge, MA 02138

Micah_Altman@Harvard.edu

ABSTRACTIn this paper we present an overview of the usability issuesrelated to the Virtual Data Center (VDC) digital library.Usability testing and heuristics were an integral part of thedevelopment process from the inception of the project andthroughout its evolution. The developers of the VDC haveused the findings from usability evaluations to identifyseveral prominent usability issues present within the digitallibrary. Test results continue to exert a positive andvaluable influence on both system redesign and theimplementation of new features.

KeywordsUsability, digital library, open source, distributed systems,World Wide Web

INTRODUCTIONThe Virtual Data Center (VDC) is an open-source, web-based digital library for the management and disseminationof social science research data. It is being developed as acollaborative effort between researchers at the Harvard-MIT Data Center (HMDC) at Harvard University and theUniversity of Michigan’s School of Information andCollege of Engineering (<http://TheData.ORG>). TheVDC is designed to facilitate the retrieval anddissemination of quantitative social science research data.It provides support for distributed and federated collectionsof data through features such as distributed security andfederated metadata harvesting [1]. It also provides tools fornumerical and graphical data analysis. The VDC usesXML metadata specificiations created by the DataDocumentation Initiative (DDI), which has its roots atInter-university Consortium for Political and SocialResearch (<http://www.icpsr.umich.edu>). By using thiswidely accepted specification, the VDC will allow librariesand data centers around the world to share collections ofdata with one another.

The usability issues present with the VDC are three-fold.First, the VDC uses all open-source software (OSS) and isitself open source. The maintainers and installers of

OSS have a notorious reputation for presupposing thatusers have high levels of technical expertise [2]. There arefew examples of usability techniques being applied to opensource projects, with the Gnome Usability Project beingone notable exception [3]. Second, the VDC is web-based,carrying with it a number of usability constraints not facedby non-web applications. Finally, the VDC is a digitallibrary. With more and more students and researchersturning to online resources rather than physical libraries toconduct their work, the importance of usable digitallibraries has risen significantly. Digital libraries, ifpresented in a usable way, can greatly enhance theproductivity of researchers by making a much largeramount of data accessible and by providing extendedfunctionality such as data analysis and download. Yet thisgreater power adds complexity to user interfaces, and bestpractices in this area are still in early development.

The goal of the VDC is increased data access and researchefficiency, and the development team has put a renewedfocus on the usability testing and enhancement of thesystem. We conducted user-needs analyses using focusgroups and surveys before commencing the project. Afterdeveloping the software to a sufficient level to meet theseneeds, the development team once again returned tousability studies to assess the progress of the VDC in adeliberate attempt to improve the usability of the system forthe next release. Areas of particular focus include improveddocumentation, navigability, metadata presentation, andsearching functionality.

INITIAL USABILITY WORK ON THE VDCIn the year 2000, the VDC development team at HarvardUniversity conducted several tests to gain preliminaryfeedback on the usability of the HMDC website, aprecursor of the VDC. While the workflow, technology,and processes were quite different than the (then) proposedVDC, the studies were important for gathering informationabout how users conducted research and how the VDCmight solve existing problems.

Focus GroupsIn January 2000, the team conducted focus groups withstudents from a digital library design class at Harvard. Theoverarching theme that could be derived from results ofthese focus groups was that the cognitive time necessary to

8

complete a task was orders of magnitude greater than thecomputation time necessary. This was illustrated by givingthe same task to an experienced research assistant and tostudents from the focus group, considered to be less-experienced, “average” users. The task that was assignedwas to find the percentage of women who had voted forClinton for President in 1996. The research assistant foundthe answer from the system in less than five minutes byfinding the data in the related study and running a cross-tabulation analysis in the system. However, only onestudent from the focus group was able to find the answer,and it took this student several hours to do so.

FindingsStudents from the focus group cited several usabilityproblems as being the reason why this simple task was sodifficult to complete. First, the language in the system wasspecialized to people who had experience with this type ofresearch data, making it difficult for the average user torecognize what they were looking for. Second, thetypology of resources available through the system wasunfamiliar to the users. Users were not familiar withfundamental differences between studies collected usingdifferent research methods. For example, the data thatprovided the answer to the question was most likely to befound in survey data but not in census data. Withoutsufficient knowledge of the collection and what existedthere, the users were not able to make educated andefficient choices about where to search for the necessarydata. Third, incomplete documentation kept the users fromlearning all that was available on the system. Sinceinformation about cross tabulations was available only in along “Frequently Asked Questions” document, many usersdid not realize that they could run such analyses on thesystem. Due to these usability burdens, the task was verydifficult for the users to complete even though the actualprocess for finding the answer was not time-consuming.

User SurveysIn November 2000, the HMDC, jointly with the HarvardGraduate School of Education, conducted extensive userstudies evaluating the existing version of the HMDCwebsite. The goals of these studies were to identify:

• Current patterns of use among those seeking data• Qualities of other sites that users find helpful• Traits of users seeking data• Usability issues within the (then) existing website

For these studies, team members conducted hour-longinterviews and questionnaires of 53 advanced researchers,professors, and graduate students with online researchexperience.

FindingsThe team members conducting the surveys reported threemain findings that resulted from these studies:

• Users are increasingly conducting data researchvia the web instead of from the physical library. In

fact, most of those surveyed conducted researchfor data primarily via the web.

• In digital libraries, users want less navigation infavor of a more direct route to the data.

• Users want raw data instead of an author’sinterpretation of that data.

This study showed the increasing importance of digitallibraries as more researchers turn to online resources toconduct their research. These online libraries provideubiquitous access to data and can also provide advancedsupport for data analysis. Interestingly, the subjects inthese studies who listed the library as their primaryresource rather than online resources stated that theavailability of reference help was the main reason that theycontinued to do research at a library. This resultunderscores the importance of developing usable digitallibraries if they are to truly enhance research and dataaccessibility. The digital library must, in essence, do thejob of not only the library, but of the reference librarian, aswell.

A minimalist approach to usable interfaces seemed to bepreferred by the users, who decried the use of frames andextensive navigation and insisted on obvious and directroutes to the data. This means making the collections andthe searching mechanisms visible and easily accessible tothe user.

An advantage of digital libraries is the ability to displaydata to the user in a number of useful formats. Users ofthese studies appreciated graphs, dynamic charts,descriptive tables, and written summaries. However, theformat that was preferred the most was raw data,accompanied by bibliographic information documentinghow that data had been used in published research. Mostusers preferred to conduct detailed analysis with their owntools, and wanted the library to provide sufficient onlineanalysis functionality to explore the data and convert it to aform that was usable by standard statistical packages.

CURRENT USABILTY WORK WITH THE VDCIn January of 2002, we began a second iteration of usabilitytesting to evaluate the current VDC. First, we conductedcognitive walkthroughs so as to thoroughly assess theusability of the system ourselves, before presenting thesystem to test users. Several prominent usability issues inrelation to the functionality and navigability of the systemwere identified by these evaluations, and the VDCdevelopment team made a round of changes andenhancements. Several months later, with a stable andenhanced VDC in place, we conducted usability tests withtest users.

Cognitive WalkthroughsOur goal in conducting cognitive walkthroughs was toevaluate the system by methodically walking through thesystem as an actual user would. At each step in a process,we considered a series of questions to analyze the usability

9

at that specific point in the process. We questioned whetheran actual user would know what to do at each step, andwhether they would be presented with appropriate feedback[4]. We evaluated basic user tasks such as logging into thesystem, searching for a study, and performing extractionson data.

FindingsAfter completing the walkthroughs, several major andminor usability issues were evident: lack of documentation,unfamiliar language, and inefficient search functionality.Some of the language used in the VDC is specific to eitherthe VDC or to digital libraries in general (e.g. “collection”and “permissions”) and could be foreign to a user.Likewise, some features supported by the VDC wereconfusing and lacked explanatory documentation. Forthese reasons, we noted a definite need for helpdocumentation to define some of the language in thesystem and to instruct the user on how to complete certaintasks. As a result of this recommendation, the VDC teamdeveloped context-specific help documentation asrecommended by Nielsen [5] and others. Hyperlinksembedded in each page are linked to the help topicsspecific to that page. Through this mechanism thedevelopers attempted to first provide the most relevant helpdocumentation to the user, while also providing access tocomprehensive documentation. In addition to the helpsystem, we re-evaluated the documentation and directions,re-structured interfaces, and added concise in-line help toseveral interfaces.

The search interface reflected several usability issues, andin one case required the removal of a feature. Initially, theVDC supported a way for users to browse studies by authorname. Although this was useful in theory, walkthroughsdetermined that inadequate adherence to metadatastandards by the metadata content providers seriouslyhandicapped the ability of the VDC to provide this featurein a useful manner. The feature was subsequently disabled,although it remains available for small collections withhigh-quality metadata.

We felt that the search form did not match usersexpectations based on other search engines. The searchfunctionality within a digital library is crucial to theusefulness of the library and to the productive use of thelibrary by researchers. Therefore, we stressed theimportance of powerful and useful search features. As aresult of these recommendations, the VDC developmentteam implemented a “quick search” feature that workssimilar to Internet search engines such as Google™, whichallow users to type keywords into a single text box toperform a search across several fields. This featurefacilitates quick searching for users who may have moregeneral search criteria in mind. The existing search formthen became an “advanced search”. Users who know moreinformation about studies for which they are searching may

specify search terms that exist in the title or the author list,for example.

User TestingSeveral months after the cognitive walkthroughs andsubsequent modifications to the VDC, we conducted formalusability tests with five graduate students from the Schoolof Information at the University of Michigan. The purposeof these tests was to expose the VDC to novices whotypified potential users of the system, and to evaluate theirsuccess with using the system. The tests were conducted atthe University of Michigan Media Union UsabilityLaboratory, which facilitated the recording of the sessionby capturing the screen onto videotape along with the testuser’s voice. Test users were given a list of tasks toattempt and were instructed to work through the tasks asthoroughly as possible. While they worked, they verbalizedtheir thoughts and explained their reasons for acting uponthe system as they did. We evaluated the test results bynoting where test users became confused, what commonmistakes they made, and what routes through the systemthey took while attempting to complete each task. Thesetests were immensely helpful because they producedempirical evidence of how typical users will interact withthe system and what usability issues may cause problemsfor the user.

FindingsOur first finding was that the documentation still neededimprovement. Most of the test users had used other digitallibraries regularly but were still unfamiliar with some of theterms used in the VDC. Almost all of the test users wereconfused by the concept of a “collection”. They asked whatthe difference was between the collections and were notsure in which collection they should search for a study.Although there was some documentation about collections,the users noted that this documentation was vague and didnot help them to understand the concept of collections. Webelieve this is, in part, because the collection functionalityhas not yet been fully utilized in the production system.These findings are consistent with the findings from initialfocus groups and demonstrate that the VDC has to do evenmore to help users and curators conceptualize collections.

Although most of the users had experience in data analysis,they still had many questions about the process ofextractions and performing cross tabulations in the VDC.They felt that the minimal explanation on the pages was notsufficient for them to feel confident performing dataanalysis. Some users suggested providing different levelsof documentation for different levels of user proficiency.For example, this would allow experienced researchers toconduct powerful data analyses, while still supportingbeginners through the process of conducting analyses.

Several test users observed that the presentation ofmetadata about a study could be improved so as to facilitatebetter searching and browsing of studies. After searching

10

for a study, the users wanted to know more about the studythan the basic metadata shown on the results page, beforecommitting to downloading the data. Users said that theywould be interested in having the ability to browseadditional metadata, and particularly to see informationsuch as dates when the study was submitted to the VDC,the location where the study took place, or moreinformation about the subjects in the study. One solution isto allow users to control the level of detail presented andthe presentation format.

The users noted that the search functionality could beimproved further. They tried to perform more powerfulkeyword searches than the system supported and wereexpecting the search to be more like other search engineswith which they are familiar. We have since added a query-builder that supports powerful search functionality such asBoolean operators, phrase searching, and stop-words.

The majority of the users found that the search view thatallowed them to browse subject headings was the mostuseful. Unless they were trying to search for a study by aparticular name, they found it to be more helpful to be ableto browse a list of subjects, and then to search for studiesunder that subject.

FUTURE WORKThe VDC development team continues to enhance thedocumentation within the system, including bothdocumentation within the pages and the separate helppages. The team is creating a glossary of terms tosupplement the help documentation. We expect this to beparticularly valuable for the terms used in data analysis andmanipulation. For longer processes, such as performingdata analysis, users may benefit from a more systematic,step-by-step description of the process within the page.Furthermore, it might be useful to provide optional linkswithin the pages that would direct beginners to more basicand detailed documentation. The team has alsoimplemented a correspondence form within the system toallow the user to communicate questions or problems to asite administrator.

The development team continues to create more clear anddirect navigation. The system now includes a history barthat leaves a trail of all of the pages that the user has visitedto get to their current location, allowing the user tobacktrack as necessary. There is also a “Recently ViewedStudies” bar, which many of the test users cited as beingvery helpful. In this bar all of the studies that the user haslooked at in the system remain visible and accessible.These and other navigation tools have greatly improved thenavigational ability of the user within the system.

The VDC development team improved the searching andbrowsing capabilities in the system. We have implementedmore powerful searches, and are adding integration with

topic thesauri. More browsing capabilities withincollections are being explored, including keywordbrowsing, since this is a feature that the test usersoverwhelmingly appreciated and found useful. And, inresponse to user demand, we are implementing cross-collection and cross-federation searches.

In response to users’ requests to see more about the studies,we have developed alternative (e.g. ‘printer-friendly’)layout options, and are developing a mechanism wherebyusers can also control the amount of detail presented invarious descriptions of studies and results sets.Furthermore, we are developing heuristics for enhancingthe metadata, which would both enhance presentation andmake searching and browsing more effective.

Through all of these tests on the usability of the VDC, wehave drawn a distinct list of crucial usability features.Overall, test users have expressed interest and enthusiasmfor the VDC. The VDC enables valuable searching and dataanalysis activities. However, in order to really harness thefull potential of a digital library such as the VDC, we mustcontinue to work with real users, and real content, and tocontinue to adapt and enhance the interface based upon realusage.

ACKNOWLEDGEMENTSSupported by NSF Award #IIS-9874747.

REFERENCES1. Altman, M., et. al. Overview of The Virtual Data CenterProject and Software. JCDL ’01: First Joint Conference onDigital Libraries

2. Nichols, David M., et al. “Usability and Open-SourceSoftware Development” in Proceedings of the Symposiumon Computer Human Interaction, (eds.) Kemp, E., et. al., 6July 2001, ACM SIGCHI New Zealand. 49-54

3. The Gnome Usability Project. Web site. <URL:http://developer.gnome.org/projects/gup/> (06/26/02)

4. Nielsen, Jakob. Usability Engineering. Academic Press,San Diego, CA, 1993.

5. Wharton, Cathleen, et. al., “The Cognitive WalkthroughMethod: A Practitioner’s Guide.” in Nielsen, Jakob, andMack, R. eds., Usability Inspection Methods, 1994, JohnWiley and Sons, New York, NY