1536-1268/02/$17.00 © 2002 IEEE PERVASIVEcomputing 33

User Testing aHypermedia Tour Guide

Information technology’s rapid evolution offers agreat opportunity to enhance people’s educationand understanding of their cultural heritage. IT cando this in many ways, from facilitating digital acqui-sition of data from pictures and relics to multime-

dia content presentations. Currently, people access culturaland arts information through interactions with desktopcomputers and other similar platforms, including stand-alone programs, CD-ROMs, Web sites, and informationkiosks. However, even in this arena, ubiquitous computingsystems1 can make an important contribution by support-ing people in the most important moment of their educa-tional experience: when they are up close to the subject,

whether they’re viewing a painting ina museum, a monument in a park, oran animal in the zoo.

Several research projects in ubiq-uitous and context-aware comput-ing have focused on computer sup-port for museum and tour guides.2,3

Nevertheless, little has been published about how endusers evaluate and accept ubicomp systems in museumand park environments.4 Here, we address that user per-spective, presenting results from experiments aimed atunderstanding how pervasive computing can support amuseum-like experience. Our emphasis is on how ubi-comp systems affect real visitors in real environments.

In our project, we tested user acceptance and evaluationof hypermedia guides run on palmtop computers atGenoa’s Costa Aquarium, the largest European aquariumand one of Italy’s most visited cultural sites, with morethan a million visitors per year. Figure 1 shows the guide,including the software we developed, which has beenavailable for rent to the public since December 2000.

Before describing our experiments and results, we dis-

Although handheld guides have been proposed as a way to enhancevisitors’ experience of museums and exhibitions, the authors describeuser studies that actually test the theory in a real-world setting.

I N T E G R A T E D E N V I R O N M E N T S

Francesco Bellotti, Riccardo Berta,Alessandro de Gloria, andMassimiliano MargaroneUniversity of Genoa

Figure 1. The hypermedia guide for the CostaAquarium in Genoa, Italy. The guide usesMicrosoft’s PocketPC, with human-computer interaction software.

cuss the experimental framework at the aquar-ium, as well as design issues and the guides’ inter-face structure. The “Museum Multimedia” side-bar offers an overview of related research.

Experimental frameworkEvaluating ubicomp systems and their

impact on a target population is difficult andcostly because we must analyze real users in areal context. Also, to evaluate continuous inter-action with computation resources, test usersmust have a fully operational, reliable, androbust tool, not just a demonstration proto-type.4 Presently, most cutting-edge technology

to support museum visits is in the prototypestage and not yet suitable for the general pub-lic.5,6 Nonetheless, performing early tests in anauthentic context is important both to verifyend-user acceptance and overall system useful-ness and to get feedback to inform futuredesign.4

34 PERVASIVEcomputing http://computer.org/pervasive

I N T E G R A T E D E N V I R O N M E N T S

Some museum professionals have expressed concern about com-

puters, worrying that Web-based art collections might entice visi-

tors away from real-world museums.1 In fact, IT has actually become a

boon to art, and the use of it in museums is rapidly growing.1–4 Elec-

tronic tools can offer a valid museum aid, providing visitors multime-

dia information about museum items, for example, or reconstructing

historical scenarios to contextualize an exhibition’s artifacts.

Research shows that computer presence in an exhibition typically

enhances visitors’ experience, offering them more interesting interac-

tions with the museum space and attracting people with the technol-

ogy’s novelty.1 In the near future, museums will likely offer personalized

visit itineraries geared toward a user’s specific needs, sensors for enhanc-

ing the viewing experience, and network access for e-commerce and

retrieving and storing museum information.3

Currently, new technologies let museums increase what they have to

offer, helping them better meet user requirements and expectations.1,4,5

Museum visitor surveys highlight three main needs: navigation aids to

support basic orientation and organization of personalized paths, infor-

mation about museum exhibits, and, for some people, specialized

information retrieval.5

Interactive systems that visitors use by themselves efficiently satisfy

these needs. Up to now, such hypermedia systems have been imple-

mented on specific Web sites6 or electronic kiosks within the museum.1

However, such solutions are stationary and cannot accompany visitors

along their paths or provide information and assistance the moment

visitors need it. Palmtop computers overcome this limitation, while also

keeping most of the merits of common hypermedia systems.

Several projects have investigated the use of handheld devices as

guide tools for tourists.7–12 These projects offer interesting insights

into computing, communications, and localization functions for pro-

viding context-aware information and services to nomadic tourists.

Augmented reality systems promise even more natural and seamless

interaction modalities with the surrounding environment.13–16

Likewise, our system aims to provide easily accessible high-quality

multimedia. Because content quality and easy accessibility are key fac-

tors in supporting knowledge acquisition, we’ve focused our efforts

on translating palmtop computers’ high computation and multime-

dia potential into a pleasant tool that is easily usable by the general

public. In this way, we hope to enhance museum visitors’ experience

through “edutainment,” combining education and entertainment.

REFERENCES

1. M. Economou, “The Evaluation of Museum Multimedia Applications:

Lessons from Research,” Museum Management and Curatorship, vol. 17,no. 2, June 1998, pp. 173–187.

2. S.T. Refsland et al., “Virtual Heritage—Breathing New Life into Our AncientPast,” IEEE Multimedia, vol. 7, no. 2, Apr.–June 2000, pp. 20–21.

3. D.S.L. Ing, “Innovation in a Technology Museum,” IEEE Micro, vol. 19, no.6, Nov./Dec. 1999, pp. 44–52.

4. S. Yamada, J.K. Hong, and S. Sugita, “Development and Evaluation ofHypermedia for Museum Education: Validation of Metrics,” ACM Trans.Computer-Human Interaction, vol. 2, no. 4, Dec. 1995, pp. 284–307.

5. B. Booth, “Understanding the Information Needs of Visitors to Museums,”Museum Management and Curatorship, vol. 17, no. 2, June 1998, pp.139–157.

6. A. Badre and A. Jacobs, “Usability, Aesthetics, and Efficiency: An Evaluationin a Multimedia Environment,” Proc. IEEE Int’l Conf. on Multimedia Comput-ing and Systems, IEEE CS Press, Los Alamitos, Calif., 1999, pp. 103–106.

7. K. Cheverst et al., “Developing a Context-Aware Electronic Tourist Guide:Some Issues and Experiences,” Proc. ACM Human Factors in Computing(CHI’00), ACM Press, New York, 2000, pp. 17–24.

8. K. Cheverst et al., “The Role of a Shared Context in Supporting Coopera-tion Between City Visitors,” IEEE Computer Graphics and Applications, vol.25, no. 4, July/Aug. 2001, pp. 555–562.

9. N. Davies et al., “Using and Determining Location in a Context-SensitiveTour Guide,” Computer, vol. 34, no. 8, Aug. 2001, pp. 35–41.

10. G.D. Abowd et al., “Cyberguide: A Mobile Context-Aware Tour Guide,”Wireless Networks, vol. 3, no. 5, Oct. 1997, pp. 421–433.

11. B.B. Bederson, “Audio Augmented Reality: A Prototype Automated TourGuide,” Conf. Companion, ACM Conf. Human Factors in Computing (CHI’95), ACM Press, New York, 1995, pp. 210–211.

12. R. Oppermann and M. Specht, “Adaptive Support for a Mobile MuseumGuide,” Proc. Conf. Interactive Applications of Mobile Computing (IMC’98),Neuer Hochschulschriftverlag, 1998; www.rostock.igd.fhg.de/veranstal-tungen/workshops/imc98/proceedings.html.

13. G.D. Abowd and E.D. Mynatt, “Charting Past, Present, and FutureResearch in Ubiquitous Computing,” ACM Trans. Computer-Human Interac-tion, vol. 7, no. 1, Mar. 2000, pp. 29–58.

14. B. MacIntyre and S. Feiner, ”Future Multimedia User Interfaces,” Multime-dia Systems, vol. 4, no. 5, Oct. 1996, pp. 250–268.

15. B. MacIntyre and E.D. Mynatt, “Augmenting Intelligent Environments:Augmented Reality as an Interface to Intelligent Environments,” Proc. Intel-ligent Environments Symposium, AAAI Press, Menlo Park, Calif., 1998, pp.93–95.

16. T. Starner et al., “Augmented Reality through Wearable Computing,” Pres-ence, vol. 6, no. 4, Aug. 1997, pp. 386–398.

Museum Multimedia

Our project’s aim was thus not to develop thebest possible ubicomp system for the museumenvironment, but rather to test a simple system’seffectiveness in a real-world setting. To this end,our experimental framework was

• A simple tour guide, consisting of a matureand reliable palmtop computer equippedwith hypermedia software

• A real museum—Genoa’s Costa Aquar-ium—which has rented the tour guides sinceDecember 2000, at a cost of 4.50 Euros each

Using this framework, we were able to analyze,from a user-centric perspective, several aspectsof continuous interaction with a mobile multi-media system. We performed extensive testingin all phases of design, development, deploy-ment, and redesign, including early tests withexperts, questionnaires, think-aloud sessions,and ethnographic observations of user behavior.

Because the market already offered severalplatforms that met our project’s needs, wefocused on designing novel human-computerinteraction software for existing hardware. Wecarefully examined and tested available devicesfor their suitability in the museum environ-ment. Among our criteria were that it be pocketsize, robust, reliable, and flexible, and that itsupport open, incremental development andnetwork connectivity. We also required a high-quality visual and audio interface and archi-tectural support for multimedia.

Based on these criteria, we chose the WinCEoperating system as the most suitable platform.As of April 2002, we also made available forrental 32 Compaq iPAQ PC H3630s, with anIntel Strong Arm 1110 206 MHz processor, 32MB RAM, 16 MB ROM, and a 240 × 320 dottouch-sensitive TFT Liquid Crystal Displaywith 4,096 colors.

Tour guide designThe idea behind automated tour guides is to

replace traditional portable audio guides witha more appealing multimedia electronic tool,which has two expected advantages:

• Flexibility—easy upgrades of both contentand interface according to user preferencesand exhibition changes

• User satisfaction—providing an enjoyableand more complete visitor experience

Both benefits are crucial in the aquarium set-ting, because most visitors are school partiesand family groups (77 percent), while the restare general visitors and specialists. The aquar-ium management’s main requirement was thatwe match the technology to an average audi-ence, which is unlikely to be familiar with mostcomputer technologies.

Because we had to combine a palmtop com-puter’s flexibility and multimedia potential withextremely simple interactions for general pub-lic use, we faced two main human-computerinteraction issues:

• Usability. Aquarium visitors have little timeand less willingness to learn how to use a newtechnological tool. Because palmtop com-puters are still somewhat rare, few visitorshave prior experience with them, and mostwill use them for only a short time (typically,30 to 60 minutes). Thus, the platform’s com-plexity must be hidden, and the guide mustbe immediately usable and require no usereffort. Essentially, the interface must be assimple and intuitive as possible.

• Information presentation. Added-value in-formation—such as how various specimenslive in their natural environment—should besynergetic with a visitor’s direct aquariumexperience. Thus, we must structure infor-mation to enhance a visitor’s direct percep-

tion in order to create a better and morepleasant comprehension of the surroundingenvironment.

Facing these problems along with a palm-top’s resource constraints is a novel and chal-lenging research area. To tackle these issues, weopted for a user-centric design methodology7 inan iterative tour-guide development processwith three key steps:

• Participatory design.8 We designed the guidewith input from technical developers, and from

aquarium experts and visitors, who helped usdefine targets and concrete scenarios.

• Usability specifications. To verify our work,we identified explicit and measurable targets.We had several goals, including that 90 per-cent of users should be able to operate theguide without asking questions; 90 percentshould be able to start, suspend, and completethe multimedia description of the first tankwithout errors; and 90 percent should be ableto understand the meaning of all visible con-trols within 30 seconds. We tested each objec-tive in early lab and field tests so that we couldtake the appropriate corrective actions.

• Contextual design. We ran early field testswith aquarium experts and users to highlightproblems and shortcomings that we’d over-looked or ignored. For example, followingthe tests, we found that some users tried toopen other programs or change system prop-erties, so we made access to the underlyingoperating system impossible.

Interface structureTo support immediate use by the general

public, we emphasized four key factors indesigning the tour guide’s interface: simplicity,low intrusiveness, support for natural interac-tion, and current-state visibility.4

As Figure 2 shows, the interface’s basic ele-ment is a multimedia card that corresponds to

each presentation subject, such as a particularfish or a fish tank containing several fish species.Each multimedia card provides users with con-tent and touch-screen buttons that let them con-trol content presentation and navigate betweentanks. All usable controls are always visible.

To make presentation control intuitive, weborrowed the CD-player interface metaphorfrom consumer electronics. We reduced tech-nology intrusiveness by sizing all touch-screenbuttons so that users can easily press them withtheir fingers. This avoids the need for a stylus,which can hinder users and easily get lost.

APRIL–JUNE 2002 PERVASIVEcomputing 35

Because the market already offered several

platforms that met our project’s needs, we

focused on designing novel human-computer

interaction software for existing hardware.

Moreover, there are no hardware buttons; usersoperate only software buttons, each of which isclearly labeled. This prevents the need for writ-ten instructions on how to use the guide.

However, large buttons consume preciousscreen real estate, and we are now testing a newversion of the guide using ergonomically shapedbuttons (see Figure 3), this lets us keep theadvantages of large buttons while reducing theimpact on screen space. According to prelimi-nary tests with users and experts, this is the mostsuitable interaction modality. That said, as weincrease the user’s options, we’ll have to resortto hardware keys or smaller buttons and a sty-lus (or both).

One of the guide’s main tasks is to assist vis-itors in exploring the museum space. For exam-ple, we could use image recognition andinfrared radio communications to trigger con-text-sensitive information.6,9 Initially, however,we decided not to implement an automatic sys-tem, because the aquarium’s visitor path is lin-ear and the fish tanks are sequential. Thus, userscan simply access information about a giventank by reading the tank’s name and numberand selecting the corresponding multimedia pre-sentation labeled with the same name and num-ber. Users can easily move through the hyper-media program using linear navigation controls(such as “previous tank” and “next tank”) thatdirectly correspond to the sequential path.Despite the lack of automation, synchronizingthe aquarium’s physical space with the pro-gram’s virtual space imposes a minimal mental

workload on visitors. The task is simple, andthey perform it using a familiar interface. More-over, opting not to automate let the aquarium’smanagement evaluate the guide’s impact at acomparatively low cost, with the option of laterupgrading the system once we verified users’needs.

To evaluate the affect of providing both on-demand contents and better support for userorientation, we implemented a second interfaceversion that offered more detailed and struc-tured information, including buttons for“Map,” “Links,” and “More info” (see Figure4a).

The map helps tourists orient themselveswithin the aquarium, showing its structure,

including bathrooms, the cafe, exits, and fishtanks. As Figure 4b shows, each tank icon isan active area that users can click to call up thecorresponding multimedia card. To enhanceorientation, we center the map on the user’sposition as determined by the current multi-media card. Users can move through the mapusing two directional buttons (left and right).

The “More info” button gives users an in-depth audio–visual or text presentation relatedto the current multimedia card. The “Link”control opens a small window that lists fishtanks related to the current one. Users can clickany list item to go to the corresponding multi-media card.

Guide contentThe guide has 44 multimedia cards covering

all aquarium tanks, for a total presentation timeof 25 minutes. To prepare the multimedia con-tent, we collaborated closely with the aquar-ium’s education experts and were guided by twomain goals:

• Strengthen the link between visitors and theenvironment. To achieve this, we showedusers photos or photo sequences, withaccompanying comments, all appropriatelytimed to offer useful information while keep-ing visitors’ attention on the fish in the tanks,rather than on the guide.

• Present places and situations that are not typ-ically accessible to aquarium visitors butmight be particularly appealing to them. Forexample, we show visitors a video of thebirth of Giotto, the aquarium’s mascot sealpup (see Figure 2).

Providing high-quality multimedia contentswas also a key requirement. However, thepalmtop’s limited memory capacity forced usto resort to lossy compressed formats foraudio, images, and videos. The programincludes a total of 76 images (1 Mbyte), 9videos (8.4 Mbytes), and 45 mono audio clips(5.3 Mbytes). We use recorded rather than syn-thesized speech, because preliminary lab testsshowed that users preferred it, particularly forthe Italian-language version. Of course, giventhe memory size constraints, increasing con-tent quantity would require that we either usea synthesizer or download content on-demandfrom a local network.

36 PERVASIVEcomputing http://computer.org/pervasive

I N T E G R A T E D E N V I R O N M E N T S

Figure 3. Snapshot of the revised guide,which includes the new, ergonomic but-tons and advanced interface elements,“Link,” “Map,” and “More Info.”

Figure 2. Snapshot of a multimedia card,highlighting three different areas. Thenavigation control area includes the cur-rent tank’s name and number, helpingvisitors synchronize the tool with theirpath. The “Previous” and “Next” buttonssupport linear navigation. The presenta-tion control area lets visitors stop or runthe program and includes the Universityof Genoa’s ELIOS Lab logo.

Evaluation 1: Overall useracceptance

We conducted user tests in two main stages.The first tests, which we ran over a 30-dayperiod in January 2001, started just after wefirst introduced the tour guide service. The sec-ond evaluation (described later) followed a yearlater.

In the first stage, we evaluated our interfacethrough a user survey containing two kinds ofquestionnaires: a 23-question complete version(CV) and a five-question short version (SV). Wealso held an interview session and observedusers’ interactions with the guide, recordingtheir reactions and comments after we encour-aged them to “think out loud.” Our goals inthe first testing stage were to qualitatively ana-lyze user acceptance in authentic use conditions,verify the experimental framework, and gatherinformation for future design.

QuestionnairesBoth the CV and SV questionnaires con-

tained multiple choice, fill in the answers, anda free comment section. (The questionnairesand results, along with a detailed characteriza-tion of our respondents, are available at www.eliosmultimedia.dibe.unige.it/acquario/tests.htm).

Aquarium personnel asked visitors to fill outthe questionnaire at the exhibition’s end, whenthey collected the tour guides. Participation wasvoluntary and anonymous. All participantsrented hypermedia guides at the beginning of

their visit, at the regular price of 4.50 Euros, andtoured the museum, without knowing they’d beasked to fill out a questionnaire at the end. Thisfact is important; it lets us analyze data capturedin authentic use conditions. Subjects couldchoose either the complete questionnaire, whichtypically takes 10 minutes, or a shorter versionif they had less time. The total number of respon-dents was 151 (103 SV and 48 CV). The totalnumber of users during the test period was 185.(We discarded 5 percent of the SVs becauserespondents didn’t understand the questions.)

Results: SVThe SV’s focus was on pleasure and usabil-

ity. Of the 103 SV respondents, 82 percentassessed the tool as usable, and 78 percentassessed it as enjoyable. For 63 percent of theusers, the guide was both enjoyable and useful,but there is no close relation between the twovariables according to the chi-square test, whichevaluates correlations between observed statis-tics. Enjoyability was related to user age (95percent confidence), with people between 25and 45 enjoying the tool less. We found no cor-relation between enjoyability and occupation.

The free comment section offered interest-ing insight into what users perceived as theguide’s shortcomings: Thirty-three percent ofrespondents said they’d like a more informa-tive tool. This highlights the need for layeringthe available information based on user pref-erences and needs. Almost all usability prob-lems were due to the fact that batteries lost

power before the exhibit ended (as reportedby 20 percent of users). This is essentially alogistics issue; some guides were erroneouslyrented out before they’d fully recharged. Only2 percent of users were unsatisfied with theirinteractions with the guide.

Results: CV The CV covered ease of use, enjoyability,

general usefulness, and value. When askedabout ease of use, 23 percent said the devicewas very simple to use, 69 percent said it wassimple, 6 percent said it was not immediatelyusable, and 2 percent said it was difficult to use.Most users—88 percent—used the tool with-out any help. For enjoyability, 17 percent saidit was high, 67 percent said it was sufficient, 15percent said it was low, and 2 percent of respon-dents found no enjoyment at all. Most respon-dents—79 percent—found the device useful,while 21 percent did not.

People in museum management commonlyfear that visitors will find such hypermediatools distracting. However, in our study thiswas true for only 8 percent of users. As forprice, 58 percent said the rental cost was ade-quate, but only 48 percent found the guide’simage and video content satisfying, suggestinga need for improvement.

Table 1 shows the relationships betweensome of the CV survey’s variables. Enjoyabilityis strongly correlated with usefulness, high-lighting tourists’ desire for a visit that combineseducation with entertainment. Enjoyability isalso correlated with all variables related to qual-ity of content (image, audio, and comments)and technology (controls and system responsetime). This suggests the importance of joiningknowledge with taste and an understanding ofhuman psychology when developing mobilemultimedia tools.

As the final rows show, user characteristicshave little correlation with the considered vari-ables, suggesting the guide’s suitability for thegeneral public. Exceptions here are familiaritywith natural science (which correlated withlength of use) and familiarity with computers

APRIL–JUNE 2002 PERVASIVEcomputing 37

Figure 4. The guide’s (a) advanced interface and (b) the corresponding map that enhances user orientation.

(a) (b)

(which correlated with perception of the tools’usefulness). Length of use is also significantlycorrelated with an absence of distraction. Thatis, people tend to use the guide less if it inter-feres with their focus on the real exhibition.

Results: Advanced interface In the first testing stage, we also tested the

more advanced interface. We field tested it ininterview and think-aloud sessions with 10users. Of the subjects, five were female and five

male, with a mean age of 29. We randomlyselected test users from among people enteringthe aquarium, independent of whether theywanted to rent the tour guide or not. We thenasked them whether they would like to partic-ipate in a scientific evaluation of the multime-dia guide and told them it would involve athink-aloud session and an interview. If theysaid yes, we gave them the guide for free and adetailed explanation of the experiment’s modal-ities and goals. Two of us then accompaniedthem on their visit, took notes about users’impressions, and conducted the interviews.

The interview’s focus was on usefulness and

acceptance of the “Map” and “Links” tools anduser preferences about the visit’s organization.We also discussed how information was pro-vided and asked users to compare hypermediasystems with audio guides. Among the presen-tation modes we asked them to discuss weretext versus audio versus audio–visual and a mapversus a textual index of attractions.

The most significant result was visitors’ sub-stantial preference for the multimedia system.The main reason for this preference was that it

offered users the possibility of easily correlat-ing the description with their direct experienceof the exhibit. The correspondence betweenimages—they preferred still images to video—audio, and reality let them, as one user sum-marized, “immediately and better understandwhat the presentation is about.” Although thisbenefit is clearly evident in a dynamic aquar-ium environment, where a single tank housesseveral fish species, it can also extend to tradi-tional museum spaces. For example, it mighthelp users recognize and study particularaspects of a given painting.

With our “More info” button, which pro-

vides in-depth content, most users said they pre-ferred audio–video content to text. In fact, beingable to listen to information let the users focustheir attention on the fish, rather than the guidescreen. Similar considerations were evident inrelation to video. Users appreciated video whenit presented special information, such as theechography of a pregnant dolphin; otherwise,they considered it less important than the realexperience. We believe that these considerationshighlight the importance visitors place on enjoy-ing and focusing on real animals, and are inde-pendent of the number of available video clips.More memory space will permit different kindsof content design and presentation, so we’ll beexploring this issue further in the near future.

These results match those from our CV ques-tionnaire, indicating a close relationship betweenlack of distraction and length of use. Users enjoytheir visits most when they can fully experiencethe environment. Hypermedia guides shouldenhance this experience, providing clues to syn-chronize visitors with the aquarium reality andoffering additional videos of special events.

Users did not generally view our “Map” toolas essential, which is not surprising given theaquarium’s linear layout. Such a service is likelyto be more useful in complex structures, such asbig museums. In the aquarium, however, mapscan give users a way to estimate the length oftheir visit, follow thematic paths, and get a struc-tural overview, including available services. Fewvisitors in our study used the map to build per-sonalized paths. However, for such purposes,

38 PERVASIVEcomputing http://computer.org/pervasive

I N T E G R A T E D E N V I R O N M E N T S

Overall, presenting additional advanced tools

such as maps and links increases the interface’s

complexity, without negatively impacting the

system’s simple usability.

TABLE 1Confidence ratings for variable correlations based on the Chi-Square test.

Enjoyability (%) Usefulness (%) Length of use (%) Distraction (%)

Price 95 Usability 90 90 Self-understanding of use 90 Length of use 95 Enjoyability 99 Usefulness 99 Distraction 95 Image quality 99 90 Audio quality 95 95 Comments quality 99 99 Control quality 90 95 Response time 99 90 95 AgeSexFamiliarity with computers 95 Familiarity with natural science 99 Motivation for using the guide

maps are better than the alternative text index,which simply has small photos representing thesubjects. Also, the map is particularly suited torepeat aquarium visitors (who constitute 40 per-cent of the aquarium’s traffic), because they arefamiliar with the exhibit and more likely to seekout particular tanks or species.

The “Link” tool was also underused becausepeople didn’t understand its benefit. After weexplained what the tool was for, most users saidthey would find it useful for building thematicpaths through the exhibition. We plan to dealwith this issue in subsequent design phases.

Overall, presenting additional advancedtools such as maps and links increases the inter-face’s complexity, without negatively impact-ing the system’s simple usability.

Evaluation 2: Impact on visitorsBased on feedback from the first experi-

ments, we upgraded the multimedia tour guide(see Figure 3). We then ran a second series oftests in January 2002 to evaluate how the guideaffected visitor behavior. In addition to ethno-graphic observations, we performed quantita-tive and qualitative measures.

Knowledge acquisition experimentMost visitors view a trip to an aquarium or

museum as an informal and unstructuredactivity that offers education, entertainment,leisure, and so on. Quantitatively evaluatingsuch an activity as a whole is difficult. How-ever, visits do let tourists directly learn fromexperiencing the subject—be it a painting oran octopus—and thus knowledge acquisitionis one of the tour’s most relevant aspects.Given this, we chose a task-centric approachto evaluate whether the multimedia tour guidesupports knowledge acquisition, as comparedto other tools such as audio guides and sta-tionary panels.

We selected test subjects randomly amongpeople entering the aquarium who agreed toparticipate in a scientific experiment involving

a 10-minute test. Of the 120 subjects, 69 weremale and 51 female, with a mean age of 27.We gave 40 test subjects a free multimediatour guide and 40 a free audio guide. Weloaded the audio guides with the same audiocontents as the multimedia tour guides, whichincluded no additional textual information.Our control group was the final 40 subjects,who toured the exhibition aided only by thesimple stationary posters and panels besidethe tanks.

Test tasks. All of our subjects were unawareof the test tasks until the end of their visit. Atthat point, we gave them two types of task: arecognition task and an essay task. The recog-nition task consisted of associating images withnames. We gave subjects a list of 10 images, rep-resenting fish or natural environments, alongwith four possible names for each image tochoose from. We gave them 90 seconds to com-plete this task, which focuses on visual knowl-edge—a main form of knowledge acquisition.We expected subjects using the multimedia tourguide would perform best, because unlike theother test users, they had access to a visual infor-mation channel.

The essay task consisted of subjects giving awritten description of the contents of threetanks, which we preselected with the help ofaquarium experts. This task had no time limit,but we did limit the amount of writing to 10lines. In this task, we privileged explicit anddeclarative knowledge and anticipated limitedadvantage (if any) for multimedia tour guideusers over those with the audio guide.

We began the test with two questions. Thefirst asked about users’ satisfaction with the visitas supported by their particular tool (the mul-timedia tour guide, audio guide, or panels andposters). The second asked how interested sub-jects were in seeking out more in-depth infor-mation about fish and the natural environment,given their experience with the visit and theirparticular tool.

Results. Our main goal in this second experi-ment was to evaluate the tools’ support of knowl-edge acquisition, including measures of efficiency,effectiveness, and satisfaction (in keeping withthe ISO’s definition of usability10,11). Efficiencyis related to a task’s completion time. We mea-sured efficiency by counting the number of cor-rect answers in the recognition task (task 1),which had a time limit (90 seconds). Effective-ness measures the overall quality of task perfor-mance in terms of accuracy and completeness.We measure effectiveness by evaluating, on a 0-to-4 scale, descriptions in the essay task (task 2).We measure user satisfaction on a 10-point scalebased on our initial question about subjects’ sat-isfaction with their visit and tool.

Table 2 shows our results. In task 1, we finda significant difference in the number of imagesrecognized (99.9 percent confidence). As weexpected with this visual task, a Bonferroni posthoc test—which compares averages amonggroups—showed that the multimedia tourguide supported users significantly better thanthe audio guides (90 percent confidence) andpanels (99.9 percent confidence). The differencebetween the audio guides and panels was alsosignificant. In task 2, the tool type influencedthe grade obtained (99.9 percent confidence).As a Bonferroni post hoc test showed, this dif-ference is only between the audio or multime-dia tour guides and the panels.

Our analysis of the essays showed interestingdifferences between user groups. Subjects usingthe audio guide tended to be more precise aboutthe particulars they heard in the presentation,whereas the multimedia tour guide tended to sup-port observation of specimens and their behav-ior. For example, only audio guide users reportedthat chameleons live in the Malagasy Forest,whereas only multimedia tour guide usersdescribed the chameleon’s skin and eyes. Wededuce from this that the audio guide tends todraw users’ attention to the contents, favoring averbal learning process, and the multimedia toolinvites visitors to improve their observation.

APRIL–JUNE 2002 PERVASIVEcomputing 39

TABLE 2Average results of the knowledge acquisition tasks, with standard deviation in parentheses.

Tool Task 1 recognition Task 2 essay Satisfaction

Audio guide 5.58 (1.83) 2.09 (0.94) 7.52 (1.26)Video guide 6.3 (0.97) 2.16 (1.02) 7.95 (0.58)Panels 4.38 (1.56) 1.22 (1.00) 6.69 (0.94)

We found a significant influence (99.9 per-cent confidence) also on the tools’ perceivedusability (“Satisfaction” in Table 2). As earlier,according to post hoc tests, differences are sig-nificant only when compared to panel-sup-ported visits.

The second test question about ongoinginterest showed that subjects using an electronictool (audio or video guide) are more prone toseek in-depth information after their visit. Weoffered respondents several options, and theycould select more than one. As Table 3 shows,most tour guide users said they’d deepen theirknowledge using multimedia tools, while oth-ers, particularly panel readers, tended to preferanother visit. As our results have already estab-lished, multimedia guides are effective tools forlearning, so we argue that people using guidesmight be keener on using multimedia tools tofind more information.

In general, although our results show thatthe tour guide supports better knowledgeacquisition, its results differ significantly fromthe audio guide only for a specifically visualtask. That said, in this experiment we pro-vided users the same amount of informationon both tools. It’s possible that, given a mul-timedia computer with significantly more con-tent (provided by downloading files from anetwork, for example), users might favor aubicomp system.

Ethnographic observationsFor this experiment, we selected subjects

randomly from among people who rented thetour guide. We asked potential subjects if theywould volunteer to take part in an ethno-graphic observation session, which wouldinvolve two of us following them during partof their tour (10 to 15 tanks) without inter-fering with their visit. Of the 10 subjects westudied, four were male and six were female,with a mean age of 30.

This test was important to evaluate howusing the tour guide impacts visitor behavior.

Among our findings were

• Using the guide tends to make the visitormore attentive but also lonelier: The ear-phones tend to acoustically isolate users, lim-iting their ability to listen to and speak withother people. This suggests the need to findnew, less exclusive modalities to conveyacoustic information.

• Video information is less invasive, because thescreen size is limited and we designed the pre-sentations not to draw attention to the screen.Often, users would see a fish on the screen,then immediately search for it in the tank.

• In general, few visitors read the stationarypanel texts, and those who do typically readthem only in part. The multimedia guideseems to further discourage reading such texts.

Using the guide also affects visitors’ pace. Inour study, visitors using multimedia or audioguides typically observed the fish in a tank untilthe presentation was over and then moved on,occasionally lingering at the tank briefly for fur-ther observations. In unassisted tours, visitors’time at the tank did not depend on the length oftext panel contents, which they typically onlyskimmed. Some visitors completely skippedsome tanks or observed others intently for longperiods, particularly those with attractivespecies such as seals, penguins, and the like.Also, in unassisted tours, visitors tend to readpanels after they observe the tank if they’reattracted to a particular specimen. In contrast,the multimedia presentation is generally trig-gered independently of specimens and can thusarouse visitor interest in curious, perhaps lessobvious aspects of a specimen, inviting furtherobservation.

These considerations suggest that we designcontent in two ways: for well-known, appeal-ing specimens, we can provide specific, on-demand information, while for lesser-knownspecies, we can target content to involve andappeal to the visitor.

A s our first field experiments show,there is strong public interest inand acceptance of computer-based personal guides that have

an elementary user interface. This simple tooloffers exhibition visitors an enjoyable experi-ence and supports learning. Moreover, itenlarges the base of IT users by introducingpalmtop computers, which are likely to playa significant role as IT and telecommunica-tions converge.

Compared with previously published find-ings, our work highlights the importance ofdeploying an application in its real-world con-text as soon as possible.12 For example, welearned from our early tests with real users theimportance of content presentation design inboth strengthening visitors’ direct experienceand tuning the multimedia content mix.

Although standard, “desktop-like” computerapplications tend to be stand-alone, complex, andrequire more training,5 ubicomp systems can becompletely integrated into visitors’ behavior, sup-porting their activity in a natural and immediatefashion. Ubicomp lets computing move from alocalized tool to a constant presence, scaling com-puter usage to time.4 For example, while previ-ous research reports an average usage time of 3.7and 7.7 minutes for stand-alone systems,5 visi-tors used our aquarium guide throughout theirvisit, for more than 20 minutes on average.

Our observations and lessons learned aboutdesign can support planning for expandingubiquitous system use in museums. Future sys-tems might, for example, provide more contentand interaction modalities, along with intelli-gent decision support, context-aware services,and enhanced human interactions amongdevice users.

As for our aquarium project, the next step is todeploy an 802.11b wireless network, which offersnearly limitless memory availability. This will letus expand the application’s content base, offeronline services (such as electronic transactions),automatically synchronize the physical environ-

40 PERVASIVEcomputing http://computer.org/pervasive

I N T E G R A T E D E N V I R O N M E N T S

TABLE 3How users would obtain further information following their visit.

Multimedia tools (%) Books (%) Another visit (%) No interest in more info (%)

Audio guide 69 49 45 6Video guide 59 30 38 3 Panels 28 28 58 33

ment and the program’s virtual space, and sim-plify guide updates through broadcasting.

ACKNOWLEDGMENTSOur research was partially funded by the EuropeanUnion’s E-Tour project (IST1999-20447), which ispart of the V Framework Programme. We thank theanonymous referees for their insightful and construc-tive comments, which greatly improved the qualityand readability of this article.

REFERENCES1. M. Weiser, “The Computer for the 21st Century,”

Scientific Am., vol. 265, no. 3, Sept. 1991, pp.94–104.

2. N. Davies et al., “Using and Determining Loca-tion in a Context-Sensitive Tour Guide,” Com-puter, vol. 34, no. 8, Aug. 2001, pp. 35–41.

3. G.D. Abowd et al., “Cyberguide: A Mobile Con-text-Aware Tour Guide,” Wireless Networks, vol.3, no. 5, Oct. 1997, pp. 421–433.

4. G.D. Abowd and E.D. Mynatt, “Charting Past,Present, and Future Research in Ubiquitous Com-puting,” ACM Trans. Computer-Human Inter-action, vol. 7, no. 1, Mar. 2000, pp. 29–58.

5. T. Starner et al., “Augmented Reality throughWearable Computing,” Presence, vol. 6, no. 4,Aug. 1997, pp. 386–398.

6. B. Schiele, T. Jebara, and N. Oliver, “Sensory-Augmented Computing: Wearing the MuseumGuide,“ IEEE Micro, vol. 21, no. 6, Nov./Dec.2001, pp. 44–52.

7. J.M. Carroll, “Human-Computer Interaction:Psychology as a Science of Design,” Int’l J.Human-Computer Studies, vol. 46, no. 4, Apr.1997, pp. 501–522.

8. A. Beck, “User Participation in System Design:Results of a Field Study,” Human-ComputerInteraction: Applications and Case Studies, M. J.Smith and G. Salvendy, eds., Elsevier, Amsterdam,1993, pp. 534–539.

9. S. Pradhan et al., “Websigns: Hyperlinking Phys-ical Locations to the Web,” Computer, vol. 34,no. 8, Aug. 2001, pp. 42–48.

10. ISO 9241-11, Ergonomic Requirements forOffice Work with Visual Display Terminals(VDTs) Part 11: Guidance on Usability, Int’lStandards Organization, Geneva, 1998.

11. E. Frokjaer, M. Hertzum, and K. Hornbaek,“Measuring Usability: Are Effectiveness, Effi-ciency, and Satisfaction Really Correlated?” Proc.ACM Conf. Human Factors in Computing(CHI’00), ACM Press, New York, 2000, pp.345–352.

12. G.D. Abowd, “Classroom 2000: An Experimentwith the Instrumentation of a Living EducationalEnvironment,” IBM Systems J., vol. 38, no. 4,1999, pp. 508–530.

For more information on this or any other computingtopic, please visit our digital library at http://computer.org/publications/dlib.

APRIL–JUNE 2002 PERVASIVEcomputing 41

the AUTHORS

Francesco Bellotti is aresearch consultant at the Uni-versity of Genoa’s Electronicsfor the Information Society(ELIOS) Laboratory. Hisresearch projects concernmobile computing, HCI, andmultimedia systems in the

automotive environment. He received a PhD inelectrical engineering from the University ofGenoa. Contact him at franz@dibe.unige.it;D.I.B.E., University of Genoa, Via Opera Pia 11a,16145 Genoa, Italy.

Riccardo Berta is a PhD can-didate in electronics and com-puter science in the Depart-ment of Biophysical andElectronic Engineering, Uni-versity of Genoa. His researchinterests include design,implementation, and evalua-

tion of innovative HCI modalities for mobiledevices. He received an MS in electronic engineer-ing from the University of Genoa; for his thesis, hedeveloped an optimizing environment for Javabytecode. Contact him at berta@dibe.unige.it;D.I.B.E., University of Genoa, Via Opera Pia 11a,16145 Genoa, Italy.

Alessandro de Gloria is aprofessor of electronics at theUniversity of Genoa, where heleads the Department of Elec-tronics and Biophysical Engi-neering’s (DIBE) ELIOS Labora-tory. His research interestsinclude virtual reality, mobile

computing, HCI, and embedded system design.He has participated and directed several Europeanresearch projects in these areas and written morethan 100 papers for international journals and con-ferences. He is a member of the IEEE. Contact himat adg@dibe.unige.it; D.I.B.E., University of Genoa,Via Opera Pia 11a, 16145 Genoa, Italy.

Massimiliano Margarone isa PhD candidate in electricalengineering in the Departmentof Biophysical and ElectronicEngineering, University ofGenoa. His research interestsinclude Web usability, HCI,and mobile computing. He

received an MS in electronic engineering from theUniversity of Genoa, with a thesis on asynchronousdigital filter development for wireless devices. Con-tact him at marga@dibe.unige.it; D.I.B.E., Universityof Genoa, Via Opera Pia 11a, 16145 Genoa, Italy.

computer.org/internet/

IEEE Internet Computingreports emerging tools,

technologies, andapplications implementedthrough the Internet to

support a worldwidecomputing environment.

In our upcoming issues,we’ll look at• Database Technology

on the Web• Global Deployment of

Data Centers• The Technology of Trust• Web Services• Mobile Applications• IPv6

... and more!