Embed Size (px)
Citation preview
Momento: Support for Situated Ubicomp Experimentation
Scott CarterFX Palo Alto Laboratory
3400 Hillview Avenue, Bldg 4Palo Alto, CA 94304
Jennifer MankoffHCI Institute
Carnegie Mellon UniversityPittsburgh, PA [email protected]
Jeffrey HeerBID and CSD
University of California, BerkeleyBerkeley, CA 94720
ABSTRACTWe present the iterative design of Momento, a tool that pro-vides integrated support for situated evaluation of ubiqui-tous computing applications. We derived requirements forMomento from a user-centered design process that includedinterviews, observations and field studies of early versionsof the tool. Motivated by our findings, Momento supportsremote testing of ubicomp applications, helps with partici-pant adoption and retention by minimizing the need for newhardware, and supports mid-to-long term studies to addressinfrequently occurring data. Also, Momento can gather logdata, experience sampling, diary, and other qualitative data.
Author KeywordsMobile Devices, Wizard of Oz, Rapid Prototyping, Experi-ence Sampling, Diary Studies
ACM Classification KeywordsH5.2 [Information interfaces and presentation]: User Inter-faces. - Graphical user interfaces. General Terms: Design,Human Factors, Experimentation
INTRODUCTIONUbicomp applications are designed to support tasks thatscale across multiple people, activities, and places. This of-ten leads to an interdependence between applications andtheir surroundings that creates a need for real-world, situatedapproaches to iterative design. However, it is challenging togather realistic data during the early stages of iterative de-sign of ubicomp applications [5].
In interviews with nine developers of mobile technology [5],and observations of four diary studies (a type of study inwhich participants keep a journal about events of interest tothe experimenter) [4], we identified the following difficultieswith situated evaluation: remote testing; adoption and reten-tion; and the need to study events that occur infrequently.Also, our interviews and observations highlighted the valueof both quantitative data, such as logs and timestamps, andqualitative data, such as that gathered from experience samp-ling studies (ESM) and diary studies, as well as integratedtools for annotating and reviewing data.
Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, orrepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.CHI 2007, 28 April - 3 May, 2007, San Jose, California, USA.Copyright 2007 ACM 1-59593-178-3/07/0004...$5.00.
In this paper, we present the user-centered development ofMomento, a tool that provides integrated support for situatedevaluation of ubicomp applications and that addresses thekey issues raised in our interviews and observations. In parti-cular, Momento can piggyback on participants’ existing net-worked mobile devices to minimize the need for new hard-ware and reduce issues of adoption and retention. Momentosupports remote testing and can support mid-to-long termstudies to address infrequently occurring data. Experimen-ters can use Momento to gather quantitative log data, ESM,diary, and other qualitative data. Momento is designed to beeasy to use, and can be configured and run by installing amobile client and associated text-based configuration file onparticipants’ devices and using a GUI desktop platform toconfigure experimental details. Momento supports monito-ring incoming information from and sending information toparticipants, and reviewing data or exporting it for analysis.
Momento’s initial requirements came from our interviewsand observations. As we developed Momento, we deployedit in several pilots and four studies involving ourselves andexternal researchers, ranging from two days to two monthsin length and including from four to fourteen users. Aftereach study, we analyzed experimenter and participant expe-riences when using Momento and used this information tofurther refine our design. In particular, studies led to chan-ges to supported mobile platforms and networking protocols,augmented support for mobile experimenters and automatedsampling, and privacy-sensitive support for post hoc annota-tion and interviews.
Momento leverages the text messaging (SMS) and mediamessaging (MMS) capabilities of mobile devices such asmobile phones to share information between end users andexperimenters, who use a desktop platform designed for ma-naging experiments. Using Momento, experimenters can re-spond to participant requests, ask participants to manuallycapture or record data, or automatically gather data from mo-bile devices. Today’s media-enabled mobile devices are rela-tively ubiquitous, and have the ability to capture and transfera variety of media and data, ranging from Bluetooth and GPSdata to audio and images. It is estimated that 785 million ca-mera phones will be shipped worldwide in 2010, comprising87% of all mobile phones [18].
Since our contributions center on the iterative, user-centereddesign of usable support for situated experimentation, so doour innovations. By situated experimentation, we mean thatMomento’s goal is to allow experimentation that takes intoaccount the context of use. Momento provides experimen-
1
Figure 1: (left) Part of an overview display for use by remote experimenterswhile a study is running. This display is especially valuable during lengthystudies with sparse data. (right) Part of a detailed view of a message.
ters with an overview display to help them manage lengthystudies with sparse data (shown in Fig. 1, left). Paired withthat, Momento also enables context-sensitive, real-time no-tifications to be sent to experimenters based on informationsensed about users. A configurable mobile client suitable formany experiments is provided, and a subset of experimentscan be carried out without any client-side installation. Fi-nally, by integrating support for collecting quantitative andqualitative data, we provide a consistent end user experience.
OverviewThe next section presents our interviews and observationsand the requirements arising from them. We then describethe current version of Momento, after eighteen months ofiterative design involving pilots and four studies. One stu-dy is used to illustrate this description. The other three stu-dies, which demonstrate different experimental paradigms,are presented in more depth in our validation section. Wereport on the end user and experimenter experience and con-clude by highlighting areas needing development.
FIELDWORKOur investigation into situated evaluation of ubicomp tech-nology began with two pieces of work: First, we observedtwo different sets of experimenters conducting diary studiesand conducted two diary studies ourselves [4]. Second, weinterviewed nine developers engaged in the iterative designof mobile ubicomp applications [5]. Here we unify those pie-ces of work and focus on their implications for the design ofa tool intended to support situated experimentation.
Observations of Diary StudiesThe primary goal of our diary study work was to understandthe impact of the ability to easily gather a range of media(such as location, photos, and audio) on the types of infor-mation that experimenters could elicit from subjects [4]. Ho-wever, as we also reported, it quickly became clear that theuses of these media led to a need for better tools.
Even though diary studies are well suited to gathering situa-ted data, when participants were asked to record photos oraudio, they could not always explain or remember what theychose to record (“that could have been anywhere,” one parti-cipant said of one of his audio recordings, “maybe the weirdwind by the train? I don’t know.”). Additional context suchas location or time could help with this. The tools typicallygiven to participants sometimes lacked basic features such
as time-stamping and did not support more complex featuressuch as lightweight, situated annotation of captured media.
The experimenters we observed often chose only to down-load data from capture devices at the start of interviews tominimize participant burden. This practice led to some pro-blems. Experimenters did not have time to review captureddata before elicitation interviews, which curtailed their abili-ty to prepare for the interview and increased the chance thatimportant themes would be missed. Some of the media cap-tured by users did not align with the goals of the study, andin one case experimenters did not address this until the expe-riment ended. Being able to view user captures in real timecould help with feedback and could also improve experimen-ters’ ability to prepare for elicitation interviews.
While our observations of diary studies highlighted someof the problems encountered with a particular experimen-tal technique, the interviews described next showed us somemore general problems facing experimenters engaged in aniterative design process.
Interviews with Mobile DevelopersWe conducted interviews with nine developers of mobiletechnologies. All nine interviewees had conducted at leastone field experiment. Interviewees had explored a range ofdata gathering techniques including diary studies and inter-action logs. Interviewees preferred to conduct usability testsremotely, both to avoid inhibiting normal behavior and be-cause of the cost of being “everywhere at the same time”[27]. In all but one case, only log data was gathered du-ring remote evaluations, and it was hard to understand thecontext of use for that data. One interviewee used event-contingent ESM and reported that this was highly effecti-ve. In this method, queries to participants are triggered bycontext, to focus feedback on moments when something im-portant occurs. The interviewee sent a query to participantsevery time they interacted with his application. This enabledhim to gather more than only log data and made the evalua-tion worthwhile. However, in general, interviewees reportedseveral problems that made it hard to conduct successful si-tuated evaluations, described next.
Adoption and retention of new technologies, even for thelength of a single study, were major problems. For userswho already own mobile devices, adding or switching de-vices was a big barrier to use. Although the developers weinterviewed wished to leverage existing devices, they werenot able to do so in their field studies. As a result, they hadto work hard “staying on top of users” to encourage them tocontinue using deployed devices and to report problems in atimely fashion, rather than simply abandoning their devices.
A third problem was gathering enough data. Some tasks maynaturally occur only occasionally (such as commuting to andfrom work), or may be difficult to sense (such as an emotio-nal response). For example, two of our interviewees conduc-ted field studies in which the events they were interested inhappened only twice per day. This led them to depend moreheavily on qualitative data such as interviews and experiencesampling, made it harder to collect the quantitative data theywould have liked, and pushed them towards longer studies.
2
Summary of Requirements: ParticipantsOur fieldwork helped us to identify usability requirementsfor participants and for experimenters. For participants, mi-nimizing the burden of experimentation can help with par-ticipant compliance, adoption, and retention. We identifiedthree main issues that can be addressed:
Unified client system A unified client system for gatheringmultiple types of data (such as self reports, logged events,and ESM) is one way of increasing consistency and mitiga-ting adoption and retention issues raised in our interviews.
Leverage existing devices Our field work confirmed thatcarrying a new device, even loaded with useful new features,is burdensome for end users. It is important to leverage exi-sting devices as much as possible. Communications with endusers can piggyback on commonly available devices such ascamera phones and devices that support text messaging.
Multiple, lightweight communication options Related tothis, it is important to provide multiple kinds of supportfor communication between participant/device and experi-menter. Our diary study observations showed the need forongoing availability of communication and asynchronouscommunication (e.g., annotation); and both our diary studiesand interviews emphasized the need for communication tobe lightweight. Given a focus on participants’ existing de-vices, options include: live phone discussions on a partici-pant’s phone (hard to manage asynchronously but allows ra-pid communication); text messaging (ideal for lightweightexperimentation with participants who are highly comforta-ble with text messaging); or a custom client (requires soft-ware installation, is easily customizable by the experimenter,can include many more types of media and can be optimizedfor participant usability).
Summary of Requirements: ExperimentersFor experimenters, it is important to support different le-vels of computer experience, to support multiple types ofdata and context, and to support monitoring and notification.Some specific requirements arising from our studies include:
Support qualitative data, quantitative data, and contextExperimenters reported valuing both qualitative and quan-titative data, including diary data, ESM data, log data, andsensed context. Integrating support for multiple types of datacan help to increase consistency and usability.
Do not require fully implemented applications Partici-pants in our observations wanted to test ideas they could noteasily implement. To support experimentation at the earlystages of design, as well as to support experimenters with li-mited coding experience, it is important not to require com-plete applications. One way to facilitate this is to support aWizard of Oz protocol in which experimenters can do someof the work normally done by an application.
Support the full experimental lifecycle The experimentallifecycle described by our interviewees included experimen-tal set-up, modifications to an experiment, running an expe-riment, and analysis and summarization of data both duringand after the experiment was run. A usable system shouldsupport this entire cycle.
Support monitoring and notification As reported and ob-served in our field work, experimental data may arrive in oc-casional bursts, reflecting the variable nature of day to daylife. When a study takes place over days or weeks, constantattention may be an inefficient, difficult way to watch for rareor uneven data. Overview displays and notifications can ad-dress this problem. For example, an experimenter might benotified when a user enters a certain space, or might monitora display for big changes in amount of activity. These pie-ces of information could help an experimenter decide whento take direct action (e.g., contact the user, go somewhere tomake observations, etc.).
Support lengthy, remote studies Our fieldwork identifieda preference for remote data gathering. Furthermore, datafrom studies that take place over days, weeks, or more mayovercome issues such as the novelty effect and learning. Ga-thering situated data over time is more feasible when experi-menters can be remote. Additionally, remote experimentati-on helps to mitigate the effects of an observer being present.
Based on these requirements, we created a tool, Momento,supporting situated experimentation. Our process involvediterative studies with the tool, run by ourselves and others.After each study, we reflected on the participant and experi-menter experience. The current version of Momento is des-cribed next, while studies run during and after the implemen-tation of Momento are described at the end of this paper.
SYSTEM DESCRIPTIONMomento consists of a set of configurable tools that can beused without writing source code – our goal is that anyonewith basic computer and mobile device experience will beable to use the tools. As shown in Fig. 2, the most importantcomponents of Momento are the clients (C) used primarilyby participants to send messages and make requests (clientsinclude fixed applications and mobile devices running eit-her standard mobile multimedia applications or a speciali-zed mobile application we built); the desktop platform usedby experimenters to configure and monitor experiments (D);and the server (S), which supports multiple experiments,handles communication between clients and the desktop foreach experiment, and provides remote access facilities.
All communication in Momento is sent as text or multi-media messages. Messages are sent to and from the ser-ver and mobile devices or fixed applications using HTTP orSMS/MMS (text or multimedia messaging). Momento com-municates with networked applications using the ContextToolkit (CTK) [9]: messages are sent to applications usingthe CTK event system and are sent back to Momento usingthe CTK services system.
ClientsParticipants, not experimenters, are the primary users of cli-ents. The most basic client facilitates data gathering, such asrequesting a photo or implicitly logging sensed context. It isalso possible to simulate an interactive application throughthe mobile client. Mobile experimenters can also use clientsto report context information from the field or to interactwith participants while mobile.
3
desktopplatform
mediamessaging
Momentoclient
ContextToolkit
fixed applications
mobiledevices
SMS/MMS
HTT
P
server
HTTP
CD
S
Figure 2: System architecture. The desktop platform (D) and server (S)communicate with clients (C) via SMS/MMS, HTTP, or the Context Tool-kit. Study data is stored for later analysis or retrieval via the web.
Data gathering The main purpose of Momento is to sup-port situated experimental data gathering. An importantpart of this is qualitative reports from users. In addition toSMS/MMS, Momento includes an easily configurable mo-bile client that can be installed on most mobile devices. Thisclient can display an information request to a user, who canthen respond by taking a photo (if supported by the device),recording audio, entering text, or sketching using a stylus.
Context Both the mobile client and fixed applications canreport on context data. The mobile client supports loggingof location, nearby people, and audio.
Simulating applications In addition to information requests,arbitrary messages may be sent to clients independently or inresponse to user input. This can be used to simulate an inter-active application. By default, the Momento mobile clientdisplays incoming messages to the participant (more sophi-sticated behavior must be hand-coded, as must any behaviorby fixed applications). Also, the client may be configured todisplay one or more buttons that a user can select. Dependingon its configuration, a button may send locally cached datato the desktop platform or may request media from the userand send it (e.g., ask the user to enter text or take a photo).
Mobile experimenters Although we support experimentalprotocols that do not require an experimenter to be mobi-le, there are still occasions when mobility is desirable. Forexample, a mobile experimenter might wish to observe par-ticipants or to record data that is hard to sense. Mobile ex-perimenters in Momento can use the mobile client to reportdata or to view live data recorded by participants. In parti-cular, messages can be forwarded to a mobile experimenterwho can optionally respond to participants. Finally, Momen-to can notify an experimenter when a participant is in a cer-tain context (e.g., in a certain location with a group of peo-ple). The experimenter may wish to act on this information,for example by making additional observations in person.
Versions of Momento’s mobile client are implemented in Ja-va and in C#. The mobile client can receive messages viaSMS/MMS from the server and can send messages backusing either SMS/MMS or HTTP over a network connec-tion, if one is available. In this way, clients on mobile devi-ces can leverage the fastest network connection available tothem (e.g., 802.11, GPRS, TCP/IP over Bluetooth, etc. ) to
Listing 1: The structure of the client configuration file# D e s c r i p t i o n o f s t u d y ( Any s t r i n g )Scr ibe4Me s t u d y# P a r t i c i p a n t ID ( Any s t r i n g )John# S e r v e r l o c a t i o n ( IP )2 3 6 . 3 4 5 . 2 . 1# C o n t i n u o u s l y b u f f e r e d d a t a t y p e s# ( name and b u f f e r l e n g t h i n s e c s )# a u d i o | b l u e t o o t h | pho to | gps [ 3 0 | 6 0 | 8 0 ]a u d i o 30# B u t t on d e f i n i t i o n s :# name [ c o l o r ] l a b e l [ media ]∗ [ a n n o t a t i o n ? ]# [ name ] , [ HSV ] , [ s t r i n g ] ,# [ a u d i o | b l u e t o o t h | pho to | gps | t e x t ]∗ , [ a n n o t a t e ]what , ‘ ‘ what happened ? ’ ’ , audio , pho to
send messages to the server, which will pass them on to theexperimenter’s desktop platform. The mobile client buffersstate so that it is robust to network and application interrup-tions (e.g., loss of connectivity, phone calls, etc.).
The mobile client is configured using a simple text file. List-ing 1 shows the basic structure of the configuration file. Theexperimenter can instruct the client to upload specific me-dia to the Momento desktop platform either when a buttonis pressed, as soon as information is detected (for discreteevents) or at certain intervals.
Listing 1 corresponds exactly to the Scribe4Me application(Fig. 3) [24]. Audio is stored in a 30-second buffer, andthe interface contains a single button (labeled “what hap-pened?”). When the participant presses the button, the appli-cation asks that a photo be taken and then transmits the photoand current audio buffer to the desktop platform.
Desktop PlatformThe desktop platform is the main interface used by experi-menters. It supports many of the activities of the experimen-tal lifecycle. Experimenters can use it to manage the speci-fication of the devices, participants, groups, locations, andrules associated with the experiment. While the experimentis running, experimenters can see an overview of commu-nications with the participants that supports peripheral mo-nitoring, manage communications by sending messages andviewing incoming messages, and receive notifications aboutactions that are needed. The desktop platform sends text andmedia messages to mobile devices with the server’s help, andcommunicates with other client applications via the ContextToolkit. All incoming and outgoing messages are stored lo-cally by the desktop platform. Experimenters may uploadthis data to the server at will. In this section we describeeach supported activity.
Specification of the experiment Specification, which typi-cally takes place during or just after the planning phases ofan experiment, includes the devices, participants, groups (ofparticipants), locations, and rules.
Participants and groups Participants are uniquely de-fined by their client ID (typically a mobile phone num-ber). Other information about participants such as cli-ent capabilities, demographics, and a calendar schedulemay also be recorded. Experimenters can create groups
4
if [ cond and/or cond ... ]
o/g near/!near locationo/g arrives/leaves locationo/g stays in location (min time)msg arrivesmsg has attachmentmsg matches regular expressiontime of day (repeat time)
then send [content]
file (random|specific)[o’s name][o’s location][o’s proximate people][incoming message]arbitrary textweb page
to [recipient(s)]
any personany group
Table 1: Rules. o = originator of the message; g = group. In the conditional,g is matched if anyone in the group is the originator.
of participants for use when rules or messages shouldapply to more than one person.Locations The IDs of Bluetooth beacons can be recor-ded in the desktop platform to support discrete semanticlocations (e.g., near a landmark or other phone). Fur-thermore, Momento can use standard serial protocolsto communicate with GPS devices to support geogra-phically defined locations (GPS coordinates).Rules Experimenters can pre-configure outgoing mes-sages using Momento’s rule system. The rules systemcan be used to automate simple, frequent actions and toconfigure ESM and event-contingent ESM studies. Ex-perimenters can configure Momento to evaluate rules atspecific times or periodically (e.g., “evaluate this ruleif it is 3pm and participant Chris is in his office”). Inaddition, each rule in Momento is evaluated when amessage arrives. Rules take the form: if [conditionaland/or conditional] then send [content] to [recipi-ent(s)] (see Table 1).
Overview and management of communications The desk-top platform’s overview mode, shown in Fig. 1 (left), dis-plays recent messages sent to and from all participants. Thetimeline, built using prefuse [14], associates one horizontalline with each participant. Red triangles, or photos (if part ofthe message), represent incoming messages, while blue starsrepresent outgoing messages. A vertical red line indicatesthe current time. Blue stars shown past this line indicate fu-ture messages scheduled using the rule system. Clicking on amessage icon opens an information panel showing messagedetails and associated media (see Fig. 1, right).
Notifications Notifications are provided automatically whenmessages arrive to help experimenters monitor incomingmessages. Notifications can allow an experimenter to multi-task and can reduce the impact of experimenter fatigue. Theyare especially helpful when data are sparse. Notifications arevisual and auditory.
ServerThe Momento server has three functions: it acts as the gate-way between the desktop platform and the cellular network,(allowing the desktop platform to run on any networked ma-chine); it manages study data; and it provides password pro-tected web access to captured media.
Gateway The server can communicate video, images, au-dio, or text-based data to and from email addresses, defaultmessaging applications on mobile phones, or the Momentomobile client. The server automatically compresses media,and sends messages to phones via a GPRS-enabled modem.
Figure 3: The Scribe4Me system caches 30 seconds of audio and sends itto an experimenter for transcription when the user presses the “what hap-pened?” button (left). When the experimenter completes the transcription,it is displayed (right) [24].
Study management The server keeps a database of all dataassociated with each study. This allows experimenters tostop and start the desktop platform and move between dif-ferent experiments.
Web host Web access is hosted by the server but can be con-trolled with the desktop platform. Data for each participantis provided separately, and a per-participant password (pro-vided by the experimenter) is required for access. The webinterface is implemented using PHP.
ExampleAs an example, consider a study we recently ran of the Scri-be4Me system (shown in Fig. 3), which allows people whoare deaf to request a transcription of recent audio [24]. Thegoals of our experiment were to learn when and whethersuch functionality would be valuable to people with a rangeof hearing impairments. For this reason, our experimenterswanted a photo showing additional context relevant to eachaudio request. Also, they used a diary protocol to ask parti-cipants to answer a fairly lengthy series of questions abouteach transcription request. This was done at the end of eachday, and depended on the ability to show a history of theday’s requests to participants. Although our original studyran with an earlier version of Momento that did not supportall of these features, here we describe how it would be donewith the current version.
Because of Momento’s ability to capture live context, in-cluding audio, and transmit it to experimenters, Scribe4Merequires no implementation, only configuration. In general,we have found this to be true of many studies that invol-ve an application that can be defined simply by transferringinformation and media between participants and a humanexperimenter. The participant device is configured to buffer30 seconds of live audio at all times. Additionally, a button,labeled “what happened?” is added that transmits that buf-fer to experimenters when pressed. This button also causes apop-up window to appear, asking participants to take a photoillustrating the reason for the request.
5
To run this experiment with Momento, only 4 actions areneeded: First, the experimenter downloads and installs themobile client on each participant’s mobile device. Second,the experimenter modifies a total of five lines of the configu-ration file (as shown in Listing 1). Third, the experimenterconfigures the desktop platform, specifying the IP of the ser-ver and the phone numbers and ID of each participant andenables web access. Fourth, the experimenter can optional-ly specify details about what context should be gathered (byediting the mobile client’s configuration file) or add rulesto the desktop platform describing when communicationsshould take place automatically.
Once configuration is complete and the experiment is run-ning, event flow would look like the following. Note that allcommunications from the participant to the experimenter gothrough the mobile client, to the server, to the desktop plat-form and all communications from the experimenter to theparticipant do the reverse, unless otherwise specified.
1. Scribe4Me participant presses “what happened”; sendsaudio and photo to experimenter.
2. Experimenter sees data and enters transcription, which issent to the participant.
3. At the end of the day, the participant reviews all of herrequests using a web interface provided by the server, andfills out a form answering a series of questions that arethen sent to the experimenter.
4. Optionally, an experimenter could also send a short-answer ESM request to the participant immediately aftercompleting a transcription or at random times during theday. The participant’s response would be shown immedia-tely to the experimenter.
SummaryMomento supports situated ubicomp experimentation bymanaging, recording, and automating different types of dataflow between participants, experimenters, and applications.However, our findings led us away from synchronous com-munication support. Our validation was conducted as partof an iterative design process. We deployed our system in-ternally and with external experimenters as we developedit. Each time, we interviewed experimenters and end users,and analyzed the data that experimenters gathered. Our fo-cus was on understanding and improving both the end userexperience and the experimenter experience.VALIDATIONTable 2 shows an overview of four experiments run usingMomento. External researchers ran two of them: a diary stu-dy of young adults’ approaches to informal learning (Infor-malLearning), and a mobile sketch-based learning applica-tion for children (PhotoSketch). We ran the other two in col-laboration with other researchers: a public awareness display(AwarenessBoard), and the Scribe4Me system. These expe-riments demonstrate Momento’s ability to support a varietyof experimental methods, a variety of data collection techni-ques, and a range of study lengths. AwarenessBoard was runwith an early version of Momento, while Scribe4Me was runabout five months later with an intermediate version, and theother two were run when Momento was nearly in its currentstate. We iterated after each one.
Stud
y na
me
Stud
y ty
pe
Num
ber
of p
arti
cipa
nts
Days
per
par
tici
pant
Num
ber
of e
xper
imen
ters
W
izar
d of
Oz
Feed
back
Q
uest
ions
Desk
top
revi
ewO
ther
app
licat
ions
Mob
ile e
xper
imen
ter
Dist
ribu
s te
d ex
peri
me
Desk
top
nter
s r
eal-
tim
eM
obile
clie
ntRu
les
Informal learning diary 12 7 4 y y y y y
PhotoSketch workshop 24 1 7 y y
Scribe4Me field study 6 7-15 2 y y y y y y
AwarenessBoard field study 14 51 3 y y y y y y y y
Table 2: How experimenters used Momento in four different experiments,including whether or not the experiments involved Wizard of Oz, feedbackor questions sent by experimenters to participants, use of the desktop to re-view captures with participants after the experiment, external applications,distributed or mobile experimenters, monitoring of the desktop during theexperiment, the mobile client, or rules.
The experiments spanned a range of methods and experi-menter conditions (see right columns in Table 2). Awareness-Board and Scribe4Me were deployed as field studies lastingweeks to months, while PhotoSketch was deployed during 1-day workshops and the InformalLearning study was a week-long diary study. Messaging between experimenters and par-ticipants varied: Scribe4Me and AwarenessBoard dependedon synchronous Wizard of Oz exchanges between experi-menters and participants; the InformalLearning study invol-ved the experimenters sending feedback to participants; andPhotoSketch required no messaging in the field. Further-more, experimenters in all experiments monitored capturedevents in real-time, but those who ran the PhotoSketch andInformalLearning experiments also used Momento for posthoc review on the desktop platform with participants. Final-ly, in the AwarenessBoard experiment, Momento communi-cated directly with participants as well as other applications.
In this section, we describe the AwarenessBoard, InformalLearning and PhotoSketch experiments in detail. Our goalis to illustrate that we were successful in providing a usableexperience to both end users and experimenters, as well asto illustrate a range of possible uses for Momento. For eachexperiment, we first provide a brief description of the ex-perimenters’ goals and describe the methods used. We thendescribe what we learned about both the participant and ex-perimenter experience and how this facilitated the iterativedesign of Momento.
Early Study of AwarenessBoardWe used Momento to help implement an early prototypeof a public display, the AwarenessBoard (shown in Fig. 4),intended to convey a history of the availability and locati-on of participating faculty members in our department. TheAwarenessBoard study was one of the earliest and most ex-tensive studies run with Momento, and it contributed grea-tly to our understanding of the usability issues facing bothparticipants and experimenters. It depended on almost all ofMomento’s features. The study lasted for two months andinvolved fourteen participants (including 12 faculty and twostudents). We were the primary experimenters, although wewere collaborating with two social scientists who had desi-gned an application to help test sociability and awareness ina distributed academic department.
6
SetupThe public display was designed to show faculty members’location availability, which was provided by Momento usingthe Context Toolkit. The mobile client was customized toshow the same information. Sensing of location and availabi-lity was done using Bluetooth beacons in Momento. Availa-bility was “sensed” using a simple heuristic that leveragedlocation information: If in a public place or alone in his orher office, a faculty member was available. If others werepresent, a faculty member was assumed to be in a meeting.Changes in sensed availability triggered SMS messages tofaculty checking whether the estimation was correct. Facultycould optionally provide a calendar indicating times they didnot wish to receive such interruptions. Experimenters moni-tored the desktop platform to ensure faculty were answeringquestions in a reasonable amount of time.
The availability heuristic and location sensing were imple-mented using Momento’s rules system. Additionally, a rulewas created that would notify one of our social scientist col-laborators when someone interacted with a public display(e.g., to view a faculty member’s history). This allowed ourcollaborator to conduct in-person observations.
In total, we recruited twelve faculty participants to use thepublic display and answer questions using the mobile cli-ent. Faculty were each given mobile phones so they couldrespond to SMS messages confirming estimations of availa-bility. The phones were in addition to their personal phones –there was too much variability in the technology they alreadycarried to piggyback on existing devices. We also recruitedtwo students who tested an SMS interface to the public dis-play. These participants could send a message to Momentorequesting the current availability of a faculty member. Re-sponses to these requests were handled by a wizard monito-ring the Momento desktop platform.
We ran pilots of the system for three weeks and ran the studyover the course of two months. We iterated and improvedboth the public display and the client based on participantfeedback over the course of the study, and made significantadjustments to the public display after the pilots and againduring a holiday break six weeks into the main study.
Participant experienceOur participants were biased towards a population that rare-ly if ever used phone-based applications or SMS/MMS, andthis led to some difficulties with the mobile client. Additio-nally, the unfamiliarity of the particular phone we gave toparticipants led to complaints that the phone was “big andbulky” or “stopped working” (the operating system on thephone we used notified users of irrelevant information andused modal dialogs to demand confirmations). As a result,users did not always keep the phone nearby. For these rea-sons, it was difficult to derive useful feedback for iteratingthe mobile client’s interface. Primarily, this experience en-couraged us to focus on improving our mobile client to pig-gyback on mobile devices that individual users already carry.
Experimenter experienceRegarding setup, implementation of the application beingtested (shown in Fig. 4) was time consuming. However, lin-
Figure 4: An awareness prototype deployed in a field setting. Location andavailability of users were sensed via mobile clients running on users’ mobiledevices as well as experimenter input.
king it to the Momento infrastructure was straightforward.A bigger difficulty arose when setting up security for theJava-based mobile client during mobile device installation.To address this, we began working on a C# mobile client.
The day-to-day effort of running this field study varied. Wefound that experimenters controlling the mobile interface tothe public display were able to monitor and respond to in-coming questions with only minimal distractions from theirother work. Also, experimenters wanted to observe partici-pants using the public display without necessarily always ha-ving to be present. To support this, they suggested havingMomento automatically generate messages that would besent to experimenters’ mobile devices whenever it was de-tected that a participant was using the display. Experimen-ters also wanted to be able to send messages to participantsonce they had been in a particular place for a certain amountof time. We added both of these features.
Because this study involved many different experimentersworking from different locations, we had to increase the so-phistication of the server-desktop platform networking. In-itially, Momento handed-off messages from the server to thedesktop platform through a networked folder, which requi-red the desktop to be on the same network as the server.
Part way through the study, we added support for experi-menters to walk through the hallways checking for partici-pant availability and sending updates to Momento via mobilephones. This and other unanticipated models for participati-on by remote and mobile experimenters led us to expandMomento’s sender-recipient model (who could send mes-sages to whom) and to enhance the rule system to supporttime-based triggers and triggers from external applicationsconnected via the Context Toolkit. With this change, expe-rimenters were able to tie events triggered by external ap-plications to messages sent to mobile experimenters. Final-ly, we modified networking between the server and mobi-le client to leverage HTTP. By supporting HTTP uploading,we could avoid slow media messaging communication whenother data carriers (e.g., 802.11b) were available.
Diary Study of Informal LearningFour external experimenters used a recent version of Mo-mento to conduct week-long diary studies of young adults’approaches to learning new technologies. The study wasconducted at two sites over the course of two months: three
7
experimenters ran six participants at one site and two expe-rimenters ran another six at the other site (one experimenterworked at both sites). All participants were 13-19 years old.Also, across both sites participants started and completed thestudy on different days. We interviewed experimenters aboutthe data resulting from their study. Our goals in observingthis study were to get feedback on the ability of Momento tosupport studies distributed across different sites and multipleexperimenters as well as back-and-forth interaction betweenexperimenter and participants during situated needs finding.
SetupParticipants used standard media messaging applications ontheir mobile phones to capture images. Participants unfami-liar with MMS were instructed to send text descriptions ofevents via SMS and, in some cases, given a digital came-ra. Later in the study, experimenters also occasionally sentfeedback to participants to indicate that they needed to fo-cus on capturing different events or simply to offer encou-ragement. At the end of the study, experimenters conductedin-person interviews with participants, using the Momentodesktop platform to review captured data. The experimentersalso reimbursed participants for any extra expenses incurredsending messages for the study.
Experimenters used the Momento server to share data acrossdifferent sites. Typically, one researcher would run the desk-top platform for a couple of days before handing off thestudy to another experimenter. To do this, an experimenterwould use the Momento desktop platform to upload the stu-dy from her personal computer to the server, and a secondexperimenter would then use Momento to download the stu-dy from the server to his personal computer.
Participant experienceThe study was in line with most participants’ habits – theywere comfortable taking photos with their phones. Theseparticipants often sent photos via media messaging to fri-ends, and were able to use the same feature to send photosto Momento. This study helped to validate the advantages ofpiggybacking on familiar devices and applications.
Initially, participants were somewhat frustrated with the lackof immediate feedback from experimenters. Experimentersattempted to mitigate this concern by sending some feed-back, positive or negative, for most captures. An unexpectedside benefit of Momento seen in this study was that parentswere pleased that their children could contribute to a studywithout being followed or watched by a stranger.
Experimenter experienceMomento facilitated many aspects of the experimental life-cycle. The experimenters were able to configure the studyrapidly and run pilots using their own devices. They werethen able to use the same settings (with the exception of theparticipants’ mobile phone information) for the actual study.The experimenters then used the interface to monitor eventsduring the study – in several cases they diagnosed problemswith participants’ mobile phones by noticing problems withcaptured data (e.g., only text from a phone that supportedMMS, or text added onto messages by the carrier network).As mentioned, experimenters also used the interface to pro-
vide feedback to participants, most commonly acknowledg-ment and encouragement.
Experimenters wanted to be able to use Momento during eli-citation interviews. Specifically, they wanted to show indivi-dual participants their own timeline of images and to scrollthrough their events in the detail view. While this was alrea-dy supported, it raised privacy concerns that we addressed:First, we augmented the timeline to allow experimenters toswitch between views that included all participants versusonly one participant. Second, we added the ability to scrollthrough incoming message details per-participant while kee-ping experimenter notes private. We also added a feature toprint out physical copies of messages (including text and pic-tures) to support elicitation interviews at which a computerwas not available.
Furthermore, experimenters felt that participants should beable to annotate media captures via the web, an issue thatalso arose in the Scribe4Me study. We added a web inter-face to the server with annotation support. Also, we observedthat some experimenters had difficulty sifting through hun-dreds of captures represented on some participants’ timeli-nes. We added an interactive table to the desktop platformlisting all incoming messages. Experimenters found the lista more useful way of navigating incoming messages than thetimeline for periods of high response rates (more than 20 perparticipant-hour). Finally, experimenters imported Momen-to’s data files into analysis programs.PhotoSketch: Supporting Informal Classroom LearningAnother group of external researchers are using the currentversion of Momento to conduct one-day workshops explo-ring a mobile learning application (PhotoSketch) for child-ren in fifth grade (ages 10-11) at an elementary school. Theresearchers have thus far conducted two workshops, invol-ving 11 and 13 participants and three and four experimen-ters, respectively. Our primary goal for this experiment wasto observe how easily Momento could be extended to dif-ferent environments. We interviewed experimenters to un-derstand the process of building a prototype application thatintegrated into a classroom scenario using Momento. We ga-thered feedback from the teachers as well.SetupThe experimenters designed the workshop as a game forstudents, held at the students’ schools. The objective ofthe game was for students to become more aware of phy-sical mechanisms around them. The students were to mo-ve around the schoolyard, photograph objects with movingparts, and annotate the photographed objects to show whichparts moved and how they moved. Afterward, students wereto share their thoughts regarding the photographs and sket-ches that they had created.
The experimenters configured the mobile client on a PocketPC device to capture a photo and annotation. The experi-menters also brought a laptop to the schools, primarily tomonitor captured events in real-time (with the help of thedesktop platform). The experimenters also ran a Momentoserver on the laptop and configured the laptop for peer-to-peer wireless networking. In this way, the mobile devicescould connect directly to the server running on the laptop
8
without depending on an external network (there was noneat the schools).
To run the workshops, experimenters divided the partici-pants into small groups and showed each group how to usethe mobile application. They then let the students take andannotate photos. During this phase, the experimenters andthe students’ teacher monitored the groups to help them re-cover from mistakes (most commonly, accidentally exitingthe application). Afterward, the experimenters reviewed cap-tured data with the students in a classroom setting.Participant experienceAlthough this experiment required participants to use a newdevice, it avoided some issues, such as carrying an extra de-vice, by being constrained to a workshop setting. In inter-views, experimenters reported that participants had little tosay about the mobile application itself, but it was clear thatwith only a brief training session they were able to use theapplication to take and annotate photos. The only problemreported was that occasionally one group’s device launcheda different application that momentarily distracted from themain task. The experimenters also noted that participantsuniversally liked using the application.
The teachers’ experiences were also largely positive. Basedon her experience with PhotoSketch, one teacher is integra-ting a series of similar workshops into her curriculum.Experimenter experienceExperimenters felt that using Momento was a large improve-ment over their fall back approach, having students manuallydraw both the object and the motion of the object. Because ofthe work involved in coordinating with schools and teachersand developing and iterating the overall workshop design,experimenters had little time to develop a mobile prototype,but were able to do so with Momento in minutes. One ex-perimenter said that the use of a mobile application “simplywouldn’t have happened” without Momento.
Experimenters wanted slightly more control configuring themobile client than was supported at the time. For example,they asked us to add support for a color scheme and lengt-hier text description. Also, for this application experimen-ters needed to configure the desktop platform, server, andclient to work on an ad hoc network. To support this, noadjustments were necessary to the desktop platform nor mo-bile client. However, the server configuration had involvedsome complicated manual processes that the experimentersinitially found too difficult to complete. To address this, westreamlined server configuration to the point that the experi-menters needed only to run four commands to configure thecore system and their study.DiscussionOverall, our studies have shown that Momento can be a po-werful tool for ubicomp experimenters and a usable tool forparticipants. Although the studies just described providedoverall validation for our concept, they also provided valua-ble feedback about the features and structure of Momento,which was a basis for iterative improvement.
In addition to the three experiments described in depth here,we ran multiple pilots and one extended study (Scribe4Me,
described earlier in the paper). All of these experiences alsocontributed to Momento’s iteration. Some of the importantfeatures of Momento that were influenced or identified du-ring iteration include:
• Increased support for piggybacking on existing devices(AwarenessBoard)
• Addition of support for time and place (AwarenessBoard)
• Expanded and more sophisticated networking support(AwarenessBoard)
• Better privacy support when viewing data on the desktopplatform (Informal Diary)
• Better support for sifting through hundreds of captures, anopposite problem to data sparsity (Informal Diary)
• Privacy-sensitive web access (Informal Diary)
RELATED WORKSituated experimentation is an area receiving much attenti-on in ubicomp and mobile device research. Laboratory stu-dies can uncover some interface and navigation flaws [3,20],especially when the physical configuration of the study em-bodies some aspects of a field setting [21]. However, asZhang et al.’s review of emerging mobile and ubicomp re-search trends shows, the difficulty of matching the realismof rich mobile contexts in laboratory settings makes fieldstudies paramount because they can unearth unexpected be-haviors and adaptations [28]. For example, Benford et al.coordinated one of the most extensive field deployments ofa ubicomp technology, Can You See Me Now?, a mobilegame in which participants raced through city streets to catchvirtual avatars that they tracked on mobile devices [2]. Theauthors compiled data from a variety of sources, includingvideos, interaction logs, voice and text communications bet-ween players, and interviews, and found that connectivityand location tracking irregularities could be co-opted andintegrated into the game as a feature rather than an error.Halloran et al. also showed that long term, situated designcan promote a sense of ownership of the technology amongststakeholders, encouraging adoption [12]. However, Zhang etal. and Davies et al. note that researchers find it difficult tomake use of real devices and to observe and collect data inthe field, issues that Momento can address [8].
From a tools perspective, many researchers have developedsystems that support the rapid creation of throw-away func-tional prototypes to be used during iterative design (e.g.,[10, 13, 22, 23]). These tools may use a mixture of Wizardof Oz, pre-existing library elements, and coding. For ex-ample, Topiary can be used to prototype location-enhancedapplications in which a wizard typically shadows a partici-pant and enters location information as the participant movesaround [23]. DART extends Macromedia Director to allowdesigners to Wizard of Oz augmented reality applications. Itprovides multimedia support for video, tracking, and sensordata [10]. iTools integrates support for evaluation into theirprototyping platform by showing the design of a task and theuser data derived from completing that task together in oneinterface [22]. A handful of other projects have taken ourapproach of using SMS and MMS to enable realistic studies(e.g., [1,6,15,16]), and other tools have been developed spe-
9
cifically to support event-contingent ESM [11,17,19]. Also,InterviewViz supports assisted photo elicitation [26]. Noneof these tools provide integrated, usable support for experi-mentation to the extent that Momento does. In addition to in-tegrating multiple experimental methods, Momento includesa collection of novel features ranging from privacy sensitivereview to live monitoring and feedback that were motivatedby our user-centered, iterative approach.
Recent work has suggested guidelines for tools that supportexperimentation. For example, in an analysis of several fieldevaluations of ubicomp applications, Crabtree et al. elabo-rated the need for tools that support side-by-side reviewsof video and log data; mechanisms to support user captu-red video; synchronization between recordings; a timeline-based visualization interface for rapid review of differentdata sources at different times; and better support for an-notation [2, 7]. Morrison et al. implemented some of theseconcepts to support participant observation and logging ofmobile application use [25].CONCLUSIONS AND FUTURE WORKWe presented the user-centered design of Momento, a toolthat provides integrated support for situated evaluation ofubicomp applications. Momento provides a desktop plat-form that connects experimenters with participants in thefield, and includes a simple, configurable application exten-ding the capabilities of mobile devices to support participantdata collection. Our validation demonstrates how Momen-to was iteratively designed to meet the needs of participantsand experimenters in four separate studies that involved avariety of tools and evaluation methods.
Currently, Momento’s primary limitation is that it does notsupport applications that require rapid synchronization orstreaming between the client and server (e.g., mobile vi-deoconferencing). In the future, we plan to add support forstreaming media as well as explore models for supportingdynamic coordination among multiple experimenters. Also,we are investigating ways to merge into one visualizationdata gathered using Momento with data gathered using othermethods, such as interview transcriptions.
Momento is open-source software and is available athttp://www.m0ment0.com/.ACKNOWLEDGMENTSThis work was sponsored in part by Intel and by NSF grantsIIS-0205644 and IIS-0511895. We thank Scott Hudson,Anind Dey, Peter Lyman, Tara Matthews, Heather Horst,Dan Perkel, Christo Sims, Jonathan Hey, Jaspal Sandhu, Su-san Fussell, and Peter Scupelli for their help.REFERENCES
1. M. Ananny, K. Biddick, and C. Strohecker. Constructing publicdiscourse with ethnographic/SMS “texts?”. In MobileHCI’03,368–373, 2003.
2. S. Benford, A. Crabtree, M. Flintham, A. Drozd, R. Anastasi,M. Paxton, N. Tandavanitj, M. Adams, and J. Row-Farr. Can you seeme now? TOCHI, 13(1):100–133, 2006.
3. A. H. Betiol and W. de Abreu Cybis. Usability testing of mobiledevices: A comparison of three approaches. In IFIP INTERACT’05,470–481, 2005.
4. S. Carter and J. Mankoff. When participants do the capturing: The roleof media in diary studies. In CHI’05, 899–908, 2005.
5. S. Carter, J. Mankoff, S. Klemmer, and T. Matthews. Exiting thecleanroom: On ecological validity and ubiquitous computing. HCIJournal. To appear.
6. S. Consolvo and M. Walker. Using the experience sampling method toevaluate ubicomp applications. Pervasive Computing, 2(2):24–31,2003.
7. A. Crabtree, S. Benford, C. Greenhalgh, P. Tennent, M. Chalmers, andB. Brown. Supporting ethnographic studies of ubiquitous computingin the wild. In DIS’06, 60–69, 2006.
8. N. Davies, J. Landay, S. Hudson, and A. Schmidt. Rapid prototypingfor ubiquitous computing. Pervasive Computing, 4(4):15–17, 2005.
9. A. K. Dey, G. D. Abowd, and D. Salber. A conceptual framework anda toolkit for supporting the rapid prototyping of context-awareapplications. HCI Journal, 16(2-4):97–166, 2001.
10. S. Dow, J. Lee, C. Oezbek, B. MacIntyre, J. D. Bolter, and M. Gandy.Wizard of oz interfaces for mixed reality applications. In Extendedabstracts of CHI’05, 1339–1342, 2005.
11. J. Froehlich, M. Y. Chen, I. E. Smith, and F. Potter. Voting with yourfeet: An investigative study of the relationship between place visitbehavior and preference. In Ubicomp’06, 333–350, 2006.
12. J. Halloran, E. Hornecker, G. Fitzpatrick, M. Weal, D. Millard,D. Michaelides, D. Cruickshank, and D. D. Roure. Unfoldingunderstandings: co-designing ubicomp in situ, over time. In DIS’06,109–118, 2006.
13. B. Hartmann, S. R. Klemmer, M. Bernstein, L. Abdulla, B. Burr,A. Robinson-Mosher, and J. Gee. Reflective physical prototypingthrough integrated design, test, and analysis. In UIST’06, 299–308,2006.
14. J. Heer, S. K. Card, and J. A. Landay. prefuse: a toolkit for interactiveinformation visualization. In CHI’05, 421–430, 2005.
15. R. M. Hogarth. Is confidence in decisions related to feedback? InK. Fiedler and P. Juslin, editors, In the beginning there is a sample:Information sampling as a key to understand adaptive cognition.Cambridge University Press, 2004.
16. S. Hulkko, T. Mattelmaki, K. Virtanen, and T. Keinonen. Mobileprobes. In NordiCHI’04, 43–51, 2004.
17. G. Iachello, K. Truong, G. Abowd, G. Hayes, and M. Stevens.Event-contingent experience sampling to evaluate ubicomptechnology in the real world. In CHI’06, 1009–1018, 2006.
18. Infotrends. Mobile Imaging Study.http://www.capv.com/home/Multiclient/mobileimaging2.html, 2006.
19. S. S. Intille, E. M. Tapia, J. Rondoni, J. Beaudin, C. Kukla,S. Agarwal, L. Bao, and K. Larson. Tools for studying behavior andtechnology in natural settings. In Ubicomp’03, 157–174, 2003.
20. A. Kaikkonen, A. Kekalainen, M. Cankar, T. Kallio, andA. Kankainen. Usability testing of mobile applications: A comparisonbetween laboratory and field testing. Journal of Usability Studies,1(1):4–17, 2005.
21. J. Kjeldskov, M. B. Skov, B. S. Als, and R. T. Høegh. Is it worth thehassle? Exploring the added value of evaluating the usability ofcontext-aware mobile systems in the field. In MobileHCI’04, 61–73,2004.
22. S. R. Klemmer, A. K. Sinha, J. Chen, J. A. Landay, N. Aboobaker, andA. Wang. SUEDE: A Wizard of Oz prototyping tool for speech userinterfaces. In UIST’00, 1–10, 2000.
23. Y. Li, J. I. Hong, and J. A. Landay. Topiary: A tool for prototypinglocation-enhanced applications. In UIST’04, 217–226, 2004.
24. T. Matthews, S. Carter, J. Fong, C. Pai, and J. Mankoff. Scribe4me:Evaluating a mobile sound translation tool for the deaf. InUbicomp’06, 159–176, 2006.
25. A. Morrison, P. Tennent, and M. Chalmers. Coordinated visualisationof video and system log data. In CMV’06, 91–102, 2006.
26. N. A. Van House. Interview viz: Visualization-assisted photoelicitation. In Extended abstracts of CHI’06, 1463–1468, 2006.
27. A. Weilenmann. Negotiating use: Making sense of mobile technology.PUC, 5(2):137–145, 2001.
28. D. Zhang and B. Adipat. Challenges, methodologies, and issues in theusability testing of mobile applications. IJHCI, 18(3):293–308, 2005.
10
