Experience sampling method: preliminary findings sunny consolvo & miriam walker intel research seattle [email protected] user-centered evaluations

experience sampling method: preliminary findings

sunny consolvo & miriam walkerintel research seattle

[email protected]

user-centered evaluations of ubicomp workshop4th int’l conference on ubiquitous computing

september 29, 2002

experience sampling method -- ubicomp 2002 2

before we get started…

this work is preliminary

wrapped up data collection this month

expect future publication


outline of talk

intro to experience sampling method (esm)

case study: the personal server

issues

wrap-up


esm: a brief introduction

also called: beeper studies time sampling

field study technique from psychology used to understand areas such as…

mood time use social interactions

we’re exploring its uses for ubicomp


esm: a brief introduction

fill out questionnaire when alerted researcher is not present alerted several times per day business as usual until alerted

duration is often 1-3 weeks


running esm: tools

need to choose tools for: alerting participant

random alertstimed alertsevent-based/user-driven

delivering questionnaire capturing participant responses


choosing tools: alerting

tools for alerting the participant include: cell phones pagers watches PDAs custom devices

some desirable abilities: specify time period during which alerts may go off specify # of alerts per time period & overall


choosing tools: delivering

tools for delivering questionnaires include: paper booklets PDAs phones (traditional or mobile)

live researcher recorded message

audio recordings (e.g., cassette player)


choosing tools: capturing

tools for capturing responses include: paper booklets PDAs phones (traditional or mobile)

live researcher recording service (e.g., Tellme)

audio recordings (e.g., cassette player) cameras


esm: some advantages

reduces effects of intrusive observerslarger sample size & time intervalreduces memory biases by asking

about current activity, feelings, etc. it’s all about now


esm: more advantages

less time consuming than observations, yet… study activities as they occur in the field study activities and emotions often not

accessible to observers not as disruptive to others


esm: some disadvantages

questionnaires must be brief participant must carry tools at all times power consumption and/or service

availability certain events may only show up in very

large samples sizes/durations more intrusive than interviews & surveys


outline of talk


case study: personal server

issues

wrap-up



personal server team was interested in: how people use information

what they wantwhen they want itwhere they want it

used esm to help collect this data


pilot study #1

4 participants lab researchers & administrative staff

palm m100s for alerting delivering capturing

original boston college ESP software for running ESM on palms

duration: 1½ - 4 days


pilot study #1: problems

palm dropped batteries fell out data lost only method of alert was beep (m100)

participants wouldn’t carry it in several situations once volume of beep was set, it couldn’t be changed

only response option: multi-choice, single answer difficult for participants to answer some questions

e.g., “where are you?”

some participants wanted a bag to carry the palm participants wanted to know about missed

questionnaires


pilot study #2

4 participants lab researchers, administrative staff, family

modified iESP software palm m500s for alerting and delivering

for capturing: 2 participants carried fold-out keyboards (for palm) 2 participants carried paper booklets all 4 carried digital cameras

for describing where & what they were doing

duration: 1½ - 4 days


pilot study #2: justifications

upgrading to m500s: ability to alert via beep and/or vibrate almost impossible for batteries to fall out

iESP software added functionality, including: freeform text entry multi-choice, multi-answer new formats for multi-choice, single answer slider (e.g., “on a scale of 1 to 5…”) ability to change volume, vibrate, etc. during study ability to check how many questionnaires have been

completed vs. asked during study


our implementation: the study

31 participants recruited from local community

pre/post-study interviews + 1 week with equipment 10 questionnaires/day 70 questionnaires total

incentive (in USD) $50 for participating and returning equipment $1 per completed questionnaire $120 total possible


the study (continued)

palm m500 to alert, deliver, & capture modified iESP software

more about capturing: also used camera free-form data entry:

paper booklet for all 31 participants graffiti / soft keyboard for all 31 participants full-sized fold-out keyboard for 9 of 31 participants


what the participants carried


type of data collected

responses to following types of questions: yes / no multiple choice, single answer multiple choice, multiple answers freeform text entry

looks a lot like survey data except multiple responses from same participant in

variety of circumstances over time


other data collected

qualitative data from interviews participants’ photos

some photos were of interest to the participant other photos documented where s/he was or what

s/he was doing


questions on the palm


more questions


participant photos: in transit


participant photos: at home


participant photos: at work


participant photos: retail


participant photos: pets


outline of talk



issues

wrap-up


our tool decisions…

chose palm because: no service problems—cost or coverage power consumption not an issue for duration know when participant fills out questionnaire probabilities and contingencies can be used

if a, then b if b, then not c d 50% of time

don’t have to transcribe data


tool decisions

used camera to see if it would help participants describe… where they were what they were doing

photos used to jog memories during post-study interviews


according to the participants… 1/31 would not participate again 2/31 would only do it for more money 28/31 would participate again

“it was cool” “I felt like a spy” “friends were jealous of the [fold-out] keyboard”

a lot to carry mixed feelings about carry bag

several claimed they would be purchasing palms only one participant wanted voice recording option

when asked, most did not respond favorably


from our perspective…

woo-hoo! we got all of the equipment back only 1/31 palms was broken of the 31 participants and 70 questionnaires:

28 participants completed 50+ questionnaires 13 of whom completed 60+ questionnaires

2 participants completed 25- questionnaires 1 participant performed a hard reset and returned

the palm with no data or software


outline of talk



issues

wrap-up


wrap up: we’ll use it again

pleased with results so far

intend to use esm again


wrap-up: pda software

original ESP software available at: http://www2.bc.edu/~barretli/esp/

iESP software (includes upgrades): launching on intel research seattle web site in Oct

or Nov 2002 if you want to be notified…

business card with “iESP notification” written on it legibly fill out notification sign-up sheet


acknowledgements

drs. lisa & daniel barrett justin silves gaetano borriello ken anderson rahul shah trevor paring roy want john light our pilot testers and our colleagues at intel research seattle


questions?


examples of study questions

where is your cell phone? in my… hand pocket/belt clip bag car home office somewhere else and with me somewhere else and not with me


examples of study questions in the last 30 minutes, which of the following would

you have liked to have access to but didn’t have available? web pages movie times restaurant reviews personal calendar telephone number(s) computer files music files shopping list “to do” list

Documents

Experience sampling method: preliminary findings sunny consolvo & miriam walker intel research seattle [email protected] user-centered evaluations