Outside the boxUser-testing context-aware mobile technology in the field – a case studyJerome Turner

• The product• Planning• The user-tests• Analysing results• Lessons learned: testing mobile apps in the field• (oh yes, the results)

The product

Content screens

Planning

• Main problem – how do we record and observe mobiles?• One solution, fixed in lab conditions:

Validconcept.com

• Another solution, Liliput. In the field but intrusive:

http://portal.acm.org/citation.cfm?id=1240907

Mobile observation

• User-lab’s kit to date, far too cumbersome• What are we interested in recording and at what cost?• Screen capture won’t record hand/stylus movement. • Is visual recording more useful than notetaking?• Do we record all screen / mobile activity if it will greatly

affect the user’s experience / usage?

• Compromise: use one discrete hat cam to record as much video/audio as possible.

• NB Interested in broad experience, not just around device

Hat cam

• Main difference, no live observer.• Tests should still be structured

– Recruit participants, for timed sessions– Use script and task to keep schedule and ensure equal

treatment throughout day– Use demographic form, tweaked to suit the tests– Use a ‘log sheet’ to take notes whilst facilitating– End of session, use

• randomised word list to tick• questionnaire, responses to the product• understanding of some technical terms • incentive sign-off sheet

Planning – from lab to the streets

• Usually, would test product first, establish expected issues during sessions, and thus make log list.

• Here, facilitator made log list based only on experience of building product / expectations of technology i.e. many tech errors included

• Also included boxes for other observational logging…• …and space for comments and notes.

Planning – facilitator’s log sheet

• Through external recruiters. • Requirements sent without wanting to give too much away• They should be between 18 and 70 years of age. • They should have some experience of using a PDA, iPhone, Blackberry-style phone,

smartphone or other similar digital mobile device. • They will need to be on their feet with me for potentially a full hour, walking around the

centre of Birmingham (at a leisurely pace). • They will need to be comfortable with wearing a baseball style cap with a camera fitted

that records their ‘point of view’ and voice. It doesn’t record their face. (if we really have trouble finding people for this, this requirement can be removed for a couple of people).

• We need 8 people for Thursday and 8 for Friday. Sessions on each day to start at 9.00, 10.00, 11.00, 12.00, 13.30, 14.30, 15.30, 16.30

• They will receive £30 in cash on the day. • It would be helpful if they weren’t people booked for us before, so they don’t know me. • They should dress expecting to be outside for the full hour, not under cover. E.g. Bring a

brolly!

Recruiting

• Meeting users – make sure you have each others’ mobile numbers and have told them what you’ll be wearing (something distinctive)

• The street: it’s harder to have everything to hand, prepare yourself 1. clipboard and pen 2. device 3. recording device/s

• The sessions should feel more relaxed than lab, but still remember script and schedule

• Additional safety announcements

The tests – meeting users

• Users asked to ‘think aloud’ (en.wikipedia.org/wiki/Think_aloud_protocol)

main recording method, for camera / mic / clipboard• Sessions were relaxed, users didn’t display signs of being

under pressure, partly due to ‘open’ nature of main task? • Users allowed to roam, even slightly off course e.g. Bullring

The tests – the session

• As in lab tests, facilitator only intervened when help required• Facilitator very aware log sheet was main observing tool, no

other observer• Facilitator was also relaxed, an effect of doing ‘own research’

rather than for a client? i.e. knowing expectations• Facilitator logged notes, also watched time, to bring users

back to start point after 45 minutes • After 7 sessions, very tiring

The tests – facilitator’s role

• It broke. After first session, found it had crashed.

• May have been because facilitator held receiver whilst wireless camera roamed slightly.

• Luckily, had fitted each participant with audio recorder and tie clip mic.

• Mic alone, not intrusive, especially as one user walked off wearing it!

• NB what is the effect of not having to wear hat cam, when they were expecting to?

The tests – the problem with hat cam

• Some collation very easy i.e. questionnaires• Collation of log sheet issues, plus other

common issues found from notes (see next) • No video at all, camera crashed both days• Day 2 of audio recordings lost, Day 1

recordings yet to be played back• Some quotes pulled out• All notes could have been typed up to

create a word cloud, but danger of words taken out of context of sentences

Analysing results

• Keep log sheet simple, issues to tick off, then a large empty area. Maybe use error, confusion boxes, but these less used on 2nd day

• What are legal issues of mobile testing in streets? E.g. user gets run over, filming in public. Add disclaimer to participation agreement and script

• Meeting strangers, wear something distinctive• In analysis, paper notes easier to refer to than time-based recordings • Still unsure of what could be learned from a working hat cam alone

How much can we see? • Space out test days or alternate facilitators to avoid tiring out. Maybe

4 sessions a day instead of 7.

Mobile testing – lessons learned

• What is effect of asking users to do something, and then telling them they don't need to? e.g. hat cam

• Get hat cam working. How can it be most useful?• What is effect of wearing our old style kit on the test?• Could shorter ‘café’ sessions be run in the field on

mobile devices?

Further research

Finally, some of the results

Words picked from word list, limited to top 30 words (wordle.net)

Words rated as ‘top 3’ from word list, no limit (wordle.net)

Logged issues. Did I get tired / bored?

Logged issues – totals

Logged issues totals

Would want to select where I'm going to.[walking to tree of life marker] "I'm intrigued now.Very interesting, learned a few things on the way’[in Bullring] 'Don't think it could give you shop info because of 3 floors [and GPS] and there are too many shopsThank you, I really enjoyed thatIt would be good to have a button for each saying 'Take me there', doing so on the map, or giving me text directions.The map is so accurate!Its got all the numbers for the hotels.Do like the history information. [tree of life] 'was totally unaware of that […] as a citizen, that's more interesting to me than the shops I didn't even realise there were that many places in the Bullring If you were a stranger and you could download this, not only would it guide you there but give you other information.It's told me its free to go into the museum. I'm a schoolteacher so this would be great, you could plan a school trip around it. You've got your very own tour guide with you! Architecture tours for architecture students? As a tourist, would pay £10 or £15 for it