DIY mobile usability testing - IA Summit 2011

we are ...

Bernard, packet core engineer at NSN

Belén, interaction designer at Intel

usability testinga process that employs people as testing participants who are representative of the target audience to evaluate the degree to which a product meets specific usability criteria.

Handbook of usability testing 2nd Ed., J. Rubin and D. Chisnell

please, stand up

everyone

sit down if you don’t have a US cellphone

with a data plan

sit down if you don’t like beer

sit down if you are absolutely terrified by the idea of being our

test subject

why recording?

memory aid

powerful communication tool

reactions

actions

dut = mut

where:

dut = desktop usability testingmut = mobile usability testing

dut = mut + afec

afec = a few extra challenges

which phone?

which context?

which connection?

which phone?

which context?

which connection?

web task success rates

feature phones 38%

smartphones 55%

touch phones 75%

Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009http://www.useit.com/alertbox/mobile-usability.html

web task success rates

feature phones 38%

smartphones 55%

touch phones 75%

Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009http://www.useit.com/alertbox/mobile-usability.html

web task success rates

feature phones 38%

smartphones 55%

touch phones 75%

Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009http://www.useit.com/alertbox/mobile-usability.html

web task success rates

feature phones 38%

smartphones 55%

touch phones 75%

Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009http://www.useit.com/alertbox/mobile-usability.html

handset usability affects test results

remember ...

test with participants’ own phones

if not possible, include training and warm-up tasks

which phone?

which context?

which connection?

field vs. lab

It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the FieldC.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006

experts disagreeaccording to our study there was no difference in the number of problems that occurred in the two test settings. Our hypothesis that more problems would be found in the field was not supportedUsability Testing of Mobile Applications: A Comparison between Laboratory and Field TestingA. Kaikkonen, T. Kallio, A. Kekäläinen, A. Kankainen, M. Cankar - Journal of Usability Studies, 2005

experts disagreeevaluations conducted in field settings can reveal problems not otherwise identified in laboratory evaluations

It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the FieldC.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006

but they all agreeevaluations in the field (are) more complex and time-consuming

testing in the field requires double the time in comparison to the laboratory

It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the FieldC.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006

Usability Testing of Mobile Applications: A Comparison between Laboratory and Field TestingA. Kaikkonen, T. Kallio, A. Kekäläinen, A. Kankainen, M. Cankar - Journal of Usability Studies, 2005

testing in the lab is better than no testing

remember ...for most software, lab testing is fine

if you must do field testing

do it late

plan and run pilot tests

be prepared (like the Scouts)

which phone?

which context?

which connection?

!"!"#$%&'("%)*%'!+&,-.$/&01'2..'3/0"$,'3#,#)4#5 #$%&"'())67"$$89::;;;<="#$%&,"%)*%<=+*

!"#$%& '&()"*+,-*.//$0,1*")(2

US Mobile Data Market Update Q4 2010 and 2010, Chetan Sharma Technology & Strategy Consulting http://www.chetansharma.com/usmarketupdate2010.htm

remember ...

do not test over wi-fi

cover participants’ data costs

where:

dut = desktop usability testingmut = mobile usability testingafec = a few extra challenges

dut = mut + afec

tsdohoeaygtrtwt = the small detail of how on earth are you going to record the whole thing

tsdohoeaygtrtwt( )

why recording?

memory aid

powerful communication tool

4 approaches to the small detail of how on earth

are you going to record the whole thing

.

1. wearable equipment

Methods and techniques for field-based usability testing of mobile geo-applications, I. Delikostidis (2007) International Institute for Geo-Information Science and Earth Observation (Enschede, The Netherlands)

allows testing in the field

but ...

difficult and time-consuming to set up

intrusive, uncomfortable and heavy

UsabilityExpo by Digia for S60 3rd and 5th ed, WinMo 6.1+ and Maemo 5http://www.digia.com/C2256FEF0043E9C1/0/405001348

2. screen-capture applications

provides high quality screen recording

but no application will support all platforms

http://www.digia.com/C2256FEF0043E9C1/0/405001348

it’s not cheap

https://shop.digia.com

and participants won’t appreciate you installing stuff on their phones

Handheld Usability, S. Weiss (page 174)

3. document cameras

Towards the Perfect Infrastructure for Usability Testing on Mobile DevicesR. Schusteritsch, C.Y. Wei, M. LaRosa - Google (CHI 2007)

Google

Photo from Nielsen Norman Mobile Usability workshop handout (London, 22 May 2009)

both video streams fed onto a laptop and recorded with Morae

webcam records participant’s face

document camera with autofocus controlled remotely from a laptop records screen and fingers

Nielsen Norman

Elmo TT-02RX Teachers Tool

popular and well documented

it’s not cheap

http://www.bhphotovideo.com/c/product/644591-REG/Elmo_1304.html

participants must keep within the camera range

phone must lay on a desk or be hold at a flat angle

http://www.flickr.com/photos/zabriskiepoint/2806511301/sizes/m/in/photostream/

4. mounted devices

Mobile Device Camera by Noldus http://www.noldus.com/human-behavior-research/accessories/mobile-device-camera-mdc#

Mobile Device Camera by Tracksys http://www.tracksys.co.uk/product-details.php?id=9

4a. ready-made mounted devices

Little Springs Designhttp://www.littlespringsdesign.com/blog/2008/Jun/usability-testing-for-mobile-devices-2/

4b. DIY mounted devices

Little Springs Designhttp://www.littlespringsdesign.com/blog/2008/Jun/usability-testing-for-mobile-devices-2/

Nick Bowmast http://www.bowmast.com/mob-device-cam/

4b. DIY mounted devices

Little Springs Designhttp://www.littlespringsdesign.com/blog/2008/Jun/usability-testing-for-mobile-devices-2/

Nick Bowmast http://www.bowmast.com/mob-device-cam/

Usability Scienceshttp://www.usabilitysciences.com/services/lab-based-usability-testing/mobile-usability-testing

4b. DIY mounted devices

Little Springs Designhttp://www.littlespringsdesign.com/blog/2008/Jun/usability-testing-for-mobile-devices-2/

Nick Bowmast http://www.bowmast.com/mob-device-cam/

Usability Scienceshttp://www.usabilitysciences.com/services/lab-based-usability-testing/mobile-usability-testing

Google Towards the Perfect Infrastructure for Usability Testing on Mobile Devices, R. Schusteritsch, C.Y. Wei, M. LaRosa - Google (CHI 2007)

4b. DIY mounted devices

natural interaction with the phone

but it’s not cheap

http://www.90percentofeverything.com/2010/05/07/quick-tip-make-your-own-iphone-usability-testing-sled-for-5/

messy to build

http://www.littlespringsdesign.com/blog/2008/Jun/usability-testing-for-mobile-devices-2/

and ...

if bulky they can prevent single-hand use

if heavy they can become tiring during long tests

easy to put togethercheaprepeatableallows holding the deviceallows one-handed usesupports all form factorsruns tests with participants’ phonescaptures screen, face and fingersgives enough video quality

easy to put together

cheap

repeatable

allows holding the device

allows one-handed use

supports all form factors

runs tests with participants’ phones

captures screen, face & fingers

gives enough video quality

easy to put together

cheap

repeatable

allows holding the device

allows one-handed use

supports all form factors

runs tests with participants’ phones

captures screen, face & fingers

gives enough video quality

http://www.wired.com/gadgetlab/2010/07/the-pencil-ipad-stand-smart-enough-to-impress-a-new-yorker/

the spirit

the ingredients

two meccano trunions (part no. A126)

two 5 and 6-hole meccano strips (part nos. 5 & 4)

one 11-hole meccano strip (part no. 2)

six meccano screws & nuts (part nos. 69 & 37h)

one 13-20mm jubilee clip

one HUE HD webcam

a second USB webcam

a USB male to female extension cable

blu tack

an allen key

a meccano wrench

a screwdriver

a Windows computer

screen recording software

You are planning a visit to the Denver Zoo after this (great) session.

Go to www.denverzoo.org and find out which bus number will bring you to the zoo from the Colorado Convention Centre.

a task

how was that?

b&b’s

easy to put together

cheap

repeatable

allows holding the device

allows one-handed use

supports all form factors

runs tests with participants’ phones

captures screen, face and fingers

gives enough video quality

b&b’s

easy to put together

cheap

repeatable

allows holding the device

allows one-handed use

supports all form factors

runs tests with participants’ phones

captures screen, face and fingers

gives enough video quality

15-model meccano set 18.33

Hue HD webcam 28.94

Philips webcam 52.08

additional meccano parts 4.90

blu tack 0.98

jubilee clips (x2) 1.55

USB cable 8.99

screwdriver 5.99

CamStudio 0.00

total (in GBP) 121.76

in USD: 196.40

b&b’s

easy to put together

cheap

repeatable

allows holding the device

allows one-handed use

supports all form factors

runs tests with participants’ phones

captures screen, face and fingers

gives enough video quality

b&b’s

easy to put together

cheap

repeatable

allows holding the device

allows one-handed use

supports all form factors

runs tests with participants’ phones

captures screen, face and fingers

gives enough video quality

weight: 125 gramsan iPhone weighs 137 grams

an iPad weighs 680 grams

I weigh 55,000 grams

a blue whale weighs 136,400,000 grams

b&b’s

easy to put together

cheap

repeatable

allows holding the device

allows one-handed use

supports all form factors

runs tests with participants’ phones

captures screen, face and fingers

gives enough video quality

Nokia E66slider with 12-key numeric keypad

Nokia E71

candybar with qwerty keyboard

BB Bold

iPhone 3Gtouchscreencandybar

Nokia N900

slider touchscreen with qwerty keyboard

b&b’s

easy to put together

cheap

repeatable

allows holding the device

allows one-handed use

supports all form factors

runs tests with participants’ phones

captures screen, face and fingers

gives enough video quality

b&b’s

easy to put together

cheap

repeatable

allows holding the device

allows one-handed use

supports all form factors

runs tests with participants’ phones

captures screen, face and fingers

gives enough video quality

we expect much of our buildings: they need to have firm foundations, solid structures, pleasing aesthetics. We should expect the same of emerging mobile systems.

Mobile Interaction Design, M. Jones and G. Marsden (2005)

thanks!belenbarrospena@gmail.comb@runningwithbulls.com

@belenpena @bernardtyers

http://belenpena.posterous.comhttp://www.runningwithbulls.com