Mobile Usability Testing Inst. f. Softwaretechnik und Interaktive Systeme qse.ifs.tuwien.ac.at, [email protected], [email protected]

Mobile Usability Testing

Inst. f. Softwaretechnik und Interaktive Systeme

qse.ifs.tuwien.ac.at, [email protected], [email protected]

TopicsI. Motivation & Definition

II. Mobile Devices

III. Challenges in Mobile Usability Testing

IV. Methods & Approaches

V. Heuristics & Guidelines

VI. Examples & Ideas

Lecture :: „Mobile Usability Testing“


Goals :: What do we want to achieve in this lecture?

2

*vgl.: Schilit and Theimer (1994)

What we want:

• Enable you to set up a Mobile Usability Test based on Quantitative as well as Qualitative Methods.

• Give you Inspirations & Ideas about what can be done to make mobile applications more usable.


Methods :: How do we want to achieve our goals?

3

*vgl.: Schilit and Theimer (1994)

How we want to do it:

• Creating an awareness for the challenges of mobile-Usability

• Compare common Usability Testing Methods

• Overview about possible Heuristics & Guidelines for Mobile Usability Testing

I. Motivation & Definition

Considerations for Mobile Testing:

• Results are strongly influenced by surrounding environment

• Results are influenced by devices used for testing

• Collected Data will be „richer“ (Gesture, Voice, …)

Definition :: „Mobile Usability Testing“

4


I. Motivation


For every dollar spent acquiring a customer you will spend $100 dollars reacquiring them after they leave because of poor usability

or bad customer service. (*)

Costs of late Usability research

6

*vgl.: MauroNewMedia (2002)


Usability Engineering in the Software Development Process

7

*vgl.: INTERACT 2001 Workshop, Jan Gulliksen, Inger Boivie)


II. Mobile Devices

II. Mobile Devices

Are used “on -the-run and for activities that may last only a few seconds or are highly context dependent” (*)

Mobile Devices :: Definition

9

*vgl.: (Vetere et al., 2003, p.1)

II. Mobile Devices

„Mobile Devices: One Generation From Useful“(*)

Tighter Integration needed:• Devices do not work together well with each other.

Synchronization with other Applications

• Different Features packed into one device do notact as one entity PDAs with GSM modules

• Ad-hoc Networking is still far from realisation Network coverage & Roaming (WLAN, UMTS, GPRS, GSM) Industry Standards (Bluetooth compatibility, vulnerability)

Mobile Devices :: Realistic View 1

10

*vgl.: (Jakob Nielsen 2003, Alertbox , Aug. 18th.)

II. Mobile Devices

Design / Interface Problems still to be solved:

• Deck-of-Card Size(*) limits the Screen Size Higher Screen-Resolutions, better viewing angle.

• One Dimensional Interfaces (Scroll wheels) are not suitablefor 2D – Screens.

• Text-input is still a great Problem.Small Devices small Buttons. new Button Alignments?


11

*vgl.: (Jakob Nielsen 2003, Alertbox , Aug. 18th.)

II. Mobile Devices

Fundamental Problems:

• Quality of Service of local Network Providers.Things that „could be done“ just can‘t because of local Service-Limitations or lack of network coverage. (Broadband, UMTS, etc.)

• Online Services must specialize for Mobile useMuch shorter Articles, more use of XML, simplyfied Navigation

• Reconsider the way email is used not just forward every mail to the PDA (Attachments, Executeables)


12


III. Challenges

III. Challenges

• 1. Device Proliferation

– Handling many different Devices, Rendering Methods

• 2. Application Modality

– Handling simultaneous voice / Data User interactions

• 3. User Mobility

– Users are likely to be distracted during use

• 4. Data Collection

– Recording eye-movement and video taping will not work everywhere

Challenges :: of Mobile Usability Testing (*)

14

*vgl.: http://www.littlespringsdesign.com/analysis/utest.html

III. Challenges

Devices can be• As small as possible, optimized for voice communication

• Quite large, optimized for data display

• Optimized for gaming

• Optimized for multimedia

Challenges :: 1 : Device Proliferation (*)

15

Applications are perceived differently• Reading News-Bulletin on a point-matrix phone display

content is forgotten 3 hours later.

• Reading News-Bulletin on a java-enabled 19“ CRT Monitor Will be stored in long-term memory


III. Challenges

Mobile Applications often combine Graphic / Data and Voice Elements.

• Difficult to test in an early stage of development(Software isn‘t fully functional / not yet written.)

• Test must be able to provide simultaneous experiences

Challenges :: 2 : Application Modality(*)

16


III. Challenges

Mobile Users are very likely to be distracted.

• Natural Environment is not always manageable/affordable to be simulated in a lab

• Distractions and „Normal anomalies“ (Waiter interrupting you in a restaurant to take your order) have to be part of the test

Challenges :: 3 : User Mobility

17

III. Challenges

Acquiring Data outside the Lab

• Eye tracking will hardly work on tiny screens and under mobile conditions.

• Recording tools interfere with the users‘ interaction with the device.(Cameras mounted on a cell phone make the user hold it in an unnatural position)

Challenges :: 4: Data Collection 1

18

“It is cold and snowing and you do not know from where your bus leaves in 5 minutes. You pick up your WAP phone to check: The mobile user runs to catch her bus, after her run three researchers with cameras and microphones..(*)“

*(vgl.: Per-Ola Rasmussen ExarbII 2003)

III. Challenges

Mobile Users interact not just with the screen and keyboard.

• Test needs to record, gestures, face-expressions, voice, body-language, etc.

• Much richer Data

(People leaning left and right while playing a Formula-1 racing Game on their Java-enabled Phone.)

• Record what is the user doing, what is he/she NOT doing.

Challenges :: 4: Data Collection 2

19




The common Methods

Methods :: Approaches to Mobile Usability Testing 1

21

Heuristic Evaluation(Nielsen & Mack, 1994)

Guidelines vs. Design. Used in early development

Cognitive Walkthrough(Rowley&Rhoades, 1992)

How or why a person would react in a certain situation. Based on assumptions on the user's mental model

*(vgl.: Tomas Lindroth, Stefan Nilsson & Per-Ola Rasmussen, ExarbII – HT2000)


The common Methods


22

Feature Inspection(Nielsen & Mack, 1994)

Does the product meet the users needs and demands? Used in middle stages of development

Consistency Inspection(Nielsen‚ 1995)

Checks consistency across multiple products from the same product – family.



The common Methods


23

Standards Inspection(Wixon et. Al. 1994, Nielsen 1995)

Ensures compliance with industry standards. Best used in middle-stages of development

Guideline Checklist(Wixon et. Al. 1994, Nielsen 1995)

Used in conjunction with other usability methods. The Checklists give the tester a basis by which to compare the application



The common Methods


24

Thinking Aloud(Nielsen‚1994)

Lets the evaluator understand how the user views the system.

Contextual Inquiry(Holzblatt & Beyer ‚1993)

Used to get a broad knowledge about the environment that you are producing for. More a discovery process than an evaluative process.



Methods :: Lab vs. natural environment

25

*(vgl.: Mason 1988, Järvinen, 1999)

The larger the number of factors that is under control in a test, the more scientific rigour is emphasized. The more natural like the test setting is, the more relevant and applicable the results will be.


The Ideal Test

• Natural situation / environment

• Application is fully functional

• All possible forms of devices are being tested

• Users are free to do what they would normally do

Users don't feel „tested“

• Tester can record every audio / visual / voice / movement / screen data

from the user without affecting the users behavior.

• Tester sees the Application the way the users see it.


26


Preparation• Define goals, methods and tasks/scenarios for the test.

Introduction, Warm-up• Introduce the test to the user

• Start with easier tasks, give time for a short warm-up phase.

Testing• Perform the actual test with as less interaction between user and tester

as possible

Methods :: General Mobile Usability Testing Process 1

27


Test situation

• Give the user time to get out of the test-situation.

• Then start reviewing his/her opinions, impressions and suggestions.

• Make sure to discuss special occurrences that may have happened during the test with the user.

Methods :: General Mobile Usability Testing Process 2

28


• 1. Device Proliferation

• 2. Application Modality

• 3. User Mobility

• 4. Data Collection

Remember :: Challenges to deal with (*)

29



For informal, problem identifying tests

• Ask participants to use the application maybe over lunch

• Offering a compensation helps „motivating“ the participant

• Don‘t forget to have them sign an informed consent statement

• Interruptions (waiter, etc.) are welcome Watch what happens when the users resumes the task and see what difficulties occur.

Methods :: Handling User Mobility 1

30


For formal, statistically precise tests

• Don‘t try to introduce distractions into the test unless you are testing with a greater number of participants

• Referring to Nielson, the marginal benefit will decrease if you are testing with more then 10 UsersThesis and formula is questioned from many researchers.

Methods :: 3 : Handling User Mobility 2

31


Chances(*):• Reproducible conditions

• Easier / more complete documentation

• Use of more sophisticated tools

Risks• Unusual environment for the user

• Restrictions due to simulation

• Non recording of the original work surrounding field (office atmosphere, disturbances, etc.)

Methods :: 3 : Simulate the natural environment? 1

32

*(vgl.: akziv. Requirements from users point of view. 2004)


Which environmental factors are reproduceable in a laboratory surrounding.

Environmental Conditions BT 041118.xls

Interactive Example

33


„Mobile Devices are build to be mobile so take them out into the field“ (*)• Take the lab to the user, not the user to the lab.

Methods :: 3 : Simulate the natural environment? 2

34

*(vgl.: akziv, „wearability“. 2004


Methods :: 3 : Lab Test vs. Field Test(*) 1

35

*(vgl.: Jesper Kjeldskov, Aalborg University Denmark)

Example Application: Using SMS Service on a PDA while walking

• In the lab: on a treadmill

• In the field: on a pedestrian street



36


Attention needed to navigate

None Conscious

Body

Motion

None 1. Sitting at a table or standing

n/a

Constant 2. Walking on a treadmill with constant

speed

4. Walking at constant speed on a

changing track

Varying 3. Walking on a treadmill with varying

speed

5. Walking at varying speed on a

changing track


Usability problems identified by the test subjects• Numbers are basically equal

• Notice that while sitting on a desk the cosmetic problemsidentified by the users were far more!


37

Techniques Total

Lab 1 Lab 2 Lab 3 Lab 4 Lab 5 Field

Critical 4 4 3 4 3 3 4

Serious 11 11 9 9 9 8 17

Cosmetic 19 8 8 8 6 12 32

Total 34 23 20 21 18 23 53



Are Lab tests superior?

• Consider Cost / benefit of different techniques and settingsTime and effort per problem found

• Can you afford NOT to find a problem?

• Costs of missing Usability:• lost of repurchases• increased calls at helpdesk • lost of repurchases• lost of brand reputation• necessary redesign in late state or next version• law suits • ……


38



Mobile devices are extremely personal. Users may pick them up, gesture, or lean back with them.(*)

• Record Device Screen + Users Face at the same time

• Use wireless tracking & recording technology(WLAN, Bluetooth, small radio cameras, etc.)

• Users should not be handicapped by the testing equipment

Methods :: 4 : Handling Data Collection 1

39

*(vgl.: Little Springs Inc. 2004)


Recording a mobile phone‘s screen and the users face at the same time with two cameras mounted on the phone(*)


40



Recording „Soft Information“

• Define methods to integrate „soft-information“E. g.: Users starts shaking the Phone to make it start the application faster; user‘s thumb tends to cover up parts of the device‘s display)

• Many qualitative information can be quantized. Measuring heart-beats / second to determine the stress-level Recording subconscious hand and leg movements. Record number of extra-applicational interactions

(user answered 2 phone calls and asked his colleague for help during the use of the application for 5. min.)


41



V. Possible Heuristics & Guidelines


Heuristics :: General Guidelines for mobile Applications 1

43


Highly functional design• Don‘t use fancy designs if they don‘t bring a real benefit for the User

Consistent usage of icons / buttons / names and labels• Consider the users mental-models when you introduce new functions

or name buttons.

Integrated content navigation• Help the user tracking it‘s way through the information, always provide

a clear exit – point.

• Consider Shortcuts


Heuristics :: General Guidelines for mobile Applications 2

44


Reduced HCI interactions• Especially when done on mobile devices interactions with the user are

often difficult and time-consuming (text-input on a mobile phone)

• Reduce Interactions by any means possible (Location based services, Heuristics, default-values, etc.)

Offer intelligent search-functions• Assist the user in finding the information, as any unnecessary

interaction makes the applications less usable for the user.


Heuristics :: Mobile Games 1

45

For Mobile Games the Rules are

a little different

• Navigation Consistency?

• The User should not feel like using his/her phone, he/she should experience the Game World

*(vgl.: NOKIASeries 60 Developer Platform 2.0: Usability Guidelines For J2ME™ Games


Heuristics :: Mobile Games 2

46

Game experience vs. Social acceptable behavior

• Sound, Light and Vibration enhance the users Game experience

Typically, games are played in locations where it is not suitable or socially acceptable to have the sound on.

• During Mobile Usab. Testing consider there are usually other people close by when the user plays the game.

*(vgl.: NOKIASeries 60 Developer Platform 2.0: Usability Guidelines For J2ME™ Games




Text Interface :: Projecting the image

48

*(vgl.: http://www.ibizpda.com)

A standard-sized Query-Keyboard is projectedby laser on any given surface.

• The Users input is recognised by a small camera in the cigarette-pack sized device.

• Note: Being announced in 1999 the product is still under development and may never reach market maturity. (2004)(*)

• www.virtualdevices.net

http://www.virtualdevices.net/


Text Interface :: Breaking the Qwerty Paradigm

49

*(vgl.: www.frogpad.com/)

Standard-sized keys aligned for one-handed use.

• Can be used under mobile conditions (doesn't need chair + desk environment as similar fold-up keyboards for PDAs)


Remote Controll :: Force Push

50

*(vgl.: Koji Tsukada, mobiquitous.com/pub/apchi2002-ubi-finger.pdf/)

Operate household devices with agesture of your fingertip.

• IR-Led points at Device to be controlled

• Touch & Acceleration Sensors combinedwith Software recognise gestures and execute command


Natural Interaction :: Stick with the basics

51

*(vgl.: http://news.bbc.co.uk/2/hi/technology/3774747.stm)

Real „Pick and Drop“

1. PDA – User picks up a file on his screen by tapping on it with a digital pen.

2. He passes on the pen to his colleague who drops the pen on his PDA-screen.

3. The file is copied.


For more information see:http://qse.ifs.tuwien.ac.at/courses/Usability/VO_Usability_Engineering.htm

Documents

Mobile Usability Testing Inst. f. Softwaretechnik und Interaktive Systeme qse.ifs.tuwien.ac.at, [email protected], [email protected]