Was it worth the hassle?

Ten years of Mobile HCI research discussions on lab and field evaluations

Jesper Kjeldskov & Mikael B. SkovCenter for Socio+Interactive Design / Department of Computer Science

Aalborg University, Denmark

2

Background

• At the first mobile HCI workshop (1998) Peter Johnson called for research on how to evaluate mobile systems

• In 2003 we reported that 71% of mobile HCI evaluations were done in the lab

• In 2004 we published a paper about evaluation in the field, asking “is it worth the hassle”?

• The “Hassle” paper caused a bit of a stir...

3

The “Hassle” paper

“Quite surprisingly, our study shows that compared to setting up a realistic laboratory study, evaluators achieved little added value when taking a usability evaluation of a context-aware mobile device into the field”

“Expensive time in the field should perhaps not be spent on usability evaluation”

“Field studies may instead be more suitable for obtaining insight needed to get the system right in the first place”

4

Fast forward 10 years

• We did a literature review of the publications citing the “is it worth the hassle” paper

• 191 citations (Feb. 2014) - obtained 165 electronically

• 12 were in a language we couldn’t read

• 7 cited for strange reasons (e.g. “mobiles have small displays”)

• 4 did not refer to the paper in the body text

• The remaining 142 were carefully read and classified

5

Looking back, and then ahead

Using lab or field

Comparing lab and field

Discussing lab and field

Beyond usability and usability evaluations

Beyond non-wild field studies

Beyond snap-shots of use

6

Using lab or field (62)Lab (23)

• Increasing realism in controlled environments

• From hi-fidelity setups to simple simulations

Field (39)

• Field experiments (17), ethnography (11), and surveys (11)

• From in-depth longitudinal studies to testing outside

Great variation and creativity in developing new techniques

7

Comparing field and lab (16)

• Some studies are similar to the Hassle study, but differ a lot in their experimental setup - making it difficult to talk about actual replication

• Great difference in what constitutes “lab” and “field”

• Lab: from seated at a table to hi-fidelity simulations

• Field: from simulations to actual real world settings

8

“We created two labs at the school, one for field testing sessions and one for laboratory testing sessions”

“If we want to compare the two settings, the field evaluation will have to be less realistic”

9

Discussing field and lab (64)

• Going beyond practical use and direct comparisons

• Typically quite strong and detailed discussions of methodology (e.g. ecological validity, and replicability)

• Arguments for and against both lab and field

• Some perhaps more “religious” than objective

10

Status in 2014

• A large body of research on lab and field evaluation

• New ways of simulating context and experimenting in-situ

• Have responded quite well to the challenge of building a body of knowledge and experience with both lab and field studies outside the common range of usability

• Is it worth the hassle? No definite answer has yet been found. Perhaps not the right question?

• From if and why to when and how?

11

Looking aheadThere are a number of points we would like to put

forward for your consideration

12

Move beyond usability and usability evaluations

• The challenge for mobile HCI research is no longer usability

• We are pretty good at designing interfaces for mobiles

• Today’s challenge is about designing services, devices and interactions that fit well into people’s complex lives, and fit with all those other technologies out there

• Designing for orchestration or digital ecosystems

13

Studies in the field should not be half-tame

• The real value of the fields is that it is real and messy, and not an amputated version of reality

• A field study that doesn’t take the researcher into an uncontrolled real world is perhaps not a good one

• When going out of the lab, we ought to make it across the parking lot all the way into the wild

• Embracing the wilderness rather than trying to tame it

14

Do longitudinal studies rather than snapshots

• The legacy from usability engineering has made us accustomed to “snapshots of use”

• Also true for many field studies (e.g. field experiments)

• Going beyond usability, and embracing the wild, we should also consider stretching the duration of our studies

• Perhaps even sacrifice direct researcher involvement?

15

• Move beyond usability and usability evaluations

• Embrace the wilderness - don’t be half-tame

• Do longitudinal studies rather than snapshots of use

To-do before 2024(preferably sooner)

16