Unifying the User Experience of a

Survey Across Multiple Devices

Jennifer Romano Bergstrom, David Hawkins, Jon Strohl

December 12, 2013

MoDevEast | McLean, VA

@forsmarshgroup @romanocog

About this talk

2

@forsmarshgroup @romanocog

1. Usability and User Experience Intro

2. Web Survey Design

3. Implications from Eye Tracking on a Mobile and Desktop

Tool

What is Usability?

3

@forsmarshgroup @romanocog

“the extent to which

a product can be

used by specified

users to achieve

specified goals with

effectiveness,

efficiency, and

satisfaction in a

specified context of

use.” ISO 9241-11

Usability vs. User Experience (UX)

4 The 5 Es to Understanding Users (W. Quesenbery): http://www.wqusability.com/articles/getting-started.html

User Experience Design (P. Morville): http://semanticstudios.com/publications/semantics/000029.php

@forsmarshgroup @romanocog

Whitney’s 5 Es of Usability Peter’s User Experience

Honeycomb

What People do on the Web

5 Krug, S. Don’t Make Me Think

@forsmarshgroup @romanocog

UX Design Failures

6

@forsmarshgroup @romanocog

• Poor planning

• “It’s all about me.” (Redish: filing cabinets)

• Human cognitive limitations

• Memory and Perception

• (activity)

• Primacy

• Recency

• Chunking

• Patterns

Patterns

7

@forsmarshgroup @romanocog

Patterns

8

@forsmarshgroup @romanocog

Patterns

9

@forsmarshgroup @romanocog

Patterns

10

@forsmarshgroup @romanocog

Mental models and repeating behaviors

11

@forsmarshgroup @romanocog

Mental models and repeating behaviors

12

@forsmarshgroup @romanocog

Web Survey Design

Why is Design Important?

14

@forsmarshgroup @romanocog

• No interviewer present to correct/advise

• Visual presentation affects responses

• While the Internet provides many ways to enhance surveys,

design tools may be misused

• Respondents extract meaning from how question and

response options are displayed

• Design may distract from or interfere with responses

• Design may affect data quality

Why is Design Important?

15

@forsmarshgroup @romanocog

http://www.cc.gatech.edu/gvu/user_surveys/

Why is Design Important?

16

@forsmarshgroup @romanocog

• Many surveys are long (> 30min)

• Long surveys have higher nonresponse rates

• Length affects quality

Adams & Darwin, 1982; Dillman et al., 1993; Haberlein & Baumgartner, 1978

• Respondents are more tech savvy today and use multiple

technologies

• It is not just about reducing respondent burden and

nonresponse

• We must increase engagement

• High-quality design = trust in the designer

17

http://www.pewinternet.org/Static-Pages/Trend-Data-(Adults)/Device-Ownership.aspx

@forsmarshgroup @romanocog

@forsmarshgroup @romanocog

18

Nielsen: The Cross-Platform Report, Quarter 2, 2012-US

Check Boxes in drop-down menus

19

@forsmarshgroup @romanocog

Radio Buttons on Mobile

20

@forsmarshgroup @romanocog

Useful Error Messages

21

@forsmarshgroup @romanocog

Error Messages Across Devices

22

@forsmarshgroup @romanocog

Consistency Across Devices

23

@forsmarshgroup @romanocog

• Predictable

– User can anticipate what the system will do

• Dependable

– System fulfills user’s expectations

• Habit-forming

– System encourages behavior

• Transferable

– Habits in one context can transfer to another

• Natural

– Consistent with user’s knowledge

Implications from Eye Tracking on a

Mobile and Desktop Tool

UX Testing Methodology

25

Regardless of the platform that the tool was tested on, participants completed tasks in this

order on Device 1, 2, and 3.

@forsmarshgroup @romanocog

Device 1

Account Registration

Tutorial

Homepage Impressions

Device 2

Log in

1st entry

2nd entry

Device 3

Log in

3rd entry

Check Entries

What is beautiful is usable!

26

Two-thirds of participants (24 of 36) had positive feedback about the appearance of the homepage.

@forsmarshgroup @romanocog

*Verbal responses to Moderator’s Guide Q4: Without making any clicks, can you tell me what are your initial impressions of the homepage?

Tractinsky, N., Katz, A. S., & Ikar, D. (2000). What is beautiful is usable. Interacting with computers, 13(2), 127-145.

• “I like the colors, and my name is up there so it’s

directed to me.”

• “It’s very clean and visually pleasing. The color

choices- aqua and blue- it’s very calming.”

• “It seems like icons are very big and clear on what

they are.”

0

2

4

6

8

10

12

14

16

Colorful/Attractive Clean/Straightforward

Initial Impressions of Homepage*

Logical entry order

27

The account creation process had a logical flow from one question to the next.

@forsmarshgroup @romanocog

*Device 1 Satisfaction Questionnaire Q4: Please rate the flow from one question to the next when entering your demographic and household

information. 1= not logical at all to 5= extremely logical

• “The questions were simple and were easy to answer.”

• “I was able to figure everything out with no trouble.”

• “It’s very easy and straightforward to fill out.”

0

5

10

15

20

25

30

35

40

Participant Responses*

Nu

mb

er

of

Re

sp

on

se

s

<=2

3

>=4

1 & 2

3

4 & 5

Logical Flow Ratings*

The TV entry process was “not difficult at all” or “only slightly difficult.”

Entries were “moderately to extremely accurate” in capturing TV viewing behavior.

Entering is straightforward and easy.

28

• “It’s easy to click with your finger, and the prompts are

easy to follow.”

• “There were direct questions and direct options.”

• “The screen is short; it's a not a multi-page app. I can

enter it all in one.”

@forsmarshgroup @romanocog

*Moderator’s Guide Q13 & Q25: On a scale of 1 to 5, how difficult was it to input your TV viewing entry, and why? 1 = not difficult at all to 5= extremely difficult

**Device 2 & 3 Satisfaction Questionnaire Q6: Please rate how accurately this app/site, when used on this device, captures your actual TV viewing . 1= not accurate at all to 5= extremely accurate

0

1

2

3

4

5

TV Entry Difficulty* TV Entry Accuracy**

Average Ratings of TV Entry Process

Tablet app Android app

Heat Maps: After being instructed to enter a TV show, fixations were

concentrated on the ‘Enter Viewing’ icon, suggesting that the

functionality of that icon is very clear.

Unified experience from paper to mobile/desktop

29

Prototype Grid

Paper Website/App

1. Name of Show 1. Viewing Type

2. Station Name 2. Name of Show

3. Channel Number 3. Station

Name/Number

4. Viewing Type 4. Time

Order of Entry

@forsmarshgroup @romanocog

Issue: Icons are not clickable and do not match user expectations.

30

• “I’m going to click on it, but it's not doing anything.

That is annoying.”

• “Do these not work?”

UX Best Practice: Reduce the amount of non-clickable real estate on the homepage. Focus on

making icons and images clickable. This will help the homepage feel more intuitive as users expect

icons and images to function as active links.

Logo should link to the homepage, similar to the way

logos work on websites.

Clicks along the bottom of the PC homepage showing that participants expected the icons to be working links.

Homepage on the iPad

website does not display

the clickable links below

each icon.

@forsmarshgroup @romanocog

Issue: Error messages do not promote recovery.

31

UX Best Practice: Specifically explain the error, and place the messages near where the error

occurs so the user can quickly fix the error and move on.

• “How do I advance to the next screen?”

• “It seems like it's stuck on the screen.”

M

Gaze Plot: After getting an error

message, the participant had to

search all over the screen to find

the missing field.

@forsmarshgroup @romanocog

Issue: The location of the ‘Next’ Button is not intuitive.

32

@forsmarshgroup @romanocog

* Couper, M.P., Baker, R., & Mechling, J. (2011). Placement and Design of Navigation Buttons in Web Surveys. Survey Practice,

4(1).

• “Where and how you click is a bit counter-intuitive. [It’s] not super

obvious which button to click to get to next sections.”

• “I feel like the ‘Next’ should be at the bottom and not the top.”

UX Best Practice: Users expect consistency and for navigation buttons to be close to the last entry

field.* Place all ‘Next’ and navigation buttons near the last entry fields.

Intuitive ‘Next’ button location Non-intuitive ‘Next’ button location

Issue: The ‘Back’ button confuses participants.

33

@forsmarshgroup @romanocog

UX Best Practice: Make the location of the ‘Next’ button consistent across all devices by placing it

in the lower right corner. This will assure users that their selections will be saved and promote a

sense of linear progression.

• “I was expecting ‘Next,’ but I

guess I have to push

‘Back.’”

• “It was hard to get

confidence that I was

proceeding in the right way.”

• “It didn't seem to flow, if you

will—you had to go back and

forth.”

Issue: The iPad website keyboard blocks the screen.

34

@forsmarshgroup @romanocog

UX Best Practice: Design the layout of the page such that important functions are always visible

while the keyboard is open.

0

1

2

3

4

5

6

7

8

9

10

Tablet PC Smart Phone

Nu

mb

er

of

iPa

d W

eb

Pa

rtic

ipa

nts

Tablet Web Users: Least Favorite Device

Issue: Workload is high when searching for entries.

35

@forsmarshgroup @romanocog

UX Best Practice: Follow users’ mental model: Arrange previous entries with the most recent at the

top.

36

@forsmarshgroup @romanocog

Assess the UX across devices

• Not necessarily the same UX across all devices

• Different issues occur on different devices

• Match user expectations and product objectives

• Conduct UX testing

Thank you!

• Twitter: @forsmarshgroup

• LinkedIn: http://www.linkedin.com/company/fors-marsh-group

• Blog: www.forsmarshgroup.com/index.php/blog

Jennifer Romano Bergstrom

@romanocog

jbergstrom@forsmarshgroup.com

MoDevEast