Online Data Collection: Tips for Effective Visual Design

Lindsey Witt-Swanson

Assistant Director

Bureau of Sociological Research

Bureau of Sociological Research

• Created in 1964 as the data collection vehicle for the Sociology Department

• Directed by a member of the Sociology faculty

• Served all UNL colleges in the past five years

• Fee-for-service organization

• Currently have 63 projects in the field

2

• Research Design Consultation• Survey, questionnaire, and sample design• Cost/Budget estimates

• Data Collection• Mail, phone, and web surveys• In-person and cognitive interviewing• Focus group facilitation• Program evaluation

• Data Processing• Entry/Verification, coding, and analysis• Transcription• Technical report preparation

• Research Support• Recruitment• IRB preparations• Trainings

3

Expertise through continuous research and education• Director: Jolene Smyth, PhD

• Co-Author of Internet, Phone, Mail, and Mixed-Mode Surveys with Don Dillman

• Active research agenda looking at minimizing survey error

• Staff• Four staff members have MS in survey research• Continuous education through UNL courses, short

courses at conferences, webinars, and internal, professional development

• Participation in professional meetings to stay informed of the latest developments in survey research methodologyand add to it through our original research

• Practical experience

4

Visual Design Matters

• Visual design impacts if and when respondents see the parts of our questionnaire.

• If a question, response option, instruction, or answer space is unseen, it will not be processed.

• If it is seen out of order, it may be processed in unintentional ways.

• It also impacts how respondents proceed through the rest of the response process.

• How do they comprehend the question?• How do they retrieve information?• How do they formulate a judgment?• How do they map their response to our questionnaire?

5

We need to be cooperative communicators

• In a self-administered survey, the visual design/layout is part of what we (the researchers) are saying.

• Respondents assume it is not irrelevant; rather, it has meaning.

• So they try to make sense of it – to read meaning into it.

• The meaning they read into it will impact their response process.

• This is… • Bad if our visual design is arbitrary and done without thought or strategy

• Good if we can harness visual design as an additional way to communicate to respondents (e.g., to help them comprehend a question).

6

An example of visual design impacting data

7

Source: Couper, Mick P., Michael W. Traugott, and Mark J. Lamias. 2001. “Web Survey Design and Administration.” Public Opinion Quarterly. 65(2):230-253.

8

Invalid responses included

things like “about 3” or

“between 4 and 5.”

11.3% provided invalid

responses

20.7% provided invalid

responses

The size of the answer box impacted respondents’ comprehension of the response task, particularly, how exact their answer should be.

Graphics can impact data

9

How would you rate your health?

Excellent

Very Good

Good

Fair

Poor 33% said excellent or very

good

41% said

excellent or

very good

The images likely impacted how respondents defined the vague word “healthy” and the information they retrieved and used to form their judgment.

Source: Couper, Mick P., Frederick G. Conrad, and Roger Tourangeau. 2007. “Visual Context Effects in Web Surveys.” Public Opinion Quarterly. 71(4):623-

634.

Picking the right question format

• If we are serious about data quality, functionality and usability should always come before branding and entertainment.

• Use the format that gets you the data you need• NOT what looks fun or entertaining

• NOT what is the most aesthetically pleasing

10

11

12

Sliders

• “Fun” features can have an impact on the data you collect

• Sliders are interactive and different

• Is this the real answer or did they not answer the question?

13

Side-by-Side Questions

• A way to get a lot of items in a small amount of space

• Very burdensome (multiple questions at once) and can train respondents to skip questions

• Will not work on mobile

14

Other Visual Design Concepts

15

• Processing Concepts• Bottom-up processing

• Top-down processing

• Preattentive processing

• Attentive processing

• Foveal view

• Useful field of view

• Figure/ground composition

• Visual Elements• Words

• Numbers

• Symbols

• Graphics

• Visual Properties• Size• Font• Brightness/contrast• Color• Shape• Orientation• Static vs. motion• Etc.

• Gestalt Grouping Principles• Proximity• Similarity• Pragnanz• Closure• Common Region• Continuity• Elemental Connectedness

Nonsubstantive Answer Options

• Don’t Know, Not Applicable, No Opinion

• Programs will automatically treat nonsubstantive answer options just like the substantive answer options

• Can impact data when it shifts the visual midpoint

16

Programming Matters: Skip and Loops

• Example Skip: Females get breast cancer questions, Males get prostate cancer questions

• Example Loop: Ask the same questions for each doctor visit in the last 30 days

• If they are wrong during administration, people will be asked the wrong questions leading to the very least missing data but worse possible breakoffs

• Skips and loops need to be checked, rechecked, and checked one more time

17

Radial buttons versus Check-all boxes• Radial buttons indicate one answer

• Boxes indicate more than one answer

• They need to be used consistently to train the respondent of what you want

18

Answer Option Values

• Best to get them right during programming to save a lot of time recoding on the back end

• Example: Lime automatically makes answer options A1, A2, etc.• Have to change to 1, 2, and so on to be usable in later analyses

• If you don’t do this before hand, you have to recode all of your questions before you can run any statistical tests

19

Check-All Programming

• Enter and then look at test data before fielding to uncover errors in the programming missed any other way

20

Checks

• Read through

• Check skip patterns and loops

• Answer the questions• Helps notice things missed in the read through

• Export the data you entered to make sure it is saving and exporting correctly.

21

Other Issues to consider

• You need to know what the program automatically collects• Qualtrics automatically collects IP addresses, and won’t let you shut it off

• Matters to IRB and respondent confidentiality

• Who owns and has access to the data?• Qualtrics: The data stays on the Qualtrics servers even after you export it

• Lime: The client owns the data and we can erase it from the Lime server because we own and maintain the server

22

Summary

• Visual Design matters• We need to think about it or our design may have unintended consequences

on our data

• Programming matters• Yes, you can fix some errors/issues on the back end, but certainly not all of

them

• Being educated about your web survey platform matters• It can dictate data ownership, confidentiality matters, and more

23

This is the tip of the iceberg

• Call BOSR• This is what we do

• We stay up-to-date on all the quirky, software update changes

• We work through these quirks with the IRB regularly

• We offer researchers two to three hours of free consultation

• Jolene will do another SBSRC short course on visual design in the spring

• Watch for announcements!

24

Thank you!

Lindsey Witt-Swanson

lwitt2@unl.edu

Bureau of Sociological Research

University of Nebraska-Lincoln

25

mailto:lwitt2@unl.edu