Upload
alexander-randall
View
237
Download
1
Embed Size (px)
Citation preview
Response option order effects.
Scale lengths and horizontal or vertical layout.
Johan MartinssonUniversity of Gothenburgwww.lore.gu.se
Overview
• 2 related but separate studies
• The general focus on effects of different ways to arrange response options on a screen (eg. ”primacy effects”, tendency to pick the first response option)
• Three factors to consider:
• scale length
• layout direction (horizontal or vertical)
• response device (PC, tablet, smartphone)
www.mod.gu.se
Why do we care?
• You want a neutral measurement instrument
• An arbitrary design choice on your part should not influence results
• An arbitrary choice of device of respondents should not influence results
• => robust estimates
www.mod.gu.se
Some background
• More and more respondents use smartphone to answer surveys (though clearly a minority)
• Smartphone compatibility
no classic gridsshorter scalesvertical layout
• Screen sizes differ
www.mod.gu.se
Some background
• More automatic adjustment to device by survey softwares (responsive design)
Different respondents get different design
Robustness of measurement instruments more important to examine
www.mod.gu.sewww.mod.gu.se
Examples
Reversing response option orders
Horizontal or vertical layout
Example of automatic mobile adjustment by survey software
www.mod.gu.se
Study 1
• Data from an opt-in sample from a survey connected to a voting advice application for the Swedish election 2014
• Field work: July to September 2014• Total n approximately 20,000
• Randomized experimental design 2x2x2: • ”Right direction” or ”wrong direction” first• 7 or 3 scale points• Horizontal or vertical layout
www.mod.gu.se
Study 1
• We had a few hypotheses... : primacy effects exist stronger effects with vertical layouts than horizontalstronger effects for smartphone users(especially with horizontal layouts)
www.mod.gu.se
Results
All respondents7-point scale with horizontal layout
PlacementFirst Last diff
Right direction 11 10 1Wrong directionn 2561 2538
www.mod.gu.se
Results
All respondents7-point scale with horizontal layout
PlacementFirst Last diff
Right direction 11 10 1Wrong direction 19 21 -2n 2561 2538
www.mod.gu.se
all respondents
7-point scale with vertical layout
Placement
First Last diff
Right direction 15
Wrong direction 26
n 2457 2455
www.mod.gu.se
all respondents
7-point scale with vertical layout
Placement
First Last diff
Right direction 15 11 4
Wrong direction 26 21 5
n 2457 2455
www.mod.gu.se
all respondents
3-point scale with horizontal layout
Placement
First Last diff
Right direction 31 28 3Wrong direction 49 45 4
n 2421 2492
www.mod.gu.se
all respondents
3-point scale with vertical layout
Placement
First Last diff
Right direction 32 28 4Wrong direction 48 47 1
n 2566 2550
www.mod.gu.se
Kind of boring results, no?
• Luckily, we discovered something more interesting by mistake!
www.mod.gu.se
7-point scale with horizontal layoutPlacementFirst Last
Right direction 11 10Wrong direction 19 21n 2561 2538
7-point scale with vertical layout
Placement
First Last
Right direction 15 11
Wrong direction 26 21
n 2457 2455
www.mod.gu.se
3-point scale with vertical layout
Placement
First Last
Right direction 32 28
Wrong direction 48 47
n 2566 2550
3-point scale with horizontal layoutPlacementFirst Last
Right direction 31 28
Wrong direction 49 45n 2421 2492
www.mod.gu.se
Study 2
• Data from an opt-in sample from a survey connected to a voting advice application for the Swedish election 2014
• Field work: July to September 2014• Total n approximately 34,000
• Randomized experimental design 5x2(x2): • Horizontal or vertical layout of the questions• 10, 7, 5, 3, or 2 scale points (response options)
www.mod.gu.sewww.mod.gu.se
Percent "Wrong direction" among everyoneHorizontal Vertical Difference
10-point scale7-point scale5-point scale3-point scale2-point scale
Results
www.mod.gu.sewww.mod.gu.se
Percent "Wrong direction" among everyoneHorizontal Vertical Difference
10-point scale 16 21 -57-point scale 20 27 -75-point scale 24 31 -73-point scale2-point scale
Results
www.mod.gu.sewww.mod.gu.se
Percent "Wrong direction" among everyoneHorizontal Vertical Difference
10-point scale 16 21 -57-point scale 20 27 -75-point scale 24 31 -73-point scale 49 49 02-point scale 58 59 -1
Results
www.mod.gu.se
Percent "Wrong direction" among PC users
HorizontalVertical Difference10-point scale 18 20 -27-point scale 21 27 -65-point scale 27 31 -43-point scale 50 50 02-point scale 60 60 0
www.mod.gu.se
Percent "Wrong direction" among smartphone users
Horizontal Vertical Difference10-point scale 14 25 -117-point scale 19 29 -105-point scale 21 33 -123-point scale2-point scale
www.mod.gu.se
Percent "Wrong direction" among smartphone users
Horizontal Vertical Difference10-point scale 14 25 -117-point scale 19 29 -105-point scale 21 33 -123-point scale 50 51 -12-point scale 59 61 -2
www.mod.gu.se
Difference depending on device
All PC Smartphone10-point scale7-point scale5-point scale3-point scale2-point scale
www.mod.gu.se
Difference depending on device
All PC Smartphone10-point scale -5 -2 -117-point scale -7 -6 -105-point scale -7 -4 -123-point scale 0 0 -12-point scale -1 0 -2
www.mod.gu.se
Conclusions and recommendations
• We observe weak but fairly robust primacy effects (as usual)
• These do not seem to depend on device or scale length much (more surprisingly)
www.mod.gu.se
First, some useless advice...
• Ban smartphones in your surveys!
• Only use horizontal scales (not even sure it is better)
www.mod.gu.se
Some less useless advice
• Use really short scales
• Randomize when you can (nominal scales)
• Split half reversed order?
• Use long questions wordings and make sure respondents are aware of all available options before allowing them to answer (but increases burden and time)
www.mod.gu.se
Read more at
www.lore.gu.se
Laboratory of Opinion Research (LORE)