Ordering Your Attention:

Response Order Effects in Parallel

Phone and Online Surveys

Frances M. Barlas ICF International

&

Randall K. Thomas

GfK Government and Academic Research

Presented at the 67th Annual Conference of AAPOR

Orlando, FL

May 17 – May 20, 2012

2

Introduction

• The order in which response options are presented

plays a role in respondents’ answer selection

process and has long been shown to affect survey

results (Schuman and Presser, 1981)

• Response order effects occur when respondents

select responses based on the order in which the

responses are presented:

o Primacy effects – higher selection of options when

they are presented early in the response set

o Recency effects – higher selection of options when

they are presented later in the response set

3

Introduction

• To help understand when and why response

order effects occur researchers have

considered a number of factors including:

oRespondents’ memory and attention span

oRespondents’ age and education

oResponse format

oMode of response presentation

4

Introduction

• Krosnick and Presser (2010) examined a

number of past studies on order effects

among categorical response options and

observed a pattern:

o Primacy effects are more likely to occur when

alternatives are presented visually

o Recency effects are more likely to occur when

response alternatives were presented orally

5

Introduction

• Response-order elaboration model -

– Order of response presentation affects the amount

that respondents are able to pay attention to and

process each response.

– Explains mode differences in response order

effects:

• Items presented visually: the degree to which

respondents consider earlier items likely interferes

with their ability to fully consider later items.

• Items presented orally: consideration of earlier

response options is disrupted by the presentation of

the next response option

6

Introduction

• Response-order effects have often been used to

explain response selection with collections of

categorical responses, but less is known about

selection of ordered responses as occurs with scales.

• Scales have an inherent order and connection

between responses –

– Presentation of one response can lead to a rapid

understanding of the nature of the scale.

– Often the nature of the scale is evident from the

question stem.

7

Introduction

• Order of responses may have less of a profound

effect when dealing with scales compared to

categorical response options due to the automated

way in which scales are processed.

• Our interest in this study was to compare a set of

questions asked in parallel phone and web surveys

that used the same response formats to examine the

strength and direction of order effects.

8

Method

Two surveys were fielded in parallel in April 2006:

1. Online survey –

• 3,937 completed interviews

• drawn from the Harris Poll Online panel based on a stratified, random selection on age, sex, region, education, income, and race strata to resemble U.S. general population.

2. Telephone survey –

• 1,008 completed interviews

• Random digit dial sampling within sampling blocks having at least 2 working numbers

• Targeting landlines, not cell phones

9

Method

Questionnaire:

• There were 27 items fielded in parallel across the two modes with a response order manipulation

• Three types of items were used:

–Political evaluation items – bipolar rating scale

–Economic confidence items – bipolar rating scale

–Expenditure likelihood items – unipolar rating scale

• Response order manipulation:

–Respondents randomly assigned to order of response presentation

– ‘Most to Least’ or ‘Least to Most’

–Assignment was independent within each of the questionnaire sections

10

Method

Political Evaluation Items

• Three approval rating items

- Approval of President George W. Bush’s handling of job as president

- Approval of President George W. Bush’s economic policies

- Approval of President George W. Bush’s defense policies and military decisions

•Responses: Strongly approve, Somewhat approve, Neither approve nor disapprove, Somewhat disapprove, Strongly disapprove

•Online response options presented vertically, one item per screen.

11

Method

Political Evaluation Items – Random assignment to 5 of 8 possible performance evaluation items

1. Democrats in Congress

2. Republicans in Congress

3. Senate Majority Leader Bill Frist

4. House Speaker Dennis Hastert

5 . Vice President Dick Cheney

6 . Secretary of State Condoleezza Rice

7. Secretary of Defense Donald Rumsfeld

8. Attorney General Alberto Gonzales

• Responses: Excellent, Pretty good, Only fair, or Poor

• Online items were presented in grid format, all items presented on a single screen.

12

Method

Economic Confidence Items

• Random assignment to 4 of 7 possible items.

• Online items presented all on same screen with response options displayed horizontally

1. How would you rate the economic condition of your region of the nation? [Very good, Somewhat good, Neither good nor bad, Somewhat bad, Very bad]

2. How would you rate the current job market of your region of the nation? [Very good, Somewhat good, Neither good nor bad, Somewhat bad, Very bad]

3. How would you rate the economic policies of the national government? [Very good, Somewhat good, Neither good nor bad, Somewhat bad, Very bad]

13

Method

Economic Confidence Items (4 of 7 assigned)

4. Compared to a year ago, how has your household’s financial condition changed? [Improved a lot, Improved somewhat, Remained the same, Worsened somewhat, Worsened a lot]

5. What are your feelings concerning the economy of your region of the nation over the next 6 months? [Very optimistic, Somewhat optimistic, Neither optimistic nor pessimistic, Somewhat pessimistic, Very pessimistic]

6. How do you think that the job market in your region of the nation will change over the next 6 months? [Much better, Somewhat better, Remain the same, Somewhat worse, Much worse]

7. Thinking about your household’s current financial situation, do you expect it to be better or worse in the next 6 months? [Much better, Somewhat better, Remain the same, Remain the same, Somewhat worse, Much worse]

14

Method

Expenditure Likelihood Items

• Random assignment to 5 of 9 possible items

• Online items presented as a grid, with items presented in rows and responses in columns, on a single screen.

15

Method

Expenditure Likelihood Items – likelihood in the next 6 months (5 of 9 assigned):

1. Buy or lease a newly manufactured car, truck, or van

2. Move to a different residence

3. Purchase a house or condo

4. Take a vacation costing more than $1000

5. Have more money to spend the way you want

6. Save or invest more money

7. Start a new business

8. Buy a boat or recreational vehicle (e.g., trailer, motor home)

9. Buy a new computer

• Responses: Not at all likely, Somewhat likely, Likely, Very likely, Absolutely certain

16

Results

• For analyses, all items were range-adjusted to be equivalent on a 0 to 1 scale to allow for greater comparability between scales of different types.

• We first conducted separate analyses for each item.

Phone Online

Number of experiments 27 27

Primacy Effects 18 16

Statistically Significant 3 4

Recency Effects 9 11

Statistically Significant 2 7

17

Results

• Next we compared means by mode and response order, after controlling for age, sex, income, education, region of the country, and race/ethnicity.

• Looking across all 27 items, there was a significant mode effect (Phone M=.357; Online M=.273; p<.001), which replicates other findings of reduced positive ratings given online.

• There was not a significant response order effect, but a marginally significant Mode X Order interaction with some tendency for a primacy effect for phone surveys for these scales.

18

Results – Overall Means

Analysis of overall means controls for age, sex, income,

education, region of the country, and race/ethnicity covariates.

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

0.55

0.6

Least to Most Most to Least

Phone

Online

19

Results –Means for Political Items

(bipolar scale)

Analysis of overall means controls for age, sex, income,

education, region of the country, and race/ethnicity covariates.

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

0.55

0.6

Least to Most Most to Least

Phone

Online

20

Results –Means for Economic Items

(bipolar scale)

Analysis of overall means controls for age, sex, income,

education, region of the country, and race/ethnicity covariates.

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

0.55

0.6

Least to Most Most to Least

Phone

Online

21

Results –Means for Expenditure

Likelihood Items (unipolar scale)

Analysis of overall means controls for age, sex, income,

education, region of the country, and race/ethnicity covariates.

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

0.55

0.6

Least to Most Most to Least

Phone

Online

22

Results – Pooled Analysis

The pooled analysis, combining all items within each specific item section showed few response order effects:

• The only significant response order effect (p<0.05) for the telephone administration occurred in the political items section, a primacy effect.

• With the online administration, there were significant differences with the political items (recency effect) and expenditure likelihood items (primacy effect), but not the economic items.

23

Results - Validity

• We used the 18 items that reflected political and economic evaluations as predictors of political party ID, using a regression approach to allow us to statistically evaluate the impact of mode and response order on validity.

• Political party identification was used as the primary dependent variable for validity purposes since it used a different measurement methodology than the measures of interest, a branching strategy – first asking for party identification, with a follow-up asking strength of identification and for independents the direction they lean. The branching strategy randomly alternated the presentation of ‘Democrat’ and ‘Republican’.

24

Results - Validity

• We found that the effect of Mode was significant (p<.001), with online measurement having significantly higher correspondence between the political and economic measures with political party identification.

• Response Order was also significant (p<.001) with the ‘Least to Most’ order having higher validity in both modes of survey administration.

25

Results: Predictive Validity- Adjusted R2

Predicting Party ID with Political and Economic Items

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

Telephone Online

Least to Most

Most to Least

26

Conclusions

• The effects of response order appeared to be modest at best in both online and telephone modalities for the items we studied.

• There was no clear pattern of primacy and recency effects by mode of survey administration, as has been observed with categorical response options.

• Means on the phone were found to be higher generally, replicating other findings that respondents are more likely to use positive categories in evaluations with a human interviewer.

27

Conclusion

• There was some evidence to support that presenting categories in a ‘Least to Most’ fashion may be associated with higher levels of validity for both telephone and online surveys.

• These findings may be qualified by the nature of the scales we used – most were evaluative bipolar scales and most were presented in a grid format online.

• More research is needed to compare response order effects with different types of scales and to further investigate the impact of mode and response order on measurement validity.

28

Thank You!

For further information:

FBarlas@icfi.com

randall.thomas@gfk.com