1 © 2008 Arbitron Inc. Arbitron Diary Service Offering Respondents a Choice of Survey Mode: Use Patterns of an Internet Response Option in a Mail Survey

1 © 2008 Arbitron Inc.

Arbitron Diary Service

Offering Respondents a Choice of Survey Mode: Use Patterns of an Internet Response Option in a Mail Survey

Robin Gentry

DC AAPOR Web Survey Methods WorkshopSeptember 10, 2009


Overview

• The Arbitron Diary Methodology

• eDiary Pilot Test

• The Revised eDiary Methodology

• Return Rate Effect

• Who Chooses the eDiary?

• Mode Effects?

• Where Do We Go from Here?


The Arbitron Diary Methodology

• Conducted in approximately 280 radio markets

• 48 one-week survey periods per year » Combined into four quarterly ratings releases

• Recruited by RDD phone survey with an ABS based CPO supplement

• Everyone 12 or older in household asked to participate

• Respondent households mailed a 7-day radio listening diary for each household member


The Paper Diary


The eDiary Pilot Test

• Purpose: Feasibility test of 3 web versions of diary

• Conducted in Fall 2003

• Recruited past diary survey households by phone for a web version of survey after confirming computer ownership/Internet usage

• Collected e-mail addresses for up to three household members» 493 households (with 763 unique e-mail addresses)


Test Methodology• E-mail sent within 24 hours explaining the survey

process

• E-mail direct link to eDiary the day before the start of the survey week

• Follow-up e-mail sent two days later reminding respondents to complete the survey» Additional follow-ups sent if no activity on account

• Thank-you e-mail sent the day after the end of the survey week reminding households to submit their completed survey

• Simulates the mail-based contacts with households receiving paper diaries


Three Versions of Web Survey• Version 1: Paper Diary Clone

» Designed to model paper diary with standard day parts and open time frames





• Version 2: Medium Assist Diary» Eliminates day parts and adds auto-completion of previous

stations and drop-down boxes





• Version 2: Medium Assist Diary» Eliminates day parts and adds auto-completion of previous

stations and drop-down boxes

• Version 3: Full Assist/Build-a-Diary» Allows the user to build the diary one line at a time. Offers lots

of assistance including look-up of stations in the local area.



Results of Test

• Diary completion rate of 38% (n=290)

» Similar across all 3 versions

• Listening Estimates» Test not designed to produce projectable audience

estimates» Given this, overall listening levels were similar to paper diary» All 3 web-page versions resulted in similar reported quarter

hours


Conclusions of Test• All three versions had similar usability

• The recruitment methodology could be improved» E-mail addresses are difficult to collect and verify over the

phone. Of 763 email addresses collected, over 100 error

messages» Respondent must proxy report for other household members» Spam blockers and virus checkers cause trouble» People don’t always read their e-mail in a timely fashion


Revised eDiary Methodology• Give respondents a choice of mode: paper or eDiary

» Send paper diaries to all household members» Include an insert with instructions for using the eDiary» Also, mention the option during recruitment and follow-up calls» Each household member can choose their preferred mode

• Chose the medium assist version (#2)» Similar to the paper survey but takes advantage of some

positive aspects of the web

• Those registered for the eDiary get personalized reminders if they provide an e-mail address at registration» In addition to the phone/mail reminders delivered to the

household


eDiary Insert


Larger Scale Test• Conducted in Winter 2005

• Households screened for computer ownership/web usage

• 173 eDiaries returned out of 2507 total returns (6.9%)

• No significant differences in listening behavior overall or by day of week or format

• Decision made to implement the eDiary as part of the standard methodology


Unexpected Return Rate Decline• Conducted live test in Winter 2007

» Approximately 30,000 households (63,000 individuals)» Half received eDiary mentions/insert, half received usual paper

diary only

• Expected no impact on return rates

• Instead, return rate was 4.2 points lower in group offered the eDiary option (56.4% vs. 60.6%)

• One possible explanation—the eDiary insert» Despite emphasizing the choice of modes in phone calls,

respondents may have thought they had to do the survey online

» Quickly developed and tested a revised insert


Revised eDiary Insert


Some Improvement• Second Test in Winter 2007

» Approximately 25,000 households (55,000 individuals)» Again, half received eDiary mentions/insert, half received

paper diary only

• Smaller return decline—3.4 points vs. 4.2 points in original test

• Other explanations» Some respondents complete the diary but forget to submit it» Callback study suggests some had technical difficulties or

confusion about registering for eDiary» These likely account for about 1 point of return decline


Is the Choice the Problem?• Similar findings for the American Community Survey*

» Tested a choice between mail/Internet response in 2000/2001» Return rate was 5.8 points lower in group offered mode choice

• ACS cited three explanations for response declines» Some respondents attempted the web survey but quit before

completion and didn’t return paper instead» Offering a choice caused respondents to put the task aside and

disrupted the response process» Notices about Internet reporting may have caused respondents

with privacy concerns

• Similar findings in Grigorian and Hoffer (2008) and Smythe, Dillman, Christian and O’Neill (in press)

*Griffin, D.H., D.P. Fisher and M.T. Morgan. “Testing an Internet Response Option for the American Community Survey.” Presented at the annual conference of the American Association of Public Opinion Research, May 17-20, 2001 in Montreal, Quebec, Canada


Who Chooses the eDiary?

• 3.8% of respondents offered the choice of modes registered for the eDiary

• 4.5% of usable returned diaries were eDiaries

• Once registered for eDiary, respondent was likely to return a survey» 90% of those registered entered at least one listening episode» 73% returned a usable eDiary» An additional 8% returned a paper diary instead


Age by Mode Choice

eDiary ReturnsN=9398

12-17

18-24

25-34

35-44

45-54

55-6465+

Paper Diary ReturnsN=234,406

12-17

18-24

25-34

35-4445-54

55-64

65+


Sex by Mode Choice

eDiary Returns

FemaleMale

Paper Diary Returns

FemaleMale


Race/Ethnicity by Mode Choice

0

10

20

30

40

50

60

70

80%

of

Ret

urn

s

eDiary Paper

Other

Black

Hispanic


Mode Effects?• We intentionally used a web design that closely

mimicked the paper diary. » Took advantage of some error checking features of the web

(e.g. time verification, checking for complete entries)» Didn’t take full advantage of all possible web enhancements

(e.g. look-up of local stations) because it might introduce mode effects

• No significant difference in amount of listening by mode» Average Quarter Hours of Listening: 73.4 for eDiaries and 73.9

for Paper DiariesDifference of 7.5 minutes over the entire week

• No difference in listening amounts or radio formats when test methodology (eDiary option) compared to control (no eDiary option)


Testing Conclusions• Don’t assume that adding a mode choice will

necessarily improve response rates» In fact, it may produce the opposite result

• Designing an instrument which mimics the paper version may reduce mode effects, but may be less user friendly

• Given the choice, relatively few respondents choose the web option

• If your target population is age 25-54, a web option may be a good compliment to your paper survey


Where Do We Go from Here?

• Mode Sequencing» Offer the Web option to those who don’t respond to the initial request

• Collect e-mail addresses from eConsenters» Likely to be web-savvy and less likely to have concerns about giving

an e-mail address

» Allows us to send a link to survey URL

• De-emphasize the mode choice» Still offer the Web option but emphasize less in materials and calls

» “Prefer the Web? Internet eDiary option now available at www.arbitrondiary.com

– Remove the insert

– Choice info printed only on letter and the diary itself


Questions?

[email protected]

(410) 312-8389

Documents

1 © 2008 Arbitron Inc. Arbitron Diary Service Offering Respondents a Choice of Survey Mode: Use Patterns of an Internet Response Option in a Mail Survey