A Tale of Two Methods: Comparing mail and RDD data collection for the Minnesota Adult Tobacco Survey III

Wendy Hicks and David Cantor

Westat

Ann St. Claire,

ClearWay Minnesotasm

Rebecca Fee,

Blue Cross and Blue Shield of Minnesota

Peter Rhode,

Minnesota Department of Health

2

Overview of Presentation

• Report results of a feasibility study comparing mail and telephone data collection methods Seeming declines in RDD response rates (Curtin et al,

2005) Changes in coverage of the telephone population

(Blumberg et al, 2006, 2007) Increasing costs associated with RDD surveys

• Findings mostly replicate those reported by Link et al. (2006)

• Some suggestion that an “all adult” selection method in a mail survey results in biased estimates for young adults (18-24 year olds)

3

Methodological Approach

• Mail survey has small sample and large confidence intervals Contrast results found in a parallel mail vs. telephone

study for which mail study selected individuals rather than addresses

Bring in similar findings from another pilot study (Health Information National Trends Survey, HINTS)

4

MATS Overview

• Minnesota Adult Tobacco Survey (MATS)

ClearWay Minnesotasm

Blue Cross and Blue Shield of Minnesota Minnesota Department of Health

• Major objective of MATS: collect a diverse set of public health data about adult pop in MN, focusing on tobacco and cigarette use

• MATS samples from two frames RDD Blue Cross member list frame

5

Overview of MATS 2007 samples by mode and frame

Telephone

(Feb – June, 2007)

Mail

(March – June, 2007)

General population samples

• targeted 7500 completes

• oversampled young adults & African Americans

• selected one individual within household

• sampled 1200 addresses

• no oversampling

• “all adults” selected to respond

BCBS member list

• targeted 5000 completes

• oversampled young adults

• selected individual BC member

• sampled 1200 members

• no oversampling

• selected individual BC member

6

USPS Mail Data Collection

• All adult respondent selection mechanism

• Mailing package included 4 identical questionnaires and return envelopes

“Each adult living at this address should complete one of the enclosed questionnaires. Please give one of the enclosed questionnaires and an envelope to each person 18 years old or older living at this address.”

• Some differences from the CATI instrument to make it more suitable to paper, self-administration

7

Response rates by frame and mode

MAIL TELEPHONE

General Population Frame

(USPS and RDD-only)

31% - 33% 41%

BCBS Member List 56% 48%

8

Nonresponse and coverage error by mode

• 9.3% weighted estimate of cell-phone only households in the USPS survey

• Generally, the demographic distributions for mail and telephone respondents parallel one another Overall, more missing data in mail than telephone

• USPS vs. RDD-only respondents (differences of 3% plus) USPS has fewer HS grads, larger underestimate relative to CPS USPS has fewer 65+, smaller overestimate relative to CPS USPS has fewer 1-adult households and more 4+ adult

households, closer distribution to CPS

9

Nonresponse and coverage error by mode

• Similar story for the BC member list, generally the distributions for mail and telephone parallel BC-M had fewer “high school grads” and “some college” BC-M had more “married” than phone respondents

10

Comparing survey estimates by mode

• Socially sensitive items (risk behaviors) Smoking prevalence, Binge drinking

• Factual items Exposure to media messages regarding tobacco use Workplace smoking policies

• Items assessing beliefs Harm in smoking an occasional cigarette Smoking increases comfort in social situations

11

Estimates of risk behaviors

USPS

%

RDD

%

BC mail

%

BC phone

%

Smoking prevalence

(current smokers)

14.32

± 4.66

17.01

± 1.39

11.90

± 2.85

10.86

± 0.92

Binge drinking 19.80

± 4.78

15.44

± 1.21

21.68

± 3.78

14.56

± 1.03

Hypothesis: mail > reporting of risk behaviors

12

Percent current smokers, by age, mode: General pop

0

5

10

15

20

25

Current smokers

(%)

18-24 25-44 45-64 65 andolder

USPS RDD

*

*

* p >0.03

13

Percent binge drinkers, by age, mode: General pop.

0

5

10

15

20

25

30

Binge drinkers

(%)

18-24 25-44 45-64 65 orolder

USPS RDD

14

Percent current smokers, by age, mode: Member frame

0

5

10

15

20

Current smokers

(%)

18-24 25-44 45-64 65 orolder

BC-Mail BC-Phone

15

Percent binge drinkers, by age, mode: Member frame

05

1015202530354045

Binge drinkers

(%)

18-24 25-44 45-64 65 orolder

BC-Mail BC-Phone

16

More suggestive evidence . . .

• Health Information National Trend Survey (HINTS) Small pilot USPS frame National sample “all adults” selection method

Replicated the smoking result: Estimates for 18-24 year old mail respondents significantly lower

17

What about other types of items?

• Factual items Reported exposure to media messages Work place policy regarding smoking

• No differences by mode, for either frame• No differences by mode for 18-24 years old

• Items assessing beliefs and attitudes Some support for age effect for one of the two items…

18

Any harm in smoking an occasional cigarette? Gen. pop.

0

20

40

60

80

100

Yes, harmful

(%)

18-24 25-44 45-64 65 orolder

USPS RDD

19

Any harm in smoking an occasional cigarette? Member list

0102030405060708090

Yes, harmful

(%)

18-24 25-44 45-64 65 orolder

BC-Mail BC-Phone

20

Summary and Discussion

• Response rate roughly comparable

• Mail mode includes coverage of cell-only population

• Demographic comparison by mode- USPS (all adult) to RDD (next birthday) – very similar

distributions, some differences in educ, adults in HH, Member list – also very similar, differences as anticipated

• Some suggestion that 18-24 year olds not representative for USPS “all adult” method Lower reports of risk behaviors in mail mode

21

Household type for 18-24 year olds, by mode

• In USPS sample Almost 80% live in family-type households About 6% live alone About 3% live in un-related households

Only 13% of 18-24 year olds were cell-only households

In the RDD sample, Almost 65% live in family-type households About 4% live alone Almost 25% live in un-related households

22

Next steps

• Incorporate methods to broaden “type” of young adult respondents Advance letter Incentive Fed-ex the second package

• Explore other respondent selection mechanism Next birthday method Other methods