Transcript
Page 1: Survey Research and Cell Phones: Is There a Problem? ·  · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research

Survey Research and Cell Phones:

Is There a Problem?Prepared for Harvard University conference on

“New Technologies and Survey Research”May 9, 2008

Scott KeeterDirector of Survey Research

Pew Research CenterWashington, DC

Page 2: Survey Research and Cell Phones: Is There a Problem? ·  · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research

Growth in the Cell-only Population

4.4%

12.6%

18.0%

29.1%

0%

5%

10%

15%

20%

25%

30%

35%

2004 2005 2006 2007

All Hispanic Ages 18-29

Source: National Health Interview Survey

Page 3: Survey Research and Cell Phones: Is There a Problem? ·  · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research

The Cell Phone Problem for RDD Surveys

• One-in-eight U.S. adults is cell-only according to June 2007 NCHS data

• The cell-only population is demographically different from the landline population

• As a result, landline surveys have experienced a sharp decline in the percentage of younger respondents interviewed in their samples

Page 4: Survey Research and Cell Phones: Is There a Problem? ·  · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research

Percent Ages 18-34

15

20

25

30

35

40

45

1990 1992 1994 1997 1999 2000 2002 2003 2004 2006

Parameter

Source: Pew Research Center surveys

Page 5: Survey Research and Cell Phones: Is There a Problem? ·  · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research

Percent Ages 18-34

15

20

25

30

35

40

45

1990 1992 1994 1997 1999 2000 2002 2003 2004 2006

Parameter

Unweighted

survey statistic

Source: Pew Research Center surveys

Page 6: Survey Research and Cell Phones: Is There a Problem? ·  · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research

Practical ConsiderationsIs it Feasible to Survey People on

Their Cell Phones?

Page 7: Survey Research and Cell Phones: Is There a Problem? ·  · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research

Yes, But It’s Expensive

• Cell interviews roughly 3x more expensive– Manual dialing– Reimbursement ($10 vs. $20)– Lower eligibility rate

• Many (42%) under age 18• Higher incidence of non-English speakers

• Cell-only interviews roughly 4-5x more expensive than landline– About 35% in cell sample are cell-only

Page 8: Survey Research and Cell Phones: Is There a Problem? ·  · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research

Interviewing Rates

Landline sample

Cell phone sample

Contact rate (2) 84% 83%

Eligibility rate 86% 45%

Cooperation rate (3) 27% 28%

Response rate (3) 23% 23%

Break-off rate 12% 10%

Source: October 2007 survey (results for three other surveys were highly similar)

Page 9: Survey Research and Cell Phones: Is There a Problem? ·  · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research

What We Did• 4 dual-frame surveys in 2006, 4 in 2007, 1 in 2008• Two surveys on politics and the campaign• One on gadgets/internet• One on economics• One of the Hispanic population

Oct 07GP

Dec 07 GP

Oct-Nov 07

Hispanic

Oct-Dec07GP

Jan-Feb 08GP

Landline N 1,507 1,089 1101 1554 1659Cell phone N 500 341 899 500 754Total 2,007 1,430 2,000 2,054 2,413

Page 10: Survey Research and Cell Phones: Is There a Problem? ·  · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research

Respondent Characteristics

Landline sample versus Cell-only adults

Page 11: Survey Research and Cell Phones: Is There a Problem? ·  · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research

Characteristics of Landline Sample and Cell-only Respondents*

12%

0% 20% 40% 60% 80%

Income <$30K

Age 18-29

Standard landline sampleCell-only

*Figures based on unweighted data

Page 12: Survey Research and Cell Phones: Is There a Problem? ·  · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research

Characteristics of Landline Sample and Cell-only Respondents*

12%46%

0% 20% 40% 60% 80%

Income <$30K

Age 18-29

Standard landline sampleCell-only

*Figures based on unweighted data

Page 13: Survey Research and Cell Phones: Is There a Problem? ·  · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research

Characteristics of Landline Sample and Cell-only Respondents*

41%

26%

13%

19%

61%

46%

21%

57%

6%

11%

48%

12%

0% 20% 40% 60% 80%

Income <$30K

Married

Hispanic

Black

Male

Age 18-29

Standard landline sampleCell-only

*Figures based on unweighted data

Page 14: Survey Research and Cell Phones: Is There a Problem? ·  · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research

How are estimates affected?

Page 15: Survey Research and Cell Phones: Is There a Problem? ·  · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research

Impact on Estimates from Including Cell Phones

37%

0% 10% 20% 30% 40%

Conservative

Republican/LeanRepublican

Approve ofPresident Bush

Republican/LeanRepublican

Standard landline sampleCombined landline + cell sample

Page 16: Survey Research and Cell Phones: Is There a Problem? ·  · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research

Impact on Estimates from Including Cell Phones

37%36%

0% 10% 20% 30% 40%

Conservative

Republican/LeanRepublican

Approve ofPresident Bush

Republican/LeanRepublican

Standard landline sampleCombined landline + cell sample

Page 17: Survey Research and Cell Phones: Is There a Problem? ·  · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research

Impact on Estimates from Including Cell Phones

28%

30%

35%

36%

27%

30%

35%

37%

0% 10% 20% 30% 40%

Satisfied withstate of nation

Approve ofPresident Bush

Conservative

Republican/LeanRepublican

Standard landline sampleCombined landline + cell sample

Page 18: Survey Research and Cell Phones: Is There a Problem? ·  · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research

Impact on Estimates from Including Cell Phones

1%

4%

9%

12%

17%

20%

22%

1%

4%

8%

11%

18%

19%

21%

0% 10% 20% 30% 40% 50%

Hunter

Paul

Thompson

Romney

Huckabee

Giuliani

McCain

Standard landline sampleCombined landline + cell sample

Based on Republican & Rep-leaning RVs(December 2007)

Page 19: Survey Research and Cell Phones: Is There a Problem? ·  · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research

Impact on Estimates from Including Cell Phones

0%

2%

3%

3%

14%

26%

46%

0%

2%

4%

3%

14%

27%

44%

0% 10% 20% 30% 40% 50%

Dodd

Biden

Richardson

Kucinich

Edwards

Obama

Clinton

Standard landline sampleCombined landline + cell sample

Based on Democrat& Dem-leaning RVs

(December 2007)

Page 20: Survey Research and Cell Phones: Is There a Problem? ·  · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research

Media Consumption Estimates

8%

8%

5%

28%

54%

40%

0% 20% 40% 60% 80%

Web: Natl newspaper

Web: local newspaper

Web: newsp yesterday

Network news

Local TV news

Newspaper (yesterday)

Standard landline sampleCell-only

Page 21: Survey Research and Cell Phones: Is There a Problem? ·  · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research

Media Consumption Estimates, 2006

11%

12%

9%

21%

43%

27%

8%

8%

5%

28%

54%

40%

0% 20% 40% 60% 80%

Web: Natl newspaper

Web: local newspaper

Web: newsp yesterday

Network news

Local TV news

Newspaper (yesterday)

Standard landline sampleCell-only

Page 22: Survey Research and Cell Phones: Is There a Problem? ·  · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research

Media Consumption Estimates, 2006

8%

9%

6%

27%

53%

40%

8%

8%

5%

28%

54%

40%

0% 20% 40% 60% 80%

Web: Natl newspaper

Web: local newspaper

Web: newsp yesterday

Network news

Local TV news

Newspaper (yesterday)

Standard landline sampleCombined landline + cell-only

Page 23: Survey Research and Cell Phones: Is There a Problem? ·  · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research

Under Age 30: Landline Sample and Cell-only Respondents*

32%

32%

12%

16%

48%

55%

0% 20% 40% 60% 80% 100%

Income <$30K

Married

Hispanic

Black

Male

Age 18-24

Landline sampleCell-only

*Figures based on unweighted data

Page 24: Survey Research and Cell Phones: Is There a Problem? ·  · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research

Under Age 30: Landline Sample and Cell-only Respondents*

48%

15%

15%

15%

62%

70%

32%

32%

12%

16%

48%

55%

0% 20% 40% 60% 80% 100%

Income <$30K

Married

Hispanic

Black

Male

Age 18-24

Landline sampleCell-only

*Figures based on unweighted data

Page 25: Survey Research and Cell Phones: Is There a Problem? ·  · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research

Under Age 30: Impact on Estimates from Including Cell Phones

27%

34%

28%

38%

25%

35%

27%

35%

0% 10% 20% 30% 40%

Conservative

Republican/LeanRepublican

Approve ofPresident Bush

Iraq: keeptroops in

Standard landline sample estimate for ages 18-29Combined landline + cell sample estimate for ages 18-29

Page 26: Survey Research and Cell Phones: Is There a Problem? ·  · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research

What’s the Potential for Bias?

• Example: % Favoring Iraq Withdrawal

• Is the combined estimate less biased?Under current conditionsIf cell-only adults uniformly favored withdrawalIf few cell-only adults favored withdrawal

Page 27: Survey Research and Cell Phones: Is There a Problem? ·  · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research

What’s the Potential for Bias?

Page 28: Survey Research and Cell Phones: Is There a Problem? ·  · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research

What’s the Potential for Bias?

Page 29: Survey Research and Cell Phones: Is There a Problem? ·  · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research

What’s the Potential for Bias?

Page 30: Survey Research and Cell Phones: Is There a Problem? ·  · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research

What’s the Potential for Bias?

Page 31: Survey Research and Cell Phones: Is There a Problem? ·  · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research

Quality of Responseslandline versus cell phone interviews

Page 32: Survey Research and Cell Phones: Is There a Problem? ·  · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research

Quality of Responses

• Are cell phone respondents more distracted?

Landline sample

Cell phone sample

Very distracted 1 2Somewhat 11 11Not too 18 14Not at all 70 74

100% 100%(n=1,507) (n=500)

Source: October survey

Page 33: Survey Research and Cell Phones: Is There a Problem? ·  · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research

Quality of Responses

• Are cell phone respondents less cooperative?

Respondent’s cooperation was…

Landline sample

Cell phone sample

Very good 78 81Good 15 13Fair 6 5Poor/Very poor 1 1

100% 100%(n=1,507) (n=500)

Source: October survey

Page 34: Survey Research and Cell Phones: Is There a Problem? ·  · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research

Sample Sizes:Will a combined design yield more

interviews with groups relying mostly on cell phones?

Page 35: Survey Research and Cell Phones: Is There a Problem? ·  · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research

Not if the cost is fixedExpected n

landline sampleExpected ncell sample

Standard Design ($100,000)Total sample 2,000 0 2,00018-29 yr olds 246 0 246

Blacks 212 0 212

Page 36: Survey Research and Cell Phones: Is There a Problem? ·  · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research

Not if the cost is fixedExpected n

landline sampleExpected ncell sample

Standard Design ($100,000)Total sample 2,000 0 2,00018-29 yr olds 246 0 246

Blacks 212 0 212

Combined Design ($100,000)Total sample 1,100 300 1,40018-29 yr olds 135 83 218

Blacks 117 45 162

Page 37: Survey Research and Cell Phones: Is There a Problem? ·  · 2013-01-07Survey Research and Cell Phones: Is There a Problem? ... May 9, 2008. Scott Keeter. Director of Survey Research

Conclusions• The risk of error from exclusion of cell-only

adults is increasing• No evidence of error yet for overall estimates• Comparable response rates for cell / landline• Cell interviewing is about 3x more expensive• Some evidence for gains in estimates for

groups relying mostly on cell phones• Budget permitting, a cell sample may be

prudent & boost credibility of findings


Recommended