Survey Research and Cell Phones: Is There a Problem? ?·  · 2013-01-07Survey Research and Cell Phones:…

  • Published on
    07-Jun-2018

  • View
    212

  • Download
    0

Embed Size (px)

Transcript

  • Survey Research and Cell Phones:

    Is There a Problem?Prepared for Harvard University conference on

    New Technologies and Survey ResearchMay 9, 2008

    Scott KeeterDirector of Survey Research

    Pew Research CenterWashington, DC

  • Growth in the Cell-only Population

    4.4%

    12.6%

    18.0%

    29.1%

    0%

    5%

    10%

    15%

    20%

    25%

    30%

    35%

    2004 2005 2006 2007

    All Hispanic Ages 18-29

    Source: National Health Interview Survey

  • The Cell Phone Problem for RDD Surveys

    One-in-eight U.S. adults is cell-only according to June 2007 NCHS data

    The cell-only population is demographically different from the landline population

    As a result, landline surveys have experienced a sharp decline in the percentage of younger respondents interviewed in their samples

  • Percent Ages 18-34

    15

    20

    25

    30

    35

    40

    45

    1990 1992 1994 1997 1999 2000 2002 2003 2004 2006

    Parameter

    Source: Pew Research Center surveys

  • Percent Ages 18-34

    15

    20

    25

    30

    35

    40

    45

    1990 1992 1994 1997 1999 2000 2002 2003 2004 2006

    Parameter

    Unweighted

    survey statistic

    Source: Pew Research Center surveys

  • Practical ConsiderationsIs it Feasible to Survey People on

    Their Cell Phones?

  • Yes, But Its Expensive

    Cell interviews roughly 3x more expensive Manual dialing Reimbursement ($10 vs. $20) Lower eligibility rate

    Many (42%) under age 18 Higher incidence of non-English speakers

    Cell-only interviews roughly 4-5x more expensive than landline About 35% in cell sample are cell-only

  • Interviewing Rates

    Landline sample

    Cell phone sample

    Contact rate (2) 84% 83%

    Eligibility rate 86% 45%

    Cooperation rate (3) 27% 28%

    Response rate (3) 23% 23%

    Break-off rate 12% 10%

    Source: October 2007 survey (results for three other surveys were highly similar)

  • What We Did 4 dual-frame surveys in 2006, 4 in 2007, 1 in 2008 Two surveys on politics and the campaign One on gadgets/internet One on economics One of the Hispanic population

    Oct 07GP

    Dec 07 GP

    Oct-Nov 07

    Hispanic

    Oct-Dec07GP

    Jan-Feb 08GP

    Landline N 1,507 1,089 1101 1554 1659Cell phone N 500 341 899 500 754Total 2,007 1,430 2,000 2,054 2,413

  • Respondent Characteristics

    Landline sample versus Cell-only adults

  • Characteristics of Landline Sample and Cell-only Respondents*

    12%

    0% 20% 40% 60% 80%

    Income

  • Characteristics of Landline Sample and Cell-only Respondents*

    12%46%

    0% 20% 40% 60% 80%

    Income

  • Characteristics of Landline Sample and Cell-only Respondents*

    41%

    26%

    13%

    19%

    61%

    46%

    21%

    57%

    6%

    11%

    48%

    12%

    0% 20% 40% 60% 80%

    Income

  • How are estimates affected?

  • Impact on Estimates from Including Cell Phones

    37%

    0% 10% 20% 30% 40%

    Conservative

    Republican/LeanRepublican

    Approve ofPresident Bush

    Republican/LeanRepublican

    Standard landline sampleCombined landline + cell sample

  • Impact on Estimates from Including Cell Phones

    37%36%

    0% 10% 20% 30% 40%

    Conservative

    Republican/LeanRepublican

    Approve ofPresident Bush

    Republican/LeanRepublican

    Standard landline sampleCombined landline + cell sample

  • Impact on Estimates from Including Cell Phones

    28%

    30%

    35%

    36%

    27%

    30%

    35%

    37%

    0% 10% 20% 30% 40%

    Satisfied withstate of nation

    Approve ofPresident Bush

    Conservative

    Republican/LeanRepublican

    Standard landline sampleCombined landline + cell sample

  • Impact on Estimates from Including Cell Phones

    1%

    4%

    9%

    12%

    17%

    20%

    22%

    1%

    4%

    8%

    11%

    18%

    19%

    21%

    0% 10% 20% 30% 40% 50%

    Hunter

    Paul

    Thompson

    Romney

    Huckabee

    Giuliani

    McCain

    Standard landline sampleCombined landline + cell sample

    Based on Republican & Rep-leaning RVs(December 2007)

  • Impact on Estimates from Including Cell Phones

    0%

    2%

    3%

    3%

    14%

    26%

    46%

    0%

    2%

    4%

    3%

    14%

    27%

    44%

    0% 10% 20% 30% 40% 50%

    Dodd

    Biden

    Richardson

    Kucinich

    Edwards

    Obama

    Clinton

    Standard landline sampleCombined landline + cell sample

    Based on Democrat& Dem-leaning RVs

    (December 2007)

  • Media Consumption Estimates

    8%

    8%

    5%

    28%

    54%

    40%

    0% 20% 40% 60% 80%

    Web: Natl newspaper

    Web: local newspaper

    Web: newsp yesterday

    Network news

    Local TV news

    Newspaper (yesterday)

    Standard landline sampleCell-only

  • Media Consumption Estimates, 2006

    11%

    12%

    9%

    21%

    43%

    27%

    8%

    8%

    5%

    28%

    54%

    40%

    0% 20% 40% 60% 80%

    Web: Natl newspaper

    Web: local newspaper

    Web: newsp yesterday

    Network news

    Local TV news

    Newspaper (yesterday)

    Standard landline sampleCell-only

  • Media Consumption Estimates, 2006

    8%

    9%

    6%

    27%

    53%

    40%

    8%

    8%

    5%

    28%

    54%

    40%

    0% 20% 40% 60% 80%

    Web: Natl newspaper

    Web: local newspaper

    Web: newsp yesterday

    Network news

    Local TV news

    Newspaper (yesterday)

    Standard landline sampleCombined landline + cell-only

  • Under Age 30: Landline Sample and Cell-only Respondents*

    32%

    32%

    12%

    16%

    48%

    55%

    0% 20% 40% 60% 80% 100%

    Income

  • Under Age 30: Landline Sample and Cell-only Respondents*

    48%

    15%

    15%

    15%

    62%

    70%

    32%

    32%

    12%

    16%

    48%

    55%

    0% 20% 40% 60% 80% 100%

    Income

  • Under Age 30: Impact on Estimates from Including Cell Phones

    27%

    34%

    28%

    38%

    25%

    35%

    27%

    35%

    0% 10% 20% 30% 40%

    Conservative

    Republican/LeanRepublican

    Approve ofPresident Bush

    Iraq: keeptroops in

    Standard landline sample estimate for ages 18-29Combined landline + cell sample estimate for ages 18-29

  • Whats the Potential for Bias?

    Example: % Favoring Iraq Withdrawal

    Is the combined estimate less biased?Under current conditionsIf cell-only adults uniformly favored withdrawalIf few cell-only adults favored withdrawal

  • Whats the Potential for Bias?

  • Whats the Potential for Bias?

  • Whats the Potential for Bias?

  • Whats the Potential for Bias?

  • Quality of Responseslandline versus cell phone interviews

  • Quality of Responses

    Are cell phone respondents more distracted?

    Landline sample

    Cell phone sample

    Very distracted 1 2Somewhat 11 11Not too 18 14Not at all 70 74

    100% 100%(n=1,507) (n=500)

    Source: October survey

  • Quality of Responses

    Are cell phone respondents less cooperative?

    Respondents cooperation was

    Landline sample

    Cell phone sample

    Very good 78 81Good 15 13Fair 6 5Poor/Very poor 1 1

    100% 100%(n=1,507) (n=500)

    Source: October survey

  • Sample Sizes:Will a combined design yield more

    interviews with groups relying mostly on cell phones?

  • Not if the cost is fixedExpected n

    landline sampleExpected ncell sample

    Standard Design ($100,000)Total sample 2,000 0 2,00018-29 yr olds 246 0 246

    Blacks 212 0 212

  • Not if the cost is fixedExpected n

    landline sampleExpected ncell sample

    Standard Design ($100,000)Total sample 2,000 0 2,00018-29 yr olds 246 0 246

    Blacks 212 0 212

    Combined Design ($100,000)Total sample 1,100 300 1,40018-29 yr olds 135 83 218

    Blacks 117 45 162

  • Conclusions The risk of error from exclusion of cell-only

    adults is increasing No evidence of error yet for overall estimates Comparable response rates for cell / landline Cell interviewing is about 3x more expensive Some evidence for gains in estimates for

    groups relying mostly on cell phones Budget permitting, a cell sample may be

    prudent & boost credibility of findings

    Survey Research and Cell Phones: Is There a Problem?Growth in the Cell-only PopulationThe Cell Phone Problem for RDD SurveysPercent Ages 18-34Percent Ages 18-34Practical Considerations Is it Feasible to Survey People on Their Cell Phones?Yes, But Its ExpensiveInterviewing RatesWhat We DidRespondent CharacteristicsLandline sample versus Cell-only adultsCharacteristics of Landline Sample and Cell-only Respondents*Characteristics of Landline Sample and Cell-only Respondents*Characteristics of Landline Sample and Cell-only Respondents*How are estimates affected?Impact on Estimates from Including Cell PhonesImpact on Estimates from Including Cell PhonesImpact on Estimates from Including Cell PhonesImpact on Estimates from Including Cell PhonesImpact on Estimates from Including Cell PhonesMedia Consumption EstimatesMedia Consumption Estimates, 2006Media Consumption Estimates, 2006Under Age 30: Landline Sample and Cell-only Respondents*Under Age 30: Landline Sample and Cell-only Respondents*Under Age 30: Impact on Estimates from Including Cell PhonesWhats the Potential for Bias?Whats the Potential for Bias?Whats the Potential for Bias?Whats the Potential for Bias?Whats the Potential for Bias?Quality of Responseslandline versus cell phone interviewsQuality of ResponsesQuality of ResponsesSample Sizes:Will a combined design yield more interviews with groups relying mostly on cell phones?Not if the cost is fixedNot if the cost is fixedConclusions

Recommended

View more >