The Variable Costs of Cell Phone Interviewing · 5/15/2011 · –To allow comparison of RDD+Cell vs. EWP vs. Cell –Most of these studies needed comparison to earlier years that

Guterbock, et al: Dual-frame to Triple Frame AAPOR: May 15, 2011

UVa Center for Survey Research 1

Center for Survey Research

University of Virginia

From Dual-Frame

to Triple Frame:

An Assessment of Coverage Bias in a Telephone Survey Design

Combining RDD, Directory-Listed And Cell Phone Samples

Presented at AAPOR 2011 Phoenix, AZ May 15, 2011


University of Virginia 2

Authors

Thomas M.Guterbock


[email protected]

Abdoulaye Diop

Qatar University [email protected]

James M. Ellis

University of Virginia [email protected]

John Lee Holmes

University of Virginia [email protected]

Trung Kien Le

Qatar University [email protected]



Overview

• Why triple-frame?

• The 2010 Behavioral Study of the NCR

• Coverage and distribution of phone service (5 segments)

• Contrast of RDD, EWP and Cell Phone frames

• Calling efficiencies

• Comparing substantive results:

– triple frame vs. RDD+Cell

• Cost comparisons

• Conclusion

mailto:[email protected]

mailto:[email protected]





Why triple-frame?

From one frame, to two, to three



From RDD to Dual-frame • ―Traditional‖ list-assisted landline Random Digit Dialing

is beset with problems – Increasing under-coverage due to Cell Phone Only (CPO)

households

– Lower working-number rates

– Declining response rates

• Dual-frame telephone surveys are now standard for many survey organizations – Combining landline RDD and cell phone RDD frames

– Most often using an ‗overlap‘ or ‗all cell‘ design

– See AAPOR Task Force 2010 for full discussion

• Adding cell phones covers the CPO‘s and favorably alters reachability of dual-phone households



Consider: Electronic White Pages

(directory listed) sample

• Previous studies have compared EWP to landline

RDD sample in statewide, regional, local studies.

– Guterbock et al. 2003, Oldendick et al. 2004.

• Guterbock, Diop & Holian 2007 explored race and

other predictors of listedness in a survey of the

National Capital Region

• Caution: These studies pre-date the surge in CPO

households





EWP vs. RDD

• Substantive results are similar, in general

• Advantages of EWP:

– More efficient

– Lower cost

– Greater geographic specificity

• Disadvantages:

– Undercoverage of minorities, lower income, renters

– African Americans less likely to have listed numbers

– Blacks therefore under-represented in EWP samples



Who needs RDD?

• There are notable similarities between characteristics of

unlisted landline households and CPO households

– Minorities, the young, renters, lower incomes

• Could the gaps in the EWP frame (undercoverage) be filled

in by inclusion of the cell phone frame?

• We proposed in 2008: EWP+Cell as a dual-frame design

alternative to RDD+Cell

– Based on analysis of NHIS data through 2006

– Just published in Social Science Research

• We presented a favorable comparison of the two designs in

three county-based surveys in Virginia (2009)



From Dual Frame to Triple Frame

• CSR has completed ten telephone surveys that use a triple-frame design – 4 metro-area based; 6 county-based

– All cell phones included—no screening for CPO

– All in Virginia or DC metro area

• Two reasons for triple-frame design: – To allow comparison of RDD+Cell vs. EWP vs. Cell

– Most of these studies needed comparison to earlier years that used landline RDD exclusively

• We are at a transitional stage in telephone sampling – Triple-frame designs are a compromise between ‗standard‘ dual-

frame design and our proposed alternative of EWP+Cell





error cost

It‘s all about. . .



The 2009

Behavioral Survey of the

National Capital Region

12 12 12

2009 Survey of

Behavioral Aspects of

Sheltering and Evacuation

in the National Capital

Region

Sponsor: VDEM Funding: U.S. DHS





13 13 13

Survey Goal

• Collect information from residents of the National Capital Region that would predict behavior in the area in the event of an emergency.

• Included factorial experiment that varied features of a ―dirty bomb‖ attack by terrorists.

– Ask how residents would respond to specific ―shelter-in-place‖ scenarios

– What variables have the most effect on behavior?

– What patterns of evacuation and shadow evacuation should be expected?

– Where would the evacuees try to go?

• The resulting data are being used to inform the decisions made by administrators in the region and beyond.

– Details in CSR‘s report, 2010.



14 14 14

Features of the Survey

• In-depth survey: average interview length 28 minutes – Fully supported Spanish language interviews as needed

• 2,609 interviews conducted by CSR, Sept-Dec 2009.

• Triple-frame sample design: – 1269 Landline RDD completes – 898 EWP (directory listed) completes – 442 cell phone completes (no screening for CPO‘s)

• RDD sample was backmatched to addresses – Advance postcard sent to EWP and backmatched RDD cases

• Weighting by ownership, race, gender, geography, and type of telephone service

• Margin of error: +/- 2.3 percentage points – After weighting



Coverage and

distribution of phone service





What percentage of

landline phones are unlisted?

• Each respondent was asked whether their landline

is listed in the directory.

– Dual users reached by cell phone were asked to report

on whether their landline is listed

• Percent unlisted can be taken directly from the

RDD frame.

• 19.9% of landline RDD completes are unlisted.

– As in our other studies, a small portion of those in the

EWP frame report their phones to be unlisted.



What is the CPO percentage?

• NHIS has been used as the ‗gold standard‘ for

weighting by phone service.

• NHIS did not provide estimates for this

geography.

• We used the locally based method of estimation

described by Guterbock 2009.

• 31.4% of cell phone respondents were CPO‘s.

• Final estimate: 15.4% of telephone HH are CPO.

– After excluding landlines with unknown listed status

2009 NCR Telephone Universe

2 CELL + ULL

18.1%

4 CELL + LLL

61.0%

5 LLL ONLY

4.2%

3 ULL ONLY

1.3% 1 CELL ONLY

15.4%



What EWP+Cell would cover

2 CELL + ULL

18.1%

4 CELL + LLL

61.0%

5 LLL ONLY

4.2%

3 ULL ONLY

1.3% 1 CELL ONLY

15.4%



Little undercoverage

• Households with no cell phone and an unlisted

landline phone (ULL only), are but 1.3% of the

region‘s telephone households.

• These are covered by the landline RDD frame, but

not covered by the EWP frame.

• EWP frame underestimates unlisted percentage.

• We weight the unlisted percentage among all

landlines to 19.9% (the unlisted percent in RDD).



Contrasting the three frames

And the matched vs. unmatched

portions of landline RDD frame



Comparing calling efficiency

Landline RDD

EWP

Cell

RDD Matched Un-

matched

combined

Completes per

hour (CPH) .69 .39 .58 .84 .54

Hours per

complete (HPC) 1.45 2.56 1.71 1.19 1.85

n of completes 968 301 1269 898 442

Calling hours 1403 772 2175 1069 819

Comparing key demographics

Landline RDD

EWP

Cell

RDD

ACS

2008 Matched Un-

matched

combined

% renter 18.7 25.3 20.3 14.9 38.4 34.3

% African-

American 16.5 27.4 19.1 13.8 23.9 27.5

% never married 19.3 22.7 20.1 16.5 38.9 --

% age 18-25 3.3 5.4 3.8 1.6 22.0 11.0

n of cases 968 301 1269 898 442



Comparing substantive results

• How are survey results affected when a triple-

frame sample is used?

• We compare our triple-frame result with the

results we would have obtained with an

RDD+Cell design.

• Both designs are post-weighted to the same

control weights: – Ownership and race (joint distribution), gender, 8 counties, type of

telephone service (CPO, LLO, dual user reached by landline, dual

user reached by cell phone) and listed status





Comparing substantive results

Triple

frame

design

RDD+Cell Difference

Would leave

immediately in

maximum hazard

19.4% 19.8% -0.4%

Has an emergency kit

prepared 32.6% 32.6% 0.0%

Strongly agree it is

very important to live

in this particular area

32.5% 31.8% 0.7%



Let‘s compare 20 variables

• Would leave scene in a

dirty bomb attack

– Minimum hazard level

– Moderate

– Maximum

• Can trust most people

• Trust local government

• Has an emergency plan

• Has an emergency kit

• Has a meeting place

• Worry about attack

– 3 levels of hazard

• High perceived risk

– Property damage; injury

– 3 levels of hazard

• Agree/Strongly agree:

– Feel at home where I live

– I have a lot in common with

neighbors

– Important for me to live in

this area



Differences for 20 variables

0 0 0 0

1

14

5

0 0 0 0 0

2

4

6

8

10

12

14

16

-5 -4 -3 -2 -1 0 1 2 3 4 5

N

u

m

b

e

r

o

f

I

t

e

m

s

Raw Percentage Point Difference: Triple-frame - RDD+Cell

Margin of error +/- 2.3%





Cost comparison



Cost factors and assumptions

• Assume that a survey of like size had been carried out with

a ‗traditional‘ dual frame design

– Same number of cell phone completions as in our triple-frame

design

– 442 cell phone completes

– 2167 landline RDD completes

• RDD uses more sampled numbers

• RDD cost per sampled number is higher

– Due to extra charge for backmatching

• Assume postcard sent to backmatched RDD cases

• Cell phone completes get $10 incentive

Triple frame costs Land-line

RDD

EWP Cell

phone

RDD

Total

N of completes 1,269 898 442 2,609

CPH .58 .84 .54 .62

Calling hours 2,175 1,069 819 4,062

Cost of calling hours $69,590 $34,210 $26,193 $129,992

Sampled numbers 14,083 6,127 9,585 29,795

Cost of sampled numbers $2,110 $674 $1,150 $3,934

Postcards 7,631 6,127 -- 13,758

Cost of postcards $3,052 $2,451 -- $5,503

Incentive Cost -- -- $4,420 $4,420



RDD+Cell costs (projected) (Note: Assumes 54% Landline RDD

back-matched to directory listing and

mailed a postcard)

Land-line

RDD

Cell

phone

RDD

Total

N of completes 2,167 442 2,609

CPH .58 .54 .58

Calling hours 3,714 819 4,532

Cost of calling hours $118,835 $26,193 $145,028

Sampled numbers 24,049 9,585 33,634

Cost of sampled numbers $3,603 $1,150 $4,753

Postcards 13,031 -- 13,031

Cost of postcards $5,212 -- $5,212

Incentive Cost -- $4,420 $4,420

Triple frame vs.RDD+Cell costs Each design:

2,609 completes

Triple

Frame

RDD+

Cell

Diff

CPH .62 .58

Calling hours 4,062 4,532

Cost of calling hours $129,992 $145,028 $15,036

Sampled numbers 29,795 33,634

Cost of sampled numbers $3,934 $4,753 $819

Postcards 13,758 13,031

Cost of postcards $5,503 $5,212 - $291

Incentive Cost $4,420 $4,420 0

Total cost $143,849 $159,413 $15,564



Cost summary

• RDD+Cell would have cost: $159,413 ($61.10 per

completion)

• Triple frame design cost: $143,849 ($55.14 per

completion)

• Substitution of EWP for some of the landline

RDD frame saved $15,564 ($5.97 per completion)

or 10.8% of the Triple Frame total.

– Greater savings could have been realized if EWP

percentage were larger relative to landline RDD





Conclusion



error cost

It‘s all about. . .



Conclusion

• For representative general population results, we need to include cell phones in our telephone surveys

• When combined with the cell phone sample frame, EWP sample frames offer – greater efficiency than landline RDD

– lower cost

• There are good reasons to retain some landline RDD sample in the mix at this transitional stage – Allows direct measurement of unlisted percentage

• There is little or no loss of accuracy when EWP is substituted for some of the landline RDD frame – and cell phones are included in the design

• Cost savings are considerable (over 10% in this study)





Conclusion

Three frames are better than two!

RDD EWP Cell



References AAPOR Cell Phone Task Force. 2010. New Considerations for Survey Researchers When Planning and

Conducting RDD Telephone Surveys in the U.S. With Respondents Reached via Cell Phone Numbers. Available online at www.aapor.org.

Guterbock TM, 2009. ―Estimating Local Phone Service and Usage Percentages.‖: How to Weight the Data from a Local, Dual Frame Sample Survey Of Cell Phone and Landline Telephone Users in the United States.‖ AAPOR paper.

Guterbock TM, Diop A, Ellis JM, Le TK, & Holmes JLP, 2009. ―Who Needs RDD–Part II: An Assessment of Coverage Bias in Dual-Frame Designs That Combine Directory-Listed And Cell Phone Samples.‖ AAPOR poster, Hollywood FL.

Guterbock TM, Diop A, Ellis JM, Le TK, & Holmes JLP, 2011. "Who Needs RDD? Combining Directory Listings with Cell Phone Exchanges for an Alternative Telephone Sampling Frame". Social Science Research 40:3 (May): 860-872. Also presented as 2008 AAPOR paper, New Orleans.

Guterbock TM, Diop A, & Holian L, 2007. ―White pages, white people: Reasons for the low listed-phone rates of African-Americans.‖ AAPOR paper, Anaheim.

Guterbock TM, Hartman DE & Hubbard RA, 2003. ―RDD vs listed: An experimental study of coverage error, costs and non-response in a statewide telephone survey. AAPOR paper, Nashville.

Guterbock TM, Lambert JH, Bebel RA, Ellis JM, & Kermer DA, 2010. Population Behaviors in Dirty Bomb Attack Scenarios: A Survey of the National Capital Region. Prepared for the Virginia Department of Emergency Management. University of Virginia Center for Survey Research, April.

Oldendick, Robert W., et al. 2004. ―Differences in an RDD and list sample: An experimental comparison.‖ AAPOR paper, Phoenix.



From Dual-Frame

to Triple Frame:

An Assessment of Coverage Bias in a Telephone Survey Design

Combining RDD, Directory-Listed And Cell Phone Samples

Presented at AAPOR 2011 Phoenix, AZ May 15, 2011

Contact: [email protected]

http://www.aapor.org/

Documents

The Variable Costs of Cell Phone Interviewing · 5/15/2011 · –To allow comparison of RDD+Cell vs. EWP vs. Cell –Most of these studies needed comparison to earlier years that