View
0
Download
0
Category
Preview:
Citation preview
1
ESRA Conference 2013
Examples of cost calculation for various mixed mode approaches: Cost drivers and their effectiveness Stefan Klug and Birgit Schrödle
DemoSCOPE Research & Marketing AG, Switzerland
2
ESRA Conference 2013
Mixed mode in (commercial) practise Mixing modes is a rare case where something cheaper can be better than
something more expensive. It is not a quality against price trade off.
But, cost reduction is the side-effect and not the primary purpose for doing mixed mode. The primary purpose is to achieve a higher response rate by leaving the choice of interview mode to the respondent.
Therefore, the trade off is between a (higher) mode error against a (lower ) non-response error.
We find that in practice “mixed mode surveying” was abused as a sales argument to shift interview mode from CATI to CAWI instead of selling the advantage of combined samples.
I believe that by introducing additional mode effect variance the total estimate is closer to reality than the single mode estimate.
If this does not hold true, it is still true that mode effects can at least be described and understood (instead of being ignored).
3
ESRA Conference 2013
Mixed mode in (commercial) practise
Mixed mode designs to study mode effect: An interpretation of mixed mode? Fixing the distribution of Web and CATI e.g. to 2’000 CATI and 2’000 CAWI.
The purpose for this is to achieve equal sample sizes to facilitate comparison and to decrease costs on a fixed base (by shifting the commercial risk to the institute).
• An even more technical variant is to randomly assign and enforce the
interviewing mode instead of leaving the choice to the interviewee. Purpose is the elimination of self-selection bias from the mode bias.
Mixed mode w ithout free choice of mode is an interpretation of
mixed mode. Such designs are applicable if the understanding of the mode effect is more important than the reduction of the non-response bias (w ith free choice of mode).
4
ESRA Conference 2013
Definition: Mode vs. Device
Mode Device
CATI
Landline Phone
Mobile Phone
CAPI
Portable PC (Laptop / Netbook)
Tablet Computer
Mobile (Smart) Phone
CAWI
Desktop PC or portable PC
Tablet Computer
Mobile (Smart) Phone
PAPI Paper
Interviewer guided control of order and pace
Self administered control of order and pace
Presence of interviewer
Visual Presentation
Presentation of stimuli
Acoustic Presentation / higher cognitive burden / help from interviewer available etc.
Drivers of mode effects (examples)
5
ESRA Conference 2013
«Views on the MR world»
There is no «CAMI»
(Computer Aided Mobile Interviewing). CATI Landline
+ CATI Mobile = Dual Frame CATI
There is only one internet:
«CAWI Desktop» +
«CAWI Mobile» = (still) CAWI
Mixed Device (in CAWI) is a must, not an
option
Resarch in Social
Networks is not a new mode but another sample
recruitment source
6
ESRA Conference 2013
Traditional Single Mode (zero node)
Telefone
Face to face
Internet
Letter of Invitation
No switching node. One contact mode assigned to each answer mode
CATI
CAPI
CAWI
PAPI
CATI
CAPI
CAWI
PAPI
Zero node Contact Mode
Answer Mode Start
Answer Mode End
7
ESRA Conference 2013
Single Node Mixed Mode
CATI
CAPI
CAWI
PAPI
Telefone
Face to face
Internet
Letter of Invitation
Mode switching at a single defined time in contact-interview flow
CATI
CAPI
CAWI
PAPI
Contact Mode
Answer Mode Start
Answer Mode End
Single node
Single node can be placed anywhere but it has a fixed position in interview.
8
ESRA Conference 2013
Multi Node Mixed Mode
CATI
CAPI
CAWI
PAPI
Telefone
Face to face
Internet
Letter of Invitation
Single node
Mode switching at multiple times in interview flow
CATI
CAPI
CAWI
PAPI
Contact Mode
Answer Mode Start
Answer Mode End
Multi node
9
ESRA Conference 2013
Classification of Research Methodes
• Costs for survey
preparation
increase
• Participation rate
increases
• Completion rate
increases
Mode Nodes Device
Single Mode Zero Node Single
Multiple
Mixed Mode Single Node Single
Multiple
Multi Node
Single
Multiple
Enforced mode vs. free choice as an additional classification dimension
10
ESRA Conference 2013
Is it (financially) worth mixing modes?
• Yes, when the extra costs for mixed mode can be compensated through mixing more expensive with cheaper interview modes
• If the number of interviews from break-even analysis is achievable,
than mixed mode has a positive cost effect
• If the extra costs for mixed mode setup are 0 than mixed mode offerings will always be cheaper than doing the more expensive mode
• For consequent mixed mode offerings a software platform is needed that reduces extra costs.
• But, mixed mode offering is not about cost saving it is about decreasing
the non-response error (at the cost of introducing mode effects).
11
ESRA Conference 2013
Mixed mode break even calculation
2 estimates + 1 simple calculation
How much higher are the (fixed) costs if doing mixed mode compared
to single mode?
Value 1: Additional (fixed) costs for mixed mode = $20’000
What is the price difference (variable costs) per interview between the most
expensive mode and the cheapest mode?
Value 2: (variable) costs per interview expensive ($40) - (variable) costs per interview cheap ($10) = $30
How many cheap interviews do I need to
compensate for additional fixed costs through mixed mode?
Calculation: Value 1 $20’000 / Value 2 $30 = # Interviews 667
12
ESRA Conference 2013
Example of mixed mode calculation
COSTS Best case Worst Case
h $ h $ h $ h $ $ $Setup 2'500$ 2'500$ 2'500$ 2'500$ 2'500$ 10'000$ Labor Costs
Programming 10 1'000$ 10 1'000$ 10 1'000$ 10 1'000$ 1'000$ 4'000$
Data base design/Sampling etc. 10 1'000$ 10 1'000$ 10 1'000$ 10 1'000$ 1'000$ 4'000$
Question wording 5 500$ 5 500$ 5 500$ 5 500$ 500$ 2'000$
Contact 10'000$ -$ 10'000$ 10'000$ 10'000$ 10'000$ Labor costs 200 10'000$
Adress Costs 10'000$ 10'000$
Interview 40'000$ 70'000$ 5'000$ 100$ 20'050$ 20'050$ Labor costs 667 33'333$ 1000 50'000$
Other costsTelephone expenses 6'667$ Travel expenses 20'000$ Incentives 5'000$ 100$
Analysis 2'000$ 2'000$ 2'000$ 2'000$ 2'000$ 8'000$ Labor Costs
Reporting 20 2'000$ 20 2'000$ 20 2'000$ 20 2'000$ 2'000$ 8'000$
Total Costs 54'500$ 74'500$ 19'500$ 14'600$ 34'550$ 48'050$ Fixed Costs 4'500$ 4'500$ 4'500$ 4'500$ 4'500$ 18'000$ Variable Costs 50'000$ 70'000$ 15'000$ 10'100$ 30'050$ 30'050$
Single Mode Mixed Mode
CATI CAPI PAPI CAWI
13
ESRA Conference 2013
$0
$10
$20
$30
$40
$50
$60
$70
$80
$90
$100
$110
$120
$130
Mul
tiple
con
tact
, CA
PI >
=75
%
Mul
tiple
con
tact
, CA
PI +
CAT
I >
=75
%
Sing
le o
r m
ultip
le c
onta
ct,
CAPI
= 5
0%
Sing
le c
onta
ct,
CAPI
>=
75%
Mul
tiple
con
tact
, CA
TI >
=75
%
Mul
tiple
con
tact
, CA
TI >
=50
%
Mul
tiple
con
tact
, CA
PI a
nd P
API
50%
Sing
le c
onta
ct,
CATI
und
CAP
I =
100%
Mul
tiple
con
tact
, CA
PI
50%
and
PA
PI 2
5%
Mul
tiple
con
tact
, CA
PI
50%
and
CAW
I >
= 2
5%
Mul
tiple
con
tact
, CA
TI u
nd P
API
>=
75%
Mul
tiple
con
tact
, CA
TI 7
5% u
nd C
AWI
25%
Sing
le o
r m
ultip
le c
onta
ct,
CAPI
<=
25%
Sing
le c
onta
ct,
CAPI
=50
%,
CATI
= 0
%
Mul
tiple
con
tact
PAP
I an
d CA
WI
>=
50%
Siin
gle
cont
act,
CAP
I <
=25
%
Sing
le a
nd m
ultip
le c
onta
ct P
API
>=
75%
Sing
le a
nd m
ultip
le c
onta
ct,
PAPI
>=
75%
Sing
le a
nd m
ultip
le c
onta
ct C
AWI
>=
75%
Sing
le C
onta
ct P
API
>=
75%
Sing
le c
onta
ct,
CAW
I >
=50
%
Total Costs per Interview Fixed Costs per Interview Variable Costs per Interview Contact costs per Interview Interview costs per Interview
Costs and mode combinations
CAPI only
Equal mix
CATI only CATI 75%, CAWI 25%
CAWI only
PAPI only
14
ESRA Conference 2013
Cost calculation scheme
$-
$20
$40
$60
$80
$100
$120
CAPI CATI PAPI CAWI
0%
25%
50%
75%
100%
Total Mode A Mode B Mode C Mode D
Total CATI CAPI PAPI CAWI
Fixed costs 18'000.0$ 4'500.0$ 4'500.0$ 4'500.0$ 4'500.0$ Fixed costs (basic) 4'500.0$ 4'500.0$ Fixed costs mixed mode 13'500.0$ 4'500.0$ 4'500.0$ 4'500.0$
Variable costs 48'775.0$ 12'500.0$ 17'500.0$ 3'750.0$ 2'525.0$ Variable costs contact (per interview) 20'000.0$ 10.0$ -$ 10.0$ 10.0$ Variable costs interview (per interview) 28'775.0$ 40.0$ 70.0$ 5.0$ 0.1$
Number of interviewsShare of contacts 200% 100% 0% 100% 0%Share of interviews 100% 25% 25% 25% 25%Number of interviews 1'000 250 250 250 250
Total costs 66'775.0$ 17'000.0$ 22'000.0$ 8'250.0$ 7'025.0$ Total costs per interviews 66.8$
15
ESRA Conference 2013
Inbound CATI in mixed mode surveys
• Inbound CATI is a logical extension of the traditional CATI mode to decrease non-response leaving the free choice to call the institute at the best time.
• It requires:
• Invitation letter showing a (unique and toll-free) number
• Full daytime coverage in the call center
• Messaging and IVR routings
• Identification of caller in the questionnaire (e.g. printed ID on invitation letter)
16
ESRA Conference 2013
SMS Invitation
• SMS invitation can be used to inform users of RDD mobile numbers about a study. Similar to an invitation letter sent to a landline phone related address.
• It requires:
• Short message text. (SMS costs are little but with high quantities they become expensive).
• Full multi-language text on a webpage.
• Response channels to “opt out”: e-mail, SMS, telephone
We found no better response rates in a test on 2’000 RDD interviews.
0%
10%
20%
30%
40%
50%
60%
No
conn
ectio
n
No
abili
ty
Not
in t
arge
tgr
oup
No
will
ingn
ess
No
avai
labi
lity
Appo
intm
ent
Succ
ess
SMS send No SMS send
17
ESRA Conference 2013
Conclusion
• Cost saving is not the primary argument to do mixed mode (if you want to save costs: do CAWI)
• Mixing mode design should be extended to the “full mixed mode” approach leaving the choice of the mode to the interviewee and allowing for multiple switching nodes
• Mixed devices is a Must more than an option due to the interviewee’s usage of these devices
• CAPI and PAPI should not be (generally) excluded from mixed mode designs
• It is recommended to use multiple contact modes
• Contact modes can be extended by Inbound CATI.
• SMS invitation can be useful (but not proven yet)
Recommended