The next challenge

Efficient and Effective Mixed- and multi-mode research

Tim Macer, meaning limited, London, UK

Presented at the Dutch Market Research AssociationAnnual Conference, Rotterdam, Netherlands6 & 7 November 2003

Agenda

1. The Rise of Multiple modes

2. The Issues

3. Technical framework

4. Survival guide

1. The Rise of Multiple Modes

OMR scanning

Face-to-face

Telephone

CATI

TCASI (IVR)

MCAPI

CAPI

CASI

OCR scanning

WAP

Evolution of today’s survey modes

TechnologyindependentTechnology based

Disk bymail

1975 1980 1985 1990 1995 2000Time line

CAWI

The rise of multiple modes

In USA, Web surveys are the undisputedreplacement for paper-based mail surveys*

Response rates falling

‘One size fits all’ model does not work ininternational research

Case studies showing that mixing modescanAchieve a better response

Remain scientifically valid

*Source: RS Owen in Quirk’s magazine, Feb 2002, p.24-26

What do we mean by multi-mode?

Multi-mode• Surveys utilizing more than one research

channel to reach different sub-samples, butconfining each sub-sample to one channel

Mixed modeSerial

• Surveys that involve successive interviewingstages, each utilizing a different mode

Parallel• Surveys that allows participants to choose

the mode and even to switch modes

LEVEL OF DIFFICULTY

Mixing Modes: some examples

Multi-country studies Web in USA

CATI in EU countries

CAPI or paper in India

Let respondent choose Contact by phone

Continue by phone or web

In parallel

In serial

The multi-mode bandwagon

0

1

2

3

4

5

6

Modessupported

Product choice (42 packages)Product choice (42 packages)

Source: Research Guide to Software 2003

Multi-mode: the challenge

“Survey organizations, whether they arein universities like mine, in private-sector organizations or in governmentorganizations, are going to have tochange dramatically in some ways inorder to do effective surveys as we bringthese new technologies online and stilluse our other technologies where theywork.”

Don Dillman, Washington State University

2. The issues

What are the problems?

How can these be resolved?

The three types of modal issues

CalibrationThe risk of differential measurement error due to

modal effect on the respondent

CoverageSampling issues—risk of differential non-response

from sub-samples for each mode

ComplexityDuplication of operational and programming effort in

addressing more than one mode

Increased cost, delays and errors from thisduplication

Calibration issues

Don DillmanTotal Design Method in 1978 to achieve consistency

between phone and mail surveys

Revised in 1999 to take into account Internet surveys

Examined response rate measurement differences inexperimental trials

Dillman’s conclusionsThere are observable and systematic differences

Disadvantages outweighed by overall improvement insample coverage, response, time and cost

Source: Dillman et al, paper at AAPOR Conference, Montreal, 2001

Source: Paper at ESOMAR Technovate, Cannes, 2003

Modal influence: calibration or coverage?

Oosterveld and WillemsAnother experimental research design mixed

CATI/Web surveys

Aimed to separate modal effect from populationeffect

Source: Paper at ESOMAR Technovate, Cannes, 2003

Modal influence: calibration or coverage?

Oosterveld and WillemsAnother experimental research design mixed

CATI/Web surveys

Research design separated modal effect frompopulation effect

Their conclusionsThe majority of differences reported in previous

studies between Web and paper can be explained bypopulation difference, not intrinsic modal effects

Mixed mode studies can be designed to have noinfluence on the answers

Source: Quirk’s magazine, July/Aug 2002, p20

Mode switching to improve coverage

Allison & O’KonisMixed Web/CATI survey of online financial services

Initial approach by CATI or Web with option to switch

88% of CATI respondents agreed to a continue theirinterview on the web

54% of them went on to complete

Different modes gave highly similar responses

Their conclusionsSwitching modes does increase response rate

But, provided that the switch is done immediately:tomorrow is too late

Modal influences observed

Presentational influencesGanassali and Moscarola have measured increased

responses when relevant visual clues presented inweb interviews

Modal influences observed

The moderating effect of the interviewer Noted by Poynter and Comely amongst others

Can lead to under-reporting, especially of sociallyunacceptable responses

After: Poynter & Comely, Beyond Online Panels, ESOMAR Technovate 2003

With interviewerOnline

Using a mobile phone whilst driving: claimed level of usage

RarelySometimesOften

Open-ended responses

Oosterveld and WillemsObserved longer and more detailed verbatim

response on the web than phone

Allison and O’KonisObserved great similarity for for phone and web

However, population was one with high internetpenetration

Noted some content differences e.g on‘technographic’ subjects which they attributed topopulation effect

Scale questions

Humphrey Taylor (2000) Observed a tendency for respondents to answer scale

questions differently on the web

Dillman et al (2001) Characterised differences between CATI and CAWI on

anchored scale questions (1=strongly agree etc)

CATI respondents favors the extremes

CAWI significantly more likely to use the entire scale

Bäckström and Nilsson (2003) Observed the same tendency between self completion on

paper and web

More research required

Differences in ‘don’t knows’

HoggMore answers recorded as ‘Don’t know’ or

‘No answer’ in Web surveys than same surveywhen interviewer-led in CATI

Recommends omitting explicit DK/NA categoriesin version displayed on the Internet

Source: Quirk’s magazine, July/Aug 2002, p90

Population effects

Non-response (non-participation)Don Dillman and others observed greater tendency

for males not to participate in CATI and females inWeb surveys

Population effects are also influential in…Open-ended responses

Rating scales

Possibly more (Oosterveld & Willems)

Operational complexity issues

Different recruitment and screeningCan’t always approach by same mode

Duplication of the survey instrumentComplete duplication of effort may be required

Problems managing multiple versions

Data HandlingNeed data in one place in one format

Problems mixing online and offline modes

Mode switchingMust be fast if response rate to be improved

Mode-appropriate texts

3. Technical framework

How should technology be supporting mixedmode research?

What are the software developers doing toprovide this support?

1. Common survey authoring tool across all modes

2. Independence of design and execution

3. Mode specific texts (not through foreign languages)

4. One common, central database for all modes

5. Auto-determine contact mode from sample

6. Efficient mode switching

7. Concealment of previous data when switching to self-com.

8. Reminders and auto-revert to previous mode

9. Single view management & reporting tools across allmodes

10. Quotas that operate across all modes

11. Question constructs that recognise different modes

12. Recording of mode at datum not case level

Framework for the ideal MM system

Suppliers contacted

Askia Askia

Mercator snap

MI Pro MI Pro Research Studio

Nebu Dub Interviewer

Opinion One CAVI

Pulse Train Bellview Fusion

Sphinx Sphinx

SPSS MR Dimensions

Who supports what?

Full

Full

Full

Part

Part

SPSSMR

FullSoon

FullFullPartPaper

FullFullFullFullFullFullFullCAWI

FullPartFullSoonFullFullFullCAPI

FullFullPartFullFullFullFullCATIlight

FullFullFullCATI

SphinxPulseTrainCAVINebuMI ProsnapAskia

The issues—according to the developers

7

0

23

0

5

10

15

20

25

Calibration

Coverage

Complexity

Citations

4

3

6

11

0

0 5 10 15

Complexity

Sampling, screeningDuplicationData handlingMode switchingOthers

Innovation: Calibration issues

Reduction of modal influenceOpinion One CAVI

• Totally consistent appearance for Web, CASI &CAPI

• Novel method for unaided questions in self-completion modes

Sphinx

• Experimental approach

Measurement of modal differencesPulse Train

• collect paradata on mode for each question

Innovation: Complexity issues

Modal independent designSPSS MR

• Modal “players”

Askia, MI Pro, Pulse Train, Nebu, SPSS MR

• Modal templates applied to same surveyinstrument

Central databaseAll apart from snap

Wizards for importing offline data in Askia

Innovation: Complexity issues

Mode switchingHandled well in Askia, Pulse Train, Nebu and Opinion

One

Email despatched automatically in Opinion One

Nebu recognises ‘static’ and ‘dynamic’ swaps

Call me button in Pulse Train linked to dialler

Recall of interviews into CATI mode in Askia, Nebu,Pulse Train

Switching in and out of paper in MI Pro

Missing features

Ability to cross-tab data by mode at adatum level

Support for systematic removal of answersfrom modes, i.e. Don’t Know and Not Stated from self-completion

Up-stream sample management

Support to simplify parallel screening

Developers need to focus more on thecalibration and coverage issues!

4. Mixed mode survival guide

Metadata standards can help

MR slow to embrace standards to alloweasy data transfer from system to system

Most focus on the interchange of collecteddata, not survey instruments

Standards allows the metadata to betransferred along with the data

Examples of metadata include:Question typeUnique question nameQuestion texts and answer texts/codesPermitted ranges of valuesRouting or filtering context

Triple-s

www.triple-s.org

First published 1994

Originated in the UK but now implementedby 30 vendors worldwide

Exchange data and metadata via exportsand imports in a generalized formatVersion 1.1 introduced XML support

New version 1.2 adds filters, weighting and multi-language support

No metadata support for survey filtering orrouting logic

SPSS Dimensions Data Model

A new open (though proprietary)metadata model for survey data

Can be licensed independently of all SPSS MRproducts (don’t have to use SPSS software)

Comes with a developers’ library of tools forbuilding applications that will read or write data viathe SPSS Data Model

Many other software companies now providingsupport for the SPSS Data Model

Metadata for survey data not survey routing andlogic

QEDML

www.philology.com.au

New multi-platformsurvey authoring tool

Exports scriptinglanguages for severalpackages, includingQuancept, Surveycraftand In2form

XML based open system,allows other languagetranslators to be added

Tips for multi-mode survey design

Design your survey to be as mode neutralas possible

Pay attention to rating scales

Consider exclusion of Don’t know/Notstated answers on self-completion modes

Ensure you can identify the mode whenanalysing your data, at each question

Standardise on the software, or at least,the data format

In summary

Modal differences do exist, but can beovercome with careful design

Issues relate to: Calibration, Coverage andComplexity

Common survey authoring and a commonresults database improve MM efficiency

Software manufacturers are largelyfocusing resolving complexity issues

Better standards, especially for surveyinstrument metadata, are needed

BibliographyAllison J & O’Konis C (2002) If Given the Choice, Quirk’s Marketing Research Review,

July/August issue, p 20.

Bäckström, C & Nilsson, C (2002) Mixed mode: Handling method differences betweenpaper and web questionnaires,http://gathering.itm.mh.se/modsurvey/pdf/MixedMode-MethodDiff.pdf

Dillman D A (1978) Mail and Telephone Surveys: The Total Design Method, Wiley

Dillman D A, Phelps G, Tortora R, Swift K, Kohrell J & Berck J (2001) ResponseRate Measurement Differences in Mixed Mode Surveys Using Mail, Telephone,Interactive Voice Response and the Internet, AAPOR Annual Conference, Montreal

Ganassali S & Moscarola J (2002) Protocoles d’enquête et efficacité des sondages parInternet, Journées E-Marketing AFM/AIM Conference, Nantes, France

Macer, T (2003) Research Software Review, The Market Research Society, London.

Oosterveld, P & Williams P (2003) Two Modalities, One Answer. ESOMAR TechnovateConference, Cannes.

Owen R S (2002) A Matter of Trade-offs: Examining the advantages and disadvantagesof online surveys, Quirk.s Marketing Research Review, February, pp 24-26.

Poynter R and Comely P (2003) Beyond Online Panels. ESOMAR TechnovateConference, Cannes

Taylor H (2000) Does Internet Research Work? Comparing online survey results withtelephone survey, International Journal of the Market Research Society, 42.1

www.meaning.uk.com