www.statistik.at We provide information

Development and Implementation of a Mixed-Mode Multipurpose Survey-Tool for Official Statistics

Marc PlateSocial Statistics

Rome15 May 2014

www.statistik.at slide 2 | April 21, 2023

Development of a new Survey-Tool

Status Quo:

CATI and CAPI using BLAISE,Mixed-Mode: Mainly alternating modes for Panels 160 CAPI Interviewer60 CATI InterviewerUp to 200.000 Interviews, in up to 20 Surveys per year

New IT-System to reach these goals:•Enable Multipurpose Surveys•Enable different kinds of Mixed-Mode Designs•Enable new mode “CAWI”

Planned first use: Household Budget Survey 2014/15 then LFS 2016

www.statistik.at slide 3 | April 21, 2023

Enable Multipurpose Design

Vision: Modularize the statistical production. Questionnaires are composed by combining modules

Official Statistics have the need to design surveys more flexibel

Q1Q2Q4Q5Q6Q7

Module 1

Module 2

Module 3

eQuestionnaire

…

Module-Repository

CAWI

CATI

CAPI

PAPI

www.statistik.at slide 4 | April 21, 2023

The Module Repository - an Example

Module: Expenditures for Education and Training

Submodule : Housing Expenditures connected to formal education

Submodule : Expenditures for other (non formal or occupational) education

Submodule: Expenditures for formal education

Submodule : Expenditures for occupational training

Module: Formal Education

Submodule: Highest level of education attained

Module: Occupational EducationSubmodule: Type of occupational education

Submodule: Place of education

Module-Owner

Module-Owner

www.statistik.at slide 5 | April 21, 2023

Enable Mixed-Mode Design

Vision: Make use of “classic” and “new” ways of data collection (CATI, PAPI, CAPI, CAWI) and supply the possibility of combining modes in a concurrent, sequential or longitudinal mixed-mode design

• We need the best balance between standardized ,automated and respondent oriented case-management• We need a combination of modes to achieve highest data quality

Centralized System for planning, conducting and monitoring field-

phase

www.statistik.at slide 6 | April 21, 2023

Monitoring the Field Phase – an exampleActivity Performance Indicator Evaluation Criteria Corrective actionTiming − median time per

page/item− median time for completionper respondent

− above a certainlevel− above or belowa certain level

− check wording, layout… of the item− consider removing theentry

Undesiredresponsepatterns

− rate of satisfycing patternsper screen/grid− rate of satisfycing patternsper respondent

− above a certainlevel− above a certainlevel

− check wording, layout… of screen/grid− consider removingentry

Error/promptmessaging

− rate of error/promptmessages per item− rate of error/promptmessages per respondent

− above a certainlevel− above a certainlevel

− check wording, layout… of item− inspect answers ofrespondent

Responsecorrections

− rate of response correctionsper item

− above a certainlevel

− check wording, layout… of the item

Recruiting − structure of referals - some referalswith frequencysmaller thanexpected

− check for errors,consider alternativerecruiting approach

Lozar-Manfreda & Belek 2013

www.statistik.at slide 7 | April 21, 2023

Websurvey – The Vision

More and more Respondents raise the wish to answer a survey online

Vision: Offer a web-questionnaire so well designed, that it works for the general population without the need of any personal contact.

Websurvey

On- and Offline Functionality

Highest data security

Webportal

List of Questiontypes

Innovative Plausibility Checks

Innovative Explanatory notes

Paradata

Integration of Lookuplists

www.statistik.at slide 8 | April 21, 2023

www.statistik.at slide 9 | April 21, 2023

Multipurpose Design / ModularisationThe Challenges

Substantial Challenges•Reasonable grouping of questions•How to make links between modules•…

Organisational Challenges•Reorganisation of responsibilities into modul-responsible, topic-responsible, questionnaire-responsible Persons.•Strong communications between departments•Module-responsible Person has to design „perfect“ questions•Project Team of one survey is not reasponsible for all questions anymore

How can modularity be accomplished without loosing sight of the overall context?

What methodological implication arise, when questionnaires are designed part by part?

www.statistik.at slide 10 | April 21, 2023

Mixed-Mode Design / CAWIThe Challenges

Known methodological issues of Mixed-Mode desing, particularly•Compareability between modes•Break in time series•How to get R to use web when we „only“ know adress and name• CAWI: Does loosing the interviewer mean loosing quality assurance? (Losing

control over who is answering, Losing means of collecting feedback on questions)

• „New“ Issue: Paradata

Need for constant Evaluation of questionnaire for all modes

.

www.statistik.at slide 11 | April 21, 2023

Please address queries to:Marc Plate

Contact information:Guglgasse 13, 1110 Viennaphone: +43 (1) 71128-7972Marc.Plate@statistik.gv.at

Thank you very much for your attention