Paradata Collection Research for Social Surveys at Statistics Canada

23-04-21Statistics Canada • Statistique Canada

Paradata Collection Research for Social Surveys at Statistics Canada

François LaflammeInternational Total Survey Error Workshop (ITSEW)

Quebec, June 2011

23-04-21Statistics Canada • Statistique Canada2

OutlineData collection organization Data collection challengesParadata

Sources Database

Paradata ResearchObjectives ScopePast research Current and future plans

Data Collection Organization

3 regions and 8 locations across CanadaHO collects quarterly and annual business survey dataAll other business moved in the regions

Interviewers (~ 1,500) collect the following dataFace-to-face: CAPI/PAPI (~ monthly 100,000 attempts)

● Concurrent surveysCATI call centres (5) (~ monthly 900,000 calls)

● Household, agriculture and business surveys● Concurrent surveys

Unionization and operational constraints

3 Statistics Canada • Statistique Canada 23-04-21

4

Data Collection ChallengesHandling sophistication and increase of data

requirementsMaintaining acceptable response ratesEnsuring highest quality of data collectedOptimizing capacity

Balancing work within and between Regional OfficesRetention of workforce

Reducing / maintaining collection costsDeveloping and deploying surveys consistently, cost-

effectively and timelyKeeping abreast of evolving survey collection

methodologies and technologies (e.g. multi-mode surveys)

Taking into account operational constraints

Statistics Canada • Statistique Canada 23-04-21


Paradata SourcesParadata is Data Collection Process InformationParadata sources

Call and contact informationAudit trail (interview key strokes)

Interviewer administrative and payroll informationInterviewer notes and observations - Not used extensively

Can be enhanced withSample design and sample unit informationCapacity and planning assumptions

● Budget and target figures

Paradata from previous cycle or supplement surveys


Paradata DatabaseParadata Database includes:

Call/attempts information for both ● Computer-Assisted Telephone Interview (CATI) surveys

● Computer-Assisted Personal Interview (CAPI) surveysInterviewer payroll information

Processed and standardized informationRaw files always available

Historical information since 2003Updated on daily basisPrior to 2006, used for reporting purposes - not for researchAudit trail kept separately

7 23-04-21Statistics Canada • Statistique Canada7

Paradata Research

Paradata can be used for : Operational research (including survey management)

●Essentially before and during data collectionMethodological research

●Historically, the focus is after data collection (e.g. non-response and measurement errors)

Often ‘grey’ zone between the two types of researchNeed to make the link between operational and

methodological research


Paradata Research Objectives

Better understand data collection processIdentify potential operational efficienciesEvaluate new data collection initiativesProvide timely feedback and information

Data collection survey management (Active Management)

Maintain and improve data qualityImprove the way surveys are conducted and

managed


Paradata Research Scope

Initial focus onOn CATI social surveys

● RDD, cross-sectional, longitudinal surveysCall and contact information

Extended toCATI agriculture surveysCAPI surveysPayroll informationAudit trail

And more recently toBusiness surveys


Past ResearchInitial analysis

Effort spent: calls and system timeReaching respondents: contact rate, sequence of calls, best

time to call, contact versus interview, etc.

Active managementCustomized reportsDashboard of key survey performance indicators

Impact of cap on callsOn response rates, survey estimates and costs

Production and cost analysisRelationship between production and costProductivity indicators and survey cost analysis


Past Research 2..3Pace of interview (PoINT)CAPI surveys - Initial investigations

Basic analysis: attempts, time spent, contact ratesParadata quality and consistencyProductivity and cost relationshipInteraction between CAPI surveys

Responsive Collection Design for CATI surveysActive managementIdentify a series of new indicators to assess data collection

quality and performance (e.g. representativity, productivity and cost, responding potential of in-progress case measure)

Implementation - two pilots surveysAnalysis


Past Research 3..3Many ad hoc research projects

Interviewer productivity by level of experienceInteraction between concurrent surveysSystem time versus non-system time, etc.

Research increased knowledge about data collection process and practicesDemonstrate potential benefits - Based on facts (empirical data)Investigate, test and implement new collection strategies and

tools - Think outside the box

Make the balance between theory and practice● Focus on operationally viable projects

Communicate and share information ● Documentation, papers, presentations, seminars, etc.


Distribution of Calls and Timeby Collection Phase More calls and system time spent after a first contact

for both respondents and non-respondents

SLID 2006 - Distribution of Calls and Time Spent by Data Collection Phase

18%

35%

8% 7%

22%

10%10%

29%

37%

4%

13%8%

0%

10%

20%

30%

40%

50%

Before f irstcontact

Betw een firstcontact &interview

Interview Before f irstcontact

After the f irstcontact

No contact

Response Non-Response

% Calls

% System Time Spent


Relationship between Production and Cost Throughout Survey Cycle

Strong relationship Most distributions have the same shapeSystem time is a good predictor for payroll hours Ratios of cost to production can be used to derive productivity indicators

Distribution of System Time and Payroll Hours by Collection Day - Survey of Labour and Income Dynamics (SLID 2007)

0

200

400

600

800

1,000

1,200

1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81 86 91 96 101Data Collection Days

Hours

Complete Interview System TimeTotal System TimeDirect Collection Payroll HoursTotal Payroll Hours


Survey Productivity Indicators Based on time Completed Interview System Time / Total System Time Ratios

Productivity ratios decrease during collection period for CATI Longitudinal CATI survey (SLID) shows larger decreases Productivity for CAPI survey is higher and more stable This ratio is affected by interview length and response rate

Distribution of Complete Interview System Time and Total Sytem Time Ratios For SLID (2008), CCHS CATI and CAPI (January-February 2008)

0%

20%

40%

60%

80%

100%

1 6 11 16 21 26 31 36 41 46 51 56 61 66 71Data Collection Days

CCHS - CAPI

CCHS - CATI

SLID

23-04-21Statistics Canada • Statistique Canada1616 23-04-21Statistics Canada • Statistique Canada16

Current and Future Research Plans

Focus on “Strategies to improve the way data collection is conducted and managed”.

Hence the research need to

Be sound and operationally viable Lead to more cost-effective collection and sample

design strategies Lead to data quality improvements


Responsive Collection Design (RCD) - ongoingFull RCD for SLID 2011 (including embedded experiment for 1st call)Improve current RCD strategy (e.g. propensity models, phase-in of

RCD, new conditions for decision making, cost-efficiency objective)RCD for CAPI surveysDocumentation

CATI cost-efficient framework (5 dimensions)Metrics used for costing and budgetingOptimal resources allocation within and between surveys (2)Collection process and practicesOperational constraints

Investigate approaches and assumptions to plan data collection for multi-mode surveys

Current and Future Research Plans 2..3


Paradata courseDescribe the paradata (e.g. type, contents, quality, etc.)Applications of paradata to plan, manage, monitor, assess and

improve the survey process Share experiencesLong and short versions

Other paradata research projectsSample coordination for CAPI surveysConsolidate and extend the use of audit TrailRCD - Theoretical frameworkSimulation and optimization projectsAd hoc research

Current and Future Research Plans 3..3


Potential Issues for Discussion

Are there important gaps in paradata research? If soWhich type of research need to be done? What are the research priorities?Any specific research with respect to TSE?

Sharing information (communication)Paradata working group, conferences/events (paradata sessions in

many international events), international network… Is it enough/too much? Is it efficient?

Potential collaboration between organizations - can it be improved?

What is the most efficient organizational structure for this type of research?

Statistics Canada • Statistique Canada 23-04-21


For more information, please contact

Pour plus d’information, veuillez contacter

François [email protected]

Documents

Paradata Collection Research for Social Surveys at Statistics Canada