Experiences in Implementing a Responsive Collection Design … Experiences in Implementing a... ·...

Éric Joyal

System Team Leader – CAI Development

Statistics Canada

eric.joyal@statcan.gc.ca

IBUC 2015, April 2015

Experiences in Implementing

a Responsive Collection

Design (RCD) for Blaise CATI

Social Surveys

2015-04-27 1 Statistics Canada • Statistique Canada

Overview

Introduction

Pre-Responsive Design Tools

Overall Responsive Strategy

Highlights

Lessons Learned

2015-04-27 2 Statistics Canada • Statistique Canada

Introduction

Harder to reach household/respondents

• Downward trend in response rates

• Increase in number of attempts

• Increase in no-contact rates

• Increase in tracing rates for longitudinal

and targeted respondent surveys

2015-04-27 3 Statistics Canada • Statistique Canada

Introduction

Need to find more efficient ways for managing

collection

• Researches and Analysis

Paradata

Collection cycle activity

• Improvement Strategies

2015-04-27 Statistics Canada • Statistique Canada 4

Pre-Responsive Design Tools

Cap on Calls

Time slices Strategies

Stratum Prioritization

Etc.

Need for a plan...

2015-04-27 5 Statistics Canada • Statistique Canada

Responsive Collection Design (RCD)

Responsive Design (RD) is an adaptive approach that uses the information available prior to and during data collection to adjust collection strategy for the remaining in-progress cases

2015-04-27 6 Statistics Canada • Statistique Canada

• Opportunity to control quality, cost and

productivity, based on responding potential

of in-progress cases, survey mode and

interaction between surveys

RCD History

Two experiments 2009-2010

• Households and the Environment Survey (HES)

• Survey of Labour and Income Dynamics (SLID)

Current projects

• Canadian Communities Health Survey (CCHS)

• General Social Survey (GSS)

• Households and the Environment Survey (HES)

• Canadian Tobacco, Alcohol and Drugs Survey (CTADS)

• Etc.

2015-04-27 7 Statistics Canada • Statistique Canada

RCD Strategy

Planning – (Pre-collection)

Initial Data Collection Phase

Responsive Design phase-in 1

Responsive Design phase-in 2

2015-04-27 8 Statistics Canada • Statistique Canada

Regular Management Special Survey Specific

(Z Groups)

INTER_E

INTER_F

SI_IP

PM

REFUS_E

REFUS_F

TRA_DEP_E

TRA_DEP_F

Z01

Z02

Z03

...

Z98

Z99

Overall RCD Strategy – Planning

Analysis of previous data collection cycle

Data collection phases and strategies

(Including staffing plans and propensity

model)

Active management tools and reports

New key indicators and communication plan

2015-04-27 9 Statistics Canada • Statistique Canada

Improvement Strategies

Time Slices Adaptive Strategies

• Based on historical data

Survey of Labour and Income Dynamics (SLID)

National Population Health Survey (NPHS)

2015-04-27 10 Statistics Canada • Statistique Canada

2015-04-27 Statistics Canada • Statistique Canada 11

Improvement Strategies

Time Slices Reactive Strategy

• Based on number of attempts

• Based on household composition

Monitor key indicators to identify start of RCD

phase-in 1

Response rate

Productivity

Cost (proportion of budget spent)

Responding potential of in-progress cases

Use strategic improvement opportunities

previously identified (e.g. new time slice strategy

and intermediate cap)

Overall RCD Strategy - Initial data

collection phase

2015-04-27 12 Statistics Canada • Statistique Canada

RCD Dashboard

2015-04-27 13 Statistics Canada • Statistique Canada

ROResponse

Rate

Cond

1Initial Current

Cond

2

% of

Budget

payroll

hours

Cond

3

% of

Budget

System

Time

Cond

4

% of regular

In-progress

cases

Cond

5

Avg. # of calls

for regular in-

progress

cases / cap

Cond

6

Sum of

conditions

EDM 52.2% 1 64.3% 41.6% 1 56.6% 1 62.4% 1 28.3% 0 7.2 0 4

HFX 52.6% 1 63.6% 45.5% 1 62.8% 1 69.8% 1 27.0% 0 7.7 0 4

SHER 46.3% 0 66.9% 58.3% 1 49.8% 0 50.7% 0 34.5% 0 3.4 0 1

STURG 64.2% 1 74.4% 46.6% 1 67.2% 1 65.6% 1 13.9% 1 11.3 1 6

TOR 52.6% 1 65.7% 53.6% 1 58.0% 1 55.4% 1 27.9% 0 6.8 0 4

WIN 62.4% 1 69.9% 46.8% 1 69.8% 1 69.0% 1 12.5% 1 13.0 1 6

Response Rate Budget and Cost Propensity of In-Progress CasesProductivity

(Average over last 5

• Decision based on survey progress in terms of response

rate, productivity, proportion of budget spent (cost) and

responding potential of in-progress sample

Overall RCD Strategy – Phase-in 1

Categorize and prioritize cases to improve overall

response rates

Probability of completion (propensity) – logistic regression model

(sampling frame and sequence of calls information)

2015-04-27 14 Statistics Canada • Statistique Canada

Monitor key indicators to identify start of RD phase-

in 2

Overall RCD Strategy – Phase-in 2

Prioritize cases to improve sample

representativity

• Priority to domain of interest with lower

response rates

• National versus regional objectives

2015-04-27 15 Statistics Canada • Statistique Canada

RCD Strategy

2015-04-27 16 Statistics Canada • Statistique Canada

Planning

Phase

RCD Phase 2 - to

improve

representativeness

No contact (after first 5 calls)

All No Contact High Priority

Sample Regular

cases High Probability

Special Cases (e.g. refusal, tracing)

Miscellaneous Miscellaneous (All

other in-progress and

special cases)

Initial Collection Phase RCD Phase 1 - to improve

response rate

Intermediate cap on calls

Highlights

Productivity higher for RCD

• Less time spent for same result

Sample representativity improved

More homogenous distribution of calls

• Less cases reaches the Cap on calls

2015-04-27 17 Statistics Canada • Statistique Canada

Lessons Learned

RCD concepts and technical issues

Communication – Active Management

• Required on-going and well planned

communication

2015-04-27 18 Statistics Canada • Statistique Canada

Lessons Learned

Staff

• Interviewer staffing required guidelines

• New analytical skills required

2015-04-27 19 Statistics Canada • Statistique Canada

Limited impact

Questions

2015-04-27 20 Statistics Canada • Statistique Canada