Selective automatic editing of mixed mode questionnaires for structural

business statistics

Vienna

21-23 April 2008

*

Jeffrey Hoogland

Roos Smit

Questionnaire changes SBS 2006

Electronic questionnaires

– Data keyed in by respondents– Cheaper– Faster

Content– Less variables, except for small enterprises– More branch specific questionnaires

Editing process

Automatic correction of obvious systematic errors

Selective interactive editing using plausibility indicators

Automatic editing of remaining records based on Fellegi-Holt principle

Automatic editing modifications I

One administration for edit rules used in interactive and automatic editing

Weights

Automatic editing modifications II

Table 1. Influence of reliability weights on SLICE solution for car trade. Variables Old weights New weights Values Old solution New solution Turnover from service 3 1 0 0 0 or 100 Turnover from retail 2 1 0 0 0 or 100 Turnover from wholesale 1 2 0 100 0 Other proceeds 1 3 0 0 0 Total proceeds 4 4 100 100 100

Automatic editing modifications I

One administration for edit rules used in interactive and automatic editing

Weights

Empty entries

Automatic editing modifications I

One administration for edit rules used in interactive and automatic editing

Weights

Empty entries

Protection status values

Automatic editing modifications III

Table 2. Imputation of missing values with SLICE 1.6. Variables Observed

values Solution if missing values are first set at zero

Solution if missing values are edited by SLICE

Housing costs - 100 60 Gas, water costs, etc. - 0 20 Transportation costs - 0 10 Communication costs - 0 8 Costs for machinery - 0 1 Advertising costs - 0 1 Total extra costs 100 100 100

Testing of SLICE I

Table 3. Test waves for SLICE 1.5 and 1.6. Wave SLICE

version Year System Data

1 1.5 2004 PC All kinds of datasets 2 1.5 2004 Acceptation environment SBS 2003 records 3 1.6 2004 PC All kinds of datasets 4 1.6 2007 PC SBS 2005 records for car trade transformed to 2006 format 5 1.6 2007 PC SBS 2006 records for car trade 6 1.6 2008 Acceptation environment Large selection of SBS 2006 records 7 1.6 2008 Production environment Eventually all SBS 2006 records meant for SLICE

Testing of SLICE II

Test wave 5– Leaving entries empty does not cause big performance changes.

Table 4. SLICE results for different treatments of missing values in test wave 5. With missing values Missing values set a zero

Result Records Percentage Records Percentage

Clean 2 1.0% 71 35.0%

Success 178 87.7% 122 60.0%

Success TooManyEditsAtNode 20 9.9% 4 2.0%

Success TakingTooLong 0 0.0% 3 1.5%

TakingTooLong 1 0.5% 1 0.5%

TooManyEditsAtNode 0 0.0% 1 0.5%

TakingTooLong TooManyEditsAtNode 2 1.0% 1 0.5%

Total 203 100.0% 203 100.0%

Testing of SLICE IV

Test wave 6

– Optimising SLICE

– Extensive changes in edit rules and weights

– Introducing protective status values

– Limitations of edit rules in SLICE

Testing of SLICE III

Test wave 6 & 7– Number of edits increased– Lower pass rate

Table 5. Number of variables, SLICE edit rules and SLICE pass rate, for SBS 2005 and SBS 2006 Small enterprises Medium-sized enterprises

Trade

Year

Variables

Edits

Edits for a DoO

Pass rate

Variables

Edits

Edits for a DoO

Pass rate

Car trade 2005 58 96 96 100% 108 167 167 100%

2006 65-68 172 132 90% 83-85 188 148 91%

Wholesale trade 2005 65 113 112 94% 110 171 170 97%

2006 69 321 112-113 95% 87-90 345 136 91%

Retail trade 2005 64-65 114 113 100% 110 170 169 98%

2006 69-70 329 118-119 81% 85-87 352 158 76%

Mode effects I

Especially large enterprises fill in an electronic questionnaire.

Filled-in electronic questionnaires contain less missing values.

Filled-in paper questionnaires are sent to SLICE more often.

SLICE pass rate for electronic questionnaires is higher than for paper questionnaires.

Mode effects II

Table 6. Percentage of paper and electronic questionnaires per trade and size class.

Size class

Trade Questionnaire Small Medium Large Total Number

Construction industry

PaperElectronic

27% 20% 16% 22% 907

73% 80% 84% 78% 3275

Car tradePaper

Electronic

25% 17% 15% 21% 398

75% 83% 85% 79% 1481

Wholesale tradePaper

Electronic

17% 14% 9% 15% 889

83% 86% 91% 85% 5243

Retail tradePaper

Electronic

23% 25% 15% 23% 494

77% 75% 85% 77% 1677

Mode effects III

Table 7. Percentage of paper questionnaires for SLICE, per trade and size class.

Size class Total

Trade For SLICE Small Medium Percentage

Construction industry No 25% 19% 21%

Yes 33% 22% 27%

Car trade No 25% 14% 20%

Yes 24% 22% 23%

Wholesale trade No 12% 9% 9%

Yes 26% 25% 25%

Retail trade No 15% 19% 17%

Yes 32% 35% 35%

Mode effects IVTable 8. Pass rate for SLICE records and differences between electronic and paper questionnaires.

Small enterprises Medium-sized enterprises

Trade Medium SLICE records Pass rate SLICE records Pass rate

Car trade Electronic 445 93% 206 92%

Paper 144 83% 57 89%

Wholesale trade Electronic 423 96% 1115 91%

Paper 147 90% 362 90%

Retail trade Electronic 206 87% 229 76%

Paper 126 75% 112 68%

Conclusions I

Electronic questionnaires have – less edit failures to be solved– less missing values– a higher pass rate for SLICE.

Nonetheless they are less likely to be selected for automatic editing.

Conclusions II

Changing SBS questionnaires results in many compelled changes for automatic editing.

Several ideas to improve SLICE solutions were tested and implemented for SBS 2006

Recommendations

Research into the quality of electronically filled in SBS questionnaires seems appropriate.

An extra edit step before SLICE to compensate for different shortcomings of SLICE.

Obtain more insight in SLICE performance in order to ensure an acceptable pass rate.