Claudio QUINTANO Rosalia CASTELLANO Gennaro PUNZO

11th July 200811th July 2008 University of Naples “Parthenope” – Faculty of Economics University of Naples “Parthenope” – Faculty of Economics

Department of Statistics and Mathematics for Economic Research Department of Statistics and Mathematics for Economic Research 11

Evaluating the Total Non-Response Evaluating the Total Non-Response Errors in the European-Union Errors in the European-Union Survey on Income and Living Survey on Income and Living

Conditions (EU-SILC): A Territorial Conditions (EU-SILC): A Territorial Quality ProfileQuality ProfileClaudio Claudio QUINTANOQUINTANO

Rosalia Rosalia CASTELLANOCASTELLANO

Gennaro Gennaro PUNZOPUNZO

European Conference on European Conference on Quality in Official Statistics Quality in Official Statistics

2008 2008 Rome, Italy – July 8-11, 2008Rome, Italy – July 8-11, 2008



AIM OF THE WORKAIM OF THE WORKEvaluating the ACCURACY of the Italian Section of EU-SILC Evaluating the ACCURACY of the Italian Section of EU-SILC data with focus on the NON-SAMPLING ERRORS deriving data with focus on the NON-SAMPLING ERRORS deriving from the several components of TOTAL NON-RESPONSEfrom the several components of TOTAL NON-RESPONSE

STEPSSTEPS

A SET OF AD HOC BASIC QUALITY INDICATORS A SET OF AD HOC BASIC QUALITY INDICATORS

(HIERARCHICAL FRAMEWORK)(HIERARCHICAL FRAMEWORK)

CLASSES OF SYNTHETIC QUALITY INDICATORSCLASSES OF SYNTHETIC QUALITY INDICATORS

BROKEN DOWN BY DIFFERENT TERRITORIAL LEVELSBROKEN DOWN BY DIFFERENT TERRITORIAL LEVELS

ONE-WAY RANDOM EFFECTS ANOVA MODEL (Singer, 1998)ONE-WAY RANDOM EFFECTS ANOVA MODEL (Singer, 1998)

RANKING ANALYSIS RANKING ANALYSIS TERRITORIAL QUALITY TERRITORIAL QUALITY

PROFILEPROFILE



EU-SILC: BACKGROUND AND MAIN EU-SILC: BACKGROUND AND MAIN FEATURES OF THE ITALIAN FEATURES OF THE ITALIAN

SEGMENTSEGMENT

PRIMARYSAMPLING

UNITS (PSU)

SECONDARY SAMPLING UNITS(SSU)

HOUSEHOLDSHOUSEHOLDS

A two-stages sampling design…A two-stages sampling design…

……drawn from municipality-drawn from municipality-registers registers

by a systematic samplingby a systematic sampling

MUNICIPALITIESMUNICIPALITIES

……STRATIFIED by STRATIFIED by demographic size, inside demographic size, inside

each NUTS2 regioneach NUTS2 region

288 STRATA288 STRATA

Self-RepresentingSelf-Representing Non Self-RepresentingNon Self-Representing First orderFirst order

Second orderSecond order



WAVES

LONGITUDINAL SUB-SAMPLES

2004

2005

2006

2007

2008

2009

INTEGRATED DESIGN BASED ON INTEGRATED DESIGN BASED ON FOUR YEARLY ROTATIONAL PANELSFOUR YEARLY ROTATIONAL PANELS

A(4) B(3) C(2) D1

B(4) D2C(3) E1

C(4) D3 E2 F1

D4 E3 F2 G1

E4 F3 G2 H1

F4 G3 H2 I1

CROSS-SECTIONALCROSS-SECTIONAL

WHOLE PANELWHOLE PANEL



Source: EU-SILC Quality Reports (2004 and 2005)

WavesWaves AA BB CC DD EE Total Total

20042004 7.9567.956 7.9937.993 7.9987.998 8.0458.045 –– 31.99231.992

20052005 –– 6.1796.179 6.1856.185 6.4146.414 8.0088.008 26.78626.786

EU-SILC SAMPLING DESIGN...EU-SILC SAMPLING DESIGN...

Table 1 – PSU stratification...Table 1 – PSU stratification...

Table 2 – Longitudinal replications in terms of Table 2 – Longitudinal replications in terms of SSU...SSU...

STRATASTRATA PSUPSU

Self-RepresentingSelf-RepresentingNon Self-Representing of First orderNon Self-Representing of First orderNon Self-Representing of Second orderNon Self-Representing of Second order

110110 3030148148

110110 6060592592

TotalTotal 288288 762762Source: Author’s elaborations on Istat data



SAMPLINGSAMPLINGUNITS UNITS

A THEORETICAL FRAMEWORKA THEORETICAL FRAMEWORK

IN-SCOPEIN-SCOPE

RESPONDENTS RESPONDENTS NONNONRESPONDENTS RESPONDENTS

NONNONACHIEVEMENTACHIEVEMENT

NONNONCONTACTEDCONTACTED REFUSEDREFUSED UNABLE TOUNABLE TO

RESPONSERESPONSE

INCORRECTINCORRECTADDRESSADDRESS

NOT LOCATEDNOT LOCATEDADDRESSADDRESS

UNABLE TOUNABLE TOACCESSACCESS

OUT-OF-SCOPEOUT-OF-SCOPE

TEMPORARILYTEMPORARILYNOT-AT-HOMENOT-AT-HOME



EU-SILC BASIC QUALITY EU-SILC BASIC QUALITY INDICATORSINDICATORS

IN-SCOPERATE

NON-ACHIEVEMENTRATE

REFUSALRATE

UNABLE-TO-RESPONSE RATE

INCORRECT ADDRESS RATE

NOT-LOCATEDADDRESS RATE

UNABLE-TO-ACCESS RATE

OUT-OF-SCOPERATE

TEMPORARILYNOT-AT-HOME

RATE



A THEORETICAL FRAMEWORKA THEORETICAL FRAMEWORK

IN-SCOPERATE

NON-ACHIEVEMENTRATE

REFUSALRATE

UNABLE-TO-RESPONSE RATE

INCORRECT ADDRESS RATE

NOT-LOCATEDADDRESS RATE

UNABLE-TO-ACCESS RATE

OUT-OF-SCOPERATE

TEMPORARILYNOT-AT-HOME

RATE

unitshouseholdsamplingtotal

unitshouseholdscopein unitshouseholdsamplingtotal

unitshouseholdscopeofout

contactsnonofnetunitsscopein

refusalsdecisive


eparticipattoabilityoflacked


tparticipannonlocatedresidual

unitshouseholdscopein

adresseslocatednot


addressesaccesstounable


addressesincorrect


eatnotytemporaril

hom



THE PROCESS OF CONTACTING THE PROCESS OF CONTACTING THE EU-SILC SAMPLE THE EU-SILC SAMPLE

HOUSEHOLDS...HOUSEHOLDS...cross-sectional sample...cross-sectional sample...

20042004 20052005

In-Scope RateIn-Scope RateOut-of-Scope RateOut-of-Scope Rate

99.3699.36 0.640.64

98.8598.85 1.151.15

Source: Author’s elaborations on Istat data

Table 3 – Eligibility RatesTable 3 – Eligibility Rates

Table 4 – Non-Contact RatesTable 4 – Non-Contact Rates20042004 20052005

Not-Located Address RateNot-Located Address RateUnable-to-Access RateUnable-to-Access RateIncorrect Address RateIncorrect Address Rate

0.850.850.240.244.064.06

0.530.530.320.322.302.30



scopeofoutaddressesincorrect

0.53%

0.32%

0.30%

DECEASED

INSTITUTIONALIZED

TRANSFERRED

Table 5 – Frame Error RatesTable 5 – Frame Error Rates20042004 20052005

Frame Error RateFrame Error Rate 4.674.67 3.423.42Source: Author’s elaborations on Istat data

-0.32

-1.76

-1.25



... AND THEIR ACTUAL ... AND THEIR ACTUAL INVOLVEMENTINVOLVEMENT

EU-SILC cross-sectional sample...EU-SILC cross-sectional sample...

20042004 20052005

Refusal RateRefusal RateUnable-to-Response Rate Unable-to-Response Rate Non-Achievement RateNon-Achievement Rate Temporarily Not-at-Home RateTemporarily Not-at-Home Rate

8.628.621.091.099.789.782.732.73

7.747.740.910.915.445.443.713.71

Table 6 – Non-Participation RatesTable 6 – Non-Participation Rates


-4.34

NON-ACHIEVEMENT RATE also includes a NON-ACHIEVEMENT RATE also includes a residual share of non-participant households, though residual share of non-participant households, though

successfully contacted, whose reasons are not specifiedsuccessfully contacted, whose reasons are not specified

-0.88

+0.98



COOPERATIONCOOPERATIONRATERATE

NON COOPERATION RATE

NON ACHIEVEMENT RATE

REFUSALRATE

UNABLE TORESPONSE RATE

householdscontactedlysuccessful

householdserviewedlysuccessful int

... A SYNTHESIS OF THE BASIC ... A SYNTHESIS OF THE BASIC QUALITY INDICATORS: A QUALITY INDICATORS: A

HIERACHICAL APPROACH HIERACHICAL APPROACH

COOPERATION RATECOOPERATION RATE



NONCONTACT RATE

CONTACTCONTACTRATERATE

INCORRECTADDRESS RATE

NOT LOCATEDADDRESS RATE

UNABLE TOACCESS RATE





CONTACT RATECONTACT RATE



RESPONSERATE

CONTACTRATE

COOPERATIONRATE

COMPLETIONRATE



householdscontactedsuccefully





householdserviewedlysuccessful

int

x





IN-SCOPEIN-SCOPE

RESPONDENTS NONNONRESPONDENTSRESPONDENTS

NONNONCONTACTEDCONTACTED CONTACTED

INCORRECTINCORRECTADDRESSADDRESS

NOT LOCATEDNOT LOCATEDADDRESSADDRESS

UNABLE TOUNABLE TOACCESSACCESS

OUT-OF-SCOPEOUT-OF-SCOPE

COOPERATING NON NON COOPERATINGCOOPERATING

NON ACHIEVEMENTNON ACHIEVEMENTREFUSEDREFUSED UNABLE TOUNABLE TORESPONSERESPONSE

SAMPLINGSAMPLINGUNITSUNITS



EVALUATING THE ACTUAL SURVEY PARTICIPATION OVER

TIMEEU-SILC cross-sectional and longitudinal samples...EU-SILC cross-sectional and longitudinal samples...

wave 2004wave 2004 AA BB CC DD TotalTotal

Cooperation RateCooperation RateContact RateContact RateResponse RateResponse Rate

79.3079.3095.2195.2175.5075.50

79.8079.8095.2395.2375.9975.99

80.1780.1794.3194.3175.6175.61

82.7182.7194.6794.6778.3178.31

80.5080.5094.8594.8576.3576.35

wave 2005wave 2005 BB CC DD EE TotalTotal

Cooperation RateCooperation RateContact RateContact RateResponse RateResponse Rate

87.0087.0097.4497.4484.7784.77

87.9787.9797.3597.3585.6385.63

86.8986.8997.2297.2284.4784.47

82.6182.6195.7195.7179.0679.06

85.9185.9196.8596.8583.2183.21

Table 7 – Synthetic Participation RatesTable 7 – Synthetic Participation Rates


COMPLETION RATECOMPLETION RATE 75.86%

82.25%

wave 2004

wave 2005

+6.86



0.00

0.02

0.04

0.06

0.08

0.10Piedmont

Aosta ValleyLiguria

Lombardy

Bolzano

Trent

Veneto

Friuli Venezia Giulia

Emilia Romagna

MarcheTuscanyUmbria

Lazio

Campania

Abruzzo

Molise

Apulia

Basilicata

Calabria

SicilySardinia

0.00

0.05

0.10

0.15

0.20

0.25

0.30Piedmont

Aosta ValleyLiguria

Lombardy

Bolzano

Trent

Veneto


Emilia Romagna

MarcheTuscanyUmbria

Lazio

Campania

Abruzzo

Molise

Apulia

Basilicata

Calabria

SicilySardinia

0.00

0.10

0.20

0.30

0.40Piedmont

Aosta ValleyLiguria

Lombardy

Bolzano

Trent

Veneto


Emilia Romagna

MarcheTuscanyUmbria

Lazio

Campania

Abruzzo

Molise

Apulia

Basilicata

Calabria

SicilySardinia

0.00

0.10

0.20

0.30

0.40Piedmont

Aosta ValleyLiguria

Lombardy

Bolzano

Trent

Veneto


Emilia Romagna

MarcheTuscanyUmbria

Lazio

Campania

Abruzzo

Molise

Apulia

Basilicata

Calabria

SicilySardinia

DOES TOTAL NON-PARTICIPATION DOES TOTAL NON-PARTICIPATION DIFFER ACROSS NATIONAL DIFFER ACROSS NATIONAL

TERRITORY? TERRITORY?

Fig. 3 – Non–Response Rates

Fig. 4 – Non–Completion Rates

Fig. 2 – Non–Contact RatesFig. 1 – Non–Cooperation Rates

wave 2004wave 2005



0.00

0.02

0.04

0.06

0.08Piedmont

Aosta ValleyLiguria

Lombardy

Bolzano

Trent

Veneto


Emilia Romagna

MarcheTuscanyUmbria

Lazio

Campania

Abruzzo

Molise

Apulia

Basilicata

Calabria

SicilySardinia

0.00

0.05

0.10

0.15Piedmont

Aosta ValleyLiguria

Lombardy

Bolzano

Trent

Veneto


Emilia Romagna

MarcheTuscanyUmbria

Lazio

Campania

Abruzzo

Molise

Apulia

Basilicata

Calabria

SicilySardinia

DOES TOTAL NON-PARTICIPATION DOES TOTAL NON-PARTICIPATION DIFFER ACROSS NATIONAL DIFFER ACROSS NATIONAL

TERRITORY? TERRITORY?

Fig. 5 – Frame Error Rates Fig. 6 – Refusal Rates

wave 2004wave 2005

In order to investigate in-depth the territorial In order to investigate in-depth the territorial perspective, its role and significance, as well as the perspective, its role and significance, as well as the effects on the total non-response process...effects on the total non-response process...

ONE-WAY RANDOM EFFECTS ANOVA MODEL ONE-WAY RANDOM EFFECTS ANOVA MODEL (NULL MODEL)(NULL MODEL)



ijjij ru ̂

with with ii = 1, 2, ... = 1, 2, ...

,,mm

andand j = j = 1, 1,

2, ... , 212, ... , 21

municipalitiesmunicipalities

ONE-WAY RANDOM EFFECTS ANOVA ONE-WAY RANDOM EFFECTS ANOVA MODEL (NULL MODEL) MODEL (NULL MODEL)

NUTS2 regionsNUTS2 regions

fixed fixed effecteffect random random

effectseffects of belonging to the of belonging to the jjthth NUTS2 NUTS2 regionregion

difference between the difference between the iithth municipality municipality and the mean within the and the mean within the jjthth region region

),0(~ 2Nu j

),0(~ 2Nrij

22

2

measuring the effectmeasuring the effect

representing therepresenting the

intraclass intraclass correlationcorrelation

degree of homogeneity degree of homogeneity within NUTS2 regions within NUTS2 regions



Outcome Rate (2004) Outcome Rate (2004) Non-ContactNon-Contact Non-CooperationNon-Cooperation Non-ResponseNon-Response

Fixed EffectFixed Effect: : InterceptInterceptRandom EffectRandom Effect::RegionRegion variance varianceResidual varianceResidual variance

0.0373 0.0373 (.00340)(.00340)

0.0006 0.0006 (.00009)(.00009)

0.0035 0.0035 (.00018)(.00018)

0.1575 0.1575 (.00812)(.00812)

0.0004 0.0004 (.00039)*(.00039)*

0.0463 0.0463 (.00240)(.00240)

0.1882 0.1882 (.00880)(.00880)

0.0062 0.0062 (.00053)(.00053)

0.0474 0.0474 (.00247)(.00247)

Outcome Rate (2005)Outcome Rate (2005) Non-ContactNon-Contact Non-CooperationNon-Cooperation Non-ResponseNon-Response


0.0207 0.0207 (.00322)(.00322)

0.0007 0.0007 (.00008)(.00008)

0.0027 0.0027 (.00013)(.00013)

0. 1448 0. 1448 (.01308)(.01308)

0.0085 0.0085 (.00107)(.00107)

0.0558 0.0558 (.00265)(.00265)

0.1627 0.1627 (.01366)(.01366)

0.0098 0.0098 (.00122)(.00122)

0.0568 0.0568 (.00270)(.00270)

Table 8 – One-way random effects ANOVA model: main resultsTable 8 – One-way random effects ANOVA model: main results


1363.02004 1165.02004

REGIONREGIONEFFECTEFFECT

1908.02005 1473.02005 1322.02005



Outcome Rate (2004) Outcome Rate (2004) Frame ErrorFrame Error RefusalRefusal Non-CompletionNon-Completion


0.0348 0.0348 (.00317)(.00317)

0.0005 0.0005 (.00008)(.00008)

0.0031 0.0031 (.00016)(.00016)

0.0625 0.0625 (.00463)(.00463)

0.0011 0.0011 (.00013)(.00013)

0.0061 0.0061 (.00032)(.00032)

0.1925 0.1925 (.00900)(.00900)

0.0071 0.0071 (.00056)(.00056)

0.0470 0.0470 (.00245)(.00245)

Outcome Rate (2005)Outcome Rate (2005) Frame ErrorFrame Error RefusalRefusal Non-CompletionNon-Completion


0.0247 0.0247 (.00315)(.00315)

0.0006 0.0006 (.00008)(.00008)

0.0024 0.0024 (.00011)(.00011)

0. 0740 0. 0740 (.00668)(.00668)

0.0012 0.0012 (.00027)(.00027)

0.0243 0.0243 (.00116)(.00116)

0.1703 0.1703 (.01373)(.01373)

0.0113 0.0113 (.00123)(.00123)

0.0564 0.0564 (.00269)(.00269)Source: Author’s elaborations on Istat data

Table 9 – One-way random effects ANOVA model: main resultsTable 9 – One-way random effects ANOVA model: main results

1389.02004 REGIONREGIONEFFECTEFFECT

2000.02005

1527.02004

047.02005 1312.02004

1669.02005



In order to inspect how far is from each Italian NUTS2 regionIn order to inspect how far is from each Italian NUTS2 regionor geographical macro area to the entire national territory or geographical macro area to the entire national territory

in terms of dissimilarities of data production process quality...in terms of dissimilarities of data production process quality...

SYNTHETIC SYNTHETIC

QUALITY QUALITY

INDICATORSINDICATORS

EXPLORING THE MAIN EXPLORING THE MAIN DIFFERENCES DIFFERENCES

IN EU-SILC PARTICIPATION IN EU-SILC PARTICIPATION ACROSS NATIONAL TERRITORYACROSS NATIONAL TERRITORY

SPATIAL SPATIAL

INDICESINDICES

RANKING ANALYSISRANKING ANALYSIS BY NUTS2 REGIONSBY NUTS2 REGIONS

BY MACRO AREASBY MACRO AREAS

COGRADUATIONCOGRADUATION KENDALL’S RANK KENDALL’S RANK

CORRELATIONCORRELATION



RANKING ANALYSISRANKING ANALYSIS

Macro Macro AreaArea

NonNonContactContact

MacroMacroAreaArea

NonNonCooperat.Cooperat.

MacroMacroAreaArea

NonNonResponseResponse

CenterCenterNorth-WestNorth-WestNorth-EastNorth-EastIslandsIslandsSouthSouth

124.78124.78100.13100.13 97.3597.35 81.8981.89 74.8674.86

North-WestNorth-WestCenterCenterNorth-EastNorth-EastIslandsIslandsSouthSouth

114.63114.63110.24110.24 89.2289.22 85.9685.96 83.9383.93

CenterCenterNorth-WestNorth-WestNorth-EastNorth-EastIslandsIslandsSouthSouth

112.25112.25111.46111.46 91.0991.09 85.7385.73 82.8582.85

Macro Macro AreaArea

NonNonContactContact

MacroMacroAreaArea

NonNonCooperat.Cooperat.

MacroMacroAreaArea

NonNonResponseResponse

CenterCenterNorth-WestNorth-WestIslandsIslandsNorth-EastNorth-EastSouthSouth

146.96146.96111.90111.90 95.1895.18 75.5775.57 54.1254.12


125.36125.36110.47110.47102.53102.53 71.4871.48 59.1059.10


122.47122.47115.90115.90 98.1798.17 75.9875.98 58.8858.88

Table 10 – Ranking of Italian geographical macro areas Table 10 – Ranking of Italian geographical macro areas (Italy=100) -(Italy=100) -

waves 2004 and 2005waves 2004 and 2005


Kendall’s rank correlationKendall’s rank correlation **80.02004 **60.02005



NON-CONTACT, OVERALL NON-RESPONSE AND, OBVIOUSLY, NON-CONTACT, OVERALL NON-RESPONSE AND, OBVIOUSLY,

NON-COMPLETION RATES SIGNIFICANTLY DIFFER BY ITALIAN NON-COMPLETION RATES SIGNIFICANTLY DIFFER BY ITALIAN

NUTS2 REGIONS ON BOTH THE WAVES (2004 AND 2005) NUTS2 REGIONS ON BOTH THE WAVES (2004 AND 2005)

CONCLUDING REMARKS...CONCLUDING REMARKS...

DIFFERENTIALS ACROSS NUTS2 REGIONS ALSO CONCERN THE DIFFERENTIALS ACROSS NUTS2 REGIONS ALSO CONCERN THE

FRAME ERRORS AND REFUSALS AS THE TWO CRUCIAL SOURCES FRAME ERRORS AND REFUSALS AS THE TWO CRUCIAL SOURCES

OF NON-CONTACTS AND NON-COOPERATION, RESPECTIVELYOF NON-CONTACTS AND NON-COOPERATION, RESPECTIVELY

NON-COOPERATION RATES SEEM TO BE STATISTICALLY NON-COOPERATION RATES SEEM TO BE STATISTICALLY

DIFFERENT ACROSS NATIONAL TERRITORY ONLY IN 2005DIFFERENT ACROSS NATIONAL TERRITORY ONLY IN 2005

A DOWNWARD TREND OVER WAVE IS REVEALED FOR THE A DOWNWARD TREND OVER WAVE IS REVEALED FOR THE

OUTCOME QUALITY INDICATORS CONSIDERED ALSO DUE TO THE OUTCOME QUALITY INDICATORS CONSIDERED ALSO DUE TO THE

PANEL FRAMEWORK OF THE EU-SILC SURVEYPANEL FRAMEWORK OF THE EU-SILC SURVEY

STAYING PUT OF THE ITALIAN REGIONS IN THE SAME (OR IN A STAYING PUT OF THE ITALIAN REGIONS IN THE SAME (OR IN A

SIMILAR) RANKING FOR THE NON-CONTACT AND NON-SIMILAR) RANKING FOR THE NON-CONTACT AND NON-

COOPERATION RATES AS WELL AS FOR THE FRAME ERROR AND COOPERATION RATES AS WELL AS FOR THE FRAME ERROR AND

REFUSAL RATESREFUSAL RATES



… … AND FURTHER AND FURTHER DEVELOPMENTSDEVELOPMENTS

URBANICITYURBANICITY POPULATION DENSITYPOPULATION DENSITY CRIME RATESCRIME RATES LACK OF SOCIAL COHESIONLACK OF SOCIAL COHESION AND SO ON...AND SO ON...

At a sub-national level, a variety of contextual factors At a sub-national level, a variety of contextual factors may influence survey participation, such as... may influence survey participation, such as...

REALLY...REALLY...

We deliberately neglected these aspects We deliberately neglected these aspects but but

we intend to examine them closely we intend to examine them closely afterwardsafterwards

SOCIO-ECONOMIC SOCIO-ECONOMIC

ENVIRONMENTENVIRONMENT

INFLUENCES ON SURVEY INFLUENCES ON SURVEY

PARTICIPATIONPARTICIPATION

(Groves and Couper, (Groves and Couper,

1998)1998)

Documents

Claudio QUINTANO Rosalia CASTELLANO Gennaro PUNZO