SAE-SPORD Project Team Statistics Research and Innovation Division Statistics Canada, Ottawa

1

Statistics Canada’s Small Area Estimation

Product: BUPF 1.0 (Best Unbiased Prediction via Filtering)

SAE-SPORD Project TeamStatistics Research and Innovation

DivisionStatistics Canada, Ottawa

(for presentation to FLMM_LMIWG Workshop on Oct 17, 2007, Vancouver, BC)

2

Project: SAE-SPORD (Small Area Estimation for Statistical Product Oriented

R&D)

Team: Avi Singh (Project Leader)

François Verret

Claude Nadeau

Pin Yuan

Acknowledgments:

Meth Res Block Fund, Labour Stat Div, FLMM-LMIWG

3

Outline

1. SAE: Introduction

2. SAE: Visual Depiction

3. Product BUPF: Description

4. BUPF Application to Labour Force Survey

5. BUPF Demonstration (GUI Sample Screen-shots)

6. Concluding Remarks and Future Work

4

1. SAE: Introduction

Direct estimates for small areas (or domains) not reliable; e.g., for provinces, annual LFS estimates of Managers in Manufacturing and Utilities (a three-digit occupation code A39) are not reliable. Here provinces could be deemed as small areas.

Data Requirements: Provincial estimates of employment by 3-digit occupation codes

5

Table 1 Monthly Total Employed (A39)(Annual Average for 2003 LFS )

Prov.Populatio

nSize

Sample Size

Direct Estimate

SE CV in %

NL 429,298 3,978 670 177 26.4

PEI 109,886 2,769 233 55 23.5

NS 758,549 5,858 1,532 292 19.0

NB 607,565 5,624 1,275 218 17.1

PQ 6,059,655 18,234 25,273 2,204 8.7

ON 9,766,566 30,373 42,447 3,178 7.5

MA 876,396 7,117 3,023 432 14.3

SK 744,431 7,295 1,963 339 17.3

AB 2,467,412 10,317 7,643 1,098 14.4

BC 3,346,181 9,636 8,676 1,228 14.2

Canada 25,165,939 101,201 92,734 4,260 4.6

6

1. SAE: Introduction …cont.

Need more sample to get more reliable estimates

A cost effective alternative-- use a model such as the common mean model; e.g., the proportion employed in A39 is common across provinces

Quality of estimates depends on the validity of the model.

7


Model provides an indirect (or synthetic) estimate at the area level.

For the common mean model, multiply the national total by the provincial population proportion to get indirect the estimate, e.g., for NL

1.7% times 92,734 = 1582

8

Table 2 Direct and Indirect (under an oversimplified model) Estimates for A39 (Annual Average for 2003

LFS )

Prov.Populatio

nPortion

Sample Size

Direct Estimat

eSAE

Indirect

Estimate

Sample Size

NL 1.7% 3,978 670 1,582 101,201

PEI 0.4% 2,769 233 405 101,201

NS 3.0% 5,858 1,532 2,795 101,201

NB 2.4% 5,624 1,275 2,239 101,201

PQ 24.1% 18,234 25,273 22,329 101,201

ON 38.8% 30,373 42,447 35,989 101,201

MA 3.5% 7,117 3,023 3,229 101,201

SK 3.0% 7,295 1,963 2,743 101,201

AB 9.8% 10,317 7,643 9,092 101,201

BC 13.3% 9,636 8,676 12,330 101,201

Canada 100.0% 101,201 92,734 92,734 101,201

9


A combination of the two estimates ( direct and indirect) may provide a reasonable estimate with adequate precision depending on the level of small area.

The direct estimate is not precise but unbiased, while the indirect estimate is generally precise but not unbiased.

10


SAE combines the direct and the indirect in an optimal way:

SAE for Area d = (shrinkage factor for d) x (direct Estimate for d) +

(1- shrinkage factor for d) x (indirect estimate for d)

If the shrinkage factor is 10%, then only 10% of direct and 90% of indirect are used for SAE. If it is 50%, then both direct and indirect have equal say in compositing the two for SAE.

11


The relative size of the shrinkage factor depends on variability in modeling error (in the indirect estimate) and sampling error (in the direct estimate).

Effective sample size for SAE is more than that for the direct estimate.

12

1: SAE: Introduction (Modeling Requirements)

Direct estimates from other small areas (termed indirect data) needed for modeling purposes; i.e., for predicting estimate for the area of interest.

Need enough small areas for adequate modeling. Subdivide provinces into subprovincial areas:• ER or ER by age by gender instead of province although it is the

province level that is of interest.

13


Beneficial to have an Auxiliary Information Source (Administrative/ Census): need true population totals at the area level for all areas.

Using auxiliary source can improve modeling with the indirect data.

14

1. SAE: Introduction (Modeling Requirements…cont.)

Examples of Auxiliary Information for LFS Application

Administrative Source• Number of employment beneficiary claims at the area level

• Number with employment income

Population Census based demographic projections• Subpopulation counts

15


The model predictor based on indirect data and auxiliary data provides an indirect estimate for the area of interest.

The model can be simple such as the common mean model which doesn’t use any auxiliary data or can be advanced.

16


All indirect estimates are biased but bias can be low if model is good.

Combining direct and indirect estimates gives rise to estimates more precise than either one.

Benchmarking (Sum of small area total estimates within a subgroup of areas equals the direct estimate of the subgroup) helps in reducing model bias.

17

1. SAE: Introduction (User Concerns)

Detailed area-level requirements may vary from user to user.

However, cannot go to a very low level for two reasons: precision of SAEs may not be adequate, and auxiliary data may not be available.

Bias concerns due to use of indirect estimates for borrowing information; models may not be perfect but one chosen with care may be useful.

SAE methodology involves a trade-off between bias and precision

18

1. SAE: Introduction (User Concerns…cont.)

External validation of SAE; can be done periodically using census.

Also, validation by ‘local area’ knowledge

Confidentiality concerns ( this may or may not be a problem because smaller the area, more the error in SAE; built-in protection)

19

2. SAE: A Visual Depiction

• However, with the usual SAE model the overall total is not preserved!

Province ER by Age by Gender

BeforeSAE

(rag level)

AfterSAE (rag level)

AfterSAE (prov.level)

NL

PEI

…

AB

BC

Canada Good! Good? Good?

For Employment in A39

20

2. SAE: A Visual Depiction...cont.

• Benchmarking ensures that the total stays the same after modeling

Province ER by Age by Gender

BeforeSAE (rag level)

AfterSAE (rag level)

AfterSAE (prov.level)

NL

PEI

…

AB

BC

Canada Good! Good! Good!

For Employment in A39

21

3. Product BUPF: Description

STC’s SAE product based on the client need identification (re: SAE Workshop in Feb ’05,see www.flmm-lmi.org for proceedings)

Main Features

• Menu-driven software system

• Sampling design is fully taken into account

• Self-benchmarking for protection against model breakdowns

• Area collapsing to include areas with no or few observations in the modeling process

• Extensive model diagnostics and evaluation of estimates Existing software (such as SAS PROC MIXED, MLwiN, WinBUGS)

are not satisfactory

http://www.flmm-lmi.org/

22

3. Product BUPF 1.0: Description

Part I : Data Preparations

Part II: Modeling Preparations

Part III: Model Selection and Diagnostics

Part IV: Small Area Estimation and Evaluation

Part V: Summary Report

23

4. BUPF Application to LFS

Empirical results presented here are still not final.

Two Main components of the product

• Modeling component (for increasing effective sample size)

• Estimation Component ( combining direct and indirect)

24

4. BUPF Application to LFS…cont

Model: Direct Estimate for Area d = True value + sampling error

True Value= Predictor + Model error

Predictor = x1β1+ x2β2+…; it gives rise to indirect or synthetic estimates.

X-variables considered: # reported income, # employment beneficiary, age-sex counts, etc. all at the small area level

25

Table 3 Direct, Indirect and SAE of Monthly Total Employed (A39)(Annual Average For 2003 LFS )

Prov.

Direct SAE IndirectSAE - Dir

Dir.Estimate

CVEstimat

e

Mod. CVMod

RRMSE

Estimate

Mod. CVMod

RRMSE

NF 670 0.264 579 14.4 603 0.229 -0.136

PEI 233 0.235 207 16.8 187 0.179 -0.111

NS 1,532 0.19 1,417 10.5 1,450 0.177 -0.075

NB 1,275 0.171 1,112 10.0 1,083 0.168 -0.128

PQ 25,273 0.087 24,962 5.6 25,381 0.081 -0.012

ON 42,447 0.075 44,355 6.3 46,255 0.081 0.045

MA 3,023 0.143 2,348 8.2 2,251 0.129 -0.223

SK 1,963 0.173 1,766 9.1 1,753 0.164 -0.100

AB 7,643 0.144 7,276 7.8 7,292 0.134 -0.048

BC 8,676 0.142 8,712 9.4 8,792 0.129 0.004

Canada 92,734 0.046 92,734 4.6 95,047 0.073 0.000

26

5. STC’s SAE Product Demonstration

BUPF 1.0 Demo BUPF 1.0 Demo

28

Part I: Data Preparations

29


30


31


32


34

Part IV: Small Area Estimation and Evaluation

35


Several unique features in the BUPF product for SAE such as self-benchmarking, domain collapsing for nonsampled domains, and extensive diagnostics.

The Graphical User Interface (GUI) for the product is useful as a systematic checklist or as a virtual analyst for efficient production; also useful for training and product demonstration.

36


Complete beta-version of BUPF 1.0; current version is only alpha or a prototype and is not suitable for production.

Plan for validation study with Census 2006.

37

For more information, please contact [email protected]

Thank you…Merci

38

Appendix

Product BUPF 1.0: Detailed Description

39

A1. Product BUPF 1.0: Description

Part I : Data Preparations• M1 : Data Specification

• M2 : Task Specification• The definition of Small Area Modeling domains (SAM

domains) is very important

• Direct estimates, population counts and auxiliary data must be available at this level

• # of SAM domains should be high enough for proper modeling

• Here, SAM domain = ER(73) by Age(4) by Gender(2)

40


Part II : Modeling Preparations• M3 : Benchmark Constraints & Baseline Model

• Self-benchmarking is important to protect against model breakdowns as no model is perfect

• Option: No BC, Global BC, Regional BC

• M4 : Domain Collapsing

• Improved alternative to leaving small sample size SAM domains outside of the model

• M5 : Variance Smoothing

41


Part III : Model Selection and Diagnostics• M6 : Model Selection

• Standard Forward and Backward procedures implemented

• M7 : Variance Component• Needed to find the proper shrinkage to move indirect to

direct

• M8 : Innovation Sequence• Makes it possible to diagnose the model with standard “iid

N(0,1)” error tests

• M9 : Model Diagnostics• Residual Plots, QQ-plots, R-square, Chi-square test for

overdispersion and for model adequacy…

42


Part IV : Small Area Estimation and Evaluation• M10 : Small Area Estimation

• M11 : Evaluation of Estimates

• Check for relative difference between direct and SAE

• Other measures

SAE (direct) (1 )(indirect)

(indirect) [(direct) (indirect) ]

: shrinkage factord

d d d d d

d d d d

43


Part V : Summary Report• M12 : Overall Summary

• Sampling Design and Data Sources (Part I)

• Input Diagnostics (Part II)

• Modeling Diagnostics (Part III)

• Ouput Diagnostics (Part IV)

Documents

SAE-SPORD Project Team Statistics Research and Innovation Division Statistics Canada, Ottawa