Rome, 8-11 July 2008

QUALITY 2008

An application to the Italian IIP

Revision analysis to detectpossible weakness in the

estimation procedures

A. Ciammola, T. Gambuti and A. Mancini

ISTAT

European Conference on Quality in Official Statistics

2/21Rome, 8-11 July 2008

QUALITY 2008

Introduction

Why revision analysis?

A case study

Next steps

Outline

3/21Rome, 8-11 July 2008

QUALITY 2008 Quality and some of its dimensions

AccuracyCloseness of the

estimate to the true (but unknown) value of the

variable to be measured

TimelinessSpan between the

reference period and the publication period

RevisionReliability measure

ReliabilityCloseness between

preliminary estimate and subsequent estimates

U

4/21Rome, 8-11 July 2008

QUALITY 2008 Revision analysis

Real-time databases collection of vintages computation of revisions

Revisions Rt = Lt – Pt Rt = (Lt – Pt) / Lt

Revision measures Size (MAR, RMAR, …) Bias (MR, T-test) Efficiency (News or Noise?, MSR, …)

5/21Rome, 8-11 July 2008

QUALITY 2008 Useful references

OECD / Eurostat Guidelines on Revisions Policy and Analysis

http://www.oecd.org Themes related to revision policy and analysis

Recommendations for revisions policy and analysis Guidelines for establishing a real-time database Recommended statistical measures Pre-programmed software for performing revisions analysis A framework for revisions policy of key economic indicators Comprehensive framework of reasons for revisions and their timing Guidelines on how to decompose total revision into different reasons

for revisions Guidelines on how to use the results from revision analyses to improve

compilation methods Case studies on the relationship between timeliness of release and

size of revisions

6/21Rome, 8-11 July 2008

QUALITY 2008

For users

Objective Availability of all the relevant information for using appropriately the estimates of short-term indicators at different stages of the revision process

provision of information about

past revisions

schedule future revisions

(statistical and definitional)

real-time databases gathering all the vintages

analysis of size, bias and efficiency

of revisions

Why do we measure revisions?

7/21Rome, 8-11 July 2008

QUALITY 2008

For producers

Underlying issues

Targets

Why do we measure revisions?

Bias in the revision process

Inefficiency in compilation of preliminary estimates

Reduction of (the size of) “avoidable” revisions

Detection of the source for bias /

inefficiency

8/21Rome, 8-11 July 2008

QUALITY 2008 A case study

Italian Index of Industrial Production (IIP)

1. Sources and timing of revisions

2. Revision analysis

3. Identification of specific sources for bias

4. Some evidences

9/21Rome, 8-11 July 2008

QUALITY 2008 1. Sources and timing of revisions

Y(t-3) Y(t-2) Y(t-1)Current Year Y(t) – Reference month

J F M A M J J A S O N D

M LR CE

A LR CE PC

M LR CE

J LR CE

J LR CE

A LR CE

S LR CE

O LR CE LR CE

N LR CE

D LR CE

J LR CE

F LR CE

First estimate Second estimate Six-month revision Annual revision

LR Late respondents CE Correction of errors PC Productivity coefficients

10/21Rome, 8-11 July 2008

QUALITY 2008

IIP - Revisions on year-on-year growth rates (raw indices)

Legend *

h=1 – after one month h=12 – after 12 months

MAR – Mean Absolute Revision RMAR – Relative MAR

MR – Mean Revision SD – Standard Deviation

Period: Jan-03 / Dec-07 h=1 h=12

# of revisions 60 48

MAR 0.142 0.246

RMAR 0.053 0.087

MR 0.075 0.083

SD of MR(HAC) 0.021 0.056

T-value 3.564 1.489

Significance of MR Yes * No *

2. Revision analysis

11/21Rome, 8-11 July 2008

QUALITY 2008

Jun Dec-03 Jun Dec-04 Jun Dec-05 Jun Dec-06 Jun Dec-07

-0.5

-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

0.5

2. Revision analysisIIP - Revisions after one month on year-on-year growth rates (raw indices)

12/21Rome, 8-11 July 2008

QUALITY 2008 2. Revision analysis

Why this systematic component? Late respondents? Correction of errors? Productivity coefficients?

(In revisions after 1 month, only July 2004, January 2005, January 2006 and January 2007 are affected)

Which sectors? All sectors? Some specific sector?

How to proceed? Simulation exercise Top-down approach Quality indicators

13/21Rome, 8-11 July 2008

QUALITY 2008 3. Identification of specific sources for bias

Simulation exercise

Removal of the effect of the productivity coefficients

Isolate sources of revisions external to the survey

Fulfil the condition necessary to compute the average contribution of each components to the IIP revisions

PS: revisions computed on y-o-y growth rates

14/21Rome, 8-11 July 2008

QUALITY 2008

IIP

Migs CND CDU CAP INT ENE

Divisions DINT,1 … DINT,j … DINT,7

Groups GINT,1 … GINT,k … GINT,20

Classes CINT,1 … CINT,m … … ... ... CINT,n ... …

Diagram describing the top-down approach

3. Identification of specific sources for bias

15/21Rome, 8-11 July 2008

QUALITY 2008

Quality indicators

Revision measures Contribution of each component to the mean revision

of the higher component

Response rates

3. Identification of specific sources for bias

16/21Rome, 8-11 July 2008

QUALITY 2008

MIGS - Revisions after one month on raw Y-o-Y growth rates

Period: Jan-03 / Dec-07  CND CDU CAP INT ENE

Weights % 22.9 6.1 23.8 35.5 11.7

MAR 0.272 0.415 0.378 0.223 0.149

RMAR 0.084 0.081 0.088 0.073 0.040

MR 0.092 0.072 0.042 0.143 -.003

Contribution to MR ° 0.019 0.006 0.010 0.047 -.003

SD of MR(HAC) 0.047 0.103 0.071 0.030 0.040

T-value 1.962 0.694 0.589 4.724 -.079

Significance of MR No * No * No * Yes * No *

Legend CND – Consumer non durables

CDU – Consumer durables CAP – Capital goods

INT – Intermediate Goods ENE – Energy

° Period Jan-04 / Dec-07 *

4. Some evidences

17/21Rome, 8-11 July 2008

QUALITY 2008

Dec-03 Dec-04 Dec-05 Dec-06 Dec-07-1.5

-1

-0.5

0

0.5

1

1.5Capital goods

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec-1.5

-1

-0.5

0

0.5

1

1.5Capital goods by month

Dec-03 Dec-04 Dec-05 Dec-06 Dec-07-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

1.2Intermediate goods

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

1.2Intermediate goods by month

Revisions after one month on raw Y-o-Y growth rates

4. Some evidences

18/21Rome, 8-11 July 2008

QUALITY 2008

Average weighted response rates

Year Estimate IIP CND CDU CAP INT ENE

2004First 91.5 93.9 94.3 90.3 88.3 97.4

Second 95.0 95.7 96.1 93.4 93.8 99.6

2005First 90.2 90.6 93.5 88.1 87.9 98.7

Second 93.3 93.1 95.4 91.3 92.4 100.0

2006First 88.7 89.0 90.4 87.4 86.4 97.3

Second 91.7 91.4 92.5 90.1 90.1 99.9

2007First 83.7 84.7 82.4 80.8 80.6 97.6

Second 87.6 88.4 86.0 85.6 84.9 99.0

4. Some evidences

19/21Rome, 8-11 July 2008

QUALITY 2008

Revisions after one month on raw Y-o-Y growth rates

Period: Jan-04 / Dec-07  S SC,INT SC,IIP

Weights % 32.3 / 11.5 67.7 88.5

MAR 0.362 0.263 0.159

RMAR 0.100 0.082 0.055

MR 0.263 0.071 0.056

Contribution to MR 0.088 / 0.030 0.047 0.049

T-value 3.985 1.407 1.766

P-value 0.000 0.166 0.084

Legend

S – Selected subset of INT (19 classes)

SC,INT – Complement of S in INT (S U SC,INT = INT)

SC,IIP – Complement of S in IIP (S U SC,IIP = IIP)

4. Some evidences

20/21Rome, 8-11 July 2008

QUALITY 2008 Next steps

Checking the stability of results over time

Experimenting possible countermeasures to biased revisions Treatment of late respondents with different

estimators Assessment of their effects on revisions

ISTAT web page on revisions Real-time database for several short-term indicators Revision analysis

21/21Rome, 8-11 July 2008

QUALITY 2008

THANK YOU!