J. KhélifInsee
July 2008
A quality report for seasonally and trading day adjusted French IIP
Page 2
Quality report on SAJ. Khélif July 2008
Scheme of the presentation
1. General background
2. Building a quality report for IIP
3. Is the tool useful ? How can it be improved ?
Page 3
Quality report on SAJ. Khélif July 2008
Background
When analyzing short term economical indicators, users comment on:
– seasonally adjusted (SA) and/or trading day (TD) adjusted series;
– these series are expected to reflect recent economical evolutions;
– producing good quality SA-TD adjusted series is an important issue.
Page 4
Quality report on SAJ. Khélif July 2008
Background
› Different problems have to be faced by the producer:1. Criteria: there are no universal criteria for “good quality” SA-TD
adjusted series;
2. Timeliness : time for analysis is very short, poor quality problems have to quickly be identified;
3. Know-how : Producers are not necessarily SA experts.
› Our aim: build a tool that helps us judge and improve the quality of our SA-TD adjusted series
– The tool had to be easy and quick to run;– The tool had to propose synthetic analysis tables;– The tool had to be based on widely accepted quality criteria.
Page 5
Quality report on SAJ. Khélif July 2008
2. Developing a tool to evaluate the quality of SA-TD series
› Select criteria among many:– There is no unique way of SA series (unobserved
components);– Each SA adjustment method comes with a series of widely
accepted tests;– We made a selection according to our own local
preoccupations.
› Synthesize the information :– our aim: check for quality with a top-down approach;– Find a way of judging the quality of aggregates;– Target those series that are “really” problematic.
› Our solution : – give grades to elementary series based on the technical
criteria chosen; – aggregate them according to their weight in the economy.
Page 6
Quality report on SAJ. Khélif July 2008
2. Developing a tool to evaluate the quality of SA-TD series
› We decided on controlling 3 main elements:
1. The quality of pre-adjustments and the Reg-Arima estimation of the raw data;
2. Idempotency : SA and TD adjusted series should not show seasonal and/or trading day patterns.
› Our notation system :
1. Quality of pre-adjustments and Reg-Arima modeling
Maximum mark : 20
2. Idempotency Maximum mark : 20
Page 7
Quality report on SAJ. Khélif July 2008
2.1 Measuring the quality of pre-adjustments and Reg-Arima modeling
5 items are checked using Tramo-Seats in Demetra output:– the quality of pre-adjustments : if outliers represent more than
5% of all the observations the series gets a mark of 4 (0 otherwise);
– White noise tests on residuals after Arima modeling: if residuals pass the tests (4 of them) we consider that the modeling is ok (16 points if all 4 are ok).
Possible solutions when the quality is poor:– When there are too many outliers : change the time span;– When the residuals are not white noise : change the model,
change the time span.
Page 8
Quality report on SAJ. Khélif July 2008
2.2 Idempotency
2 items are checked in order to make sure that SA-TD adjusted series show no residual seasonality (TD effects):
– Global TD effects detection test (maximum mark=8);– The X11-Arima F-test for identifiable seasonality (maximum
mark=8);– The weight of the irregular component (maximum mark=4).
Possible solutions when the quality is poor:– The series should not have been TD adjusted;– The pre-adjustments and Reg-Arima modeling is problematic
and should be re-run;– The seasonality is changing and nothing can be done at first.
Page 9
Quality report on SAJ. Khélif July 2008
3. The IIP quality report for April 2007
The quality of SA & TD adjustments seems ok for the IIP:- Idempotency is almost always ok- Whereas the Reg-Arima modeling is of poorer quality.
WeightQuality of the reg-Arima modeling
Idempotency
(mark on 20 points) (mark on 20 points)
210 524 17.5 18.0 17.7 17.7
163 949 18.1 19.0 18.5 14.4
18 577 16.0 16.7 16.4 1.4
30 706 16.7 20.0 18.3 2.7
4 650 12.0 20.0 16.0 0.4
8 323 17.5 20.0 18.8 0.7
12 200 17.1 20.0 18.6 1.1
5 533 18.6 19.9 19.3 0.5
19 144 19.1 20.0 19.6 1.8
40 166 18.2 17.5 17.9 3.4
7 931 19.4 20.0 19.7 0.7
18 076 17.0 15.6 16.3 1.4
14 159 19.2 18.4 18.8 1.3
73 932 18.4 19.1 18.8 6.6
8 134 17.8 19.2 18.5 0.7
4 303 16.6 19.7 18.2 0.4
7 603 19.5 15.7 17.6 0.6
22 151 17.0 19.8 18.4 1.9
21 282 19.3 19.3 19.3 2.0
10 459 19.9 19.4 19.7 1.0
27 999 14.7 13.1 13.9 1.8
33 797 16.0 18.3 17.2 -
EB Agricultural and food industries
EC Consumer-goods industries
C2 Printing and reproduction of recorded media
C3 Pharmacy & perfumes
F4 Manufacture of chemicals
C1 Manufacture of wearing apparel
E2 Machinery and equipement
E3 Electrical & electronic equipment
EF Intermediate-goods industries
F1 Manufacture of non-metallic mineral products
C4 Durable consumer goods
ED Automotive industry
EE Producer-durables industries
E1 Ships, railways & aerospace equipment
ContributionAverage
mark
EH Construction
Manufacturing Industry
Industry
F5 Manufacture metal products,
F6 Manufacture of electrical equipment
EG Energy
F2 Manufacture of textiles
F3 Manufacture of wood, paper & paper products
Page 10
Quality report on SAJ. Khélif July 2008
3. The IIP quality report for April 2007
› 33 elementary indices (out of 120) didn’t pass at least one white-noise test;
› The quality results weren’t satisfactory for wearing apparel & machinery equipment (decreasing production, new surveys);
› An example of what happens when you don’t change your Arima models often enough: the automotive industry in November 2005.
IPI de la branche automobile
90
95
100
105
110
115
120
janv-
00
mar
s-00
mai-
00
juil-0
0
sept
-00
nov-
00
janv-
01
mar
s-01
mai-
01
juil-0
1
sept
-01
nov-
01
janv-
02
mar
s-02
mai-
02
juil-0
2
sept
-02
nov-
02
janv-
03
mar
s-03
mai-
03
juil-0
3
sept
-03
nov-
03
janv-
04
mar
s-04
mai-
04
juil-0
4
sept
-04
nov-
04
janv-
05
mar
s-05
mai-
05
juil-0
5
sept
-05
Série CVSCJO avec anciens paramètres
Série CVSCJO avec nouveaux paramètres
Page 11
Quality report on SAJ. Khélif July 2008
3. The IIP quality report for April 2007
› Idempotency: checking certain properties
– The SA-TD adjusted series must not have seasonal/trading day patterns ;
– Same criterion for the irregular component;– The irregular component should not explain most of the series
evolutions.
› In April 2007, for the IIP :– 25 times out of 120 we detected residual seasonal and trading day effects
‐ The series are unstable (recent surveys)‐ This can happen when particular events close to TD effects happen (some firms close during the summer, change in dates of school vacations, long week-ends…).
– The average weight of the irregular component is almost always very low (inf. to 80%)
– Examples : electricity production was not adjusted for TD effects whereas it should have been ; unstable elementary indexes in “machinery & equipment” ; problems with firms closing in august since 2004 in the automobile industry.
Page 12
Quality report on SAJ. Khélif July 2008
3. Is the tool useful ? How can it be improved ?
› In September 2007, the report was implemented again– we improved the global quality of the IIP;– we corrected some options (electricity production).
› But the tool can be improved– It should be easier to implement ;– It should make more use of X12 quality tests;– It should be harsher on certain criteria (the outlier problem);– It should take other criteria into account (revisions);– It should be made useful to users.