1

Composite indicators

Andrea Saltelli andrea.saltelli@jrc.it

OECD WORKSHOP ON INDICATORS OF REGULATORY MANAGEMENT SYSTEMS

EXPERT MEETING

2-3 April 2009

BERR CONFERENCE CENTRE LONDON, UNITED KINGDOM

2

Course based on:

Joint OECD-JRC

handbook.

•5 years of preparation,

•2 rounds of consultation

with OECD high level

statistical committee,

•finally endorsed March

2008 with one abstention

3

How many composites are around? Searching “composite indicators” on Scholar Google:

Scholar Google

October 2005 992

June 2006 1,440

May 2007

1,900

October 2008 3,030

February 2009 3,300

4

• Analytic work based on CI‟s

From Jean Philippe Cotis and Romain Duval, Competitiveness, Innovation and economic growth Istanbul 2007, Conference ‘Measuring and Fostering Progress’

See http://www.oecd.org/dataoecd/20/52/38859413.pdf

5

See www.education-economics.org

From Ludger Wößmann Contribution of Education and Training to Innovation and Growth Symposium on the Future Perspectives of European Education and Training for Growth, Jobs and Social Cohesion, Brussels, 2007

6

7

• Composite indicators and the media

8

Use: Rankings from the Economist web

edition, March 30, 2009

Our rankings Business …

Economics Big Mac index: A light-hearted guide to whether

currencies are at their “correct” level

All Economics rankings »

Education …

Living…

Politics… Global peace index: The world's most and least

peaceful countries

All Politics rankings »

Risk…

Technology…

9

Economically literate press does have appetite for statistic based narratives.

10

Jean Pisani-Ferry and Andre‟ Sapir argument on league tables

11

“[…] civil societies learn from the experience of others. Such policy learning can be enhanced by initiatives that facilitate cross country comparison and benchmarking. A telling example in this respect is […] PISA.”

12

“Transparency benefits the democratic process as it empowers national electorates to review the performance of their own governments and it helps focus the debate on key areas of underperformance. The use of league tables facilitates this process.”

13

Tito Boeri and the World

Bank’s Doing Business

report, speaking at the

Italian Rai3, March 16

2009

14

A ten-pillars snapshot of

doing business in

different countries

181 economies, from

Afghanistan

to Zimbabwe

15

16

65

53 83

75

58

84

53

128

60

156

27

“Italy lost six position

in a year”, says Boeri,

then he goes into the

details. Paying taxes

and enforcing contracts

…

Ranks

for Italy

17

International university rankings: How reliable are these rankings?

THES World University Rankings

Jiao Tong ranking of World Universities

What can we do

to improve our

position on the

international

scene?

18

• Can good practices help ? Ten steps to build Composite Indicators

19

Ideally, a composite indicator should be based on:

-a solid theoretical framework,

-underlying data of good quality,

-a defensible construct.

20

A good technical preparation for a CI can make it more robust (to uncertainties in data, weights,…) more resilient (remain relevant over time), more defensible (in dialogue with stakeholders…)

21

Step 1. Developing a solid theoretical framework

What is badly defined is likely to be badly measured …

Examples …

The challenges are:

• To integrate a broad set of (probably conflicting) points of view while keeping within a manageable construct

• To have a community of peers (individuals, regions, countries) willing to accept the theoretical framework.

22

‘’Doing Business provides a quantitative

measure of regulations for starting a business,

dealing with construction permits, employing

workers, registering property, getting credit,

protecting investors, paying taxes, trading

across borders, enforcing contracts and closing

a business […]’’.

Examples …

23

‘’Doing Business does not measure all aspects of the

business environment […]. It does not, for example,

measure security, macroeconomic stability,

corruption, the labor skills of the population, the

underlying strength of institutions or the quality of

infrastructure […]’’.

Examples …

24

The Canadian Council on Learning has developed the Composite Learning Index

(http://www.ccl-cca.ca/). The framework:

Learning to Know Learning to Do Learning to Live Together Learning to Be[*] **+ Pillars from Jacques Delors’ Task Force: UNESCO's International Commission on Education for the Twenty-first Century.

Jacques Delors

25

I. Learning to Know involves developing the foundation

of skills and knowledge needed to function in the world. This includes literacy, numeracy, general knowledge and critical thinking.

II. Learning to Do refers to the acquisition of applied

skills. It can encompass technical and hands-on skills and knowledge, and is closely tied to occupational success.

26

III. Learning to Live Together involves developing

values of respect and concern for others, fostering social and inter-personal skills, and an appreciation of the diversity of Canadians. This area of learning contributes to a cohesive society.

IV. Learning to Be refers to the learning that helps develop

the whole person ― mind, body and spirit. This aspect concerns personal discovery, self-knowledge, creativity and achieving a healthy balance in life (~Maslow’s top).

27

Step 1. Developing a solid theoretical framework

After Step 1. the developer should have… • A clear understanding and definition of the multidimensional phenomenon to be measured. • A nested structure of the various sub-groups of the phenomenon. • A list of selection criteria for the underlying indicators, e.g., input, process, output.

28

Step 2. Selecting indicators

A composite indicator is above all the sum of its parts…

Excerpt: The strength of a composite indicator can largely depend on the quality of the underlying data. *…+. The theoretical framework should guide the choice of the underlying indicators. The selection process can be quite subjective and therefore should involve stakeholders. Moreover, depending on availability of data, certain indicators cannot be used and proxies need to be considered.

29

Step 3. Multivariate analysis

Analysing the underlying structure of the data is an art …

a1 1

a2 0.85 1

a3 0.83 0.87 1

a4 0.77 0.81 0.89 1

a5 0.56 0.55 0.72 0.72 1

a6 0.64 0.61 0.66 0.73 0.56 1

b1 0.50 0.72 0.69 0.63 0.40 0.54 1

b2 0.61 0.80 0.75 0.74 0.62 0.54 0.86 1

b3 0.43 0.39 0.54 0.54 0.39 0.48 0.10 0.29 1

b4 0.35 0.26 0.41 0.30 0.31 0.37 0.16 0.24 0.75 1

b5 0.70 0.49 0.48 0.45 0.60 0.31 0.09 0.38 0.21 0.13 1

b6 0.55 0.75 0.78 0.77 0.59 0.57 0.77 0.83 0.44 0.24 0.29

a1 a2 a3 a4 a5 a6 b1 b2 b3 b4 b5

Component Initial Eigenvalues

Total % of variance Cumulative %

1 7.242 60.35 60.35

2 1.523 12.69 73.04

3 1.178 9.82 82.86

4 0.554 4.61 87.47

5 0.512 4.26 91.73

6 0.385 3.20 94.94

7 0.242 2.01 96.95

8 0.131 1.09 98.04

9 0.098 0.82 98.04

10 0.064 0.54 99.40

11 0.043 0.36 99.76

12 0.029 0.24 100.00

0

1

2

3

4

5

6

7

8

0 2 4 6 8 10 12 14

Component number

Eig

enva

lues

30

Step 4. Imputation of missing data.

The idea of imputation could be both seductive and dangerous … Almost all datasets contain missing data.

Country a1 a2 a3 a4 a5 a6 b1 b2 b3 b4 b5 b6

AT 0.97 0.78 0.84 0.53 0.72 0.34 0.42 0.18 0.49 0.19 0.85 0.02

BE 0.97 0.72 0.82 0.59 0.86 0.45 0.43 0.18 0.49 0.18 0.87 0.02

BG 0.75 0.31 0.54 0.20 0.61 0.16 0.03 0.01 0.17 0.05 0.44 0.01

CY 0.88 0.47 0.67 0.42 0.69 0.20 0.12 0.07 0.44 0.06 0.54 0.01

CZ 0.95 0.71 0.77 0.40 0.77 0.31 0.22 0.09 0.31 0.08 0.87 0.01

DE 0.95 0.78 0.90 0.61 0.80 0.47 0.52 0.24 0.52 0.19 0.76 0.05

DK 0.97 0.84 0.93 0.80 0.39 0.36 0.33 0.62 0.21 0.93 0.04

EE 0.94 0.62 0.74 0.39 0.78 0.26 0.13 0.07 0.24 0.11 0.93 0.03

EL 0.93 0.60 0.52 0.37 0.72 0.35 0.08 0.06 0.71 0.01

ES 0.94 0.49 0.76 0.49 0.90 0.27 0.16 0.08 0.30 0.11 0.81 0.02

EU27 0.93 0.63 0.80 0.49 0.77 0.34 0.29 0.15 0.41 0.14 0.77 0.03

FI 0.99 0.81 0.97 0.70 0.91 0.47 0.19 0.15 0.53 0.15 0.91 0.03

FR 0.96 0.57 0.82 0.52 0.89 0.35 0.43 0.11 0.76 0.02

31

Three common approaches to deal with missing data: •case deletion (removes either country or indicator from the analysis) •single imputation (e.g. Mean/Median substitution, Regression, etc.)

•multiple imputation (e.g. Markov Chain Monte Carlo algorithms).

Trying to minimize bias and to keep „expensive to collect‟ data that would otherwise be discarded.

32

Step 5. Normalisation of data

Avoid adding up apples and oranges …

Ranking

Standardization

Re-scaling

Distance to reference country

Categorical scales

Cyclical indicators

Balance of opinions

33

Step 6. Weighting and aggregation

The relative importance of the indicators can become the substance of a negotiation …

Weights based on statistical models

Principal component/Factor analysis

Data envelopment analysis

Regression approach

Unobserved components models

34

Step 6. Weighting and aggregation

The relative importance of the indicators can become the substance of a negotiation …

Weights based on opinions: participatory methods

Budget allocation

Public opinion

Analytic hierarchy process

Conjoint analysis

35

Space of alternatives

Including/ excluding variables

Normalisation

...

Imputation Weights

Aggregation

Performance index

Italy Greece Spain

10

20

30

40

50

60

Step 7. Robustness and sensitivity

Uncertainty analysis can be used to assess the robustness of composite indicators …

36

Step 8. Links to other variables

Composite indicators can be linked to other variables and measures

• Comparing effectively complex dimensions:

Figure 1. Relationship between the Composite Learning Index and the Economic

and Social Well-Being Index in Canada.

y = 0.7691x + 20.249

R2 = 0.6979

40

50

60

70

80

90

100

40 50 60 70 80 90 100

Composite Learning Index

Ec

on

om

ic a

nd

So

cia

l W

ell

-be

ing

37

Step 9. Back to the details De-constructing composite indicators can help extend the analysis …

Ad hoc clustering

38

The four-quadrant model of the Composite Learning Index (Canadian Council on Learning)

Step 10. Presentation and dissemination

A well-designed graph can speak louder than words …

39

• In house CI development

40

The Alcohol Policy Index (New York Medical College)

Concept: (WHO report)

Results

Policy message Sensitivity analysis

Published in

PLoSMedicine

41

Figure 1. Relationship between the Composite Learning Index and the Economic

and Social Well-Being Index in Canada.

y = 0.7691x + 20.249

R2 = 0.6979

40

50

60

70

80

90

100

40 50 60 70 80 90 100

Composite Learning Index

Ec

on

om

ic a

nd

So

cia

l W

ell

-be

ing

The Composite Learning Index (Canadian Council on Learning)

Results

Policy message Sensitivity analysis

CCL Report

Framework

Scenario

Pillar Structure Normalisation Weighting Aggregation

CLI Preserved z-scores FA within pillar, Regression

weights to Factors, FA pillars,

Regression weights to pillars

Linear

S1 Preserved z-scores FA within pillar, FA pillars Linear

S2 Preserved Min-max FA within pillar, FA pillars Linear S3 Not preserved z-scores FA all indicators Linear S4 Not preserved Min-max FA all indicators Linear S5 Preserved z-scores FA within pillar, EW pillars Linear S6 Preserved Min-max FA within pillar, EW pillars Linear S7 Not preserved z-scores EW all indicators Linear S8 Not preserved Min-max EW all indicators Linear S9 Preserved z-scores EW within pillar, EW pillars Linear S10 Preserved Min-max EW within pillar, EW pillars Linear S11 Preserved z-scores FA within pillar, FA pillars Geometric

S12 Preserved Min-max FA within pillar, FA pillars Geometric

S13 Not preserved z-scores FA all indicators Geometric

S14 Not preserved Min-max FA all indicators Geometric

S15 Preserved z-scores FA within pillar, EW pillars Geometric

S16 Preserved Min-max FA within pillar, EW pillars Geometric

S17 Not preserved z-scores EW all indicators Geometric

S18 Not preserved Min-max EW all indicators Geometric

S19 Preserved z-scores EW within pillar, EW pillars Geometric

S20 Preserved Min-max EW within pillar, EW pillars Geometric

S21 Preserved Raw data FA within pillar, FA pillars Multi-criteria

S22 Not preserved Raw data FA all indicators Multi-criteria

S23 Preserved Raw data FA within pillar, EW pillars Multi-criteria

S24 Not preserved Raw data EW all indicators Multi-criteria

S25 Preserved Raw data EW within pillar, EW pillars Multi-criteria

42

Environmental Sustainability Index ESI

and Environmental Performance Index EPI.

http://epi.yale.edu/Home

43

Uncertainty & sensitivity analysis

http://epi.yale.edu/

Home

Ran

k 1-1

0

Ran

k 11-2

0

Ran

k 21-3

0

Ran

k 31-4

0

Ran

k 41-5

0

Ran

k 51-6

0

Ran

k 61-7

0

Ran

k 71-8

0

Ran

k 81-9

0

Ran

k 91-1

00

Ran

k 101-1

10

Ran

k 111-1

20

Ran

k 121-1

30

Ran

k 131-1

40

Ran

k 141-1

49

Ran

k 1-1

0

Ran

k 11-2

0

Ran

k 21-3

0

Ran

k 31-4

0

Ran

k 41-5

0

Ran

k 51-6

0

Ran

k 61-7

0

Ran

k 71-8

0

Ran

k 81-9

0

Ran

k 91-1

00

Ran

k 101-1

10

Ran

k 111-1

20

Ran

k 121-1

30

Ran

k 131-1

40

Ran

k 141-1

49

Switzerland 31 30 20 11 6 Viet Nam 5 10 18 29 20 8Sweden 63 25 10 Nicaragua 8 21 28 14 10 14Norway 55 31 6 Saudi Arabia 11 13 23 16 10 11Finland 81 16 Tajikistan 9 6 14 19 18 19 11Costa Rica 81 16 Azerbaijan 10 23 9 24 26 5Austria 15 19 16 21 18 Nepal 5 9 8 15 28 14 13 6New Zealand 98 Morocco 14 20 15 13 15 16Latvia 25 39 26 6 Romania 14 10 33 16 11 5Colombia 74 18 5 Belize 9 29 16 13 14 6France 15 26 30 14 13 Turkmenistan 5 6 14 8 19 25 11 9Iceland 11 15 5 14 15 9 10 13 Ghana 11 14 16 10 10 13 9 8 5Canada 58 33 9 Moldova 8 10 13 30 31Germany 14 40 21 20 Namibia 16 16 25 18 16United Kingdom 11 44 29 11 Trinidad & Tobago 6 20 11 23 8 13 8 5Slovenia 9 18 25 23 8 10 Lebanon 5 15 13 13 5 5 6 20 8 11Lithuania 16 20 14 9 8 9 6 9 Oman 10 25 18 24 5 10Slovakia 15 21 14 25 5 6 8 Fiji 6 15 13 19 14 20 5Portugal 20 46 16 11 5 Congo 9 23 18 13 13 13 9Estonia 56 34 Kyrgyzstan 5 6 15 13 30 15 9Croatia 16 19 23 10 6 9 5 5 5 Zimbabwe 21 11 15 10 15 13 6Japan 6 38 35 14 5 Kenya 18 15 11 11 11 11 6 8 6Ecuador 63 26 5 South Africa 14 16 20 11 16 11Hungary 6 6 13 16 20 6 10 9 Botswana 5 13 18 18 19 16 6Italy 6 28 24 16 13 5 Syria 16 11 21 13 30 6Denmark 8 9 6 15 13 14 8 6 11 Mongolia 5 13 9 25 18 15 10Malaysia 31 48 15 5 Laos 10 8 9 10 6 10 9 19 11Albania 9 11 6 13 10 16 5 6 9 5 Indonesia 9 10 11 23 19 14 5 6Russia 9 33 43 9 Côte d'Ivoire 10 13 15 20 8 11 9 6Chile 16 46 25 8 Myanmar 6 5 16 24 26 15 5Spain 5 30 18 19 14 11 China 9 13 9 25 19 13 5Luxembourg 9 15 16 20 26 5 5 Uzbekistan 14 16 29 29 11Panama 73 20 Kazakhstan 5 15 16 36 24Dominican Rep. 18 54 21 6 Guyana 6 10 19 15 20 20 5Ireland 5 16 13 15 13 13 9 5 Papua New Guinea 6 10 14 9 11 20 20 8Brazil 5 20 29 24 11 Bolivia 8 10 23 13 20 11Uruguay 11 15 9 8 9 10 9 14 Kuwait 9 5 15 28 41Georgia 8 8 19 15 16 10 13 5 United Arab Em. 10 9 36 19 11 11Argentina 10 23 28 24 11 Tanzania 11 13 16 9 11 11 8 6 6United States 5 23 19 24 13 8 Cameroon 6 10 6 13 23 23 15Taiwan 20 13 19 16 10 13 Senegal 5 6 9 16 30 24 6Cuba 5 24 29 19 13 5 Togo 8 6 18 18 18 18 10Poland 5 11 20 35 15 Uganda 8 5 5 6 11 20 21 11 10Belarus 11 10 10 18 16 16 13 Swaziland 6 16 31 24 15Greece 8 18 14 19 15 5 10 6 Haiti 10 21 23 30 10Venezuela 5 11 36 25 18 5 India 11 15 31 25 13Australia 30 30 14 10 9 Malawi 9 13 13 14 11 15 9 6 8Mexico 11 15 34 28 6 Eritrea 6 13 16 16 25 18Bosnia & Herzegovina 5 10 11 24 9 6 8 14 6 Ethiopia 6 8 9 8 9 25 26 5Israel 5 31 19 19 13 5 6 Pakistan 23 9 26 18 18Sri Lanka 19 36 16 16 10 Bangladesh 9 18 24 48South Korea 6 14 14 19 9 8 13 8 Nigeria 6 5 13 15 24 23 6 5Cyprus 10 9 25 14 28 6 Benin 10 11 10 14 13 9 11 13Thailand 8 30 35 11 11 Central Afr. Rep. 13 14 16 38 13Jamaica 8 15 24 11 11 9 10 5 Sudan 10 34 46 6Netherlands 9 11 14 10 21 9 11 9 Zambia 10 10 14 9 21 21 11Bulgaria 5 19 25 15 8 10 6 Rwanda 6 11 18 11 18 5 13 6 9Belgium 13 6 11 6 6 16 10 13 9 Burundi 9 8 15 9 18 29 11Mauritius 6 9 19 18 8 16 15 Madagascar 8 13 16 20 21 15Tunisia 5 10 10 10 14 19 18 9 Mozambique 6 6 9 11 14 18 21 9Peru 15 30 18 30 Iraq 11 26 60Philippines 6 13 26 21 16 9 5 Cambodia 8 15 11 31 28Armenia 6 13 19 8 16 18 8 6 Solomon Islands 16 81Paraguay 11 18 20 18 9 8 5 6 Guinea 13 14 23 36 6Gabon 6 35 28 16 5 6 Djibouti 8 18 35 39El Salvador 5 6 13 16 9 10 9 8 9 8 5 Guinea-Bissau 15 14 28 19 15 5Algeria 5 5 15 26 24 11 6 5 Yemen 6 29 63Iran 11 23 26 18 16 Dem. Rep. Congo 13 29 26 23Czech Rep. 9 8 15 11 13 19 15 10 Chad 8 16 33 40Guatemala 10 16 23 26 14 8 Burkina Faso 9 6 18 43 25Jordan 8 14 24 20 6 14 8 Mali 5 18 36 41Egypt 19 21 24 13 10 6 Mauritania 9 25 40 24Turkey 18 15 18 16 9 6 6 Sierra Leone 11 18 70Honduras 9 28 20 15 13 8 5 Angola 19 79Fyrom 5 5 15 10 18 21 13 6 5 Niger 6 19 73Ukraine 8 15 6 23 11 10 13 10

Legend:Probability between 5 and 15% Source: JRC calculationsProbability between 15 and 30% Notes:Probability between 30 and 50% 1. Numbers express probabilities for the country rankProbability greater than 50% 2. Countries listed based on the 2008 EPI scores from highest to lowest (left to right)Probability lower than 5% is not shown

2008 EPI: Uncertainty analysis for 149 countries

44

Results

Sensitivity analysis

The Knowledge Economy Index

(FP6 - DG RTD)

0

5

10

15

20

25

30

Sw

eden

Denm

ar

Luxem

b

Fin

land

US

A

Japan

Unite

d

Neth

erl

Irela

nd

Austr

ia

Belg

ium

Fra

nce

EU

15

EU

27

Germ

an

Slo

veni

Esto

nia

Malta

Cypru

s

Spain

Czech

Latv

ia

Italy

Gre

ece

Lith

oua

Hungar

Port

uga

Slo

vaki

Pola

nd

20

04

Kn

ow

led

ge

Ec

on

om

y In

de

x (

ran

k)

45

What are the levels of Civic Competence of young people in Europe?

84 indicator index

IEA data on:

youth knowledge,

skills,

attitudes,

values and beliefs

towards

citizenship

Results: Newer democracies perform better on Attitudes towards participation and

Citizenship values.

Older and more stable democracies perform better on Social Justice and Cognitive

tasks on civic knowledge and skills.

46

COMPARATIVE ANALYSIS OF

INNOVATION PERFORMANCE of THE

EUROPEAN COUNTRIES

THE 2007 SUMMARY INNOVATION INDEX (SII)

Data from Eurostat Science and

Technology Indicators and Community

Innovation Survey (CIS)

Country scores, analysis of trends, and

variability across countries and possible

underlying reasons

Nordic countries, Germany and UK are

ahead of the US.

0.0

8

0.2

3

0.2

3

0.2

4

0.2

5

0.2

5

0.2

6

0.2

6

0.2

7

0.2

9

0.3

1

0.3

3

0.3

3

0.3

5

0.3

6

0.3

6

0.3

6

0.3

7

0.4

4

0.4

7

0.4

7

0.4

8

0.4

8

0.4

9

0.5

0 0.5

3

0.5

5

0.5

7

0.5

9

0.6

0

0.6

1

0.6

2

0.6

4

0.6

7

0.7

3

0.1

8

0.1

9

0.4

5

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

TR RO LV BGHR PL SK PT EL HU LT M TES CY IT SI CZ NOAU EECA EU BE FR NL AT IE IS LU US UK DEJP DK IL FI CH SE

AU

IL

SKEL LT

PT

LU

ATIE

FR

NL

BE

UKUS

DE

JP

CH

DK

HU

CYES

MT

EE

ITCZSI

IS

CA

HR

PL

LV

RO

BG

SE

FI

NO

TR

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

-4.0 -3.0 -2.0 -1.0 0.0 1.0 2.0 3.0 4.0 5.0 6.0

Average grow th rate of SII (2003-2007)

2007 S

um

mary

Innovatio

n Index

Sw eden Innovation leaders Innovation follow ersModerate innovators Catching-up countries Turkey

Dotted lines show EU performance.

47

Other sources:

Saisana M., Saltelli A., Tarantola S. (2005) Uncertainty and Sensitivity analysis techniques as

tools for the quality assessment of composite indicators, Journal of the Royal Statistical Society,

A 168(2), 307-323.

Saltelli, A., 2007, Composite

indicators between analysis and advocacy Social Indicators Research, 81 , 65-77.

Brand DA, Saisana M, Rynn LA, Pennoni F, Lowenfels AB, 2007, Comparative Analysis of Alcohol Control Policies in 30 Countries, PLoS Medicine, 4(4), 752-

759.