Advocacy, analysis and quality. The Bermuda triangle of ... · Advocacy, analysis and quality. The...

Advocacy, analysis and quality. The Bermuda triangle of Statistics

59th ISI World Statistics Congress

25-30 August 2013 - Hong Kong Special

Administrative Region, China

Session STS023

Statistics and policy

Andrea Saltelli & Michaela Saisana

Joint Research Centre, European

Commission

About indicators

Content

• Statistical indicators for

policy between

modernity and post-

modernity;

• their use for analysis and

advocacy;

• how quality can save the

• when things go wrong

The example of composite

indicators

What official statistics are to the consolidation of the modern

nation state (Hacking, 1990), composite indicators are to the

emergence of post-modernity.

Leibnitz ‘philosophical godfather of Prussian official statistics’ to the Prince Frederik of

Prussia 1700; 56 categories to ‘measure the power of a state’ (the first scoreboard); first

proposal for a statistical office …

Modernity and post modernity; from positivism to constructivism; how ‘Matters of fact’ are

established; Shapin and Shaffer, Latour, …; the emergence of a plurality of norms and views;

the Human Development Index (HDI, 1990) and the explosion of indices…

Composite indicators and post-Modernity

Statistics for policy: three models

A rational-positivist model for the use of indicators and policy

(good quality statistics underpin good policies)

Discursive-interpretive model (statistics contribute to a process

of framing of and focusing on an issue among the many

competing for public's attention)

Strategic model (statistics is used by parties competing for a

given constituency).

see Boulanger, P-M., Political uses of social indicators: overview and application to

sustainable development indicators. International Journal of Sustainable Development,

10 (1,2):14-32, 2007.

Contexts

Composite Indicators

Apples and Oranges

Composite indicators as an object populating a

multidimensional space whose main axes are

advocacy, analysis and quality.

Composite indicators sit between analysis and

advocacy, but quality discriminates the plausible

from the rhetorical.

Advocacy, analysis and quality

These three dimensions (advocacy, analysis and quality) are

not independent from one another.

Most developers adopt for transparency and simplicity linear

aggregation procedures to build composite indicators which

are fraught with considerable difficulties.

In this case quality may suffer at the expenses of advocacy.

THE ROLE OF COMPOSITE INDICATORS FOR

MEASURING SOCIETAL PROGRESS

Ubiquitous; 6-fold increase in 7 y

Statistics' best known face (to general public & media)

Can provide analytic input to policy

Features of CI

October 2005 992

June 2006 1,440

May 2007

October 2008 3,030

September 2009

August 2010 5,240

May 2011 5,900

October 2012

Searching

“composite

indicators” on

Scholar Google:

Ubiquitous; 6-fold increase in 7 y

August

2013: 9,130

1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012

Search www.scopus.com query: TITLE-ABS-KEY("composite indicator*") OR TITLE-ABS-KEY("composite index") OR TITLE-ABS-KEY("composite indices")

Papers

An example, about Côte d’Ivoire ,

Economist October July 2013

One of the indicators in the ruling justly category is

control of corruption, an area in which Côte d’Ivoire fares

particularly poorly. The World Bank’s most recent

corruption rankings, from 2011, put it 38th out of 49

African countries. Transparency International ranked it

130th out of 176 countries last year in its Corruption

Perceptions Index, ahead of Nigeria and Guinea but well

behind neighbouring Liberia, Burkina Faso and Ghana.

Statistics' best known face (to general public & media)

Fortune of CI

Caveats in use

More ICT + More statistical literacy + More appreciation of complexity

“the role of statistical indicators has increased over the last two

decades”

What the Stiglitz report says about CI’s

Report by the Commission on the Measurement of Economic Performance and Social Progress, 2009,

Joseph E. STIGLITZ, Chair, Columbia, University, Amartya SEN, Chair Adviser, Harvard University,

Jean-Paul FITOUSSI, Coordinator of the Commission, IEP,www.stiglitz-sen-fitoussi.fr

The Stiglitz report, on page 65, mentions: […] a general

criticism that is frequently addressed at composite

indicators, i.e. the arbitrary character of the procedures

used to weight their various components.

Adding: […] The problem is not that these weighting

procedures are hidden, non-transparent or non-replicable

– they are often very explicitly presented by the authors of

the indices, and this is one of the strengths of this

literature. The problem is rather that their normative

implications are seldom made explicit or justified.

Caveats in use and construction

It is possible to disentangle evidence based policy from policy based evidence?

see Benoît GODIN on eugenics and the birth of R&D stats: The Culture of

Numbers: From Science to Innovation, INRS, Montreal, Canada,

Communication presented to the Government-University-Industry Research

Roundtable (GUIRR) US National Academy of Sciences, Washington, May 21,

… but many other data based stories as well: Tobacco & health, capital

punishment & crime rate …

•Oreskes, N., Conway E. M., 2010, Merchants of Doubt, Bloomsbury Press

•Leamer, E. E., Tantalus on the Road to Asymptopia, 2010, Journal of

Economic Perspectives, 24, (2), 31–46.

Caveats

Quality can become the new organizing principle which “enables

us to manage the irreducible uncertainties and ethical

complexities” (*).

We call this approach ‘sensitivity auditing’ of a composite

indicator (+).

(*) Funtowicz, S.O. and Ravetz, J.R. (1994). The worth of a songbird: Ecological economics as a

post-normal science. Ecological Economics 10(3), 197-207.

(+) Saltelli A, Guimarães Pereira A, van der Sluijs JP, Funtowicz S., 2013, What do I make of

your Latinorum? Sensitivity auditing of mathematical modeling, to appear, Foresight and

Innovation Policy, arXiv:1211.2668 [physics.soc-ph].

Analysis

CI for evidence based policy

Can provide analytic input to policy

Suggestion: CI as a useful tool within the open method of coordination

Ratings and Rankings 20

The Alcohol Policy Index (New York Medical College)

Concept: (WHO report)

Results

Policy message Sensitivity analysis

Published in

PLoSMedicine

Advocacy

6th dimension of the Rule of

Law Index

(World Justice Project)

• 83 survey questions

• 97 countries (20 EU)

RoL; blue bars = EU countries

7th dimension of the Rule of

Law Index

(World Justice Project)

• 56 survey questions

• 97 countries (20 EU)

RoL; blue bars = EU countries

25 18 December 2013

Corruption Perceptions

Index (Transparency

International)

• 13 sources

• 176 countries (27

CPI; blue bars = EU countries

26 18 December 2013

Canada

Seychell

Ease of Doing Business Rank Paying Taxes

Ease of Doing Business Rank Starting a Business

Dealing with Construction Permits Getting Electricity

Registering Property

Getting Credit

Protecting Inv estors

Paying Taxes Trading Across Borders Enf orcing Contracts

Resolv ing Insolv ency

World Bank Index ‘Ease of Doing Business’ + one of its sub-indices:

‘Paying taxes’; blue bars = EU countries

Paying taxes

Quality

International works on the quality of composite indicators have been

ongoing since 2003 (OECD and JRC).

Quality

Construction E.g. University System at the regional

scale;

Validation E.g. Corruption Perceptions Index

2012, Rule of Law 2012;

Methodology Sensitivity analysis & auditing, multi-

criteria methods, statistics and policy;

Training E.g. WEF, Genève April 2013;

Istanbul July 2013.

Testing (composite) indicators: two approaches

Michaela Saisana, Andrea. Saltelli,

Stefano Tarantola (2005),

Uncertainty and sensitivity analysis techniques as

tools for the quality assessment of composite

indicators,

J. R. Statist. Soc. A 168(2), 307–323.

Paolo Paruolo, Michaela Saisana,

Andrea Saltelli, (2013),

Ratings and rankings: Voodoo or Science?,

J. R. Statist. Soc. A, 176 (3), 609–634.

Sensitivity analysis

First: The invasive approach

Michaela Saisana, Béatrice d’Hombres,

Andrea Saltelli, Rickety numbers: Volatility of

university rankings and policy implications

Research Policy (2011), 40, 165-177

Space of alternatives

Including/

excluding variables

Normalisation

Missing data Weights

Aggregation

Country 1

Country 2 Country 3

Second: The non-invasive approach

Comparing the weights as assigned by developers with

‘effective weights’ derived from sensitivity analysis.

Non invasive Sensitivity analysis

University

Rankings

Comparing the internal coherence of ARWU versus THES (2008) by testing the weights declared by developers with ‘effective’ importance measures. ARWU=Academic ranking of world universities;

THES=Times Higher Education Supplement

declared weight importance

THES X1_Academic opinion: 6354 academics 40% X2_Recruiters’ opinion: 2339 recruiters 10% X3_Full-time equivalent faculty/student ratio 20% X4_Total citation/full time equivalent faculty 20% X5_Percentage of full-time international staff 5% X6_Percentage of full-time international students 5%

Issues with THES: a) ‘Opinion’ variables’ weight overall: >60% instead of 50 b) Faculty/student ratio: 10% instead of 20%

Life expectancy, 33%

Adult literacy, 22%

Enrollment education, 11%

GDP per capita, 33%

Life expectancy, 33%

Education, 33%

GNI per capita, 33%

HDI 2010 more coherent than HDI 2009

Two applications of non invasive sensitivity analysis

…the Bermuda

angle From: Ecological Footprint: Unneeded Footwork

Mario Giampietro and Andrea Saltelli,

Under revision for Ecological Indicators, April 2013

Ecological Footprint

- The implausible accuracy (Earth overshoot day = August 20!)

- Offsetting a flow with a stock (Kg of CO2 per year versus

square meters of land)

- The anti-trade bias (Stiglitz report p. 71)

- The total dependence upon energy related pressures (but

only sinks!)

- Paradoxical policy implications (e.g. in Agriculture)

… a rhetorical device?

Advocacy, analysis and quality. The Bermuda triangle of ... · Advocacy, analysis and quality. The...

Documents

BERMUDA TRIANGLE/DEVIL'S TRIANGLE

Bermuda Triangle

BERMUDA TRIANGLE

BERMUDA TRIANGLE