Embed Size (px)
Citation preview
Advocacy, analysis and quality. The Bermuda triangle of Statistics
59th ISI World Statistics Congress
25-30 August 2013 - Hong Kong Special
Administrative Region, China
Session STS023
Statistics and policy
Andrea Saltelli & Michaela Saisana
Joint Research Centre, European
Commission
About indicators
Content
• Statistical indicators for
policy between
modernity and post-
modernity;
• their use for analysis and
advocacy;
• how quality can save the
day;
• when things go wrong
The example of composite
indicators
What official statistics are to the consolidation of the modern
nation state (Hacking, 1990), composite indicators are to the
emergence of post-modernity.
Leibnitz ‘philosophical godfather of Prussian official statistics’ to the Prince Frederik of
Prussia 1700; 56 categories to ‘measure the power of a state’ (the first scoreboard); first
proposal for a statistical office …
Modernity and post modernity; from positivism to constructivism; how ‘Matters of fact’ are
established; Shapin and Shaffer, Latour, …; the emergence of a plurality of norms and views;
the Human Development Index (HDI, 1990) and the explosion of indices…
Composite indicators and post-Modernity
Statistics for policy: three models
A rational-positivist model for the use of indicators and policy
(good quality statistics underpin good policies)
Discursive-interpretive model (statistics contribute to a process
of framing of and focusing on an issue among the many
competing for public's attention)
Strategic model (statistics is used by parties competing for a
given constituency).
see Boulanger, P-M., Political uses of social indicators: overview and application to
sustainable development indicators. International Journal of Sustainable Development,
10 (1,2):14-32, 2007.
Contexts
Composite Indicators
Apples and Oranges
Composite indicators as an object populating a
multidimensional space whose main axes are
advocacy, analysis and quality.
Composite indicators sit between analysis and
advocacy, but quality discriminates the plausible
from the rhetorical.
Advocacy, analysis and quality
These three dimensions (advocacy, analysis and quality) are
not independent from one another.
Most developers adopt for transparency and simplicity linear
aggregation procedures to build composite indicators which
are fraught with considerable difficulties.
In this case quality may suffer at the expenses of advocacy.
Advocacy, analysis and quality
THE ROLE OF COMPOSITE INDICATORS FOR
MEASURING SOCIETAL PROGRESS
Ubiquitous; 6-fold increase in 7 y
Statistics' best known face (to general public & media)
Can provide analytic input to policy
Features of CI
October 2005 992
June 2006 1,440
May 2007
1,900
October 2008 3,030
September 2009
4,420
August 2010 5,240
May 2011 5,900
October 2012
7,650
Searching
“composite
indicators” on
Scholar Google:
Ubiquitous; 6-fold increase in 7 y
August
2013: 9,130
0
50
100
150
200
250
300
350
1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012
Nu
mb
er o
f p
ap
ers d
iscu
ssin
g c
om
po
sit
e i
nd
icato
rs
Year
Search www.scopus.com query: TITLE-ABS-KEY("composite indicator*") OR TITLE-ABS-KEY("composite index") OR TITLE-ABS-KEY("composite indices")
Papers
Year
An example, about Côte d’Ivoire ,
Economist October July 2013
One of the indicators in the ruling justly category is
control of corruption, an area in which Côte d’Ivoire fares
particularly poorly. The World Bank’s most recent
corruption rankings, from 2011, put it 38th out of 49
African countries. Transparency International ranked it
130th out of 176 countries last year in its Corruption
Perceptions Index, ahead of Nigeria and Guinea but well
behind neighbouring Liberia, Burkina Faso and Ghana.
Statistics' best known face (to general public & media)
Fortune of CI
Caveats in use
More ICT + More statistical literacy + More appreciation of complexity
“the role of statistical indicators has increased over the last two
decades”
What the Stiglitz report says about CI’s
Report by the Commission on the Measurement of Economic Performance and Social Progress, 2009,
Joseph E. STIGLITZ, Chair, Columbia, University, Amartya SEN, Chair Adviser, Harvard University,
Jean-Paul FITOUSSI, Coordinator of the Commission, IEP,www.stiglitz-sen-fitoussi.fr
The Stiglitz report, on page 65, mentions: […] a general
criticism that is frequently addressed at composite
indicators, i.e. the arbitrary character of the procedures
used to weight their various components.
Adding: […] The problem is not that these weighting
procedures are hidden, non-transparent or non-replicable
– they are often very explicitly presented by the authors of
the indices, and this is one of the strengths of this
literature. The problem is rather that their normative
implications are seldom made explicit or justified.
Caveats in use and construction
But:
It is possible to disentangle evidence based policy from policy based evidence?
see Benoît GODIN on eugenics and the birth of R&D stats: The Culture of
Numbers: From Science to Innovation, INRS, Montreal, Canada,
Communication presented to the Government-University-Industry Research
Roundtable (GUIRR) US National Academy of Sciences, Washington, May 21,
2010.
… but many other data based stories as well: Tobacco & health, capital
punishment & crime rate …
•Oreskes, N., Conway E. M., 2010, Merchants of Doubt, Bloomsbury Press
•Leamer, E. E., Tantalus on the Road to Asymptopia, 2010, Journal of
Economic Perspectives, 24, (2), 31–46.
Caveats
Quality can become the new organizing principle which “enables
us to manage the irreducible uncertainties and ethical
complexities” (*).
We call this approach ‘sensitivity auditing’ of a composite
indicator (+).
(*) Funtowicz, S.O. and Ravetz, J.R. (1994). The worth of a songbird: Ecological economics as a
post-normal science. Ecological Economics 10(3), 197-207.
(+) Saltelli A, Guimarães Pereira A, van der Sluijs JP, Funtowicz S., 2013, What do I make of
your Latinorum? Sensitivity auditing of mathematical modeling, to appear, Foresight and
Innovation Policy, arXiv:1211.2668 [physics.soc-ph].
Advocacy, analysis and quality
Analysis
CI for evidence based policy
Can provide analytic input to policy
Suggestion: CI as a useful tool within the open method of coordination
Ratings and Rankings 20
The Alcohol Policy Index (New York Medical College)
Concept: (WHO report)
Results
Policy message Sensitivity analysis
Published in
PLoSMedicine
Advocacy
Ratings and Rankings 23
6th dimension of the Rule of
Law Index
(World Justice Project)
• 83 survey questions
• 97 countries (20 EU)
RoL; blue bars = EU countries
Ratings and Rankings 24
7th dimension of the Rule of
Law Index
(World Justice Project)
• 56 survey questions
• 97 countries (20 EU)
RoL; blue bars = EU countries
25 18 December 2013
Corruption Perceptions
Index (Transparency
International)
• 13 sources
• 176 countries (27
EU)
CPI; blue bars = EU countries
26 18 December 2013
0
20
40
60
80
100
120
140
160
180
200
Sin
gap
ore
Unit
ed S
tate
s
Den
mark
Norw
ay
Unit
ed K
ingdo
m
Aust
rali
a
Fin
land
Mala
ysi
a
Sw
eden
Icela
nd
Irela
nd
Canada
Ger
many
Est
onia
Saudi
Ara
bia
Japan
Lat
via
Lith
uania
Sw
itze
rlan
d
Aust
ria
Port
ugal
Net
her
lands
Belg
ium
Fra
nce
Slo
ven
ia
Cyp
rus
Chil
e
Peru
Spain
Colo
mbia
Slo
vak
Repu
bli
c
Mex
ico
St.
Lucia
Hung
ary
Pola
nd
Luxem
bourg
Sam
oa
Ghana
Czech
Repu
bli
c
Bulg
aria
Aze
rbaij
an
Turk
ey
Rom
ania
Ital
y
Seychell
es
Baham
as,
The
Gre
ece
Cro
ati
a
Jam
aic
a
Chin
a
Malt
a
Para
gu
ay
Centr
al A
fric
an R
epu
bli
c
Ease of Doing Business Rank Paying Taxes
Ease of Doing Business Rank Starting a Business
Dealing with Construction Permits Getting Electricity
Registering Property
Getting Credit
Protecting Inv estors
Paying Taxes Trading Across Borders Enf orcing Contracts
Resolv ing Insolv ency
World Bank Index ‘Ease of Doing Business’ + one of its sub-indices:
‘Paying taxes’; blue bars = EU countries
Paying taxes
Quality
International works on the quality of composite indicators have been
ongoing since 2003 (OECD and JRC).
Quality
2008
Quality
Construction E.g. University System at the regional
scale;
Validation E.g. Corruption Perceptions Index
2012, Rule of Law 2012;
Methodology Sensitivity analysis & auditing, multi-
criteria methods, statistics and policy;
Training E.g. WEF, Genève April 2013;
Istanbul July 2013.
Testing (composite) indicators: two approaches
Michaela Saisana, Andrea. Saltelli,
Stefano Tarantola (2005),
Uncertainty and sensitivity analysis techniques as
tools for the quality assessment of composite
indicators,
J. R. Statist. Soc. A 168(2), 307–323.
Paolo Paruolo, Michaela Saisana,
Andrea Saltelli, (2013),
Ratings and rankings: Voodoo or Science?,
J. R. Statist. Soc. A, 176 (3), 609–634.
Sensitivity analysis
First: The invasive approach
Michaela Saisana, Béatrice d’Hombres,
Andrea Saltelli, Rickety numbers: Volatility of
university rankings and policy implications
Research Policy (2011), 40, 165-177
Sensitivity analysis
Space of alternatives
Including/
excluding variables
Normalisation
Missing data Weights
Aggregation
Country 1
10
20
30
40
50
60
Country 2 Country 3
Sensitivity analysis
Second: The non-invasive approach
Comparing the weights as assigned by developers with
‘effective weights’ derived from sensitivity analysis.
Non invasive Sensitivity analysis
University
Rankings
Comparing the internal coherence of ARWU versus THES (2008) by testing the weights declared by developers with ‘effective’ importance measures. ARWU=Academic ranking of world universities;
THES=Times Higher Education Supplement
declared weight importance
THES X1_Academic opinion: 6354 academics 40% X2_Recruiters’ opinion: 2339 recruiters 10% X3_Full-time equivalent faculty/student ratio 20% X4_Total citation/full time equivalent faculty 20% X5_Percentage of full-time international staff 5% X6_Percentage of full-time international students 5%
Issues with THES: a) ‘Opinion’ variables’ weight overall: >60% instead of 50 b) Faculty/student ratio: 10% instead of 20%
HDI
2009
declared weight importance
Life expectancy, 33%
Adult literacy, 22%
Enrollment education, 11%
GDP per capita, 33%
Non invasive Sensitivity analysis
HDI
2010
Life expectancy, 33%
Education, 33%
GNI per capita, 33%
Non invasive Sensitivity analysis
declared weight importance
HDI 2010 more coherent than HDI 2009
Non invasive Sensitivity analysis
declared weight importance
39
Two applications of non invasive sensitivity analysis
…the Bermuda
angle From: Ecological Footprint: Unneeded Footwork
Mario Giampietro and Andrea Saltelli,
Under revision for Ecological Indicators, April 2013
Ecological Footprint
Ecological Footprint
- The implausible accuracy (Earth overshoot day = August 20!)
- Offsetting a flow with a stock (Kg of CO2 per year versus
square meters of land)
- The anti-trade bias (Stiglitz report p. 71)
- The total dependence upon energy related pressures (but
only sinks!)
- Paradoxical policy implications (e.g. in Agriculture)
… a rhetorical device?
END
