Embed Size (px)
Citation preview
The European Commission’s science and knowledge service
Joint Research Centre
The Innovation Output Indicator
Dániel Vértesy
COIN CoP 2017 – Composite Indicators & Scoreboards Community of Practice: 2nd Annual Meeting 09-10/11/2017, Ispra (IT)
What does the IOI measure?
Innovation output: “the extent to which innovative ideas can reach the market”
• A composite index of 4 components (5 variables): • Published by DG RTD since 2013 • Methodology developed with JRC/COIN support
Innovation Output Indicator (IOI)
PCT KIABI COMP
GOOD SERV
DYN
02468
101214
JP IL SE FI CH DE
DK NL
AT
US FR EU BE IS UK SI
NZ
NO IE IT LU ES EE HU
MT
CZ CY LT PT TR BG HR PL EL SK LV BR
RO
MK
"2016" (2014)"2011" (2009)"2015" (2013)
05
101520253035
IL NZ LU CH IE IS MT
SE
UK NL
US CY JP DK FI BE
NO DE AT FR EU IT SI
CZ EE ES EL HU
HR LV PT BR
BG PL SK LT RO TR MK
"2016" (2016)"2011" (2011)"2015" (2015)
PCT PCT patent applications per billion GDP (in PPS) KIABI Share of employment in knowledge-intensive activities in business industries (in %)
Components
GOOD Exports of medium- and high-technology products as a share of total product exports (in %) SERV Knowledge-intensive services exports as percentage of total services exports (in %)
01020304050607080
JP HU
MT
DE
SK
CZ
EUx
CY FR UK AT
EU MK SI
SE
RO IL US IT IE LU CH PL NL
BE
ES DK EE FI HR PT TR LT LV BG BR EL NZ
NO IS
"2016" (2016)
"2011" (2011)
"2015" (2015)
0
20
40
60
80
100
IE LU NO BR
EUx
NL
SE
DK
DE JP UK EU CH BE FR IL US CY FI IS LV IT HU EE RO EL AT PT CZ
BG PL SK SI
MT
ES TR NZ
MK LT HR
"2016" (2015) "2011" (2010)"2015" (2014)
Components
DYN Employment in fast-growing enterprises in the top 50% most innovative sectors as a percentage of total employment (in %)
Components
0
2
4
6
8
10
IE HU SK
MT
UK IL BG SE
BR PL NL
NZ
CZ LV NO EU DE
DK FR LU LT PT ES CH EE SI
AT FI HR
RO IT BE
CY
"2016" (2014)"2011" (2012)"2015" (2013)
Why measure innovation output? • Innovation key to Europe 2020 strategy:
make the EU a “dynamic, sustainable, knowledge-based (‘smarter’) economy”
• The political mandate from the European Council:
Complement the Lisbon agenda’s 3% R&D [input] target with output-oriented indicator(s)
• Aim to complement the Innovation Union Scoreboard, and its Summary Innovation Index (SII), which measures overall performance, by zooming in on a reduced set of dimensions
“ Each of them, alone, yields only a partial view of the innovation phenomenon, but together they provide if not for a perfect, at least for a clearer assessment
”
High-level panel on the measurement of innovation proposed a short list of indicators:
EC (2010)
Why multiple indicators?
• Single indicator: hourly labor productivity: indirect measure • Short list of indicators offering a clearer picture of the innovation phenomenon
• No ideal world:
• Existing data: not good enough • Proxies based on data produced by official statistics are available
• Policy demand for a concise picture
How: a composite index or a list of indicators?
• Disagreement: use a composite index or not?
First solution: a complex formula of 2 indices (sectoral innovation performance, sectoral innovation dynamics)
• Make use of innovation coefficients computed by OECD and Eurostat’s new employment in high-growth enterprises data
Against: HLP’s reservations: not easy to interpret
or... if there is a composite, HLP advised to: • keep the number of indicators limited to ensure transparency; • Ensure a relative independence of components; • Make possible to monitor the policies addressing the various components
Stakeholder reservation: Too complicated; insufficient coverage!
How: a composite index or a list of indicators?
• Political decision to go for a composite index
• Inter-service Task force (DG RTD, ECFIN, ESTAT, GROW, JRC, TRADE, etc.) established to develop an index from a list of 4 components
For: Summarize an admittedly multi-dimensional phenomenon Useful for delivering policy messages (factors hampering R&I system performance) Easy to interpret
JRC team’s role in developing the index
• Building indicator • Improve robustness
• Statistical validation: • checking conceptual and statistical soundness; • Perform global sensitivity analyses (MC simulations,
alternative weights, DEA, etc.) • Computing scaling factors:
• Decision to address statistical imbalances across dimensions using effective equal weights
Updates and refinements
• Communication challenge: align with EIS • Similar measures: avoid different message • Frequency of publication: align or not?
• Refinements: “if you want to measure change, better not change
the measurement” • Increase country coverage (given data availability) • Indicators refined
Results
The EU in international comparison
0
20
40
60
80
100
120
140
IL JP US EUx NZ BR
201620112015
JRC - COIN Statistical Support
Balancing components (Computing scaling coefficients) for annual updates
JRC - COIN Statistical Support
Balancing components (Computing scaling coefficients) for annual updates
EU28
AT
BE
BG CYCZ
DEDK
EEELES
FI
FR
HRHU
IEIT
LTLU
LVMT
NL
PLPTRO
SE
SI
SK
UK
CH
ISNO
IL
JP
NZ
US
MKTR BR
EU28
AT
BE
BG CYCZ
DEDK
EEEL
ES
FI
FR
HRHU
IEIT
LTLU
LVMT
NL
PLPT RO
SE
SI
SK
UK
CH
ISNO
IL
JP
NZ
US
MKTR BR
EU28
AT
BE
BGCY CZ
DEDK
EEEL
ES
FI
FR
HRHU
IEIT
LTLU
LVMT
NL
PLPTRO
SE
SI
SK
UK
CH
ISNO
IL
JP
NZ
US
MKTR BR
EU28 ATBE
BG
CY
CZDEDK
EEEL ES
FIFR
HRHU
IE
IT
LT
LU
LV
MT NL
PLPT
RO
SE
SI
SK
UK
CHIS
NO
IL
JP
NZ
US
MKTR
BR
EU28AT BE
BG
CY
CZDEDK
EEEL ES
FIFR
HR HU
IE
IT
LT
LU
LV
MT NL
PLPT
RO
SE
SI
SK
UK
CHIS
NO
IL
JP
NZ
US
MKTR
BR
EU28ATBE
BG
CY
CZDEDK
EE ELES
FIFR
HR HU
IE
IT
LT
LU
LV
MTNL
PLPT
RO
SE
SI
SK
UK
CHIS
NO
IL
JP
NZ
US
MKTR
BR
EU28
ATBE
BG
CY
CZ
DE
DK
EE
EL
ES
FI
FR
HR
HU
IE
IT
LT
LU
LV
MT
NL
PLPT
RO
SE
SI
SK
UK
CH
IS
NO
IL
JP
NZ
US
MK
TR
BR
EU28
ATBE
BG
CY
CZ
DE
DK
EE
EL
ES
FI
FR
HR
HU
IE
IT
LT
LU
LV
MT
NL
PLPT
RO
SE
SI
SK
UK
CH
IS
NO
IL
JP
NZ
US
MK
TR
BR
EU28
ATBE
BG
CY
CZ
DE
DK
EE
EL
ES
FI
FR
HR
HU
IE
IT
LT
LU
LV
MT
NL
PLPT
RO
SE
SI
SK
UK
CH
IS
NO
IL
JP
NZ
US
MK
TR
BR
EU28
ATBE
BG
CY
CZDEDK
EE
EL
ESFI
FR
HR
HU
IE
IT
LT LULV
MT
NLPL
PT
RO
SE
SI
SKUK
CH
ISNO
IL
JPNZ
US
MKTR
BR
EU28
ATBE
BG
CY
CZDEDK
EE
EL
ESFI
FR
HR
HU
IE
IT
LT LULV
MT
NLPL
PT
RO
SE
SI
SKUK
CH
ISNO
IL
JP NZ
US
MKTR
BR
EU28
ATBE
BG
CY
CZDEDK
EE
EL
ESFI
FR
HR
HU
IE
IT
LT LULV
MT
NLPL
PT
RO
SE
SI
SKUK
CH
ISNO
IL
JPNZ
US
MKTR
BR
PCT
KIABI
COMP_EUR
DYN(imp)
4
6
8
10
4 6 8 10
0
5
10
0 5 10
2
4
6
8
2 4 6 8
2
4
6
8
2 4 6 8
Statistical coherence
PCT KIABI COMPEU DYNimp N.Obs. 234 234 234 234 Min 3.5 2.1 2.5 1.7 Max 9.4 10.1 7.5 9.0 Mean 5.0 5.0 5.0 5.0 Std. Dev. 1.5 1.5 1.1 1.5 Skewness 1.2 0.9 -0.1 0.3 Kurtosis 0.5 1.4 -0.7 -0.4
Correlation PCT 1.000 KIABI 0.583 1.000 COMPEUR 0.524 0.414 1.000 DYNimp. 0.112 0.123 0.286 1.000
JRC - COIN Statistical Support
Developing alternatives Alternative ways of measuring components – i.e., DYN
EU28
SE
SK
BE
CZLTMTES
BG
UK
NL
LU
PTPLRO
FI
LV
SI
HU
IE
DE
CY
AT
EE
DK
EL
FR
HR
IT
CH
NO
IS
NZ
US
MXCL
KR
JP
CAAU
IL
RSMKTR CN
BRRUZA010
020
030
040
0P
CT
App
licat
ions
per
mill
ion
pop
0 5 10 15PCT Applications per bln GDP(PPS)
Year: '2016' (2014); R-squared = 0.961
JRC - COIN Statistical Support
Robustness analyses when revising components Sensitivity analyses to reveal impact of changing modelling assumptions
• Moderate, positive association:
• SII: relatively stronger influence of science and R&D-based innovation activities;
• IOI: relatively stronger influence of firm dynamics
Comparing scores: IOI vs SII
EU
AT
BE BG
CY
CZ
DE DK
EE EL ES
FI FR
HR
HU
IE
IT
LT
LU
LV
MT NL
PL
PT RO
SE
SI
SK
UK
CH
IS NO
IL
R² = 0.37 60
70
80
90
100
110
120
130
140
150
0.1 0.3 0.5 0.7 0.9
Inn
ovat
ion
Ou
tpu
t In
dic
ator
Summary Innovation Index
• Strong positive correlation:
• SII: high-growth, innovative firms component receives stronger relative weight
• Still some differences: consider Sweden, Germany
Comparing scores: IOI vs ‘Employment Impacts’ of EIS
EU
AT
BE BG
CY
CZ
DE DK
EE EL ES
FI FR
HR
HU
IE
IT
LT
LU
LV
MT NL
PL
PT RO
SE
SI
SK
UK
CH
IS NO
IL
R² = 0.82 60
70
80
90
100
110
120
130
140
150
0.1 0.3 0.5 0.7 0.9
Inn
ovat
ion
Ou
tpu
t In
dic
ator
Summary Innovation Index: Impacts Employment (4.1)
• Moderate, positive association • Gives a different picture • Caveat:
• Not all innovation is R&D based; • No linear relationship, • considerable time lag
• Yet, high spenders feel “not rewarded” (i.e. AT)
Output vs Input: IOI vs R&D expenditure
AT BE BG
CH
CY
CZ
DE DK
EE ES
EUx FI FR
UK
EL
HR
HU
IE
IL
IS
IT
JP
LT
LU
LV
MT NL
NO
PL
PT RO
SE
SI
SK
TR
US
R² = 0.42 60
70
80
90
100
110
120
130
140
150
0.0 1.0 2.0 3.0 4.0
Inn
ovat
ion
Ou
tpu
t In
dic
ator
"2
01
6"
Gross R&D expenditures per GDP (2015)
Results
Limitations, debates
• Structural differences across countries: • i.e., what do exports measure? • Indicators affected by geography, natural resource endowments
(<> smart specialization?)
• Economic impact of innovation output: where is it realized? • transnational companies vs. national borders
• DYN: conceptually as well as statistically different from the rest of the indicators • International comparison limited: different benchmarks
• Is there a validator: a single measure of innovation? • HLP: hourly labour productivity
Challenge for innovation indicators
Advocacy tool
• 15+ years of experience (think of SII; GII!)
• Innovation policy (more than technology!) now high on the agenda “innovate out of the crisis”
• Advocacy for data improvements
Analytical tool
• Contents of “the rest of the reports” • Data challenges:
• Established official statistics (i.e., R&D); • Improving official statistics:
(i.e., CIS surveys, 20 years); • Emerging data sources: experimental mainstream /admin, balance-sheet data, ‘big data’ (i.e., introduction and diffusion of new products, their consumption)/ => make it useful for policy monitoring
Welcome to email us at: [email protected] [email protected]
THANK YOU COIN in the EU Science Hub https://ec.europa.eu/jrc/en/coin COIN tools are available at: https://composite-indicators.jrc.ec.europa.eu/
The European Commission’s Competence Centre on Composite Indicators and Scoreboards
References
EC (2010) Elements for the setting-up of headline indicators for innovation in support of the Europe 2020 strategy: Report of the High Level Panel on the measurement of Innovation 2010.
