METHODOLOGY REPORT - European Commission · KETs Observatory Phase II: Methodology Report 1 Introduction 1.2 WP1: Assessment of the reliability and validity of the KETs Observatory

METHODOLOGY REPORT

The views expressed in this report, as well as the information included in it, do not necessarily reflect the opinion or position of the European Commission.

METHODOLOGY REPORT

KETs Observatory Phase II Contract nr EASME/COSME/2015/026

Authors: Mark Lengton (PwC), Kristina Dervojeda (PwC), Anton Koonstra (PwC), Vadym Volosovych (Erasmus University Rotterdam)

Coordination: EUROPEAN COMMISSION, Executive Agency for Small and Medium-sized Enterprises (EASME), Department A – COSME, H2020 SME and EMFF, Unit A1 – COSME; DG for Internal Market, Industry, Entrepreneurship and SMEs, Unit F.3 - KETs, Digital Manufacturing and Interoperability

European Union, June 2017.

Executive summary ...................................................................................................... 5

1. Introduction ............................................................................................................... 6

1.1 KETs Observatory Phase II ................................................................................. 6

1.2 WP1: Assessment of the reliability and validity of the KETs Observatory

methodology framework ............................................................................................ 7

1.3 Objectives and structure of the current report ...................................................... 7

1.4 Status of the proposed methodological changes ................................................. 8

2. Observatory map ...................................................................................................... 9

2.1 Rationale of the KETs Observatory and its indicators .......................................... 9

2.1.1 Indicators to measure the deployment of KETs ................................................ 9

2.1.2 Operationalised sub-indicators ....................................................................... 10

2.2 Schematic overview of all included indicators .................................................... 11

2.2.1. Addressing differences in data due to methodological changes ..................... 13

3. Conceptual level ..................................................................................................... 14

3.1 The second phase of the KETs Observatory aims to improve upon a number of

concepts at a conceptual level ................................................................................ 14

3.2 Reducing expert bias by relying on Input-Output tables at the technology

diffusion stage ......................................................................................................... 14

3.2.1. Proposed solutions ........................................................................................ 14

3.2.2. Illustrative example of the approach .............................................................. 16

3.2.3. Benefits, drawbacks and challenges of the approach .................................... 18

3.2.4. Final conclusions ........................................................................................... 19

3.3. Expanding the KETs Observatory to increase international coverage ............... 19



3.4. Exploring an alternative measure to increase coverage of the KETs Observatory

beyond manufacturing industries ............................................................................. 20



4. Technology generation & exploitation approach ...................................................... 22

4.1 Methodological considerations for the technology indicators ............................. 22

4.1.1. Identified challenges of technology indicators ................................................ 22

4.1.2. Proposed solution .......................................................................................... 23


4.2. Methodological considerations for the turnover indicators ................................ 23

4.2.1. Identified challenges of turnover indicators .................................................... 24

4.2.2. Proposed solution .......................................................................................... 24


4.3 Methodological considerations for the production indicators .............................. 26

4.3.1. Identified challenges of production indicators ................................................ 26

4.3.2. Proposed solution to address the challenges at hand .................................... 26


4.4. Methodological considerations for the trade indicators ..................................... 27

4.4.1. Identified challenges of trade indicators ......................................................... 27



4.5. Expanding the KETs Observatory to estimate KETs-ICT .................................. 28

4.5.1. Proposed approach ....................................................................................... 28


4.6. Methodological considerations for the composite indicators ............................. 29

4.6.1. Identified challenges of composite indicators ................................................. 30

4.6.2 Proposed solution to address the challenges at hand ..................................... 30

4.6.3 Final conclusions ............................................................................................ 30

5. Technology diffusion approach ............................................................................... 31

5.1. Methodological considerations for production and demand indicators .............. 31

5.1.1. Identified challenges of production and demand indicators ............................ 31



5.2. Methodological considerations for employment indicators ................................ 31

5.2.1. Identified challenges of employment indicators .............................................. 32



KETs Observatory Phase II: Methodology Report Executive summary

Executive summary

This report presents an overview of the key methodological considerations and proposed changes for the

second phase of the KETs Observatory, which takes place in 2016-2018. Based on the methodology

outlined in this report, the first set of updated indicators is expected to be available after the summer

2017, with the remaining indicators scheduled to be calculated by March 2018.

This report specifically contains the results of the assessment of the KETs Observatory methodology of

phase one (2013-2015), as well as suggestions for improvement and an analysis of their feasibility. The

report also addresses methodological considerations related to the expansion of the KETs Observatory,

and specifically the inclusion of an international perspective for the production and demand indicators,

and the inclusion of “ICT-KETs” components and intermediary systems.

Overall, it has been concluded that the concept underlying the KETs Observatory is valid and of added

value. A number of advancements to the methodology have been proposed for the second phase, to

address the key challenges. The considered advancements are as follows:

The technology indicators need to be expanded beyond patent data. While the patent indicators

were considered appropriately calculated, there was a common agreement among experts that there

is a need to look beyond patents, as they do not capture the full scale of technology generation.

Consequently, the feasibility of incorporating data on scientific publications produced by parallel

studies needs to be explored. The intention is to use raw data to calculate time series on the number

of scientific publications in KETs, operationalised into sub-indicators.

Expert bias in the methodology can be reduced by taking a more data-driven approach. Our

analysis identified a strong need to reduce dependency on and bias from experts in identifying and

weighting the relevant statistical codes. To address this challenge, our proposal is to use Input-Output

tables, which was supported by the experts. Using detailed 6-digit Input-Output tables, unavailable at

the time the methodology of the first phase was designed, a more data-driven approach could be

employed to capture the deployment of KETs in the wider economy.

Turnover indicators can be calculated differently to address the assumed direct link between

patenting activity and commercialisation and production indicators. For the second phase, we

aim to follow an alternative approach to address the issues related to the weighting scheme.

Specifically, we suggest calculating turnover using publically available business statistics at the

aggregate sector level and linking them to the KETs-based product production data (8-digit

PRODCOM). Experts unanimously welcomed the approach, acknowledging that there is a strong

need to address the assumed causal link between patenting activity and commercial activity.

In order to introduce an international perspective for production and demand indicators, the

second phase will rely on harmonised international databases. Our research concluded that

harmonised data sources contain the most complete and comparable data that is available to date.

We therefore propose to rely on the UN COMTRADE database for obtaining trade data at the

international level (available at 6-digit level) and the OECD databases for obtaining the industry

statistics (available at 4-digit level).

The KETs Observatory will be expanded to measure the technology generation and

exploitation of ICT-enabled KETs industries. Instead of following the expert-based approach to

identify the reliance on (i.e. “use of”) ICT in KETs/AMT, experts concluded that an Input-Output

approach could also be applied here. Following their suggestions, we aim to further develop and

employ an Input-Output approach for identifying and selecting relevant statistical codes. To take into

account expert views, we also aim to validate the final list of identified industries with experts from the

field.

KETs Observatory Phase II: Methodology Report 1 Introduction

1. Introduction

This report represents an overview of the key methodological considerations and proposed changes for the

“KETs Observatory: Phase II” (contract nr. EASME/COSME/2015/026), prepared by PwC EU Services

(hereafter “PwC”) for the Executive Agency for Small and Medium-sized Enterprises (hereafter “EASME”) and

the Directorate General for Internal Market, Industry, Entrepreneurship and SMEs (hereafter “DG GROW”) of

the European Commission (hereafter “the Commission”).

The report aims to provide an overview of the overall methodology for the second phase of the KETs

Observatory, particularly addressing the proposed advancements of the methodology developed for phase

one1. In the current chapter, we briefly address the context of this report, its objectives and structure.

1.1 KETs Observatory Phase II

The KETs Observatory is an online monitoring platform that aims to provide the EU, national and regional

policy makers and business stakeholders with quantitative and qualitative information on the deployment of Key

Enabling Technologies (KETs2) both within the EU 28 and in comparison with other world regions (e.g. East

Asia, North America). The first phase of the KETs Observatory (2013-2015) covered the following years and

indicators for KETs components: patents (2000-2012), production & demand (2003-2013), and business (2005-

2013) and trade indicators (2002-2013).

The objective of the second phase of the KETs Observatory is to update, expand and develop the database

and activities of the KETs Observatory for the time period 2016-2018. It has to address the identified

challenges and lessons learned from the previous activities, and propose and implement new solutions to those

challenges, as well as expand and advance these previous activities.

The overall objective of the second phase is to continue providing business stakeholders, as well as

European and national and regional policy makers with reliable, regularly updated and comprehensive data and

analysis on the deployment of KETs. Specifically, the KETs Observatory will be promoted as a practical

tool for the elaboration and implementation of Smart Specialisation Strategies in the EU regions,

providing valuable insights into the performance of the EU regions in terms of industrial deployment of KETs

and thus helping regional stakeholders with their entrepreneurial discovery process.

The abovementioned overall objective can be operationalised into four specific objectives:

Specific Objective 1 (corresponding to Work Package 1): Assessment of the reliability and validity of

the KETs Observatory methodology framework;

Specific Objective 2 (corresponding to Work Package 2): Updating and expanding the KETs

Observatory database: collection of data and calculation of indicators;

Specific Objective 3 (corresponding to Work Package 3): Production of Country Profiles and

Analytical reports on promising KETs-based products; and

Specific Objective 4 (corresponding to Work Package 4): Dissemination and communication of the

KETs Observatory activities.

1 See: IDEA Consult et al. (2015) “Key Enabling Technologies (KETs) Observatory: Methodology report”. 2 Namely Nanotechnology, Micro-/Nanoelectronics, Photonics, Industrial Biotechnology, Advanced Materials and Advanced Manufacturing Technologies


1.2 WP1: Assessment of the reliability and validity of the KETs Observatory methodology framework

The objective of Work Package 1 (hereafter “WP1”) was to review, assess and improve, wherever

necessary, the methodology developed during the first phase of the KETs Observatory. The two main

tasks of WP1 implied improving the robustness of the methodology of the “technology generation and

exploitation” approach and its indicators (Task 1.1); and improving the robustness of the methodology of the

“technology diffusion” approach and its indicators (Task 1.2).

The work that has been done during the feasibility study and the first phase of the KETs Observatory already

led to a solid methodological base. We fully acknowledge that the existing methodological base should be

cherished and used as a platform to continue building further. There are, however, several methodological

points that require additional attention, such as, for example, composite indicator, the turnover data being

presented at headquarters-level of companies, SMEs being less represented in some data while they form the

majority of companies for many KETs (e.g. in nanotechnology, photonics), assumptions on the existence of

direct correlation between the share of patents in KETs and KETs-related company turnover etc. Furthermore,

the existing methodological framework needed to be expanded to include an international perspective

for production and demand indicators, as well as ICT-KETs technologies.

Multiple ways exist of tackling the abovementioned challenges and achieving the objectives set. There was a

clear need to reach a consensus among stakeholders on how to proceed, what strengths to build on and

what assumptions should be tolerated or challenged. For that, stakeholder engagement and validation were

embedded in this stage. There was a need to make sure the refined framework is ‘accepted’ by the

stakeholders as only then the outputs of the KETs Observatory are likely to be widely used.

We therefore organised the process of assessment of robustness of the KETs Observatory methodology

framework in two stages: (1) internal assessment by the research team, complemented by inputs from key

informants (external experts and stakeholders); and (2) external validation by stakeholders during a dedicated

validation workshop.

1.3 Objectives and structure of the current report

This report contains the results of the assessment of the KETs Observatory methodology of phase one, as well

as suggestions for improvement and the analysis of their feasibility. The report also addresses the

methodological considerations related to the expansion of the KETs Observatory, and specifically the inclusion

of an international perspective for the production and demand indicators, and the inclusion of “ICT-KETs”

technologies.

The remainder of this report is structured as follows. Chapter 2 contains a comprehensive KETs Observatory

map, representing an overview of all indicators to be included in phase two. For each of the indicators, we also

provide specific data sources, and identify whether each indicator has in any way been modified when

compared to phase one, or whether it is a new indicator. The objective of this overview is to provide a full

outline of the quantitative content of the second phase of the KETs Observatory, as well as, in a summative

way, to present the differences with regard to the first phase.

Chapter 3 presents our proposed methodological changes at the conceptual level for the KETs Observatory as

a whole. This chapter introduces a number of changes for the overall conceptual approach of the KETs

Observatory, the specific challenges that are addressed, and the proposed solutions and conclusions.

Chapter 4 contains the detailed results of the methodology assessment exercise for the technology generation

& exploitation approach. For each indicator/sub-indicator, we provide a short description, identified challenges,

as well as proposed solutions and conclusions regarding the feasibility of those solutions.


Chapter 5 provides the detailed results of the methodology assessment exercise for the technology diffusion

approach. It is structured according to the same principle as Chapter 4.

1.4 Status of the proposed methodological changes

This document reflects the current state of the validated methodological changes, based on an in-depth

assessment and expert validation. Given a high complexity and a broad scope of the KETs Observatory

methodology, it is, however, possible that in the course of implementation, new factors and challenges will be

discovered. That may require readdressing some of the methodological points and approaching the engaged

experts again for their consultation and feedback. Consequently, the methodology revision process should be

viewed as a continuous iterative task that requires sufficient flexibility and agile communication with

stakeholders. If at any point in the course of the project, the proposed methodological changes proof to

be unfeasible, we may choose to revert to the methodology of phase one or consider implementing

alternative solutions.

KETs Observatory Phase II: Methodology Report 2 Observatory map

2. Observatory map

This chapter presents a comprehensive overview of the indicators that will be included in the second

phase of the KETs Observatory. Section 2.1 presents the rationale of the KETs Observatory and the

indicators, followed by Section 2.2 providing a schematic overview of all included indicators and sub-

indicators, as well as a concise overview of changes.

2.1 Rationale of the KETs Observatory and its indicators

The KETs Observatory rests on two complementary approaches, the “technology generation and

exploitation” approach and the “technology diffusion” approach. While the technology generation

and exploitation approach looks at the ability of countries to generate and commercialise new

knowledge, the technology diffusion approach investigates the impact of KETs on the wider

economy. The combination of both approaches aims to provide insight into the ability to transfer new

knowledge and technology into value added and growth.

Specifically, the KETs Observatory indicator framework aims to address the so-called “valley of

death” when commercialising new technology. While technology indicators report on the

development of new technology (patenting), production and trade indicators identify the extent of

successful commercialisation of this new technology and hence indicate whether the “valley of

death” could be crossed. The technology diffusion approach, in turn, goes beyond this perspective

and looks at the potential of successfully commercialised new technology (KETs-based products) to

trigger innovation and competitiveness across many industries. Figure 2-1 depicts the current

conceptual framework used by the KETs Observatory.

FIGURE 2-1: Conceptual framework of the first KETs Observatory3

2.1.1 Indicators to measure the deployment of KETs

To measure both technology generation and exploitation, and technology diffusion, different sets of

indicators were developed. To measure technology generation and exploitation of KETs, the

following types of indicators are included in the KETs Observatory:

Technology indicators that measure the ability to produce new technological knowledge

relevant to industrial application;

Production indicators that measure the relevance and dynamics of the production of KETs

components;

3 IDEA Consult et al., (2015). Key Enabling Technologies (KETs) Observatory: Methodology report.


Trade indicators (import/export) that measure the ability to produce and commercialise

internationally competitive products based on new technological knowledge;

Turnover indicators that measure the ability of industries/businesses to compete in the

market for KETs based components and to transfer new technologies and innovations to

industrial applications;

Employment indicators that reveal a country’s performance with regard to KETs-related

employment (not displayed in the KETs Observatory website);

Composite indicators which are calculated on the basis of single indicators. They describe

and analyse the performance of a country in a given KET.

To measure technology diffusion of KETs, the following types of indicators are included in the KETs

Observatory:

Production and demand indicators that show to what extent the EU countries use the

potential of KETs to improve its competitiveness by manufacturing KETs-based products

and applying them in the production of manufacturing goods, both in the sectors that

produce KETs as well as, and more importantly, in other industry sectors;

Employment indicators that reveal a country’s performance with regard to KETs-related

employment in the wider economy. Employment data is calculated based on the production

data multiplied with country and KETs specific estimates for employment per Euro of gross

output (the inverse of productivity).

2.1.2 Operationalised sub-indicators

For each indicator used within the KETs Observatory, specific sub-indicators are calculated to reflect

different perspectives of the concepts. While the overall structure of the indicators implies that each

group of indicators has different sub-indicators, the KETs Observatory has operationalised each

indicator – with the exception of trade and demand indicators - in the form of the following sub-

indicators:

Country significance (i.e. how important is a certain KET in a country’s total patent activity,

exports, production and turnover);

Shares of patents, production, in total export and in turnover (i.e. how important a country is

for European or global patent activity, exports, production and turnover for the relevant KET);

Medium-term dynamics (i.e. how KETs activities have changed over the past decade);

KET specialisation, indicating the relative significance of a particular KET;

The absolute trade balance per KET (for trade indicators only);

Export shares (for the demand indicators of the technology diffusion approach only);

Import shares (for the demand indicators of the technology diffusion approach only).

For the trade indicator, a fifth sub-indicator to assess countries’ trade performance was added: the

absolute trade balance per KET. For the demand indicators of the technology diffusion approach,

two sub-indicators were added: export and import shares.


2.2 Schematic overview of all included indicators

A structured overview of all indicators, including the relevant sources that will be employed as well as

whether the indicators need to be adjusted in comparison to the previous phase, or whether they are

new to the KETs Observatory, is presented in Table 2-1 below.

TABLE 2-1: Schematic overview of the included indicators

Type of indicators

Indicators Sub-indicators Sources Comparison to KETO-1

Technology generation and exploitation approach

Technology indicators

Patents Country significance;

KET specialisation;

Share in patents;

Medium-term dynamics.

PATSTAT (country level)

REGPAT (Patents; regional level)

Unchanged

Scientific publications Country significance;

KET specialisation;

Share in publications;


DG RTD study on KETs-related scientific publications (by Leiden University)

New

Production indicators

Production Country significance;

KET specialisation;

Share of production;

Medium-term dynamics

Eurostat PRODCOM (Europe)

OECD (international)

Adjusted:

International expansion of database in the form of using harmonised OECD data (new).

For Europe, the detailed PRODCOM data will be used (unchanged).

Trade indicators

Trade Country significance;

KET specialisation;


Medium-term dynamics;

Trade balance.

UN COMTRADE (6-digit)

Unchanged

Turnover indicators

Turnover Country significance;

KET specialisation;

Share in turnover;


PRODCOM (8-digit)

OECD STAN

Eurostat SBS and PRODCOM

Adjusted:

Method to calculate (new; see section 4.2).

Consequently changed source data from Bureau van Dijk’s Orbis database to OECD’s STAN database for Structural Analysis


Type of

indicators Indicators Sub-indicators Sources Comparison to KETO-1

(4-digit), Eurostat’s Structural Business Statistics (SBS) from (4-digit) and PRODCOM (8-digit).

Technology diffusion approach

Production and demand indicators

Production Country significance;

KET specialisation;


Medium-term dynamics


OECD STAN (international)

Expert assigned KETs-intensity weights for products

U.S. input-output accounts (US Bureau of Economic Analysis)

Adjusted:



The KETs diffusion is computed by applying the detailed 6-digit US input-output table to 4-digit industry aggregates (new).

Demand Country significance;

KET specialisation;

Share in demand;

Medium-term dynamics;

Export quotient;

Import quotient.


UN COMTRADE (6-digit)


Adjusted:



Employment indicators

Employment Country significance;

KET specialisation;

Share in demand;




Adjusted:



To calculate employment at international level, we will apply the KET-specific calculated inverse of productivity for Europe (based on confidential PRODCOM data) to the relevant sectors


Type of

indicators Indicators Sub-indicators Sources Comparison to KETO-1

in the international databases (new).

2.2.1. Addressing differences in data due to methodological changes

Some of the changes described in Table 2-1 imply adjustments in the underlying data and/or

calculation method. This implies that in some cases, data from the current phase cannot be directly

compared to the previous phase. A “trend break” can be incorporated in the KETs Observatory,

signalling a change in the methodology. We note that this is the common approach for official

statistical offices when they incorporate changes in the methodology. Alternatively, especially to

enable comparisons in country profiles, there may be a need to recalculate the data for the previous

period, following the new approach.

KETs Observatory Phase II: Methodology Report 3 Conceptual level

3. Conceptual level

This chapter presents the results of the in-depth analysis of the methodology at the conceptual level. Section

3.1 presents the overall changes we propose to apply to the methodology. Sections 3.2 – 3.4 provide further

details on the suggested advancements of the methodology at the conceptual level.

3.1 The second phase of the KETs Observatory aims to improve upon a number of concepts at a conceptual level

The work that has already been conducted in previous phase of the KETs Observatory should be appreciated

for creating a solid methodological foundation to build upon. Overall, the first phase has resulted in a valuable

tool that can be used to extract data for policy and decision making purposes. Still, some challenges were

identified that will be addressed through the following proposed advancements:

Reducing expert bias by relying on Input-Output tables at the technology diffusion stage;

Expanding the KETs Observatory to increase international coverage;

Exploring an alternative increasing coverage of the KETs Observatory beyond manufacturing

industries.

The following sections describe in greater detail the underlying challenge the improvements aim to address, the

various solutions that were proposed, and our final conclusions on the approach that will be adopted within

phase II of the KETs Observatory.

3.2 Reducing expert bias by relying on Input-Output tables at the technology diffusion stage

The methodology of the first phase heavily relies on expert opinion for assessing the relative contribution of

KETs to the competitiveness of an industry. This introduces subjectivity to the statistics, as it relies on expert

opinion to identify which (and to what extent) PRODCOM codes are included and affected. Furthermore, the

strong reliance on PRODCOM prohibits international expansion of the Observatory database due to

unavailability of the data outside of Europe.

To address these challenges, we propose to adopt a data-driven approach that partially relies on Input-

Output tables. Although such an approach was considered at an early stage of the KETs Observatory, at the

time it was not considered feasible due to the lack of granularity of the data. The Input-Output tables were only

available at 2-digit classification level, which was rightfully considered not granular enough to identify the KET-

relevant content. Meanwhile, more detailed tables have become available. More specifically, the publication in

November 2014 of a 6-digit classification Input-Output table for the United States provides opportunities that

can now be exploited by the KETs Observatory.

3.2.1. Proposed solutions

The current KETs Observatory methodology distinguishes between the two approaches to represent the

technological value chain: the technology generation and exploitation approach and the technology diffusion

approach. For the first, experts are only asked to consider whether a product contains KET-technology and/or

is produced using KET-technology. Although this still relies on expert opinion, we argue that a panel of

experts can reliably identify whether the products are linked to KETs, especially when considering KET

components. Therefore, we suggest to keep in place the experts selection and validation of statistical

product codes for the technology generation and exploitation approach.

Expert bias was introduced to a larger extent in the technology diffusion approach, where experts are also

asked to rate on a scale of 0-3 to what extent KETs impact the competitiveness of production sectors. Instead


of using a 4-point scale, we propose to limit expert judgement to a simple 2-point scale to represent the

diffusion of KETs. Specifically, the two expert ratings 2 and 3 that represent larger impact of KETs would be

replaced by a single rating “Yes, the competitiveness of the sector is substantially impacted by KETs”, whereas

the two lower expert ratings 0 and 1 would be recoded to the rating “No, the competitiveness of the sector is not

substantially impacted by KETs”.

While we acknowledge that some variance is lost in consolidating the expert ratings, we note that they

implied a level of accuracy that cannot be convincingly verified or substantiated. Since it is difficult for

experts to judge whether the impact of KETs on the competitiveness of another industry is, e.g., “low,”

“moderate,” or “substantial” the minor differences in judgement can change the weight of that product code as

significantly as 67% (e.g. 2: moderate impact, weighted 0.33 vs. 3: high impact, weighted 1). The sensitivity of

the current approach to expert opinion is, therefore, of rather extreme nature, as judgement error of small scale

(e.g. moderate vs high impact; low vs. moderate impact) can result in large differences in the implied impact of

KETs onto competitiveness of certain industries and countries. In contrast, an assessment of whether there is a

clear impact (“yes” or “no”) is often easier to answer, implying that we remove some of the sensitivity

attributable to the expert bias. Our concern was shared by experts from the validation workshop, who

unanimously agreed that the approach geared toward accuracy at a more consolidated level was preferred

over the subjective variance.

After consolidating the expert opinions to the level of the narrowly defined industrial sectors in order to

measure their “KETs intensity”, we aim to use Input-Output (I-O) tables to quantify the extent to which

KETs are diffused in these sectors. The I-O tables show how industries interact; specifically, they show how

industries provide input to, and use output from each other to produce Gross Domestic Product. These

accounts provide detailed information on the flows of the goods and services that comprise the production

processes of industries. Specifically, we would take advantage of a detailed “use” table from the United States

for the benchmark year 20074 compiled by the Bureau of Economic Analysis, U.S. Department of Commerce.

The table shows the inputs to industry production and the commodities that are consumed by final users. This

will allow us to trace the use of goods in the value chain and distinguish between intermediate and final use.

The latter was emphasised but not properly addressed in the previous phase of the Observatory because the

reliance on the product-level production data in PRODCOM does not allow separating the part of the total

production for a given product that is used as the input to further production processes as it is conceptually

implied by the diffusion stage.

Since the most detailed I-O table available is defined at an aggregated sectoral level that can be matched to

the 4-digit NACE industrial classification, we will first compute the KET-intensity of every 4-digit sector using the

average of the simplified expert weights. The weights are defined at 8-digit product level and the averaging may

be performed by simple averaging of weights across all products in a given 4-digit sector or by weighted

average, where the weights are defined in terms of the value of the output of the products that fall into the

category with substantial impact of KETs. Whereas the former can be computed without relying on confidential

data, the latter – although more accurate – requires the use of the confidential Eurostat data on the production

of certain subset of products in a given country-year. Even though this needs to be further discussed with

Eurostat at this stage, we note that this does not constitute the use of different data in comparison to the

previous phase. The project team will verify the feasibility of it with Eurostat as soon as possible. Moreover, the

data we would request from Eurostat would be already aggregated to a 4-digit sector by them and, hence, the

individual 8-digit production would remain confidential. The alternative approach that would rely on simple

counts of KTEs-relevant product codes identified by the Observatory experts relative to the total number of

products in a given 4-digit industry – although not preferred – can be explored regardless of having access to

the confidential data.

After calculating the KET-intensity of the industries, the Input-Output tables will be used to determine the share

of inputs produced by each of the KET-intensive industries that are supplied to or used by the rest of the

4 Statistics prepared at the 389-industry level of aggregation are available only for estimate year 2007. The table is available at https://www.bea.gov/industry/xls/io-annual/IOUse_After_Redefinitions_PRO_2007_Detail.xlsx


industries in the economy in the final goods production. This, in turn, will be used to obtain the KET-relevant

industrial outcome (e.g., production) interpreted by us as the extent of the “diffusion” of KETs to the wider

economy.

An additional benefit of the proposed approach is the ease with which it can be applied using data from

harmonised international databases, allowing the calculations to be performed outside Europe.

3.2.2. Illustrative example of the approach

To better understand how the Input-Output approach works, it is worthwhile to illustrate it with a methodological

example. We suggest considering the following observed supplier-user relationship between two sectors

implied by the I-O table.

We propose the following steps for calculating the relevant indicators:

1. Use the experts’ identified KETs-relevant product codes within 29.31, as developed in the previous

phase, and, possibly, product-level data from Eurostat PRODCOM database to compute the KETs

intensity of 29.31 as the share of KETs in total intermediate output of 29.31. This can be done in two

ways:

KET Intensity2931 =∑ (𝑝|𝐾𝐸𝑇)𝑝∈2931

∑ 𝑝𝑝∈2931 (1)

where (p|KET) is the indicator representing the product that is classified as “Yes, the competitiveness

of the sector is substantially impacted by KETs” in a sector 29.31, and p is the index of every product

identified in PRODCOM classification system in sector 2931.

or

KET Intensity2931,c,t =∑ (𝑝|𝐾𝐸𝑇)×𝑃𝑟𝑜𝑑𝑢𝑐𝑡𝑖𝑜𝑛𝑝,𝑐,𝑡𝑝∈2931

∑ 𝑃𝑟𝑜𝑑𝑢𝑐𝑡𝑖𝑜𝑛𝑝,𝑐,𝑡𝑝∈2931 (2)

where Productionp,c,t is output of product p in country c in year t, and the rest of the variable is defined

as in equation (1).

The difference between the measure (1) and (2) is that in (1) we rely only on the expert-determined

KETs-relevant PRODCOM product codes in sector 29.31, divided by the total number product codes as

defined by PRODCOM in 29.31. For (2) we use the observed country-year production of the KETs-

relevant PRODCOM products in sector 29.31 divided by the total production in sector 29.31. In other

words, the second equation takes into account the relative production value of the KETs part, capturing

the relative economic weight of products within a sector as well as time and country variation. The

production data in equation (2) is confidentially available at the 8-digit level within Eurostat. Therefore,

cooperation with Eurostat to compute these weights is needed. Nevertheless, equation (1) can be used

in case the weights cannot be obtained through Eurostat. The project team will verify the feasibility of it

with Eurostat as soon as possible.

29.31 Manufacture of electrical & electronic equipment for motor

vehicles

(Acting as supplier of inputs)

29.10 Manufacture of motor vehicles

(The user of inputs and maker of the final good)


2. Based on the input-output table, we will then compute linkages between the input supplier sector 29.31

and the user (final production) sector 29.10, represented by the following coefficients often used in

academic literature on the technology spillovers5:

𝜶𝟐𝟗𝟑𝟏,𝟐𝟗𝟏𝟎: 29.31’s deliveries of inputs to 29.10 relative to the 29.31’s deliveries of inputs to all

industries. This represents supplier-industry significance for given user-industry, relative to

supplier-industry significance for the total economy. Important to note: this proportion is

computed by excluding products supplied for final consumption.

𝝈𝟐𝟗𝟏𝟎,𝟐𝟗𝟑𝟏: 29.10’s uses of inputs from 29.31 relative to the 29.10's overall uses of intermediate

inputs coming from all supplier sectors. This depicts the user-industry’s reliance on inputs from a

given supplier-industry, relative to the user-industry’s total input uses.

3. We will then compute the economy-wide measures of KETs diffusion for a given sector in two

possible ways, taking the prospective of a given supplier-sector or user-sector.

a. KETs diffusion from a given supplier sector, such as 29.31, to all the users in rest of economy

as

KET Intensity2931,c,t × (∑ 𝛼2931,𝑠 [Outputs,c,t

∑ Outputs,c,t 𝑠 𝑖𝑓 𝑠≠2931]𝑠 𝑖𝑓 𝑠≠2931 ) (3)

where Outputs,c,t represents some industry-level outcome of sector s, taken from the publicly-

available Eurostat SBS or OECD STAN databases, such as final production, turnover, or value

added in county c in year y. The term in brackets represents the “domestic demand” for the

intermediate inputs supplied by the sector like 29.31.

b. KETs diffusion to a given user sector 29.10 from all supplying sectors in the economy, as

∑ 𝜎2910,𝑠𝑠 𝑖𝑓 𝑠≠2910 KET Intensitys,c,t (4)

4. Finally, the economy-wide diffusion measures can be obtained by adding up the measure 3a in

equation (3) over all supplier sectors and adding up the measure 3b in equation (4) over all user

sectors.

Using Input-Output tables for KETs proves to be technically feasible Our detailed feasibility check confirmed that it is foremost possible to match most NACE sectors with the proxies of KETs intensity to the I-O table. This implies that from a technical point of view, the methodology can be applied. Out of the 112 unique 4-digit NACE codes for which we can compute the KETs relevant content, we can match 100 sectors from the I-O table for which the required coefficients from step 2 of our methodology are defined. This implies that the sectors are adequately covered at the 4-digit level in our approach. Moreover, if we only focus on the manufacturing sectors (2-digit NACE sectors no. 10-32), then out of 212 4-digit NACE codes in I-O table in manufacturing, we can identify the KETs intensity of 108 of these sectors, following the assessment of experts.

5 Following Javorcik (2004) (Smarzynska Javorcik, Beata. 2004. "Does Foreign Direct Investment Increase the Productivity of Domestic Firms? In Search of Spillovers Through Backward Linkages." American Economic Review, 94(3): 605-627) and Blalock and Gertler (2008) (Blalock, Garrick and Paul J. Gertler, 2008. “Welfare gains from Foreign Direct Investment through technology transfer to local suppliers," Journal of International Economics 74 (2), 402-421) who studied the productivity spillovers from foreign firms in supplier or user industries to domestic firms. In our case, the intention is to compute the spillovers from KETs in supplier sectors


3.2.3. Benefits, drawbacks and challenges of the approach

To fully grasp the impact of adopting an I-O approach instead of the current expert-based approach, the pros

and cons need to be carefully weighed. Table 3-1 presents the benefits and drawbacks of both approaches.

TABLE 3-1: Benefits and drawbacks of the different technology diffusion approaches

Expert-based approach (KETO1) Input-Output approach (KETO2)

Benefits

Detailed 8-digit product level approach to identify the diffusion of KETs at the maximum level of granularity available in Europe.

Makes an attempt to account for the degree of the enhanced competitiveness of industries as a result of KETs.

Validated by technical experts.

Eliminates strong reliance on expert opinion, while still taking into account their expertise.

Considers the use of KETs in industries regardless of the degree of their impact on competitiveness, eliminating bias in assessing the degree of the technology’s impact on competitiveness.

Validated by both data and experts.

Approach can be applied outside of Europe, allowing for international comparison.

Accepted approach in the academic literature for assessing technology spillovers to other industries.

Drawbacks

Complete reliance on confidential data.

High sensitivity of expert opinion on resulting data, with minor deviations (e.g. low or medium impact) causing high fluctuations (e.g. +- 24% between “low” and “medium”).

Limited number of technical experts with in-depth knowledge on KETs that are also able judge the impact on competiveness of high number of detailed non-KETs related product industries.

Approach is based on confidential 8-digit production data only available in Europe and cannot be used for international databases. As a result, no international comparison is possible.

Assessing the degree to which KETs impact the competitiveness of specific product industries is prone to discussions and subject to expert opinion only.

Reliance on confidential data to estimate the KET intensity of industries.

Less variance in the degree of KETs impact on competitiveness of final products due to reducing expert rating to a simple “yes/no”.

Less granularity in the data representing the concept of economic activity (4-digit industry as opposed to 8-digit product-level data).

Assumptions required on the intensity of KETs in 4-digit industry data.

International comparison is possible, but only under the assumption that the KETs intensity of sectors outside Europe is comparable to what is estimated for Europe (given that these are calculated on the basis of confidential 8-digit product level data).

I-O-tables reveal interlinkages between more traditional sectors, which require additional calculations before they can be used to identify the KET specific interlinkage. We note that this drawback is addressed through our approach, which takes into account the KET intensity of the sectors that are identified in the I-O-tables (see 3.2.2)

Clearly, where the key difference in both approaches lies is in the way it deals with expert opinion. Although it

can be argued that data computed at a more aggregated level (4-digit vs 8-digit) makes it more difficult to

identify the KET-relevant content, it needs to be understood that an expert-based approach using more

granular data is not by definition more accurate. On the contrary, experts on methodology unanimously

agreed at the validation workshop that it is unlikely that experts are able to accurately assess the

competitiveness impact of KETs for each of the thousands of individual 8-digit products separately.

Our proposed solution thus sacrifices some of the granularity of the data in favour of following a more

objective, data-driven approach. While we potentially lose some of the ability to capture the KET-relevant

content by focusing on more aggregated data, experts argued that an I-O approach is likely to result in a more

accurate quantification of the diffusion of KETs along the value chain, from inputs to final products, at the

aggregate level aimed by the Observatory. It was argued that the objectivity of a data-driven approach is much

preferred due to its consistency in assessing the diffusion of KETs. Moreover, the expert assessment is still

taken into account in reduced form to identify the relevant KETs industries while the aggregation of the expert

opinions to the narrow 4-digit industry sectors still results in considerable across-industry, across-country, and

over time variation in KETs intensity. Based on the confidential 8-digit product level data, we seek to calculate


the relative share of KETs in the 4-digit industry sectors which implies that we still intend to make use of the

level of granularity available where considered fit-for-purpose.

Nevertheless, we acknowledge that neither approach is free from constraints. In case of the I-O approach, it

is particularly important to control for ‘double counting’ of KETs-based products. First of all, we note that we can

distinguish between intermediate and final use in the I-O tables. We agree, however, that the linkages in

traditional I-O tables do not correct for the KET-specific linkages, which are more easily (though also not

perfectly) captured at the 8-digit product code level. To address this, we propose to estimate the KET intensity

of the 4-digit industry sectors by taking into account the confidential 8-digit product code data (i.e. the 8-digit

PRODCOM data confidentially available at Eurostat). By multiplying with the estimated KET-intensity, we aim to

take into account the KET-specific nature of the industries as well as the expert assessment and validation.

3.2.4. Final conclusions

Experts supported the idea of a data-driven approach instead of a pure expert opinion-based approach. An

added benefit of using an I-O approach is that it is more focused on quantifying the “use” (i.e. dissemination) of

technology, instead of having experts try to capture the effect on competitiveness (i.e. the use of a UV-coating

in packaging vs. the competitiveness effect of using a UV-coating in packaging). Therefore, we aim to proceed

with implementing an approach using input-output tables to capture the diffusion of KETs for the

second phase of the KETs Observatory.

We will first pilot our approach before finalising it, to demonstrate its feasibility. The results can then be

compared with the existing approach and offered to experts for their validation and for making the final

conclusion on whether the new approach will be fully implemented during the second phase.

3.3. Expanding the KETs Observatory to increase international coverage

There is a need to expand the KETs Observatory with international data to allow for comparison between

Europe and other regions. The approach taken in phase one of the KETs Observatory makes the international

expansion of the database highly challenging due to the strong focus on detailed data available only within

Europe (e.g. confidential PRODCOM data at 8-digit product level).


Our research concluded that harmonised data sources contain the most complete and comparable data that is

available to date. Our suggestion therefore is to focus on using the UN COMTRADE database for obtaining

trade data at the international level (available at 6-digit level) and the OECD databases for obtaining the

industry statistics (available at 4-digit level). The latter implies, however, that with the exception of patent

data, there is an extremely low chance of obtaining data at a comparable level of detail for the non-OECD

countries (i.e. China, Singapore, Taiwan, India, Brazil, Russia, and South Africa). Further research has

concluded that such data also cannot be obtained directly from statistical offices in the countries in question, for

example, China. To this end, it is reasonable to focus on data available in internationally accepted and

harmonised databases.


Overall, experts agreed that the proposed approach is sensible. Although each approach will have its

advantages and disadvantages, by focusing on harmonised data from the OECD, a comparable international

data set can be developed. Nevertheless, it is crucial to be explicit at this point of the limitations of the

approach:

The OECD does not cover all countries of interest (specifically, it does not cover China, Singapore,

Taiwan, India, Brazil, Russia, and South Africa). Our analysis, however, also revealed that the required


data cannot be obtained outside the OECD databases, such as directly from the statistical offices.

Therefore, taking another approach here is not expected to yield better results.

With respect to patent data, all OECD countries are included in PATSTAT and share the same patent

application procedure under the Patent Cooperation Treaty. However, there is a certain time lag in it, as

companies that apply for patents internationally tend to submit for a wide range of countries at the time

of application. Only at a later stage, they decide for which markets (or regions) the patent is relevant. It

is therefore only after 30 months that one will be able to see if there is a real interest in a

particular country. Experts could not, however, identify a more feasible approach.

International quality standard in patents may differ across regions. Experts provided an example

of China, where applicants typically receive (large) cash bonuses for applying for patents. Moreover,

Japanese law, for example, causes inventors to patent parts of their innovation separately. This has

resulted in some breakthrough technologies that have not been patented as such, but are scattered

across a multitude of patents (according to experts sometimes referred to as “sashimi patents”). To

take into account the quality of patents, experts argued it would be good to focus on PCT patents. Also,

it would benefit the approach to consider triadic patents. However, the drawback of this approach would

be the time lag in publishing triadic patent data. Therefore, we propose to follow the approach set out in

the previous phase of the KETs Observatory and retrieve the relevant international data using OECD,

PATSTAT and UN COMTRADE.

3.4. Exploring an alternative measure to increase coverage of the KETs Observatory beyond manufacturing industries

Phase one of the KETs Observatory focused on manufacturing sectors and production data, which implies that

its methodology does not take into account employment in non-manufacturing industries (e.g. R&D and

services activities).


Two alternative approaches were analysed in-depth for their feasibility to expand the coverage beyond

manufacturing industries:

Selecting the relevant statistical codes outside manufacturing industries and using the data to calculate

the deployment of KETs in non-manufacturing industries;

Using employment multipliers to roughly estimate employment generated outside the manufacturing

industries as a direct result of manufacturing activities.

Based on our research, we concluded that data as currently collected by statistical offices all over the world

strongly limits the extent to which we can estimate employment for KETs in non-manufacturing sectors. Data

collected for manufacturing industries (i.e. NACE C) combined with production data at product level

(PRODCOM) can be disaggregated to a much higher degree, allowing us to better capture the KET-relevant

data. Data from other sectors, such as the services industries (i.e. NACE H-N) cannot be disaggregated to the

point where it would be possible to focus on KETs specific services. We concluded that this approach would not

be feasible due to data limitations.

The use of employment multipliers has also been investigated. However, such multipliers need to be

considered as a rough approximation only. Given the specific and detailed approach the KETs Observatory

takes on estimating the deployment of KETs as accurate as possible, applying employment multipliers would

greatly dilute the level of accuracy and create a false sense of accurate coverage beyond manufacturing

industries. We propose that employment multipliers can be used to gain a rough approximation, but that

they should therefore not be reported in the KETs Observatory.



The results of our analysis were presented at the validation workshop, where experts agreed that adopting

either of the alternatives would not be of added value. Concluding, a feasible alternative approach to address

the challenge was not identified. As a result, the methodology as developed in the previous phase of the

KETs Observatory will be continued with respect to the focus on data from manufacturing industries.

However, with implementation of our new approach for calculating turnover indicators (see section 4.2),

we will be able to gain rough insights into how turnover from production activities relates to overall

firm turnover. Whereas the former is calculated using the value of sold production (in euros as, for example,

defined by PRODCOM), the latter includes revenue from all activities of firms that are active in the

manufacturing industries, including marketing and R&D-efforts. By directly comparing the two, we will attempt

to provide insight in the extent to which KET-focused companies active in the manufacturing industries

engage in non-production activities.

As the information this statistic provides is more informative of nature, rather than a subset of the indicators

employed in the KETs Observatory, it is best presented and qualified in the (country) reports. While reporting

the (trends in) turnover of KETs, it can thus be presented as an extra qualification of the turnover. This would

also prevent us from ‘overcrowding’ the KETs Observatory portal with a new set of indicators only intended to

give an approximation of the relative share of production vs. non-production based turnover.

We note, however, that this approach will not capture research and service activities of firms that are active in

the non-manufacturing sectors. Moreover, the feasibility of the approach can only be fully assessed once the

new turnover indicators are correctly calculated. For this reason, we will re-assess the feasibility and

implementation of this approach upon calculating the new turnover indicators, as well as the best place to

present this within the KETs Observatory.

KETs Observatory Phase II: Methodology Report 4 Technology generation & exploitation

4. Technology generation & exploitation approach

This chapter presents the key results of the methodological assessment for the technology generation and

exploitation approach. Per group of indicators (technology, turnover, production and trade), we present a short

description of what they collectively aim to measure, the identified challenges in the methodology of the

indicators, a proposed solution (where applicable), and final conclusions on how to proceed.

4.1 Methodological considerations for the technology indicators

The KETs Observatory includes indicators to measure the generation of technology. The main purpose is to

measure the ability of countries to generate KETs-relevant technology. The original framework focuses on

measuring patent applications as a proxy for the generation of technology. Based on our assessment and on

expert suggestions, we propose to include indicators on scientific publications in addition to patent statistics.

4.1.1. Identified challenges of technology indicators

The identified challenges can be grouped in two types of concerns that need to be addressed:

Comprehensiveness of the indicators to measure knowledge generation in KETs;

Representativeness of the data used for calculating the indicators.

Comprehensiveness of the included indicators

Although it is generally accepted in the scientific literature that patent data captures part of knowledge

generation, there are more possible outcomes of R&D-activities that cause patent statistics to offer an

incomprehensive measurement of knowledge generation activities. An often cited phenomenon is that not

all companies patent their technology, for example, in order not to reveal their technology to competitors.

Moreover, not all knowledge generated in research (including academia) is patented, although the knowledge is

often published and further built upon. Experts at the validation workshop agreed that by looking at patents

only, a considerable part of knowledge generation is missing in the statistics.

Representativeness of the data used for calculating the indicators

As the methodological framework currently only includes patent statistics, an in-depth assessment was made of

the representativeness of the data used for calculating these indicators. For the KETs Observatory, only filed

patent data was considered. Although granted patents may better reveal actors’ true intention to commercialise

(as opposed to application data, which also reflects patent filing strategy), there is a time lag between the

moment when technology is developed and patented (i.e. patent applied for) and subsequently granted. Using

patent application data thus has the benefit of providing a timely measurement of when technology was

generated, in which case it better grasps the outcome of knowledge creation activities. Its drawback is that not

all of the patent applications may result in granted patents, implying that not all filed patents are initially filed

with the intention to commercialise over time. If patents are granted and held over time, this would in theory

better reflect expectations regarding the (future) commercialisation of technology. Patent applications therefore

capture the outcome of knowledge creation activities at an earlier stage, whereas data on granted patents is

more explicitly linked to the (future) commercialisation of technology (i.e. constituting a closer tie between

generation and exploitation of technology).

Furthermore, the methodology of the first phase uses applicant location instead of inventor location for the

country level analysis, as this would best reflect the location where the decision is made to commercialise

technology. Inventor location was preferred for regional level analysis, as this best informs the regional origin of

the innovation. The differentiated approach raised questions among stakeholders in the previous phase, as it


needed to be considered whether one should be preferred over the other or whether they should be continued

as is.

4.1.2. Proposed solution

To improve the comprehensiveness of indicators, we propose to incorporate data on scientific publications

in KETs. The data is currently generated in parallel to the KETs Observatory in a study conducted by CWTS

(Leiden University)/KU Leuven using a bibliometric approach. The project team will explore the feasibility of

incorporating the data of this study into the second phase of the KETs Observatory.

With respect to the representativeness of patent data, we conclude that the approach taken by the previous

consortium appears valid. Applicant data is – in our view – a correct representation of the location where a

patent is held and most likely commercialised. Inventor location, in contrast, is typically based on personal

addresses. Although the same caveat applies to patent data at regional level, where currently inventor location

data at regional level is used, we agree that it is an approximation to trace the origin of an innovation. In

addition, we argue that the technology indicators should be foremost focused on measuring the output of

knowledge creation activities (i.e. generation). Focusing on patent application data is consistent with this notion

and timely available, hence no changes are proposed with respect to the patent data.


The proposed solutions were supported by experts at the validation workshop. This has the following

implications for the second phase of the KETs Observatory:

No changes will be made to calculate patent indicators as adopted in the previous phase of the KETs

Observatory. We will follow the approach as adopted in the first phase6.

The project team will explore the feasibility of incorporating the data on scientific publications

produced by CWTS (Leiden University)/KU Leuven into the second phase of the KETs

Observatory. Depending on the feasibility, we will use the raw data to calculate time series on the

number of scientific publications in KETs, operationalised through its sub-indicators: Country

significance in scientific KET publications; Specialisation in scientific KET publications; Share in

scientific KET publications; and the Medium-term dynamics.

Pending on the feasibility of integrating the data we will also explore whether the data on scientific

publications can be taken into account in the composite indicators for technology generation.

4.2. Methodological considerations for the turnover indicators

The turnover indicators in their current form measure turnover of firms with KET-activities at headquarter level,

which is used as a proxy for measuring the ability of industries/businesses to compete in the market for KETs-

based components and to transfer new technologies and innovations to industrial applications. As the turnover

is presented at the headquarters level, it is argued that it reflects where decision power with regard to

commercialising KETs-based components is located.

From a conceptual perspective, turnover can be considered a measure of economic activity from a revenue

point of view for the entire product and services portfolio of firms. Turnover indicators therefore provide insight

into the ability of firms to generate revenue from products and services.

6 See IDEA Consult et al., (2015) “Key Enabling Technologies (KETs) Observatory: Methodology report”


4.2.1. Identified challenges of turnover indicators

Turnover can be considered relevant for the KETs Observatory if a good approximation can be obtained of the

KETs-relevant part of turnover of companies. In practice, it is challenging to both identify the companies

that are engaged in KETs-relevant activities and to determine the share of their revenue that stems

from these activities. In order to do so, a selection of KET-relevant NACE-codes was used to identify firms in

Bureau van Dijk’s Orbis database that are potentially involved in KETs-relevant activities. This selection of firms

was then further refined by considering patent activity in KET-relevant patent codes. Finally, the revenue of the

resulting list of firms was weighted proportionally to the share of KET-relevant patents a company applies for to

obtain the KET-related revenue of their portfolio.

The methodological approach described above can be linked to several attention points:

(1) Only the turnover of firms that apply for at least 10 KET-relevant patents per year is included.

(2) For each firm, its’ turnover is weighted proportionally by the share of KET-relevant patent

applications in their total number of patent applications.

(3) Decisions on innovation and strategy are not always taken at headquarters only, implying that

company location is a poor reflection of where the decisions are made.

The above has strong implications for the completeness and reliability of the estimates. First of all, it assumes

that there is a one-to-one link between generating technology, patenting technology and

commercialising technology. In practice, however, companies often opt not to patent part of their technology

so as not to reveal it to their competitors.

Secondly, it assumes that all patents are equal. A company with 20 minor KETs-relevant patents is

considered to be highly relevant, whereas a company exploiting a handful of highly valuable patents is not

included. Moreover, as the turnover is weighted by the share of KET-patent applications in the total company’s

patent portfolio, it assumes that the commercial value of all patents is equal and proportional.

Thirdly, the current approach does not take into account the time lag in the commercialisation of

technology. Put differently, it assumes that a patent application in a given year will result in the weighted share

of turnover in that same year. This assumption has a profound effect on the accuracy of the estimate, as it

could mean that a company that patents technology in one year and commercialises the technology over the

years to come could be filtered out of the sample and not be taken into account.

Fourth, it fails to take into account knowledge transfer, as it only focuses on applicants that commercialise.

It therefore explicitly excludes commercialisation activities of patented technology developed outside a

company, e.g. technology developed in a consortium with academia where a university ultimately holds the

patent. Breakthrough innovations are often developed in collaboration, which also typically imply innovations

with the highest commercial value.

Finally, the KETs Observatory already measures sold production value of KETs-based products, based on

confidential 8-digit PRODCOM data. As such, there is partial overlap between the turnover of KET-relevant

production companies on the one hand and the sold KET relevant production of companies on the other.

4.2.2. Proposed solution

We propose to calculate turnover using publically available business statistics at the aggregate sector

level and linking them to the confidential KET-based product production data (8-digit PRODCOM).

Instead of trying to correctly identify companies active in KETs and the relevant share of their revenue related

to KETs on the basis of patent activity, we can follow an alternative bottom-up approach that utilises the


detailed production data available to us to estimate the KET-relevant activities of the manufacturing sectors.

Since the first four digits of the PRODCOM data coincide with the relevant NACE codes, we can extract total

turnover in the relevant manufacturing sectors from public business statistics.

To identify the KETs-relevant revenue, we will proportionally weight the sector revenue by multiplying with the

share of KETs-relevant production value (at 8-digit PRODCOM) in the total production value (i.e. the total at 4-

digit PRODCOM). This requires us to build the weighted ratios (which are at aggregate level) on the basis of

data available confidentially to Eurostat. While this needs to be further discussed with Eurostat at this stage, we

note that this does not constitute the use of different data in comparison to the previous phase.

Alternatively – e.g. in case confidentiality of the production data prohibits us from following this approach – we

can weigh the number of the KETs-relevant PRODCOM product categories in a given NACE sector relative to

the total number of product categories in the sector and use this to weigh the sector-level outcome. The

advantage of the latter is that we do not require involvement from Eurostat for using confidential PRODCOM-

level data, meaning that we will be able to calculate regardless of confidentiality of data. The clear

disadvantage, however, is that we would not take into account the different volumes of the underlying sectors

and treat them all as equal. As a result, the data presented will be less accurate than in the case of basing the

ratios on the confidential 8-digit production data available to Eurostat.

The advantage of this approach is that we eliminate the causal link between patenting and commercial value.

Therefore, the turnover calculated with our newly proposed approach is expected to provide a more realistic

approximation of the turnover of KETs-focused companies. Moreover, it allows us to gain a rough

understanding of manufacturing vs. non-manufacturing revenue, as we can compare the value of sold

production to the approximated turnover of companies in this sector. This provides additional insight in

manufacturing vs. non-manufacturing revenue that cannot be obtained with the current turnover indicators due

to the focus on HQ-location and the existing link with patenting activity7.


Experts underlined a need to develop a new approach to reduce the potential biases related to the

representation of the KETs-relevant turnover at the firm level. The approach as outlined above was well

received by experts at the validation workshop, and overall considered feasible to implement. Although we

assume the share of non-production turnover to be proportional to the share of production-related turnover in

the manufacturing sectors, experts agreed that this is a more accurate reflection of KETs-based revenue than

in its current form. A drawback of the approach, however, is that the detailed (8-digit) production data is only

available within Europe. To overcome this caveat, we found agreement among experts to apply the ratio at

European level to countries outside of Europe8.

Therefore, the following adjustments were developed for the methodology for calculating turnover indicators:

Turnover will be calculated by weighting aggregated business statistics (at 4-digit NACE-level) by the

share of KET-specific production value (derived from 8-digit PRODCOM codes) in total corresponding

production at 4-digit PRODCOM (to correspond to the 4-digit NACE-level).

Since we no longer require the level of granularity provided by Bureau van Dijk’s Orbis database, the

sectoral turnover (at 4-digit NACE) will be extracted from OECD’s Structural and Demographic

7 Our suggestion for how to include the data in the KETs Observatory was already presented in section 3.3.2 of this report and is not repeated here. 8 It was considered as an alternative to weight turnover outside of Europe proportionally to the trade volume of KET-relevant 6-digit HS-codes in the total 4-digit sector trade volume. This, however, would require a differentiated approach for EU vs. non-EU, rendering data incomparable across regions. Combined with the notion that it considered to result in little added value compared to the approach outlined above, it was not adopted as a suitable alternative.


Business Statistics (to allow for international comparison). OECD was preferred over Eurostat as this

allows for direct international comparison.

To calculate internationally comparable turnover statistics, we aim to apply the weighted shares as

calculated for Europe to countries outside Europe as well.

4.3 Methodological considerations for the production indicators

Production indicators for the technology generation and exploitation approach measure the relevance and

dynamics of the production of KETs components. Production output can be considered as one of the key

economic measurements for economic activity and are highly relevant for measuring the deployment of KETs.

Moreover, the KETs Observatory focuses on measuring the value of production, which is defined by Eurostat

as the value of production sold9.

4.3.1. Identified challenges of production indicators

Due to the transverse nature of KETs, the production of KETs components is typically not caught in single

production codes. While overall we conclude that the production indicators are relevant and sound, the

identification of the relevant production codes is challenging and subjected to expert opinion.

To understand, one first needs to consider that production statistics are structured in accordance to a

classification of production codes. Each manufactured good is assigned a production code, which can be used

for all sorts of purposes, including regulatory and monitoring activities. Statistical offices collect data on these

specific product codes, in particular, production data. Subsequently, the data collected at a product category

level can be further aggregated to a sectoral level, providing insight into the production of goods within

manufacturing sectors.

The key challenge for measuring KETs-based production is that even at its most granular level – e.g.

PRODCOM 8-digit statistics that are only confidentially available – product codes cannot always be

exclusively related or directly linked to KETs. Nevertheless, a sound approximation can be obtained at the

detailed product category level to ascertain that a set of product codes (mostly) relates to KETs components.

Moreover, with a narrow focus on KETs components in mind, the KETs Observatory already takes a

conservative approach to prevent overestimating the KET-relevant production values.

Despite the narrow focus, in-depth knowledge of both KETs components and of product categories is required

to correctly identify those codes that are relevant. To address this challenge, the KETs Observatory has relied

on expert opinion to correctly identify the relevant detailed production codes. This gives rise to a potential bias

in the statistics due to the strong focus on expert opinion, which has been described and addressed in more

detail in section 3-2 of this report.

4.3.2. Proposed solution to address the challenges at hand

The key challenge we identified for the production indicators relates to possible expert bias in the statistics only.

Proposals to further limit expert bias in the overall approach of the KETs Observatory have already been

discussed in Section 3-2 of this report and also apply here. To prevent repetition, they are not described here in

detail.

9 More specifically, the Explanatory Notes that accompany Eurostat’s methodology for the PRODCOM database explicitly refer to this as “[…] calculated on the basis of the ex-works selling price obtained during the reporting period [..]”.



The production indicators are overall solid and of added value to the KETs Observatory. They will therefore

also be concluded in the upcoming phase. However, the selection of relevant statistical codes is strongly based

on expert opinion, which may introduce expert bias. To limit this bias, we have proposed and validated

suggestions as described in section 3-2 of this report. We note, however, that experts at the validation

workshop agreed this is not a concern for the technology generation and exploitation approach. Due to its

narrow focus and clear approach towards identification, there was a general agreement that experts should

be able to determine whether a product category contains KET-components, or whether it was produced

using KETs. It was argued that re-validation of the list with experts would be sufficient for the technology

generation and exploitation to address the challenges in the upcoming phase of the KETs Observatory.

Consequently, we aim to adopt this approach.

4.4. Methodological considerations for the trade indicators

Trade statistics showcase the import and export contribution of countries in the total European export of KETs

components and intermediary systems. As such, they measure the ability to produce and commercialise

internationally competitive products based on new technological knowledge. By considering the import/export

balance, they also consider to what extent firms are competitive on the global market. The indicators can be

considered as highly relevant, as they measures the (cross-border) deployment of KETs from an interaction

point of view. Moreover, the trade balance provides an indication of the competitiveness of countries in KETs.

4.4.1. Identified challenges of trade indicators

Similar to the production indicators, the reliability is highly dependent on the relevant selection of

statistical codes. As the trade data are calculated on the basis of this selection, similar reservations hold. As

such, the discussion in sections 3-2 and 4-3 of this report also apply here and have not been repeated.


The key challenge we identified for the trade indicators relates to possible expert bias in the statistics only.

Proposals to further limit expert bias in the overall approach of the KETs Observatory have already been

discussed in section 3-2 of this report and also apply here. To prevent repetition, they are not described here in

detail.


Similar to our final conclusions on the production indicators, trade indicators were overall considered solid

and of added value to the KETs Observatory. They will therefore also be concluded in the upcoming phase.

Moreover, similar to our assessment in sections 3.2 and 4.3.3, experts argued that re-validation of the list with

experts would be sufficient for the technology generation and exploitation to address the challenges in the

upcoming phase of the KETs Observatory. Consequently, we will adopt this approach and calculate the trade

indicators in line with the existing KETs Observatory methodology10.

10 For a detailed description of the methodology for calculating the indicators, please refer to IDEA Consult et al., (2015). Key Enabling Technologies (KETs) Observatory: Methodology report.


4.5. Expanding the KETs Observatory to estimate KETs-ICT

The current phase seeks to expand the database with measuring the digital dimension of Advanced

Manufacturing. The impact and added value of digital technologies on manufacturing can be indicatively

captured through the following three different dimensions:

First dimension: innovation in products through “ICT-inside”, i.e. embedding ICT in any product and

artefact;

Second dimension: transformation in processes, i.e. “digital manufacturing”. This includes digital

innovations affecting the full product lifecycle (from product design and simulation tools to automation

and shop floor controls to logistics and supply chain management to product tracking and recycling);

Third dimension: disruptive changes in the business models. Digital technologies (Big Data, Internet

of Things) are radically modifying the business of well-established industries such as automotive,

lighting or textiles.

Considering that this expansion is only foreseen for the “technology generation and exploitation” approach, the

third dimension will not be covered by this phase.

4.5.1. Proposed approach

In order to capture the value creation of digital technologies in KETs, it is crucial to identify those KETs-

relevant PRODCOM codes that are impacted by digital technologies, either through the first or the second

dimension. This ensures that we measure the value creation within KETs that can be (partially) attributed to

digital technologies.

Based on unanimous agreement among experts on exploring input-output tables for limiting expert bias in

selection relevant statistical codes for particularly the diffusion of KETs, it was argued that a similar approach

can be conducted for estimating the diffusion of ICT in Advanced Manufacturing. Such an approach was

favoured over identifying relevant statistical codes with industry experts. Consequently, we propose to identify

the use of ICT in Advanced Manufacturing with the help of input-output tables.

For ICT, we will use the statistical codes as identified in the 2007 OECD ICT sector definition11. The sector

definition is presented in Table 4-1 below. Using the identified sectors, we will then consider for each of the

identified Advanced Manufacturing sectors what the inter-industry trade is to consider the use of ICT in

Advanced Manufacturing.

Table 4-1: The 2007 OECD ICT sector definition12

NACE Rev. 2 code OECD ICT sector description

ICT manufacturing industries

261 Manufacture of electronic components and boards

262 Manufacture of computers and peripheral equipment

263 Manufacture of communication equipment

264 Manufacture of consumer electronics

268 Manufacture of magnetic and optical media

ICT trade industries

4651 Wholesale of computers, computer peripheral equipment and software

11 OECD (2011). Guide to Measuring the Information Society 2011. Paris: OECD. 12 OECD (2011). Guide to Measuring the Information Society 2011. Paris: OECD.


NACE Rev. 2 code OECD ICT sector description

4652 Wholesale of electronic and telecommunications equipment and parts

ICT services industries

5820 Software publishing

6110 Wired telecommunications activities

6120 Wireless telecommunications activities

6130 Satellite telecommunications activities

6190 Other telecommunications activities

6201 Computer programming activities

6202 Computer consultancy activities

6203 Computer facilities management activities

6209 Other information technology and computer service activities

6311 Data processing, hosting and related activities

6312 Web portals

9511 Repair of computers and peripheral equipment

9512 Repair of communication equipment

The ICT sector definition also allows us to distinguish between the first and second dimension. Experts argued

during the validation workshop that the first dimension typically relates to the ICT hardware (i.e.

components) that would be used in KETs. The second dimension, in contrast, is more on the software and

services side, where digital transformation is created through software improvements, new algorithms and

delivering new digital solutions. As such, it can be argued that the use of input from ICT manufacturing and

trade industries is to be considered in the first dimension, whereas the use of input from ICT services industries

is more related to the second dimension.


Instead of following an expert-based approach to identify the reliance on (i.e. “use of”) ICT in KETs/AMT,

experts concluded that an I-O approach could also be applied here. By tracing the use of ICT in the relevant

product sectors, an approximation of the degree to which the selected industries rely on ICT can be obtained.

Following their suggestions, we will further develop and employ an input-output approach for identifying and

selecting relevant statistical codes. To take into account expert views, we also intend to validate the final list of

identified industries with experts from the field.

4.6. Methodological considerations for the composite indicators

The composite indicators are used to measure the overall performance of countries in KETs and build upon all

groups of indicators of the technology generation and exploitation approach. In line with the “Deployment Value

Chain” (as described in detail in section 2.1 of this report), composite indicators are calculated separately

for each stage of the technology generation and exploitation approach (i.e. Technology, Production and

Demand, Trade, and Turnover). This leads to four different types of composite indicators, each providing insight

in a different aspect of the exploitation of KETs. Furthermore, the calculation is performed for all KETs

separately, resulting in a total of 24 indices (4 composite indicators for 6 KETs equals a total of 24 indicators).

To ensure that the composite indicators presented in the KETs Observatory are meaningful, we assessed to

what extent the composite indicators are calculated in accordance to the "Handbook on Constructing

Composite Indicators: Methodology and User Guide" (OECD, 2008). Using their “Checklist for building a

composite indicator” we scrutinised the current methodology used for calculating the composite indicators.

Overall, we find that the composite indicators are correctly calculated, that the previous consortium

made defendable and sensible choices with respect to calculating the indicators, and that no particular


challenges can be identified. In fact, the previous consortium also relied on the OECD (2008) guideline to

construct the indicators.

4.6.1. Identified challenges of composite indicators

The challenge with composite indicators in general is that the methodology employed can have profound

effects on its accuracy, relevancy and comparability (both between groups and over time). Scrutinising the

methodology with the OECD (2008) guidelines on constructing a composite indicator did not reveal particular

weaknesses in the methodological approach that is already in place. Moreover, by calculating the indicators

for each stage separately (i.e. technology, production, trade and turnover), the indices provide a high degree of

freedom for interpretation.

4.6.2 Proposed solution to address the challenges at hand

Based on our analysis above, we see no need to change the methodology at this point in time.

4.6.3 Final conclusions

Based on our assessment, we conclude that the methodology for calculating the composite indicators is

structurally sound. We therefore suggest to continue the approach as outlined in the methodology report

developed for the previous phase13.

13 IDEA Consult et al. (2015). Key Enabling Technologies (KETs) Observatory: Methodology report.

KETs Observatory Phase II: Methodology Report 5 Technology diffusion approach

5. Technology diffusion approach

This chapter presents the key results of the methodological assessment for the technology generation

and exploitation approach. Per group of indicators (technology, turnover, production and trade), we

present a short description of what they collectively aim to measure, the identified challenges in the

methodology of the indicators, a proposed solution (where applicable), and final conclusions on how to

proceed.

5.1. Methodological considerations for production and demand indicators

The production and demand indicators show to what extent the EU countries use the potential of KETs

to improve its competitiveness by manufacturing KETs-based products and applying them in the

production of manufacturing goods, both in the sectors that produce KETs as well as, and more

importantly, in other industry sectors.

5.1.1. Identified challenges of production and demand indicators

As discussed throughout the report (e.g. sections 3-2, 4-3 and 4-4), the methodology of the first phase

heavily relied on expert opinion. For the technology diffusion approach, this is an even greater

concern, as experts are asked to assess the relative contribution of KETs to the

competitiveness of an industry. To calculate this contribution, selected PRODCOM codes are

weighed proportionally (on a 4-point scale). The selection and weighing process relies on expert

judgment to assess the contribution of the deployment of KETs to the competitiveness of a selected

PRODCOM code. This introduces subjectivity to the statistics, as it relies on expert opinion to

identify which (and to what extent) PRODCOM codes are included. Moreover, it can create

inconsistencies over time, as experts may not value PRODCOM codes in a consistent way over longer

periods or because expert panels change over time. Furthermore, it may be more difficult for experts

to identify relevant PRODCOM codes outside their sectors (i.e. for the technology diffusion

approach) than it is within (i.e. for the technology generation and exploitation approach).


The identified challenges relate to expert bias. Our proposed solution (i.e. complementing the

approach with Input-Output tables instead of relying on expert opinion only) is described in more detail

in section 3-2 of this report and is not described in detail here to prevent repetition.


Experts at the validation workshop welcomed the suggestion to complement the approach with input-

output models. Similar to the above, this has already been described in more detail in Section 3-2 of

this report. Therefore, we follow the expert supported approach described there and apply input-

output tables to reduce expert bias in production statistics. Furthermore, in line with the changes

described in section 3-4 of this report, the data will be internationally expanded by calculating the

relevant statistics on the basis of OECD’s Structural and Demographic Business Statistics.

5.2. Methodological considerations for employment indicators

The employment indicators reveal a country’s performance with regard to KETs-related employment.

Although presented in the various reports produced by the KETs Observatory, they are not presented

on the website. Nevertheless, the indicators can be considered as a measurement of one of the key

KETs Observatory Phase II: Methodology Report 5 Technology diffusion approach

economic impacts of KETs in the wider economy, as it includes employment generated in the core of

the KETs (i.e. the KETs components industries) as well as employment generated in industries that

are enabled by KETs.

5.2.1. Identified challenges of employment indicators

Although the indicators are meant as an all-inclusive measurement of KETs(-enabled) employment,

the focus on data from manufacturing industries prevents it from presenting the broad scope

of employment generated by KETs. As such, employment in research and services industries are

not included in the KETs Observatory. This needs to be taken into account when interpreting the data.


To increase coverage of the KETs Observatory on employment in research and services industries,

we have already described in Section 3-4 of this report that we considered the following approaches:

Select the relevant statistical codes outside manufacturing industries and use the data

to calculate the deployment of KETs in non-manufacturing industries.

Use employment multipliers to roughly estimate employment generated outside the

manufacturing industries as a direct result of manufacturing activities.

As described in more detail in section 3-4, we concluded that both approaches would not be

feasible due to data limitations and accuracy of the results.

We also note that the changes to the methodology with respect to reducing expert bias will directly

affect the estimated employment in the technology diffusion approach. As such, we emphasise a need

to carefully analyse the output of the newly calculated employment indicators based on the adjusted

data that will be generated for the production statistics. We will therefore compare the results with the

latest industry-specific data available to assess the accuracy of the employment estimated within the

KETs Observatory.


Based on the expert validation and our research, we conclude that the employment indicators will

be calculated similarly to the previous phase. In addition, we will re-assess the accuracy of the

estimates based on the newly developed production statistics, for which we will compare the

estimated employment per KET with key industry data available (including employment data for

photonics and micro- and nanoelectronics). Examples of such sources include, but are not limited to:

Photonics (2017). Jobs and Growth in Europe: Realizing the Potential of Photonics. PPP

Impact Report 2017.

Photonik Forschung Deutschland (2015). Photonics Industry Report: Current Situation 2015.

Photonik Forschung Deutschland (2013). Photonik Branchenreport 2013.

Tematys (2013). Photonics Ecosystem in Europe.

International Trade Administration, (2016). Top Markets Report: Semiconductors and Related

Equipment.

Oxford Economics (2014). Enabling the Hyperconnected Age: The role of semiconductors.

Semiconductor Industry Association Global Sales Reports (2006-2015)

The Eisenhower School Industry Study Reports (2015-2017) on Advanced Manufacturing.

The Eisenhower School Industry Study Reports (2015-2017) on Biotechnology.

Documents

METHODOLOGY REPORT - European Commission · KETs Observatory Phase II: Methodology Report 1 Introduction 1.2 WP1: Assessment of the reliability and validity of the KETs Observatory