Final Evaluation of the ARTEMIS and ENIAC Joint Undertaking … · 2017-10-11 · systems market to grow from $3.43 Billion in 2016, to $4.71 Billion by 2021. The Unmanned Aerial

FR

Digital

Single Market

Final Evaluation of the

ARTEMIS and ENIAC Joint Undertaking (2008-2013)

Operating under FP7

FINAL REPORT A report prepared for the European Commission

DG Communications Networks, Content & Technology by

Haydn Thompson, Rapporteur

Expert group Emilio Lora-Tamayo, Chairman

Werner Damm

Jean-Luc Dormoy Leonard Hobbs

Margriet Jansz Tomasz Kosmider

Wolfgang Pribyl

June 2017

2

DISCLAIMER

By the European Commission, Directorate-General of Communications Networks, Content & Technology.

The information and views set out in this publication are those of the author(s) and do not necessarily reflect the

official opinion of the Commission. The Commission does not guarantee the accuracy of the data included in this

study. Neither the Commission nor any person acting on the Commission’s behalf may be held responsible for

the use which may be made of the information contained therein.

ISBN 978-92-79-69634-3 doi:10.2759/271765

Copyright © 2017 – European Union. All rights reserved. Certain parts are licensed under conditions to the EU.

Reproduction is authorised provided the source is acknowledged.

3

4

Table of Contents

ABSTRACT ................................................................................................................................. 5

1. EXECUTIVE SUMMARY ..................................................................................................... 6

1.1 EXTENDED EXECUTIVE SUMMARY.......................................................................................... 8

1.2 KEY FINDINGS ...................................................................................................................... 9

2. INTRODUCTION ............................................................................................................... 19

2.1. PURPOSE OF THE EVALUATION ........................................................................................... 19

2.2. SCOPE OF THE EVALUATION .............................................................................................. 20

3. BACKGROUND TO THE INITIATIVE .......................................................................... 21

3.1. DESCRIPTION OF THE INITIATIVE, OBJECTIVES AND RELEVANCE ....................................... 21

3.2. ARTEMIS JU ................................................................................................................... 22

3.3. ENIAC JU ........................................................................................................................ 25

3.4. BASELINE .......................................................................................................................... 28

4. EVALUATION QUESTIONS ............................................................................................ 30

5. METHOD/PROCESS FOLLOWED ................................................................................. 36

5.1. PROCESS/METHODOLOGY ................................................................................................. 36

5.2. LIMITATIONS – ROBUSTNESS OF FINDINGS ....................................................................... 37

6. IMPLEMENTATION OF ARTEMIS AND ENIAC JOINT TECHNOLOGY

INITIATIVES .......................................................................................................................... 38

6.1. IMPLEMENTATION OF THE ARTEMIS JU ........................................................................... 38

6.2. IMPLEMENTATION OF THE ENIAC JU .............................................................................................. 44

7. ANSWERS TO THE EVALUATION QUESTIONS ...................................................... 50

7.1. ARTEMIS - MAIN ACHIEVEMENTS AND EFFECTIVENESS OF IMPLEMENTATION................... 51

7.2. ENIAC - MAIN ACHIEVEMENTS AND EFFECTIVENESS OF IMPLEMENTATION ....................... 55

7.3. EU ADDED VALUE ............................................................................................................. 83

7.4. COHERENCE ...................................................................................................................... 86

7.5. RELEVANCE ....................................................................................................................... 93

8. CONCLUSIONS .................................................................................................................. 99

5

Abstract

This report presents the results of the final evaluation of both the ARTEMIS and ENIAC Joint Undertakings which were set up in 2008 with the overriding aim to contribute towards the development and implementation of strategic research programmes in the areas of nanoelectronics and embedded systems, respectively. The evaluation covers the period 2008-2016 considering three main aspects:

Effectiveness: progress made towards meeting the objectives set.

Efficiency: extent to which the JUs were managed and operated efficiently.

Research Quality: extent to which the JUs enabled world-class research that helped Europe to establish a leadership position globally, and how it engaged with a wider constituency to open the research to the broader society.

The panel gathered evidence via an extensive review of relevant documentation and from interviews with a wide range of stakeholders and interested parties from industry, research institutes and universities, the European Commission, Member State Public Authorities, and participants in the JU programmes. The results will be used to inform the European Parliament and Council, national authorities, the research community and other stakeholders on the final outcome of the JUs, to improve the implementation of the H2020 ECSEL JU and provide input for the framework programme after 2020.

Ce rapport présente les résultats de l'évaluation finale d'ARTEMIS et d'ENIAC, deux "Entreprise Commune" (Joint Undertaking - JU) mises en place en 2018, dont le but essentiel est de contribuer au développement et à la mise en œuvre des programmes de recherche stratégiques, respectivement dans les domaines des systèmes embarqués et de la nanoélectronique. L'évaluation couvre la période 2008-2016 et s'attache aux trois critères suivants :

Efficacité: Avancement vers les objectifs prédéfinis, Efficience: Qualité du management et de la mise en œuvre de la "Entreprise Commune", Qualité de la recherche: Mesure dans laquelle le "Entreprise Commune" a aidé l'Europe à

établir une recherche de niveau mondial, et dans quelle mesure il a engagé un plus large spectre d'acteurs afin d'ouvrir plus largement la recherche dans la société.

Le panel d'évaluateurs a recueilli des données via un examen détaillé de documents et des entretiens avec un large spectre d'acteurs et de parties prenantes, venant de l'industrie, d'instituts de recherche et d'universités, ainsi que de la commission européenne, d'autorités publiques d'états membres, et de participants au programme du JU. Les résultats seront exploités pour informer le parlement européen, le conseil européen, les autorités nationales, ainsi que la communauté de recherche et autres parties prenantes sur le résultat final du JU. Les résultats permettront d'améliorer la mise en œuvre d'ECSEL et fourniront une contribution au programme-cadre après 2020.

6

1. EXECUTIVE SUMMARY

The ARTEMIS and ENIAC JU’s have strongly supported the embedded systems and semiconductor domains strengthening European industry in key application areas which generate billions of Euros turnover for Europe and are responsible for 10’s of millions of jobs across Europe. They have pushed innovation that have led to new fabrication techniques, new products, new applications and pre-standardisation activities for Platforms. This is strategically important for Europe with the on-going digitisation of industry. McKinsey [1] estimates that digitisation will potentially add €1 trillion to the GDP in Europe. Sectors that crucially rely on embedded systems and semiconductor markets include:

Automotive - The EU is among the world’s biggest producers of motor vehicles, and the sector represents the largest private investor in research and development (R&D) within Europe. The sector provides jobs for 12 million people and accounts for 4% of the EU’s GDP. Manufacturing accounts for 3 million jobs, sales and maintenance for 4.3 million, and transport for 4.8 million. The global car fleet is predicted to double from currently 800 million vehicles to over 1.6 billion vehicles by 2030. Markets and Markets predicts that the global traffic management market will grow from $4.12 Billion in 2015 to $17.64 Billion by 2020 and the self-driving car market will grow from $42 Billion in 2025 to $77 Billion by 2035.

Rail - The overall rail sector in the EU, including the rail operators and infrastructure managers, employs approximately 1.8 million people with an estimated 817,000 dependent individuals. The European rail supply industry employs nearly 400,000 people and is a top exporter, accounting for nearly half of the world market for rail products with a market share of 84% in Europe and a total production value of €40 billion (2010). Markets and Markets predicts that the railway management system market will grow from $29.27 Billion in 2016 to $57.88 Billion by 2021.

Aerospace - The European aerospace industry is a world leader in the production of civil and military aircraft, helicopters, drones, aero-engines, and equipment, exporting them all over the world. Aerospace within the EU provides more than 500,000 jobs and generated a turnover of €140 billion in 2013. The commercial aircraft market is expected to grow steadily to 2035. The aircraft flight control system market projected to grow from $11.85 Billion in 2016 to $16.59 Billion by 2021, and the aircraft health monitoring systems market to grow from $3.43 Billion in 2016, to $4.71 Billion by 2021. The Unmanned Aerial Vehicle market was estimated to be $13.22 Billion in 2016 and is projected to reach $28.27 Billion by 2022 with opportunities in software ($12.33 Billion by 2022) and services ($18.02 Billion by 2022). The Air Traffic Management (ATM) market is projected to grow from $50.01 Billion in 2016 to $97.30 Billion by 2022.

Manufacturing - The manufacturing sector accounts for 15.0% GDP and provides around 33 million jobs in Europe. Europe is a front runner in manufacturing excellence with the vision of smart and connected factories swiftly becoming a reality. The industrial control and factory automation market, comprising control system manufacturers, field components manufacturers, system integrators, and software manufacturers, is projected to reach $153.30 Billion by 2022. By 2025 additive manufacturing is expected

to create a €6.3 billion opportunity in the consumer electronics, automotive and aerospace industries.

Health - Health care and long-term care expenditure accounted for 8.7% of GDP and about 15% of total government expenditure in the EU in 2015. Spending is rising faster than GDP and it is estimated that it will reach 16% of GDP by 2020 in OECD countries. The health sector accounts for 10% of all employment and is expected to grow by a further 1.8 million jobs up to 2025. Life expectancy currently increases with “one weekend per week” in Europe. The ageing population and prevalence of chronic diseases will increase public health and care budgets significantly due to the need to provide long-term care driving the need for new solutions. The healthcare IT market is projected to reach $280.25 Billion by 2021 from $134.25 Billion in 2016. The global medical device connectivity market is projected to reach $1.34 Billion by 2021 and the telehealth market is projected to reach $9.35 Billion by 2021.

Table 1 Market Opportunities for Embedded Systems and Semiconductors (Source: THHINK [2]) Going forward the strategic merger into the ECSEL JU of the ARTEMIS and ENIAC JUs, combined with the EPoSS Smart Systems Integration community, brings together all the key players necessary to have a strong impact on the Electronics Components and Systems market. The tri-partite organisation of ARTEMIS, ENIAC and ECSEL allows strategic alignment of the Member States, Industry and the European Commission to push its key mission, thus supporting the findings of the Digitising European Industries Summit in Rome in March 2017.

7

ARTEMIS and ENIAC have successfully funded 119 projects with €630 million of EC funding leveraged with €912 million of national contributions. Industry has also contributed €2.46 billion of funding resulting in €4 billion being targeted at research and innovation in the two sectors. 1420 entities (ARTEMIS) and 1384 entities (ENIAC) have participated with 1000’s of researchers being engaged on projects. Already ENIAC has been highly successful in supporting the interests of the micro/nanoelectronics sector such as in supporting the development of FDSOI, a key advanced low-power technology, and in keeping production capabilities for advanced silicon processes as well as the resulting know-how in Europe. Projects within ARTEMIS such as CRYSTAL demonstrate the potential for creating success stories in the higher system levels, nowadays referred to as (Systems of) Cyber-Physical Systems, underlying Smart Mobility, Smart Energy, Smart Home, Smart Health, Smart Production (Industry 4.0) or Smart Cities.

As ARTEMIS and ENIAC have formally concluded recommendations can only be made with respect to the current on-going ECSEL JU. Here there may also be lessons that can be learned and adopted for the operation of JUs in general. To unfold potential and fully achieve the key mission of Digitising European Industry the following critical points should be addressed going forward in ECSEL:

a. There is a need to develop a global overarching strategy supported by commitment from all parties involved for Electronic Components and Systems that addresses the vertical integration chain

b. There is a need to synchronise national activities, harmonise participation rules, funding rates and procedures

c. There is a need to place greater emphasis and target resources on coverage of the value chain, particularly with respect to systems

d. Instruments should be created targeted at encouraging more SME and start-up participation in order to more closely meet the goals sets within H2020 (20% of allocated budget allocated to SMEs)

e. There is a need to put in place appropriate metrics and compulsory follow up to assess the impact of projects to justify funding commitments from the EU, Member States and Industry

In addressing these critical points we recommend:

a. Strengthening the involvement of system industries, in particular, by enforcing that the ECSEL Lighthouse projects are driven by systems companies

b. At the highest political level a discussion should be initiated to harmonise and synchronise the Member State participation rules, funding rates and procedures wherever possible, adopting best practice as the guiding principle

c. Development of an industry driven top-down overarching strategy concentrating on the integration of CPS and electronics with a focus on the key application domains such as automotive, health, energy, etc.

d. Capitalise on the innovation capabilities of start-ups and SMEs by fully integrating them into innovation ecosystems for the key targeted market segments organised around Lighthouse projects, such as by developing open platforms

e. Although it is clear that the JU outcomes are resulting in impact, in order to more accurately measure impact and success of projects appropriate metrics should be collected during project execution and also subsequently after a project is completed. In order to benchmark the results at an international level it is important to also provide assessment from international experts.

8

1.1 Extended Executive Summary

The ARTEMIS and ENIAC Joint Technology Initiatives (JTIs) and Joint Undertakings (JUs) were among the first to be established under the FP7 framework programme in February 2008. The intended role and objective of both ARTEMIS and ENIAC JUs was to support Research and Innovation (R&I) in Information and Communication Technologies (ICT) with the remit to develop and implement strategic research programmes in the areas of embedded systems (ARTEMIS) and nanoelectronics (ENIAC). Both initiatives were tri-partite public-private partnerships (PPPs) jointly funded by industry, research organisations, participating Member States and the European Commission (representing the EU). This bringing together of funding allowed the JUs to tackle problems that could not be addressed by single funding sources alone and by so doing create significant impacts. At the same time the JUs were to act as a catalyst to align fragmented research efforts across Europe under a common agenda and increase and leverage private and public investment in the related sectors in Europe. The overall aim was to strengthen Europe's future growth, competitiveness and sustainable development, in their respective fields by enabling greater coherence of R&D across Europe. The JUs ran for 6 years until 2013 when they were combined into the ECSEL JU. This report provides a final assessment of the ENIAC and ARTEMIS JUs over the period 2008-2013 considering:

Effectiveness: progress made towards meeting the objectives set

Efficiency: extent to which the JUs were managed and operated efficiently

Research Quality: extent to which the JUs enabled world-class research that helped Europe to establish a leadership position globally, and how it engaged with a wider constituency to open the research to the broader society

In answering all of these questions conclusions and recommendations are put forward which will be used to inform the European Parliament and Council, national authorities, the research community and other stakeholders on the final outcome of the ARTEMIS and ENIAC JUs under FP7. Additionally, these will be used to improve the implementation of the ECSEL Joint Undertaking operating under Horizon 2020 and will also be used to provide credible and evidence-based input for the design of the next generation of Joint Undertakings. The results of the evaluation will also be used as input to the debate on the future research and innovation policy for the overall research framework programme after 2020. To perform the evaluation an Expert Panel was formed consisting of independent experts specifically chosen to provide both a deep knowledge of the embedded systems and nanoelectronics fields, as well as general expertise in R&D strategy and management. The Evaluation Panel analysed relevant documentation and interviewed representatives of the two JU communities including industrial participants, representatives of national public authorities, and staff of the European Commission.

1.1.1 Importance of Electronics Components and Systems Industries

Cyber-Physical Systems (CPS) are a pervasive enabling technology, which are impacting all industrial sectors and almost all aspects of society. Emerging industrial platforms such as the Internet of Things (IoT), Industrial Internet [3] and Industrie 4.0 [4] are triggering a “gold rush” toward new markets and are creating societal-scale systems, combining computational, physical and human components. Within Europe this is being driven by the Digitising European Industry initiative [5]. The aim of this is to establish next generation digital platforms and re-build the underlying digital supply chain. Here initiatives such as ARTEMIS, ENIAC and now ECSEL have a pivotal role to play supported by other initiatives funded under Framework 7 and the H2020 Work Programme. Current activities under H2020 are addressing platform-related projects and large-scale integration, testing and experimentation pilots. The aim of the H2020 large-scale pilots, which are co-funded by Member States, is to remove cross-border obstacles that currently prevent large-scale testing and

9

experimentation thus blocking the full deployment of technologies into the market. This is particularly relevant for areas such as autonomous connected vehicles and connected smart factories. There are also aims to develop Europe-wide facilities for experimentation to foster rapid development of ICT standardisation leading to new standards.

At a component level there is increasing competition in the semiconductor market, particularly from China [6]. The Ensuring Long-Term U.S. Leadership in Semiconductors report [7] by the President's Council of Advisors on Science and Technology (PCAST) (January 2017) states that the "concerted push by China to reshape the market in its favour, using industrial policies backed by over one hundred billion dollars in government-directed funds, threatens the competitiveness of the U.S. industry and the national and global benefits it brings“. The report concludes that: "only by continuing to innovate at the cutting edge will the United States be able to mitigate the threat posed by Chinese industrial policy and strengthen the U.S. economy”. The report recommends a three pillar strategy to (i) push back against innovation-inhibiting Chinese industrial policy, (ii) improve the business environment for U.S.-based semiconductor producers, and (iii) help catalyse transformative semiconductor innovation over the next decade. Delivering on this strategy will require co-operation among government, industry, and academia. The same considerations hold true for Europe.

1.2 Key Findings

In terms of leveraging investment the ENIAC and ARTEMIS JUs have been successful in increasing the private and public investment in the two sectors. 119 projects have been funded with €630 million of EC funding leveraged with €912 million of national contributions. Industry has also contributed €2.46 billion of funding overall resulting in €4 billion being targeted at research and innovation in the two sectors. The financial commitment by Member States and industry increased as the JUs adopted an approach to funding larger-scale projects. In the case of ENIAC large-scale pilot projects have been performed to develop new manufacturing processes. This has kept European companies at the forefront in key microelectronics and nanoelectronics areas. The embedded systems domain covered by ARTEMIS is driven by applications and here the concentration has been on development of software and hardware that can be used in multiple application domains. This diversity of applications made it more difficult for Member States to commit to funding ARTEMIS in the early years. The adoption of larger-scale projects at higher TRL levels in the later calls of ARTEMIS renewed Member State interest and more importantly led to a significant increase in commitment from Industry. A number of industry driven projects, e.g. CRYSTAL, EMC2, have been funded which have had a high impact.

Another major goal of setting up the ARTEMIS and ENIAC JUs was to foster collaboration by providing a focus for research efforts for key stakeholders such as industry, including small and medium-sized enterprises (SMEs), national authorities, the academic community and research centres. Both JUs have made considerable efforts to align their activities with other programmes which have similar goals, e.g. CATRENE in the case of ENIAC. The EC has also endeavoured to create linkages between projects funded under relevant areas, e.g. Smart CPS with ARTEMIS and ECSEL. This has also been supported via the organisation of a number of joint events to bring the two communities together. Even so, the analysis of the projects undertaken and their coherence with other EC and National programmes still shows overlaps and a lack of linkage. Surveys of JU project participants also confirms that despite clear synergies more work is needed to increase interaction.

There is a pressing need to clearly define an overarching strategy that can then be used to help align activities with other national programmes. This was recommended at an interim review of the two JUs which highlighting the need for a top-down strategy rather than a bottom-up strategy. This was

10

partially achieved with the move towards pilot lines in the case of ENIAC and large application driven projects in ARTEMIS but still more work needs to be done. Looking to the future under ECSEL there are opportunities to align with the PENTA EUREKA initiative and also with other JUs and initiatives that are application oriented, e.g. Factories of the Future, Robotics, the EIP on Active and Healthy Ageing, Big data and IoT. The proposed ECSEL Lighthouse projects that will cluster activities into umbrella domains, e.g. smart factories and smart mobility, are seen as a very good opportunity for bringing research in specific domains together.

Engagement with the SME community which accounts for 99% of companies across Europe (20.7 million), 2/3 of European jobs and 85% of new jobs, was also a goal of the two JUs. In this respect there has been some success with 29% of ARTEMIS and 27% of ENIAC applicants being SMEs. The area of embedded systems represented by ARTEMIS has a much higher SME industrial base and as a consequence 19% of the ARTEMIS budget was allocated to SMEs (close to the Horizon 2020 target of 20%). Notably ENIAC is dominated by a few larger companies and as a result a small proportion of budget was allocated to SMEs (10%) with a preference to fund pilot line initiatives driven by large industry. With the combination of ARTEMIS and ENIAC into ECSEL the current annual level of SME funding is 12-13.5%. Here there is a need to support SMEs more strongly and this could be done by a number of means, therefore, it is recommended that the allocation of funding to support smaller scale experiments (e.g. €50K – €100K) is considered to encourage easy access to ECSEL and provide connection mechanisms to larger companies.

The ARTEMIS and ENIAC JTIs were two of the first JUs to be established in 2008 and considering operations it was to be expected that there would be some initial problems. This valuable experience has been exploited in the organisation of new JUs but something which is unique to ARTEMIS and ENIAC is the adoption of a tri-partite funding strategy. Considerably more effort is required to co-ordinate Member States which leads to a number of administration complexities. In order to justify their funding commitments the Member States played an active role in the governance of the two JUs, defining strategy, funding priorities and also in the selection of projects. As a result of funding obligations some Member States also perform their own project monitoring in addition to that performed by the JU. The involvement of many Member States across Europe presents challenges. Elections change governments in a number of these every year with consequent changes in national funding priorities. This makes it a significant challenge to co-ordinate, reach consensus and guarantee long-term funding within the JUs. The creation of Multi-Annual Strategic Plans for the JUs has addressed this to some extent but it has still not been possible to gain commitment from the majority of Member States for multi-annual funding. Notably there is a need to develop trust to decrease the additional administrative reporting burden on project leaders imposed by some Member States. Here there is a need to harmonise and synchronise Member State participation rules, funding rates and procedures. Although there was engagement at CEO level in ENIAC, for ARTEMIS there was less CEO engagement especially from application domains, e.g. aerospace, automotive, etc., and it was noted that Board Members spent considerable time addressing lower level decisions, leaving less time for high-level strategic discussions. Participation of high-level industry representatives would add credibility to the strategies formed at a European level and would also help in the gaining support for Member State funding.

Key to justifying the existence of the JUs and the funding allocated to them by the EC, Member States and Industry is the impact that the projects performed have had in their domains. The high level goal of ARTEMIS was that ARTEMIS technology outcomes would be used in 20% of the embedded systems in the world. For the ENIAC JU the target was to create 250,000 jobs in Europe. Both of these are challenging goals and practically attainment of these impacts is hard to measure. Some projects have yet to finish and notably it will take time for the outcomes to be transferred into products and translate into jobs.

11

The evaluation panel thus considered other impacts that could be measured:

Engagement with the community

Inclusion of SMEs

Number of patents and innovation outputs

Success in attracting public/private funding

Number of projects initiated, success stories and evidence of impact from projects

Working Groups established to support the Community

This identified a number of positive impacts including good engagement with the community, positive SME participation, filing of a number of patents (ARTEMIS 28, ENIAC 209), dissemination via publications (ARTEMIS 1460, ENIAC 2381), good success in achieving industrial funding, in particular, the funding of 119 industry relevant projects and the establishment of a vibrant set of Working Groups in the case of ARTEMIS. Additionally, the participation across Europe of 1420 entities (ARTEMIS) and 1384 entities (ENIAC) with 1000’s of full time researchers being engaged on projects, resulting in additional further direct and indirect employment, is also seen to be a catalyst for the two communities.

For the ARTEMIS JU the monitoring of impact had been addressed in rigorous manner by setting up a Working Group specifically to address monitoring of KPIs. Targets were also set such that projects delivered cross-domain re-use and interoperability for different product categories and application domains, or promised a reduction of system design costs and development cycles for both hardware and software. In many cases the projects expected commercial impact within a 3 to 5 years’ time. The development of SW design tools, environments and general purpose architectures for integration and reuse across sectors are important in the automotive, aerospace, factory automation, industrial processes, smart buildings, energy production and medical/healthcare sectors. Shorter development cycles and increased interoperability enable shorter time-to-market for new products and services in the CPS domain and the Internet of Things market. Also for ARTEMIS a number of success stories could be cited such as the AUTOSAR standard for automotive which was supported by activities performed. This is now used world-wide. Projects such as EMC2 (mixed criticality applications), CRYSTAL (design environment integration) and ARROWHEAD (cooperative automation) have addressed pan-European industry issues developing frameworks, platforms for interoperability, tools and methods to cope with the ever increasing complexity of smart digital systems. Examples of exploitation of outputs, company growth and world leadership can be seen in companies such as TTTech that provides core safety-critical data bus technologies to Airbus, Boeing and NASA.

In the case of ENIAC there had been no formal follow up of impact metrics, however, there is evidence of impact in a number of areas. Funding from ENIAC had allowed AMS to transform itself from a foundry with commodity products into a specialist for producing sensors and sensor systems, Infineon Technologies Austria had developed a 300mm fab. capability and was now a world leader in power electronic discretes and modules, ST Microelectronics had exploited project outcomes in its

piezoMEMS market and deep sub-micron pilot fab capabilities, and ASML had become a leader in lithography selling equipment around the world.

In this section a number of recommendations for the future are made. As the ARTEMIS and ENIAC JUs have now formally finished the recommendations made apply directly to the ECSEL JU that combines the ARTEMIS and ENIAC JUs. Some recommendations also have relevance across JUs.

12

Strategic Recommendations

Recommendation 1

Develop overall ECS strategy

Action: JU with EC and Industry CEOs

Although the MASP and Annual Work plans set the short term goals of ECSEL well, there is a need to clearly define a long-term, top-down research, development and innovation strategy in Electronic Components and Systems (ECS) to co-ordinate activities within the ECSEL JU while still leaving room for bottom-up initiatives. The overall strategy should concentrate on the integration of different value chains (equipment, semiconductor, system integration, photonics, software) and large-scale pilot experiments driven by real needs in key application domains such as automotive, health, energy, etc. CEO level industry leaders, helped by the European Commission and ECSEL JU as facilitators, should create a "Digital Leaders Group" (on the model of the ELG – Electronics Leaders Group), to develop a strategy for digital industry in Europe. There should be a clear link with activities in the (DEI) Digitising European Industry initiative where CEOs of application domain companies are already engaged. It is important to understand the concerns of CEOs, provide information in their language and develop an accessible value proposition for technology. There is also a need to provide a forum at CEO level for cross-company, cross-domain priorities. In the next phase ECSEL needs to have greater levels of vertical integration to fulfil its mission to cover the full ECS domain and avoid dominance by single technology areas. In particular, there needs to be a clear value proposition for all pilot line activities that is driven from vertical Lighthouse projects.

These issues should be addressed in the proposal evaluation and selection criteria to improve the match of the project portfolio to strategic European aims. This is to ensure optimum coverage of key areas defined in the overarching EU ECS strategy.

Recommendation 2

Closer community integration

Action: IAs

It is still early days with respect to the integration of activities within ECSEL. In the future there needs to be greater integration of the AENEAS, ARTEMIS-IA and EPoSS communities, that each provide different perspectives but overlap on technologies, to focus on integrated vertical roadmaps so that the European benefits of investing are realised. The three organisations have a leading position in their respective domains and each produce valuable Strategic Research Agendas. However, considering the whole community these should be better integrated into a single roadmap.

Recommendation 3

Emphasise vertical integration

Action: JU

For selection of proposals to be funded the vertical integration aspect should be a dominant evaluation factor to promote greater cross-value chain co-operation. This should be supported by providing funding rates that are related to TRL levels to distinguish between pilot line activities and systems-level integration. In particular, there should be TRL-aligned funding rates in Lighthouse projects. Moreover, it is important that proposals include measurable target results, so that the improvement over the state-of-the-art can be evaluated and compared across different technological areas like nanoelectronics, CPS and smart systems.

13

Recommendation 4

Harmonise Member State participation rules

Action: EU with Member States

At the highest political level a discussion should be initiated towards harmonisation and synchronisation of the Member State participation rules, funding rates and procedures wherever possible, adopting best practice as the guiding principle. Also at the highest political level efforts should be made to encourage Member States to commit to a multi-annual funding system to provide stability and a longer-term focus. This recommendation was made at the 2nd Interim Review of the ENIAC and ARTEMIS tri-partite PPPs. The issues have proved to be difficult to address and still remain a barrier. Thus the recommendation has been strengthened to emphasise the need for very high-level engagement for resolution.

Recommendation 5

Increase SME and start-up involvement

Action: JU

In terms of numbers the engagement with SMEs is very positive. The percentage of budget allocated to SMEs was 19% for ARTEMIS and 10% for ENIAC. Given the importance of SMEs the three Industrial Associations and JU should play a more active role in facilitating the engagement with SMEs. The JU should put in place a strategy to improve SME engagement and the level of funding being allocated to SMEs. The success of this should be monitored. This could be done by a number of means such as the allocation of funding to support smaller scale experiments (e.g. €50K - €100K) to encourage easy access to ECSEL and provide connection mechanisms to larger companies. A plan for supporting start-ups should be considered with actions to align national and regional investment in Europe for innovation such as the ARTEMIS Centres of Innovation Excellence.

Recommendation 6

Engage with CEOs across the Systems Community

Action: JU

As highlighted in Recommendation 1 in order to define a global overarching strategy there is a need for CEO engagement. Going beyond this there is a need to better engage the CEOs of systems companies to encourage participation in ECSEL addressing key issues, not only strategy, but also funding levels, project types and success rates in order to make ECSEL an attractive proposition for industry. All parameters such as funding levels, project types and success rates should be revisited to ensure the achievement of innovation across complete vertical value chains targeting European needs.

Recommendation 7

Explore synergies with other JUs and LSPs

Action: JU

The JU should explore synergies with the Large-Scale Pilots and other relevant JUs to maximise the benefits of work and ensure coherence with other initiatives. Here the working groups set up by EPoSS addressing sectors such as automotive, healthy living, manufacturing and robotics may have a role to play.

14

Operational Recommendations

Recommendation 8

Promote ECS Strategy

Action: JU

Based on an overall top-down Electronic Components and Systems (ECS) strategy targeted calls should be made to address sectors and Brokerage Events should be used to promote the sectoral integration of projects around topics of importance to Europe, in which European companies have a chance to remain/become leading global players or gain market share.

Recommendation 9

Monitor impact post project

Action: JU

Although it is clear that the ARTEMIS, ENIAC outcomes are resulting in impact, in order to more accurately measure impact and success of projects appropriate metrics should be collected during project execution and also subsequently after a project is completed (e.g. 12 months after project end). Although it is difficult to give quantitative metrics for the impact of complex projects there is a need to gather appropriate metrics from projects to support an assessment of impact against the goals of the JU. It is recommended that some budget is reserved for post project evaluation and this should be mandatory for future projects funded under the ECSEL JU. This recommendation was made previously at the 2nd Interim Review of ENIAC and ARTEMIS. ECSEL representatives highlighted that in practice that it had not been possible to follow this up in ECSEL due to lack of resource. Thus the need for budget allocation to support this and the mandatory nature of the review is highlighted.

Recommendation 10

Benchmark internationally

Action: JU

In order to benchmark at an international level it is important to also provide assessment from international experts. It is recommended that reviewers from other geographies such as Asia and the US should be considered during project selections with a view to ensuring that the projects being selected are truly world class. If such is the case, then the project goals will deliver impacts which will enable those stakeholders to win on the global stage. Care needs to be taken, however, in selecting the reviewers so as to avoid any conflicts of interest which might result in competing international industries gaining insights into European ideas. This could be achieved by allowing stakeholders a veto on international reviewer selections.

Recommendation 11

Trace reuse of project results

Action: JU

For traceability of impact and to show how projects build upon previous results it is important to maintain a log of how project results are used from project to project. This has been done to a certain extent but it would also be useful to also indicate the "share of reuse", i.e. an assessment of funding reuse with respect to the full budget. Additionally, this should apply to reuse both into and from other H2020 programmes. This will become more important as Lighthouse projects are initiated via ECSEL which will cluster activities funded by different sources. This recommendation was made at the Interim Review of ENIAC and ARTEMIS but is re-iterated as it will become more important as ECSEL engages in more collaborative activities.

Recommendation 12

Reduce

The administrative burden should be reduced so that the JU Governing Boards can spend more time addressing strategic issues. This was recommended at the 2nd Interim Review of ENIAC and ARTEMIS and

15

administration and concentrate on strategy

Action: JU

interviews of ECSEL board members highlighted that progress had been made to reduce overhead. However, this was still considered to be a barrier and although AENEAS, ARTEMIS-IA and EPoSS have very good industrial representation from key companies, higher level involvement is encouraged.

Recommendation 13

Reduce management overhead for participants

Action: JU

Specific support mechanisms should be developed to enhance project management processes. A key recommendation is that management costs should be 100% funded by the EC for all JTI projects. This recommendation was made in the 2nd Interim Review of ENIAC and ARTEMIS but was not taken up and has not been implemented in ECSEL. Survey respondents highlighted that management overhead of engaging with ARTEMIS, ENIAC and ECSEL was a barrier to engagement, so it is still considered to be an issue. Additionally, it is recommended this time that lessons learned should be collected from projects with the aim of improving management effectiveness by analysing project communication process models, tools and practises among project participants, identifying best practices and introducing them into the project management schemes to streamline processes. The outcomes of this work could also be exploited by other JUs.

Recommendation 14

Streamline review and reporting processes

Action: JU, Member States

Projects should be subject to a unique review and reporting process to avoid an unnecessary overhead. Efforts should be made to harmonise the financial reporting process, remove duplication and encourage Member States to develop trust in the level of reporting provided to the JU. A level of trust may be partially achieved via improving metrics collection and via impact assessment of projects. This recommendation was made at both the 1st and 2nd Interim Reviews of ENIAC and ARTEMIS. This has been addressed but it is acknowledged that this is a complex issue and it has yet to be resolved. Notably survey respondents highlight this as being a key issue which merits the recommendation to still consider ways of simplifying procedures.

Recommendation 15

Harmonise re-imbursement rates

Action: Member States

There is a need to harmonise re-imbursement rates at a national level as the current system makes engagement of actors from some countries in ECSEL unattractive. Much has already been achieved by the Public Authorities Board (PAB) in bringing the Member States together, but this issue remains largely unresolved. Here there needs to be further action from Member States.

As well as the previously described recommendations, a number of observations were also made, the learnings from which could be applied to other JU programs.

Observations

Observation 1 It is clear that the JU approach has been successful in bringing together EU, national and industrial actors to pursue common goals. The two JUs have engaged 1420 entities (ARTEMIS) and 1384 entities (ENIAC) with 1000’s of full time researchers working on projects. Originally initiated with goals to strengthen the embedded system industry and the semiconductor industry it was recommended at the second interim review that an integrated strategy

16

should be pursued with a view towards merging the two initiatives in ECSEL. Subsequently ECSEL has made great progress towards integrating the Electronic Components and Systems domains but still work is needed to achieve the ultimate goal of creating a global strategy across Europe.

Observation 2 The ENIAC and ARTEMIS JUs have been successful in increasing the private and public investment in the two sectors. 119 projects have been funded with €630 million of EC funding leveraged with €912 million of national contributions. Industry has also contributed €2.46 billion of funding overall resulting in €4 billion being targeted at research and innovation in the two sectors. Notably the financial commitment by Member States and industry increased as the JUs adopted an approach to funding larger-scale projects.

Observation 3 For ARTEMIS there has been good engagement across the embedded systems community driven by end user applications in safety-critical systems, automotive, manufacturing, aerospace, telecommunications, industrial software and medical technologies. Many projects result in demonstrators of technologies at a high TRL and a number of projects e.g. CRYSTAL, EMC2, have had a high impact.

Observation 4 ARTEMIS has engaged key European players including major aircraft companies such as EADS and THALES, medical electronics companies such as Philips, automotive drivetrain specialists AV LIST, car manufacturers such as Daimler, Volvo and Fiat, and well-known research organisations across Europe. There is also strong participation from semiconductor manufacturers such as Infineon and NXP who embed their electronics into a number of applications. Notably Fraunhofer, Infineon, STMicroelectronics, NXP Semiconductors, CEA, TU Eindhoven, TNO and TU Delft are part of the core network of both the ARTEMIS and ENIAC JUs. 51% of the surveyed ARTEMIS participants [37] assess that investments have been made by their organisation as a result of their participation in the projects.

Observation 5 For ENIAC the KET initiated pilot lines had resulted in 9 projects focused on consolidating technologies on 200mm platforms, strengthening 300mm infrastructure and enabling the future transition to advanced CMOS technologies such as FDSOI. In assessing the impact on the nanoelectronics sector, 88% believed that the work of the ENIAC JU would create commercialisable innovative outputs. In regards to job creation, 75% believed that the projects would have a high impact.

Observation 6 In ENIAC there is a strong participation from semiconductor manufacturers and research centres in the field with a lower participation of end users. A notable end user company is Philips from the medical domain where there is interest in miniaturised electronics for in-vitro applications.

Observation 7 Interviews performed by CARSA [37] highlighted that 71% of beneficiaries and stakeholders believe that the ARTEMIS and ENIAC JUs have brought together the best competencies in the industry across the EU. The cross-disciplinarity and trans-nationality of the players is perceived as a strength which fosters innovation across the regions. Additionally, respondents highlighted that the technology readiness level and manufacturing readiness is substantially increased in ENIAC and ARTEMIS projects.

17

Observation 8 In surveys the added value of the JU’s was rated positively for both ENIAC and ARTEMIS. The most positively evaluated aspects are related to the JUs’ contributions to “dialogue between researchers and industry” and “achievement of critical mass”.

Observation 9 The SME participation was 29% in ARTEMIS and 27% in ENIAC while the share of the budget to SMEs was 19% for ARTEMIS compared to 10% in ENIAC. It is also notable that research organisations (HEIs & RES) received a comparatively higher share of funding in ARTEMIS than in ENIAC. The embedded systems domain is characterised by a high number of SMEs and start-ups. Surveys (See CARSA survey [37]) rated the efforts to involve SMEs more positively for ARTEMIS than for ENIAC. In the case of ENIAC it was highlighted that the market structure is geared towards larger companies.

Observation 10 In terms of impact ARTEMIS-IA questionnaires in 2014 [48], highlighted that the ARTEMIS JU had had a significant and pivotal business impact on reducing development costs (53% of respondents), had contributed to reduced time-to-market (41%) and had developed a new generation of products (38%). Here the ARTEMIS JU has been successful in bringing organisations together to develop prototypes and demonstrators. In 2014, 95% of the respondents indicated that a key aim was to build application prototypes [48].

Observation 11 Surveys indicated that the end user involvement in ARTEMIS was perceived as being very strong. Operation at higher TRLs has led to an industry focus bringing together the necessary actors to address specific applications. Many of the projects envisage providing R&D results with commercial impact within a 3 to 5 years’ time in the automotive, avionics, space, factory automation, healthcare and Internet of Things application areas. Therefore, ARTEMIS has been successful in bringing together the embedded systems community around a number of core domains to tackle “big issues”.

Observation 12 The perceived added value of ARTEMIS was rated positively with a number of benefits being highlighted such as development of a platform approach, the ability to focus work to get a fast impact in a newly emerging technology area, and the ability to uniquely perform large-scale projects. Representatives from public and private sector also believed that the programme encouraged a better dialogue between researchers and industry in the sector, which enhanced cross-border and interregional co-operation.

Observation 13 For ENIAC interviewed representatives considered that two of the most important added values of the programme were in fostering a greater dialogue between all involved actors and significantly enhancing cross-border co-operation. Both public and private representatives highlighted that the programme ensured a better dialogue among Member States and between industry and researchers. In terms of cross-border co-operation ENIAC also provided a means of extending national clusters to EU-scale ecosystems and of creating a critical mass at the ecosystem level.

Observation 14 Both ENIAC and ARTEMIS surveyed participants believed that investments by their organisation had been facilitated due to their participation. In the case of ENIAC several representatives could cite examples where investments were made related to the large-scale pilot lines; a specific example being the production of Vertical-Cavity-Surface-Emitting Lasers (VCSEL) by Philips in Ulm

18

based on the success of the ENIAC project ViDaP. For ARTEMIS there is some factual evidence that projects have led to significant increases in productivity and reduced time-to-market after the adoption of innovative tools and methodologies. Surveyed ARTEMIS representatives highlighted increased European collaboration on areas such as platforms, on a pan-European ecosystem, new communication frameworks (e.g. TTP and FlexRay), reference architectures and tool chains. For ARTEMIS a good example of success is the company TTTech which has strategically moved into different sectors with the support of a number of EU projects and notably via the EMC2 and CRYSTAL projects. Similar examples of success for ENIAC can be cited such as AMS, Infineon Technologies Austria, ASML and Carl Zeiss SMT.

Observation 15 The support from the JUs is a key enabler. The data produced by CARSA [37] indicated that 88% of respondents believed that the research work would not have been completed were it not for the funding received from the ENIAC JU and 78% agreed that the topics being pursued would require further support from the JU.

Observation 16 ENIAC projects led on average to 4.7 patent applications per €10 million of EU funding which is higher than the Horizon 2020 benchmark of 3 patent applications per €10 million of EU funding. 28 patents were filed and ARTEMIS projects led on average to 1.54 patent applications per €10 million of EU funding. Here it should be noted that ARTEMIS has had less than half the EU funding that was put into ENIAC and patenting plays a different role in the two areas, leading to different patenting strategies. ARTEMIS projects produced 1460 scientific publications whilst ENIAC projects produced 2381 scientific publications.

19

2. INTRODUCTION

2.1. Purpose of the Evaluation

This report presents a final evaluation of both the ARTEMIS and ENIAC Joint Undertakings [8, 9] considering three main aspects:

Effectiveness: progress made towards meeting the objectives set

Efficiency: extent to which the JUs were managed and operated efficiently

Research Quality: extent to which the JUs enabled world-class research that helped Europe to establish a leadership position globally, and how it engaged with a wider constituency to open the research to the broader society

The evaluation covers the entire implementation period of ARTEMIS and ENIAC Joint Undertakings (2008-2016). These were set up in 2008 to address the objectives of the political and socio-economic situation in 2006-2007 with the overriding aim to contribute towards the development and implementation of strategic research programmes in the areas of nanoelectronics and embedded systems, respectively. The key roles envisioned for the ARTEMIS and ENIAC JUs were to achieve an effective co-ordination and synergy of resources and funding from the 7th Framework Programme [10], industry, national R&D programmes and intergovernmental R&D schemes. By bringing together funding sources and key stakeholders the intention was to strengthen Europe's future growth, competitiveness and sustainable development in their respective fields. Two previous interim evaluations of the JUs have been performed [11, 12] and recommendations have been adopted to take into account that the original specific objectives of the ARTEMIS and ENIAC Joint Undertaking in 2008 have changed significantly during the period of programme implementation. Other recommendations made by stakeholders such as the High Level Group on Key Enabling Technologies (KET) [13] have also been taken into consideration as well as the implications from new emerging fields such as Industry 4.0 [14] and the Internet of Things (IoT) [15]. The Evaluation Panel comprised a mix of independent experts (See Annex 2). The backgrounds of the experts were specifically chosen to provide both a deep knowledge of the embedded systems and nanoelectronics fields, as well as general expertise in R&D strategy and management. The Evaluation Panel drew upon both published information (See Annex 1) and a wide range of interviews with representatives of the two JU communities including industrial participants, representatives of national public authorities, and staff of the European Commission (See Annex 5). The final evaluation considers the objectives and their relevance, implementation, the main achievements, effectiveness of implementation, performance of the JUs, European added value and coherence. It also considers how well previous recommendations from the interim evaluations were addressed and highlights any lessons learnt. The key findings of the Evaluation Panel are summarised in Conclusions and Recommendations. The results of this evaluation will be used to inform the European Parliament and Council, national authorities, the research community and other stakeholders on the final outcome of the ARTEMIS and ENIAC JUs under FP7. Additionally, the results of this final evaluation will be used to improve the implementation of the ECSEL Joint Undertaking [16] operating under Horizon 2020 [17]. It will contribute to the formulation of the 2018-2019 ECSEL JU Annual Work Plans and serve as a basis for the ex-ante impact assessment of the next generation JUs. More specifically, the results of the evaluation will provide credible and evidence-based input for the design of the next generation of

20

Joint Undertakings and will also feed into the debate on the future research and innovation policy for the overall research framework programme after 2020.

2.2. Scope of the Evaluation

The evaluation covers the progress made with respect to the specific objectives of the JUs as set out at their foundation and also examines the follow-up and implementation of recommendations from previous evaluations. The approach taken has been to address 8 key evaluation criteria covering:

Background of the initiative, objectives and relevance - Considering the situation before the set-up of the Joint Undertakings, the context and background to their establishment, the policy and regulatory framework used, and assessment of the intervention logic, the relevance of its objectives and the interactions between different measures. Implementation of the JUs - Considering the calls launched, participation patterns by country, region, thematic topics and beneficiary organisation types, success rates, and budget share considering EU, National and Industrial contributions, average grant sizes, number of beneficiaries and distribution of funds by country, region, activity type and thematic area. Main achievements and effectiveness - Considering the various outputs and results produced and the extent to which the scientific, technical and industrial outputs affected the longer-term economic and societal impact of EU-funded research and innovation.

Effectiveness of Implementation - Considering the research agenda, work programme, process for call publication, evaluation, selection, and contract/budget negotiation, and the participation of the best European players and their relevance to the JU specific areas.

Joint Undertaking's performance – Considering the mission, legal structure, governance and operation modalities, financing, Strategic Research Agenda, operational effectiveness, efficiency, ease and timeliness of budget execution and overall satisfaction of participants.

The lessons learned from the previous evaluations (ARTEMIS – ENIAC) - Examining the follow-up and implementation of recommendations from previous evaluations.

European added value - Analysing the leverage effect of each JU in attracting additional finance and the scale of resources involved.

Coherence - Considering the coherence with other interventions, e.g. FP7 ICT or H2020 LEIT with similar objectives, the relation with other Union funding programmes and synergies with similar national, international, national and intergovernmental programmes to encourage optimal use of resources and avoid unnecessary duplication.

Synthesis, conclusions and recommendations - Considering the validity and justification for the creation of the JU, attainment of objectives and contribution to overall Framework Programme objectives, openness, transparency, effectiveness and efficiency of implementation, added value, leveraging of funds, and synergy with other policies.

The Panel has consulted a wide range of documentation and interviewed key stakeholders to gather relevant evidence to rigorously address each of these questions. The outcomes of this work is described in the following sections.

21

3. BACKGROUND TO THE INITIATIVE

3.1. Description of the Initiative, Objectives and Relevance

The Joint Technology Initiatives (JTIs) are public-private partnerships that perform industrial research at the European level [18]. They were set up in 2007-2008 under the Seventh Framework Programme (FP7) in five strategic areas — aeronautics and air transport, public health, fuel cell and hydrogen technologies, embedded computing systems and nanoelectronics. The aim of the JTIs are to bring together industry, the research community, Member States, regulators and the EU to define and implement common research agendas and invest in large-scale multinational research activities. By bringing resources and stakeholders together in this way the aim is to strengthen competitiveness through scientific excellence, industry led research, openness and innovation. They support the development and implementation of research and innovation activities of strategic importance to the European Union's competitiveness and industrial leadership and/or address specific societal challenges. The JTIs were set up as Joint Undertakings (JUs) under Article 187 of the Treaty on the Functioning of the EU (TFEU) [19], which states that “the Union may set up Joint Undertakings or any other structure necessary for the efficient execution of Union research, technological development and demonstration programmes”. They are “Union bodies” under Articles 208 and 209 of the EU Financial Regulation [20]. The European Commission, as a co-founding member, was responsible for setting up the JUs. Once they had built up their legal and financial framework and demonstrated their capacity to manage their own budgets, they were granted autonomy. Under the FP7 framework programme the ARTEMIS and ENIAC Joint Technology Initiatives (JTIs) and Joint Undertakings (JUs) were among the first to be established in February 2008 by the Council Regulations 72/2008 and 74/2008 [21, 22]. These two JUs had the remit to develop and implement strategic research programmes in the areas of embedded systems (ARTEMIS) and nanoelectronics (ENIAC), respectively. Both are tri-partite public-private partnerships (PPPs) jointly funded by industry, research organisations, participating Member States and the European Commission (representing the EU). This tri-partite model was adopted because of the pre-existing transnational “EUREKA Clusters” ITEA2 [23] and CATRENE [24] in the ARTEMIS and ENIAC technical fields.

Initially funded under FP7, they were continued under H2020 following a review of existing JU experience. The European Commission published a series of proposals for Council regulations on public-private and public-public partnership initiatives under Horizon 2020 and in May 2014, the Innovation Investment Package [25] was officially adopted by the EU Member States. This set out the framework for a new generation of public and private partnerships, which will pool research and innovation investments of more than €22 billion. The package includes nine Joint Technology Initiatives that organise their own research and innovation agenda and award funding for technology specific projects on the basis of competitive calls.

The JUs bring together EU, national and private resources, know-how and research capabilities, with the aim of addressing major issues by sharing knowledge, achieving critical mass, scale and scope. The aim is to help the EU to become a world leader in developing breakthrough technologies with high innovation potential. Running since 2008 with calls for projects being issued annually a total of 119 projects have been funded. The final projects funded by the two JUs should be completed by December 2017.

22

3.1.1 Council Regulations

The “formal specifications” for the ARTEMIS and ENIAC JTIs are contained in the Council Regulations [21, 22] that provide the legal basis for each. The regulations set out the administrative requirements to ensure the probity of the operation of the JTIs. Additionally the regulations contain both operational and outcome-oriented objectives. Operational Objectives The operational objectives outline the details of how the Public Private Partnerships should be formed and the operational processes to be adopted:

Regular review and updating of the research agenda and a Multi-Annual Strategic Plan

The preparation and implementation of an Annual Work Programme

Calls for, evaluation and selection of proposals

Funding and oversight of selected projects Outcome-Oriented Objectives

Co-ordinate resources and funding from all sources in order to make a significant contribution to the European Research Area by achieving greater coherence of R&D across Europe

Achieve higher efficiency by harmonising procedures and removing uncertainty as to the availability of national budgets (as has been experienced in EUREKA) and (“when added value can be created”) integrating related EUREKA activities into the JTIs

Increase overall private and public investment in the two sectors

Contribute beyond Research & Technology Development to the research and innovation ecosystem - encompassing the participation of SMEs, the enhancement of education and training, and contribution to standards

3.1.2 Role of ARTEMIS and ENIAC JUs

The role of both ARTEMIS and ENIAC JUs was to support Research and Innovation (R&I) in Information and Communication Technologies (ICT), as well as for combining national and Union funding. A central element of their implementation was increasing and leveraging private and public investment in the related sectors in Europe. They were to achieve effective co-ordination and synergy of resources and funding from the 7th Framework Programme, industry and national R&D programmes, thus contributing to strengthening Europe's future growth, competitiveness and sustainable development, in their respective fields. Finally, they were to foster collaboration between all stakeholders such as industry, including small and medium-sized enterprises (SMEs), national authorities, academic and research centres, by establishing consistency in and focuses on research efforts.

3.2. ARTEMIS JU

The ARTEMIS JU addresses the "embedded everywhere" revolution which is driven by device miniaturisation, where more and more functionality is available for less and less cost; by cheap and pervasive networking technologies; and by digital convergence between formerly distinct technology families and industrial sectors. The vision for the future that ARTEMIS promotes is a world supported by intelligent embedded systems for the benefit of mankind where all systems, machines, and

23

objects will become digital, self-managed, interconnected resources. This can be exploited in a number of ways with embedded systems being used to provide intelligent support in everyday objects, such as furniture, clothes, vehicles, buildings, roads and smart materials. Benefits to society will be improved quality of life, reduction of pollution and energy savings making life healthier and more secure for European Citizens, and also more comfortable for Europe’s ageing population.

The ARTEMIS JU is a tri-partite public private partnership that was established in February 2008 under Council Regulation (EC) No 74/2008 of 20 December 2007 [26] as a Community body based in Brussels. It is a Joint Undertaking between the EU represented by the Commission, Member States and the embedded systems industry association ARTEMIS-IA. The aim of the ARTEMIS JU was to implement significant parts of the Strategic Research Agenda put forward [27] by ARTEMIS-IA co-funded by industry, research organisations, participating Member States and the Commission's ICT programme. The ARTEMIS JU was responsible for maintaining the Strategic Research Agenda with the remit to cover embedded systems and tools. The ARTEMIS JU also initiated, managed and co-ordinated research activities through open calls for proposals and via funding research projects.

3.2.1 ARTEMIS Strategic Research Agenda

The ARTEMIS SRA [28] is to maintain a strong technological capability in both supply and application of embedded systems by overcoming fragmentation in the embedded systems supply base for components and tools. 3.2.2 Objectives of ARTEMIS JU

The objectives [8] of the ARTEMIS JU are defined as:

Define and implement a “Research Agenda” for the development of key technologies for Embedded Computing Systems across different application areas in order to strengthen European competitiveness and sustainability, and allow the emergence of new markets and societal applications

Support the implementation of the R&D Activities by awarding funding to participants in selected projects following competitive calls for proposals

Promote the involvement of SMEs in its activities in line with the objectives of the Seventh Framework Programme

Promote a public-private partnership aimed at mobilising and pooling Community, national and private efforts, increasing overall R&D investments in the field of Embedded Computing Systems, and fostering collaboration between the public and private sectors

Achieve synergy and co-ordination of European R&D efforts in the field of Embedded Computing Systems including, when added value can be created, the progressive integration in the ARTEMIS JU of related activities currently implemented through intergovernmental R&D schemes (such as EUREKA [29]).

3.2.3 Expected Benefits

One of the major ARTEMIS goals was to significantly impact the embedded systems market, both in Europe and throughout the world. The expectation was that by 2016, more than 20% of embedded systems deployed in the world would be based on ARTEMIS-JU results encompassing hardware,

24

software and systems design for embedded systems. A key driver was to address system design and implementation productivity with the aim to close the design productivity gap between requirement and capability.

The ARTEMIS JU set out a number of targets for 2016 [30]:

reduce the cost of system design from 2005 levels by more than 20%

achieve over 20% reduction in development cycles - especially in sectors requiring

qualification or certification

manage a complexity increase of more than 30% with 10% effort reduction

reduce the effort and time required for re-validation and recertification after change by 20%

or more

achieve cross-sectorial reusability of embedded systems devices that will be developed using

the ARTEMIS-JU results

increase the number of European SMEs engaged in the embedded systems supply chain,

from concept through design and manufacture, delivery and support by 40% more European

SMEs under the aegis of ARTEMIS-JU engaged in the embedded systems supply chain, than

there were in 2005

3.2.4 Funding

The overall contributions for ARTEMIS were anticipated to be €2.6 billion [8], comprising:

Member States: €745million

EC: €410 million

Private sector (industry): more than the sum of contributions of EC and Member States

3.2.5 ARTEMIS Intervention Logic Diagram

Figure 1 ARTEMIS Intervention Logic Diagram

25

Figure 1 shows an Intervention Logic Diagram for ARTEMIS. This highlights the general objectives set out for EC Joint Undertakings, the ARTEMIS specific objectives and the activities undertaken by ARTEMIS to address the objectives. Strong arrows reflect a direct impact on the objective and weaker arrows reflect an indirect impact on the objective. The main output indicators are also shown which have been used as a basis for this evaluation. The activities of the JU highlighted in yellow were also complemented by the activities of the ARTEMIS Industry Association shown in grey, notably the setting up of Working Groups. In terms of timing the activities to co-ordinate European R&D and set the strategic research agenda were a continuous activity with annual plans being issued. Likewise the mobilisation of funding and promotion of SME participation was an ongoing effort. Calls were made every year and a key criteria is the success of these calls and the quality of the proposals selected. The outcomes in terms of patents, innovations and publications have been analysed. The activity of the ARTEMIS-IA has been considered with emphasis on outreach to SMEs and metrics collection. The latter is considered particularly important in assessing the impact that funded projects will make in the future.

3.3. ENIAC JU

At the Lisbon European Council in March 2000 [31] the European Union set an ambition for 2010 “to become the most competitive and dynamic knowledge-based economy in the world, capable of sustainable economic growth with more and better jobs and greater social cohesion.” It was realised that in order to achieve this it was necessary to structure, optimise and integrate innovation efforts within given domains. Nanoelectronics as a core technology is exploited across a number of sectors including multimedia, telecommunications, transport, health, environment, industrial processing, etc. It is thus of very high strategic importance for European Industry but operates in a market with very rapid technological development and strong global competition. In response, the European Nanoelectronics Initiative Advisory Council (known as ENIAC) [9] was established comprising a wide membership of actors steered by a Steering Group made up of senior experts from semiconductor manufacturing companies, equipment and materials suppliers, application/system integrators, research organisations, academia, Member States, Regions, EUREKA and other public authorities, financial organisations, etc. The aim of ENIAC was to put forward a Strategic Research Agenda outlining future research and innovation priorities necessary to support the further development of a competitive nanoelectronics industry in Europe. In 2008 the ENIAC JU was established as a tri-partite Public-Private Partnership under Council Regulation (EC) No 72/2008 of 20 December 2007 [32]. This Joint Undertaking between the EU represented by the Commission, Member States and the nanoelectronics industry association AENEAS [26] had the remit of maintaining the Strategic Research Agenda and a strong technological capability to enable European industry to remain competitive in the global market. 3.3.1 Strategic Research Agenda

The goal of the ENIAC Strategic Research Agenda [33] was to enable European industry to remain competitive in the global market, by maintaining a strong technological capability even though there are few European semiconductor manufacturers.

26

3.3.2 Objectives of ENIAC JU

The objectives [34] of the ENIAC JU were defined as:

Define and implement a Research Agenda for the development of key competences for nanoelectronics across different application areas in order to strengthen European competitiveness and sustainability and allow for the emergence of new markets and societal applications

Support the activities required for the implementation of the Research Agenda, notably by awarding funding to participants in selected projects following competitive calls for proposals

Promote a public-private partnership aiming at mobilising and pooling Community, national and private efforts, increasing overall R&D investments in the field of nanoelectronics, and fostering collaboration between the public and private sectors

Achieve synergy and co-ordination of European R&D efforts in the field of nanoelectronics including, when added value can be created, the progressive integration into the ENIAC Joint Undertaking of the related activities in this field currently implemented through intergovernmental R&D schemes (EUREKA)

Promote the involvement of SMEs in its activities in line with the objectives of the Seventh Framework Programme

3.3.3 Expected Benefits

The expected benefits [34] of the ENIAC JU activities were defined as:

Support and contribute to raising European, Member State and private R&D investment in the nanoelectronics sector and to improving its effectiveness, by demonstrating a common vision and a consistent reference framework at EU level for both R&D funding and infrastructure initiatives. This common reference will help concentrate efforts and resources and avoid fragmentation, thus contributing to restructuring and optimising nanoelectronics research in Europe in the spirit of the ERA.

Accelerate the generation of new knowledge, innovation and the uptake of research and technologies, improving the competitiveness and productivity of the nanoelectronics sector

Contribute to removing obstacles at EU, national and regional levels, for co-ordination and facilitate/accelerate the market penetration of new technologies. Technology demonstration will form an important element in this process.

Maintain an appropriate balance between innovative and policy oriented research while contributing to aligning, in a coherent and consistent way, Research and Technology Developments with European policies and regulatory frameworks.

Make the EU more attractive both for researchers and industrial investment. ENIAC will also act as an early warning system to alert policy makers to the changing needs of the sector and the consequences for society, for example in terms of skill shortages or research infrastructure deficiencies.

Increase public awareness, understanding and acceptance of the technologies concerned and the research policy choices necessary to maximise the benefits for all stakeholders.

27

3.3.4 Budget

The overall contributions for ENIAC were anticipated to be €2.8 billion [9], comprising: • Member States: €800 million • EC: €440 million • Private sector (industry): more than the sum of contributions of EC and MS.

3.3.5 ENIAC Intervention Logic Diagram

Figure 2 ENIAC Intervention Logic Diagram Figure 2 shows the Intervention Logic Diagram for ENIAC. This highlights the general objectives set out for EC Joint Undertakings, the ENIAC specific objectives and the activities undertaken by ENIAC to address the objectives. Strong arrows reflect a direct impact on the objective and weaker arrows reflect an indirect impact on the objective. The main output indicators are also shown which have been used as a basis for this evaluation. The activities of the JU highlighted in yellow were also complemented by the activities of the AENEAS Industry Association shown in grey, notably the setting up of Working Groups. In terms of timing the activities to co-ordinate European R&D and set the strategic research agenda were a continuous activity with annual plans being issued. Likewise the mobilisation of funding and promotion of SME participation was an ongoing effort. Calls were made every year. The success of these calls and the quality of the proposals selected has been considered. The output of the JU in terms of patents, innovations and publications has been analysed along with additional activities performed by AENEAS.

28

3.4. Baseline

The baseline situation that drove the set-up of the ENIAC and ARTEMIS JUs in 2008 was the political and socio-economic situation in 2006-2007. It is unique and has enabled the Commission, Member States and Industry to define a common agenda for Europe via pooling resources.

3.4.1 ARTEMIS

Figure 3 The Embedded Systems Market (Source: European Commission)

Europe is very strong in the embedded systems area as shown in Figure 3 with capture of 30% of the global market, however, there is no single strategy for the development of the sector across Europe. The area is a clear European strength and there is a desire to develop standard platforms that can be used across domains. The ARTEMIS JU has contributed towards development and implementation of a strategy for embedded systems and there has been success in federating projects for embedded systems. The large-scale demonstrators developed in ARTEMIS have already supported the production of results that are on the market, e.g. the standard for automotive – AUTOSAR [35]. This was tested in large-scale demonstrators prior to introduction. A key benefit has been in getting co-investment in technologies bringing together, investment in research from industry, Member States and the European Union. Moving forward there has been a change from addressing embedded systems to considering the layers above those shown in Figure 3. These upper levels are characterised by societal-scale networks of Cyber-Physical Systems and Humans, sometimes also called Systems of Cyber-Physical Systems.

29

3.4.2 ENIAC

In the micro-electronics area prior to ENIAC there had been no overriding strategy. Although a lot of pre-competitive R&D was performed there was a lack of communication between actors, particularly RTOs. Now, however, there is a collective vision and an ambition to co-invest together at Member State level on the development and implementation of strategic research programmes in the areas of nanoelectronics.

The first 2 years were very difficult for ARTEMIS and ENIAC – this was because it was necessary to create a climate of trust and confidence to align strategies. This alignment has been easier for the micro-electronics area as there are not too many actors and there is an international roadmap for development of the technology. The embedded software area is more complex to bring together with many diverse applications. The added value of coming together is to make it possible to support large-scale federating projects that no one country could support by itself.

In the micro-electronics domain ENIAC confirmed leadership in equipment with a huge investment in fabrication technology. The ENIAC pilot lines were formulated using demand coming from industry and the academic community for research into production and nanoelectronics. The combined investment in each pilot line is equivalent to 2 years EU money for the area and in general 1 Euro drives 4 Euros from other partners in leveraging. There was also a need to structure research and development moving from lab. to fab. Here Europe has leadership in low power electronics. As ENIAC became more established funding naturally aligned itself around 5 main tracks in micro-electronics and the communities have structured themselves around these.

Going forward with the integration of the ARTEMIS and ENIAC JUs into ECSEL the expectation is that this will further enhance linkage within the value chain to support Europe’s competitiveness.

30

4. EVALUATION QUESTIONS

The evaluation has been configured to address 8 key evaluation questions that were defined in the Terms of Reference [36] provided to the evaluation panel. Evaluation Question 1: Background of initiative, objectives and relevance The JTIs represent the stepping stone towards setting up public-private partnership in research at the European level. They bring together EU, national and private resources, know-how and research capabilities, with the aim of addressing major issues by sharing knowledge, achieving critical mass, scale and scope. In this way, they help the EU to become a world leader in developing breakthrough technologies with high innovation potential. The public-private partnership is one of the FP7 implementation modalities where all involved partners commit to support the development and implementation of research and innovation activities of strategic importance to the Union's competitiveness and industrial leadership or to addressing specific societal challenges. As a first step, the Expert Group will study the regulatory framework and provide context and background information concerning the setting up of the ARTEMIS and ENIAC Joint Undertaking. The Expert Group will briefly summarise what was the situation before the approval and the set-up of the Joint Undertaking, present a brief description of the initiative, its objectives and the problems it intended to solve. The Expert Group will assess the intervention logic, the relevance of its objectives and whether the objectives are consistent with the strategic context and with the challenges that had been identified. If possible, the Expert Group will summarise this information in an intervention logic diagram as presented in the evaluation guidelines of the Better Regulation package, showing how different measures were expected to interact with each other. Evaluation Question 2: Implementation of ARTEMIS and ENIAC Joint Technology Initiative The Expert Group shall analyse how the Joint Technology Initiative ARTEMIS and ENIAC set by the Council Regulations 74/2008 and 72/2008 respectively was implemented. Information about different participation patterns of European research actors and about the distribution of funds among beneficiaries provides important information in order to assess if the ARTEMIS and ENIAC JU has reached the main research actors in Europe and, also, highlight the main research and structural trends. In this context, the Expert Group should analyse and assess the following aspects:

Presentation of an overview of calls launched during the period 2008-2013. Participation patterns broken down by country and region where possible; what trends and

specificities can be identified (for example participation trends of specific country groups like EU12)?

Participation patterns per specific thematic topic broken down by type of beneficiary organisations: universities, research organisations, industrial participation (large companies and SME).

Competition for funding; What were the success rates in terms of successful proposals, activity types of applicants and budget share – identification of areas/research topics where proposal success rates are above or below average?- are there any types of applicants showing significantly higher or lower success rates?

EU contribution – what has been the distribution of funds, broken down by country and region where possible, activity type of beneficiaries, and thematic area?

National contributions – what has been the distribution of funds, broken down by country and region where possible, activity type of beneficiaries, and thematic area?

What is the average grant size in terms of budget and number of beneficiaries (overall and by call and research topic)?

31

Evaluation Question 3: Main achievements and effectiveness of implementation Main achievements Direct achievements focus on concrete outcomes of ARTEMIS and ENIAC JU interventions. Information about different forms of direct achievements of ARTEMIS and ENIAC JU funded research projects is crucial to assess whether the JTI ARTEMIS and ENIAC reached its goals. It also represents the core of an evidence-based analysis of funded projects. The Expert Group should present, examine and assess various outputs and results produced under the ARTEMIS and ENIAC JU interventions (“what was produced through this intervention”). The Expert Group should aim at defining and capturing the lasting effects of JTI ARTEMIS and ENIAC interventions in the wider context of the longer-term economic and societal impact of EU-funded research and innovation. Notably, it should examine the extent to which scientific, technical and industrial outputs produced by JTI ARTEMIS and ENIAC interventions generated socio-economic effects and other impacts and helped to tackle relevant societal challenges. Effectiveness of implementation The Expert Group should analyse and assess the progress towards meeting the objectives set by the ARTEMIS and ENIAC JU, including how all parties in the public-private partnerships live up to their financial and managerial responsibilities and keep an open non-discriminatory attitude towards a wide community of stakeholders. As a minimum, the Expert Group should address the following non-exhaustive list of topics:

Assessment of the programme administration lifecycle and setting up an research agenda- from definition of the work programme and publication of Calls, to evaluation, selection, negotiation, contract/budget engagement?

Are all stakeholders relevant to the specific area of ARTEMIS and ENIAC JU involved (industry, research organisations, academia, public authorities, users, regulators, consumers etc.)?

Have the actions attracted and allowed a satisfactory level of participation of the best European player’s active in their specific areas?

Evaluation Question 4: ARTEMIS and ENIAC Joint Undertaking's performance in 2008 - 2016 ARTEMIS and ENIAC mission and governance The Expert Group will review and take stock of the legal basis in order to address the key mission and governance issues in the context of this evaluation. As a minimum, the following issues need to be analysed: Legal structure, governance arrangements and operation modalities. The Expert Group shall analyse and assess:

the clarity of the overall legal framework and the modalities for the implementation of the JTI programme;

the governance structure and processes of operation modalities including the participation of Member States/National Funding Authorities;

the robustness of the monitoring and control system – including the level of supervision/control within the JU;

appropriateness of the available capabilities to monitor progress – the use of funding; and the dissemination strategy.

32

Source of financing The Expert Group should present and assess the sources of financing (Members' financial contributions, any other financial contributions and revenues), the principle of financing the running costs of ARTEMIS and ENIAC JU and the financing of the research activities. Strategic Research Agenda (SRA) setting How the SRA scope and priorities were developed? Was the process transparent? Did it include the relevant stakeholders? Were the roles and responsibilities of the various stakeholders involved in the process clear and transparent? Who had the final ownership and responsibility of SRA? Operational effectiveness In this task the expert group will evaluate the actual operations of the ARTEMIS and ENIAC JU and assess if they are in line with the respective Council regulations. The overall approach in answering this evaluation question should focus on assessing the link between the Joint Undertaking's mandate/responsibilities, its governance and the actual activities and performance. As a minimum, the following questions should be addressed:

To what extent does the Joint Undertaking operate according to the legal framework establishing it?

To what extent has the Joint Undertaking led to improved management of the programme and better services to the stakeholders and addressees as compared to the alternative options?

What is the overall satisfaction of beneficiaries with the services provided by the Joint Undertaking?

Operational efficiency Efficiency will consider the relationship between the resources used and changes generated. The Expert Group will evaluate the operational efficiency of the ARTEMIS and ENIAC JU based on an analysis and interpretation of the indicators related to:

Timely execution of the functions. The Expert Group will have to analyse management performance indicators, such as time to grant, time to pay and average evaluation cost per proposal.

Cost-efficiency of the management and control arrangements. Management efficiency for this purpose is defined as the ratio between inputs (staff) and outputs (the budget managed by the Joint Undertaking). The analysis will cover i) the ratio between the administrative and operational budget (%) and ii) budget “per head” (million €). In addition, calculation of the average project management cost per running project (staff FTE * standard staff cost) has to be calculated.

Budget execution of commitment and payment appropriations during the reference period. Simplification and reduction of the administrative burden for participants.

33

Evaluation Question 5 - What are the lessons learned from the previous evaluations? The previous external evaluations covered the operation of the ARTEMIS and ENIAC JU, from 2008 to 2013. The evaluation provided the following key recommendations:

The ARTEMIS and ENIAC JTI SRA and work programmes need to reflect more strongly a coherent European perspective, linking to an overarching European Electronic Components and Systems research, development and innovation strategy

The ARTEMIS and ENIAC Industrial Association should play a more active role in the definition of the overall objectives and strategy of the JTI and should engage more actively with stakeholders so as to promote and facilitate participation in project proposals, especially by SMEs, and to develop and keep up to date the Strategic Research Agenda

ARTEMIS and ENIAC JTI project reviews, including a final post-project review that should be held, should monitor more closely and rigorously the actual and planned exploitation of project results, and the measures put in place by project partners to achieve such planned exploitation

ARTEMIS projects should build, where appropriate, on previously developed ARTEMIS technology, making reference to what has been funded before and demonstrating, in addition to novelty, the appropriate re-use of previous project results combined with a suitable progression to higher TRL levels. The proportion of funding for projects targeting generic applications and services (Applications projects) should be increased.

ENIAC and CATRENE calls for, and selection of, proposals should be more closely aligned (e.g. by the use of common and/or complementary calls), with the relevant funding awarding bodies retaining some flexibility over the assignment of the most appropriate funding stream

The evaluation also formulated the following recommendations for the future ECSEL JU:

The ARTEMIS and ENIAC JTIs, along with the European Technology Platform (ETP) on Smart Systems Integration (EPoSS), should be integrated into a single organisation (an ECS JTI)

Construct a proposed new, integrated JTI, or indeed any future JTI, as a PPP body as defined in the financial regulation

Focus the JU Governing Boards on strategic issues and reduce its administrative burden in order to attract participation from high-level industry representatives

Member State participation rules, funding rates and procedures should be harmonised and synchronised wherever possible, adopting best practice as the guiding principle

The JUs should explore and develop appropriate mechanisms to create an “early warning system” to identify potential delays, or restrictions to the availability, of funding from Member States

Member States should commit to a multi-annual funding system Take steps during the proposal evaluation and selection process to improve the match of the

project portfolio to strategic European aims and to ensure optimum coverage of key areas defined in the overarching EU ECS strategy and the work plans derived from such a strategy.

Specific support mechanisms for enhancing the project management processes in JTI projects should be developed and implemented. Management costs should be 100% funded by the EC for all JTI projects

JTI projects should be subject to only one (i.e. the JU) project review and reporting process Appropriate metrics for measuring the impact and success of JTI projects should be developed

and used for both current and future JTIs

34

The Expert Group will examine the follow-up and implementation of these recommendations and assess the extent to which the identified shortcomings in implementation have been addressed to date in the implementation modalities of Horizon 2020. Evaluation Question 6 - European added value The EU-added value relates to changes that can be reasonably attributed to an EU intervention, rather than other factors. The Expert Group should help determine the added value of the EU intervention via the JTI ARTEMIS and ENIAC, compared to what could be achieved by the Member States alone at national/or regional levels. To answer this question at the JU level, the Expert Group should analyse, among others, the leverage effect and assess the scale of resources involved and the ARTEMIS and ENIAC JU's ability to leverage additional investments in research and innovation. By leverage effect we refer to the ARTEMIS and ENIAC JU ability to attract additional finance and multiply its own, mainly EU resources, including additional activities, i.e., activities of the industry outside the work plan of ARTEMIS and ENIAC JU that nevertheless are in support of JU objectives. The leverage effect is defined as the total amount of funds leveraged through an Art.187 initiative, including additional activities, divided by the respective EU contribution to this initiative. Evaluation Question 7 - Coherence The question of “coherence” is mainly relevant at programme/general objectives level, i.e. FP7, and it was fully addressed in the FP7 ex-post evaluation. Taking into account the specificities of the public-private partnership, the Expert Group should look at how well the intervention worked: i) internally within FP7 and ii) with other EU policies and interventions. As regards external coherence, the Expert Group should look at the following aspects:

To what extent was the ARTEMIS and ENIAC JTI coherent with other interventions such as ICT as part of the FP7 that have similar objectives?

What was the relation with other Union funding programmes? (i) complementarity, (ii) synergies or (iii) potential overlaps?

If applicable and feasible, examine whether synergies with similar national, international, national and intergovernmental programmes have been sought after and/or developed to encourage optimal use of resources and avoid unnecessary duplication?

Evaluation Question 8 – Synthesis, conclusions and recommendations The Expert Group shall synthesise the work done under the previous tasks, draw conclusions and provide recommendations. The Expert Group must make judgements based on the evidence and analysis available. These judgements should be as specific as possible and based on clearly defined criteria. The evaluation shall draw conclusions on:

Relevance of the ARTEMIS and ENIAC JU. The Expert Group shall analyse and conclude whether the initial identification of tasks entrusted to the Joint Undertaking was valid and sufficient to justify the creation and existence of this public-private partnership.

Whether and how the ARTEMIS and ENIAC JU contributed to the overall objectives of FP7? The Expert Group shall respond to the question on whether the objectives set in the Council Regulation establishing ARTEMIS and ENIAC Joint Undertaking were reached during the reference period (effectiveness).

35

Whether and how this intervention worked together with other policies and helped to mutually reinforce the effects of all policy interventions (“coherence”)?

The Expert Group shall analyse, assess and draw conclusions if the public-private partnership was implemented in an open, transparent, effective and efficient way.

The Expert group shall analyse, assess and draw conclusions if the public-private partnership demonstrated the added value and generated the necessary leverage.

The Expert Group should also identify the key elements explaining success or highlight the areas for further improvement. On this basis, the Expert Group should also assess the extent to which the identified shortcomings in the ARTEMIS and ENIAC Joint Undertaking under FP7 implementation have been addressed to date in the design and implementation modalities of Horizon 2020.

36

5. METHOD/PROCESS FOLLOWED

5.1. Process/Methodology

The panel comprised 7 Experts and a Rapporteur who worked over a 6 month period to gather evidence from an extensive review of relevant documentation and from interviews with a wide range of stakeholders and interested parties from industry, research institutes and universities, the European Commission, Member State Public Authorities, and the JTIs themselves, including both participant and non-participants of the JTI programmes.

A number of surveys were launched to gather the views of stakeholders. These include surveys initiated by the EC, by the JTIs themselves and also via an independent study commissioned from CARSA [37]. The results from the various surveys were analysed and cross-referenced to assess their validity. Additionally, face-to-face interviews were held with representatives of key stakeholder groups, including Interviews with the Presidents of ARTEMIS and ENIAC and interviews with Public Bodies. These interviews typically drew attention to particular aspects or activities of the JTIs that are running well, or to areas where there is a need for improvement. The findings from such interviews were critically examined by the panel, compared against each other and with factual evidence, such as JTI statistics and reports, before drawing appropriate conclusions in relation to the main evaluation criteria (effectiveness, efficiency and quality).

The panel also carried out, as an integral part of the evaluation process, a thorough examination and analysis of ARTEMIS and ENIAC (and related) documentation.

JTI strategic research agenda and related mission statements

JTI annual work programmes and annual activity reports

JTI project databases, project reports are project review documentation

Reports of JTI Governing Board, Public Authorities Board and Industrial Association meetings

Horizon 2020 and Key Enabling Technology reports

Council regulations establishing JUs

Council regulation establishing FP7 / Horizon 2020

1st and 2nd Interim Evaluations of JUs under FP7

CORDA database

Court of Auditors (CoA) and European Parliament recommendations

Factual Support Study performed by CARSA for the evaluation of the ARTEMIS and ENIAC and ECSEL JUs under FP7 and Horizon 2020

Other evaluation studies, such as FP7 ex-post evaluation and etc.

A full list of the documents used as part of this evaluation is given in Annex 1. Panel members also attended ARTEMIS and ECSEL Brokerage Events to gauge interest in the JU programmes. The panel also drew, as part of its evaluation analysis, on its own tacit and expert knowledge of the respective R&D and industrial domains of the ARTEMIS and ENIAC JTIs. In summary, the Final Evaluation of the ARTEMIS and ENIAC JTIs was based on a thorough examination and analysis of an extensive set of relevant documentation, on the gathering and analysis of a wide cross-section of stakeholder views, and, importantly, on an informed debate amongst panel members. Each recommendation is thus the outcome of a group judgement and debate, drawing upon numerous, different, but complementary and relevant, data sources and analyses. A definitive description of the evidence base used for the evaluation is given in Annex 4.

37

5.2. Limitations – Robustness of Findings

Although every effort has been made to verify the information used in this report, some inconsistences were noted in the output from studies used as source material and published data from the JU’s. In these cases the inconsistent figures were referred to the relevant JUs for corroboration. In many cases the databases held for funded projects and for proposal applicants do not contain information on the partner types (e.g. Large Industry, Mid-Cap or SME) or the domain of the partner. This makes analysis of engagement with SMEs and domains represented difficult. In general due to the fact that the impact from many projects is realised 2-3 years’ after project end it is not possible to assess completely this criteria. In this respect recommendations have been made with respect to collecting metrics which will allow impact to be monitored in future.

38

6. IMPLEMENTATION OF ARTEMIS AND ENIAC JOINT TECHNOLOGY INITIATIVES

6.1. Implementation of the ARTEMIS JU

Figure 4 Breakdown of Projects and Participant Types Funded under ARTEMIS Period 2008-2013 (Source: ARTEMIS-IA)

Over the period 2008-2013 the ARTEMIS JU launched 6 calls and funded 56 projects as shown in Figure 4 engaging 1420 participant entities. The breakdown of participant types indicates that larger Industrials contributed more to projects with a fairly even split of SME’s and Universities/Institutions/NP also participating. In terms of country participations, Spain, Italy and Germany accounted for the highest number of participations with also strong participation from some smaller EU Member States (in particular Sweden and Austria). The UK which is a major player in EU Framework Programmes in general, only had limited participation in ARTEMIS. The participation rates show that funding was mainly provided by the private sector with significant funding provided by the Member States of around double the JU contribution from the EC.

Figure 5 Key National Actors in ARTEMIS Depicted by Centrality Measure (Source: European Commission, Calculations AIT [37])

39

In Figure 5 the core participating countries are shown as network nodes using two measures of centrality (degree of centrality and betweenness centrality).

The degree centrality of an actor is proportional to the degree of the corresponding node. The degree is defined as the number of links incident upon the node, which equates to the number of distinct partners the corresponding organisation has across the projects in which it participates. Network actors with high degree centrality are well-connected and thus may be able to access knowledge and resources more readily than less central actors. The degree of centrality is thus a measure of an actor’s power.

The betweenness centrality is calculated using the shortest paths between pairs of network nodes. A node has a betweenness centrality value proportional to the number of such paths which include the node. Actors with high betweenness centrality are thus positioned in the network so as to favour their acting as coordinators or brokers, with other actors dependent upon them to make connections in the network.

From a country perspective, the ARTEMIS project portfolio is constituted by a core of 9 countries, which collaborated significantly with each other. Among the participating countries, particularly high interactions take place between Spain, Germany, France, Sweden and Austria. Central and Eastern European Member States are only to be found at the periphery of the network. While some of these countries (e.g. Greece and Portugal) show some strong collaborations with countries from the network core, others are only loosely connected.

6.1.1 ARTEMIS Project Types

Table 2 ARTEMIS Sub-programmes (ASPs) and Innovation Pilot Projects (AIPPs) (Source: ARTEMIS

JU, Annual Activity Report 2013 [37]) Two types of projects were funded: ARTEMIS Sub-programmes (ASPs) and ARTEMIS Innovation Pilot Projects (AIPPs) as shown in Table 2. Notably more ASP projects were initiated in the first years of the JU (2008-2009) and this reduced over the last 3 years. Despite the fall in absolute numbers the funding levels increased in 2012-2013 indicating that fewer larger projects were being undertaken. This reflects a change in funding priorities with the introduction of the new ARTEMIS Innovation Pilot Projects (AIPPs) for the 2012 Call. The three AIPPs selected for funding accounted for nearly 60% of ARTEMIS budget for the years 2012 and 2013:

(AIPP1) “Critical Systems Engineering Factories”

(AIPP4) “Production and Energy Systems Automation”

(AIPP5) “Computing platforms for embedded systems”

40

Figure 6 ARTEMIS Projects by Sub-programmes (2008-2013) (Source: https://www.artemis-ju.eu; Total number of projects 2008-2013 is 56)

Over the period 2008-2013 the ARTEMIS Sub-Programmes (ASPs) ASP 1 (methods and processes for safety-relevant embedded systems) and ASP5 (computing platforms for embedded systems) accounted for the greatest number of projects selected between 2008 and 2013 (See Figure 6). 6.1.2 Project Funding

As the ARTEMIS JU introduced a different type of project in 2012 it is useful to consider the funding over the first 4 calls of the project (2008-2011) and remaining calls (2012-2013) separately.

Period 2008-2011

Figure 7 ARTEMIS Funding Allocations by ASP (2008-2011) (Source: CARSA [37]) The distribution of investment over the 8 ARTEMIS Sub-Programmes between 2008 and 2011 is shown in Figure 7. Here the concentration of funds on ASP1 and ASP5 can be clearly seen. In total over the first 4 calls these two work areas accounted for around 58% of the total allocated funds. This dominance is explained in the ARTEMIS Annual Activity Report 2012 p.11 [39] which highlights that ASP1, in particular, is highly important for the Transport and Medical industries. As a consequence it attracted larger projects that had direct relevance to industry. ASP5 which addresses the high technical complexity of low-power, multi-core platforms, is also a key area attracting many small-scale academically driven projects.

ASP1 ASP2 ASP3 ASP4 ASP5 ASP6 ASP7 ASP8

41

Period 2012-2013

Figure 8 ARTEMIS Funding Allocations by ASP and AIPP (2012 and 2013 calls) (Source: Second Interim Evaluation of ARTEMIS and ENIAC JTI. Source: ARTEMIS Annual Activity Reports 2012 [39]

and 2013 [38].) There was a significant increase in ARTEMIS funding from the private sector in 2012 (See Figure 4) attracted by the introduction of the AIPPs. The funding allocated to the AIPPs and ASPs is shown in Figure 8. Here the change in strategy to fund larger AIPP projects can be clearly seen with these accounting for the majority of the budget. The ARTEMIS IA carried out a survey of the private sector in its report Business Impact and Metrics p.41 [40]. This highlights that respondents were particularly interested in the following areas: automotive, aircraft and aerospace, smart grids, medical and health applications and energy efficiency.

Figure 9 Breakdown of ARTEMIS Funding 2008-2013 (Source: ARTEMIS Data) A breakdown of funding is given in Figure 9. The Council Regulation dictated that the European Union would contribute up to €410 million and the Member States €745 million. The funding levels fell

16%

30% 53%

1%

ARTEMIS JU Member States Private Sector Running Costs

42

short of this and the EU contributed €190.3 million of which €9.3 million was used to cover running costs. The Member States contributed €343.1 million representing 30% of total costs. The private sector provided strong support for ARTEMIS contributed €609.5 million representing 53% of the costs. 6.1.3 ARTEMIS Success Rates

The success rates for ARTEMIS proposals reduced in the period 2008 to 2010 but increased steadily in the period 2011-2013. This corresponded to an increase in funding from industry and the concentration on fewer larger AIPPs. The average success rate for ARTEMIS proposals was 26%. 6.1.4 National Contributions

The initial ARTEMIS funding target was €2.5 billion for R&D. The total eligible cost for the 6 calls between 2008 and 2013, however, only amounted to €1.1 billion [38] falling short of the target. Although industry has strongly supported ARTEMIS and has invested according to the original plan the national Member States have invested significantly less than planned over the period. The Member States and EU financial commitments were around €343 million and €181 million respectively over the period 2008-2013.

Figure 10 ARTEMIS National Funding 2008-2013 (€M) (Sources: Second Interim Evaluation of ARTEMIS and ENIAC JTIs; ARTEMIS Annual Activity Report 2013 [38], (p.18); ARTEMIS Annual

Activity Report 2012 [39] (p.16), ECSEL Annual Activity Report 2015 [41] (p.22-23)). The level of National funding from Member States is shown in Figure 10. After initial stronger commitments in 2008 and 2009 funding reduced and finished at a lower level following a renewed increase in funding in 2012. There was an increase in ARTEMIS activities in the last two years of the programme with the introduction of ARTEMIS Innovation Pilot Projects (AIPPs) for the 2012 Call. The concept of very large projects, closer to market was positively welcomed by Member States, resulting in more commitment for 2012 [39]. This led to a significant increase in 2012 of total eligible costs to €282 million up from €142.14 million in 2011. The decrease in funding between 2012 and 2013 is a result of the programme coming to an end and by the fact that only four projects were selected in 2013. Notably one AIPP project, EMC2, accounted for 57% of the €163.36 million allocated [38]. The lack of funding commitment from Member States has been indicated by ARTEMIS as an important source of problems. This led to the development of ARTEMIS Innovative Pilot Projects in 2012 and to the introduction of a new voting system to ease decision making. The ARTEMIS Annual

43

Activity Report 2012 also highlighted the need for more efficient funding under ECSEL: “While ARTEMIS has a strong industrial lead, a future programme should engage the strategic considerations of the participating states much earlier in its definition process” [39]. It was noted in the Second Interim Evaluation of ARTEMIS and ENIAC JTIs [42] that a critical issue was that Member States experienced difficulties with respect to engaging in multi-annual commitments and ensuring a good level of funding. Project reporting procedures were also time consuming and some Member States insisted on having a separate technical review process resulting in duplication and burdensome administrative procedures for project partners. A recommendation was thus made in the Second Interim Evaluation to harmonise and simplify the Member States participation rules [42] which was supported by the European Commission [43]. The ARTEMIS Annual Activity Report 2013 highlights that the collaboration between the Member States and other ARTEMIS partners improved thanks to the establishment of a new voting system. This gave participating countries a better say in the projects to be selected for financing in Call 2013 [38].

Figure 11 Breakdown of National Funding in ARTEMIS in €M (2008-2013) (Source: ARTEMIS Public

Authorities Board Decisions, available at: https://www.artemis-ju.eu N.B. The amounts provided by the Brussels Region- Belgium (€245,568), Estonia (€300,915 ) are too

low to be displayed properly.) A breakdown of overall funding contributed by each Member State is shown in Figure 11 showing strong support from Germany, Italy, Spain, the Netherlands and Finland for ARTEMIS with Austria, France and the UK contributing around €20 million each.

44

6.2. Implementation of the ENIAC JU

Figure 12 Breakdown of Projects and Participant Types Funded under ENIAC period 2008-2013 (Source: ENIAC JU)

Between 2008 and 2013 ENIAC launched 9 calls and provided funding to 63 projects [44], as shown in Figure 12, with a total eligible cost amounting to nearly €2.9 billion. In total 1384 participant entities have engaged over the period 2008-2013. The breakdown of partner types indicates a greater balance between large industry and Universities/Institutions/NP partners contributing to projects but with less participation from SMEs. The participation rates show that funding was mainly provided by the private sector with significant funding provided by the Member States of around double the JU contribution from the EC. France, the Netherlands and Germany lead in terms of participations with other strong participation from Sweden and Austria. The UK had a limited participation in comparison to its participation within EU Framework Programmes in general.

Figure 13 Key National Actors in ENIAC Depicted by Centrality Measure (Source: European Commission, Calculations AIT)

Compared with ARTEMIS, in ENIAC interactions are concentrated among five countries as shown in Figure 13, which build the core of the network. Interactions are highest between Germany, France

45

and the Netherlands. Austria, in which Infineon is a major actor, closely interacts with Germany. As in ARTEMIS, Central and Eastern European Member States are only to be found at the periphery of the network.

6.2.1 ENIAC Projects

The ENIAC MASP [45] identifies 8 key work areas that address socio-economic and technological challenges. Work areas (1-2) are domains in which European industry needs to build on its leading position. Work areas (3-5) are domains in which Europe needs to position itself at the forefront of emerging markets and work areas (6-8) address the technological enablers in three main technological research domains. Socio-economic challenges and application domains

Work Area 1 - Automotive and Transport: Innovation in the car industry is today largely based on micro and nanoelectronics, and innovation elements in this area cover application such as sensors, power electronics and embedded multi-core processors as key-enablers for sustainable, energy-efficient, and safer products.

Work Area 2 - Communication and digital lifestyle: This work area addresses modern and mobile communications as well as broadband tools that will be key elements for many new applications - such as medical and the future of Internet.

Work Area 3 - Energy efficiency: All the smart energy solutions (smart buildings, smart appliances, smart grids, etc.) use semiconductor-based technologies such as sensors, communication chips, microprocessors, etc. Therefore, this work area targets a huge and emerging market in the domains of energy efficient houses, remote metering and power control, alternative energy sources, energy efficient motor controls for private and industrial applications.

Work Area 4 - Health and ageing society: This domain addresses the need for new applications and services derived from technological R&D in medical electronics, intelligent drugs, measurements and diagnostics, ambient-assisted living and person-centric health management.

Work Area 5 - Safety and Security: Innovation in this work area spans from a wide range of application domains such as transportation, the supply of energy in which there is an increased need to ensure consumers´ safety and protect their privacy and private data.

Three strongly interdependent technology directions are also described in the ENIAC MASP [45] that support the development of the semiconductor industry:

miniaturisation (i.e. “More Moore technologies”)

diversification and differentiation (i.e. “More-than-Moore technologies”)

heterogeneous integration (i.e. integration of components of different origins and technologies in a single package).

46

These are supported by:

Work Area 6 - Design technologies: Increasing levels of system integration, involving the combination of complex IP and different technologies in a single chip or package imply the capability to handle complex architectures and require advanced tools, design flows and methodologies. Miniaturisation and complexity progress more rapidly than design capability leading to a “design gap”. This work area aims to support increasing efforts to develop adequate design capabilities to design new tools and methodologies.

Work Area 7 - Semiconductor process and integration: This work area aims to support research in advanced CMOS so as to master and access the latest technologies, to ensure manufacturing competitiveness and secure further growth in new applications and European lead markets

Work Area 8 - Equipment, materials and manufacturing: This work area aims to support the introduction of innovative materials and process technologies so as to improve competitiveness in manufacturing. Sophisticated advanced technology requires the development of specific metrology, characterisation and failure-analysis methodologies, tests and tools.

Figure 14: ENIAC Projects by Sub-programmes (2008-2012) (Source: http://www.eniac.eu; AENEAS Annual Work Programme 2013, p.7

N.B.: Some ENIAC projects pursue targets and objectives that encompass several work areas or sub-programmes that can be double-counted in the chart above. Figures for the 2013 calls are not readily

available.) Figure 14 shows a breakdown of the number of projects that have been funded in the various work areas. This shows a concentration of projects in the Energy Efficiency, Semiconductor Process and Integration and Equipment, Materials and Manufacturing areas.

47

6.2.2 Project Funding

Figure 15 Eligible Costs per MASP Area in the Projects Selected for Funding over the 2008-2013 Period (amount as per JU financial commitments) (Source: ENIAC Annual Activity Report 2013 [44],

p.19) A breakdown of funding against Work Areas is shown in Figure 15. The total eligible costs over the 2008-2013 period were around €2.9 billion. Three sub-programmes accounted for 78% of these costs with 35% of funding going to “Equipment, materials and manufacturing”, 27% to “semiconductor processes and integration” and 16% to “energy efficiency” respectively. Other research areas, e.g. automotive and transport, communication, health care, security and design have a much smaller share of funded R&D effort. There is also a concentration on More-than-Moore technology rather than More-Moore technology topics. The Second Interim Evaluation panel highlighted that projects reflecting the actual value-chain may be more appropriate for the European semiconductor industry with the aim of getting design competence and niche manufacturing closer to the product and more targeted to application-based activities [42].

Figure 16 ENIAC Cumulative Total Eligible Costs (€M) between 2008 and 2013 (Sources: Second Interim Evaluation of ARTEMIS and ENIAC JTIs; ENIAC Annual Activity Report 2013 [42] (p.7))

As shown in Figure 16 allocated funding initially lagged behind planned targets. In 2012 there was a significant improvement and in 2013 the actual funding exceeded the planned amount.

48

Figure 17 Breakdown of ENIAC Funding 2008-2013 (Source: ENIAC Annual Activity Report 2013 [42]) The Council Regulation dictated that the Union had to contribute up to €450 million, of which €10 million was to cover a third of the ENIAC JU running costs. In fact ENIAC only requested €5.6 million for running costs and so €4.4 million was re-dedicated to research activities. Overall, between 2008 and 2013 the EU´s contribution of €444.4 million was 15.4% of the total eligible costs as shown in Figure 17. The private sector, which legally had to provide at least 50% of the total research cost, actually provided €1829.5 million which accounts for 63.4% of the total eligible costs. The private sector also provided funds through AENEAS to cover 2/3 of the ENIAC JU running costs. Over the 2008-2013 period, total national contributions amounted to €570.6 million, representing 19.8% of the total eligible costs engaged in the programme. 6.2.3 ENIAC Success Rates

The success rates for ENIAC proposals reduced in the period 2008 to 2010 but increased in 2011. This corresponds to an increase in funding and development of the pilot lines. In 2013 there were no calls. The average success rate for ENIAC proposals was 46% which was far higher than for ARTEMIS. 6.2.4 National Contributions

Figure 18 ENIAC Cumulative National Grants 2008-2013 (€M) (Sources: Second Interim Evaluation

of ARTEMIS and ENIAC JTIs [42]; ENIAC Annual Activity Report 2013 [44] (p.7)) There was strong and continued support from the Member States for ENIAC as shown in Figure 18. In response to the introduction of the new ENIAC pilot lines [46, 44] there was a significant increase in national funding commitments for the years 2012 and 2013. From the outset the ENIAC JU created a

49

framework in which National Public Authorities could ensure that their policies and priorities were taken into account. Within ENIAC Member States could provide upfront guidance to Industry in formulating strong proposals aligned with national and European policies and priorities. This strong role in decision making is reflected in the ENIAC Annual Activity Report 2013 [44]: “Member States were supposed to provide approximately two-thirds of the public funding in the projects, their majority is needed to adopt the Multi-Annual Strategic Plan and the Annual Work Programmes, and they decide upon the project selection for funding. In addition, they bring essential in-kind contributions to the execution of the programme by taking a leadership position in the financial transactions, including pre-financing, cost recognition, grant payments to the participants, recoveries etc.” As a consequence of this close involvement the Member States significantly increased their financial commitments: Contributions from Member States amounted to €57.82 million in 2008, €120 million in 2011 and reached €171 million in 2013.

Figure 19 Breakdown of National Funding in ENIAC in €M (2008-2013) (Source: www.eniac.eu

N.B. The amounts provided by Estonia (€0.3 million) are too low to be properly displayed) A breakdown of overall funding contributed by each Member State is shown in Figure 19 showing very strong support from France with major contributions from Germany, Belgium, Netherlands and Italy.

50

7. ANSWERS TO THE EVALUATION QUESTIONS

Overall the ENIAC and ARTEMIS JUs selected 119 projects for funding engaging €630 million of EC contributions and leveraging €912 million of national contributions and creating €4 billion of research and innovation. It is estimated by ARTEMIS/ENIAC that each Euro contributed by the EC resulted in 6.4 Euros of research and innovation activity in Europe [47]. By 2014 59 projects had been completed generating 179 patents, 11 trade secrets, 12 trademarks, 1004 exploitable foreground intellectual property items and 3099 publications. These numbers are expected to increase when all projects are completed.

The two JUs have adopted different approaches to impact assessment. ARTEMIS set a challenging goal targeting the use of ARTEMIS technology outcomes in 20% of the embedded systems in the world. The attainment of this impact is hard to measure as it will take time for outcomes to be transferred into products and some projects have yet to finish. ARTEMIS has set up a Working Group specifically to address monitoring of KPIs. The ARTEMIS project portfolio follows targets set out in the Strategic Research Agenda:

Most projects promise to deliver cross-domain re-use and interoperability for different product categories and application domains. SW design tools and environments are important in the automotive, aerospace, industrial processes and medical/healthcare sectors.

Many projects promise a reduction of system design costs and development cycles for both hardware and software with the use of general purpose architectures that facilitate integration and reuse across sectors.

Shorter development cycles and increased interoperability result in a shorter time to market for new products and services and many projects expect commercial impact within a 3 to 5 years’ time. Application areas include the automotive, avionics, space, factory automation, healthcare, Smart Buildings and Infrastructures, Electro-mobility and Energy Production in the CPS domain with wider-scale deployment more generally in the Internet of Things market.

The ENIAC JU also set a challenging target of creating 250,000 jobs in Europe. In the case of AENEAS, however, there is no follow up of impact metrics.

To provide rigour the panel considered the following output indicators in the assessment of the impact of the JUs.

Engagement with the community

Inclusion of SMEs

Number of patents and innovation outputs

Success in attracting public/private funding

Number of projects initiated, success stories and evidence of impact from projects

Working Groups established to support the Community These categories align with the outcomes identified in the Intervention Logic Diagrams for the ARTEMIS and ENIAC JUs and allow traceability of the outcomes back to the objectives of the JUs and the overall FP7 objectives set out in the Council Regulations.

51

7.1. ARTEMIS - Main achievements and Effectiveness of Implementation

7.1.1 ARTEMIS - Main Achievements

7.1.1.1 Engagement with the Community

A key aim of the ARTEMIS JU was to bring together the fragmented embedded systems community to tackle areas that will lead to a greater impact. In total 1417 organisations have participated in ARTEMIS over the period indicating that there has been success in bringing together the community. The CARSA survey results for ARTEMIS [37] support this assertion. An overwhelming majority (84%) of the surveyed JU representatives confirmed that the calls under ARTEMIS JUs have enabled research to be performed that would otherwise not have been possible. This is via providing increased overall funding for projects, by allowing projects to be performed that are larger and with a greater scope, and by enabling increased international co-operation. The area of platform building in the embedded systems domain was also highlighted. These findings are also supported by respondents to ARTEMIS-IA questionnaires in 2014 [48], which stated that the ARTEMIS JU had had a significant and pivotal business impact on reducing development costs (53% of respondents), had contributed to reduced time-to-market (41%) and had developed a new generation of products (38%). Here the ARTEMIS JU has been successful in bringing organisations together to develop prototypes and demonstrators. In 2014, 95% of the respondents indicated that a key aim was to build application prototypes [48]. Overall surveys indicated that participants were keener to engage with ARTEMIS and that the end user involvement was perceived as being very strong. Operation at higher TRLs has led to an industry focus bringing together the necessary actors to address specific applications. Many of the projects envisage providing R&D results with commercial impact within a 3 to 5 years’ time in the automotive, avionics, space, factory automation, healthcare and Internet of Things application areas. According to the ARTEMIS-IA study [48], ARTEMIS has had specific impacts in the 5 following application domains: “Security and Safety” (41% of the respondents), “Transport and mobility” (40%), “Energy efficiency” (26%), “Health and well-being” (16%) and Future Factories (16%). Here Security and Safety mainly refers to safety-critical applications in the transportation sector. Therefore, ARTEMIS has been successful in bringing together the embedded systems community around a number of core domains to tackle “big issues”.

7.1.1.2 Inclusion of SMEs

High technology SMEs in the embedded systems domain are active in inventing and innovating new products and in providing supporting tool development, technology validation and co-development. A particular strength of European SMEs is in the take-up of new technologies to increase competitiveness and in providing added value via products and services, as well as in designing and producing such systems. ARTEMIS has created Centres of Innovation Excellence (CoIE) and Tool Platforms to support this. Notably ARTEMIS has initiated dedicated actions to include SMEs in the innovation pilots. Projects encourage SMEs to participate and this is facilitated with forums and workshops. In most cases large enterprises introduce SMEs into projects. As a consequence of these actions between 2008 and 2013 ARTEMIS displayed a good level (29%) of SME participation. In terms of requested EU contribution allocated to SMEs, ARTEMIS with 19% displays a higher share of funding allocated to SMEs than ENIAC (10%). It is also notable that research organisations (HEIs & RES) received a comparatively higher share of funding in ARTEMIS than in ENIAC.

52

This is also backed up by participant perception and surveyed representatives (See CARSA survey [37]) assessed the efforts to involve SMEs more positively for ARTEMIS than for ENIAC. This was more noticeably positive for private representatives (57% strongly agree while 43% somewhat agree). Public respondents were less convinced with 13% strongly agreeing and 60% somewhat agreeing that the ARTEMIS program significantly contributed to SME involvement. The embedded systems domain is characterised by a high number of SMEs and start-ups.

Although improvements during the implementation of the JUs have been identified, complicated procedures and related administrative burden are still the main barrier for SME involvement. Interviewees highlighted that understandably SMEs preferred that a larger partner took on project co-ordination as the costs are not fully covered and the relative burden is much smaller as part of their operation. It was noted, however, that once engaged in one project SMEs are eager to work on additional projects indicating that involvement is seen as beneficial and that the programme was achieving its goal of building an ecosystem.

7.1.1.3 Number of Patents and Innovation Outputs

Number of Projects

EU Public Funding (€M)

Patents Produced

Average number of patents per 10 €M of EU funding

Scientific publications

Average number of scientific publications per €10 million of EU funding

42 181.4 28 1.54 1460 80.49

Table 3 Results of ARTEMIS Completed Projects Up to and Including 2015 (Source: ECSEL Annual Activity Report 2015 [41], p.33)

Table 3 shows that a total of 42 projects launched under ARTEMIS came to successful completion up to and including 2015. Over this period 28 patents were filed and ARTEMIS projects led on average to 1.54 patent applications per €10 million of EU funding. This compares to the Horizon 2020 benchmark of 3 patent applications per €10 million of EU funding, reflecting the patenting practices in this part of the industry. ARTEMIS projects produced 1460 scientific publications giving an average of 80.49 scientific publications per €10 million of EU funding (a far higher level than ENIAC projects, also reflecting different communication practices in the areas covered by ARTEMIS and ENIAC).

7.1.1.4 Success in Attracting Public/Private Funding

The initial ARTEMIS funding target was €2.5 billion for R&D. The total eligible cost for the 6 calls between 2008 and 2013, however, only amounted to €1.1 billion [38] falling short of the target. Although industry has strongly supported ARTEMIS and has invested according to the original plan the national Member States have invested significantly less than planned over the period. The Member States and EU financial commitments were around €340 million and €181 million respectively over the period 2008-2013. In ARTEMIS, domains such as safety-critical systems, automotive, manufacturing, aerospace, telecommunications, industrial software and medical technologies have benefitted from a leverage effect and facilitated investments. 51% of the surveyed ARTEMIS participants [37] assess that investments to their organisation have been as a result of their participation in the projects funded by the JUs.

53

7.1.1.5 Number of Projects Initiated, Success Stories and Evidence of Impact from Projects

Over the period 2008-2013 the ARTEMIS JU launched 6 calls and funded 56 projects engaging 1420 participant entities. The projects address major societal challenges, such as affordable healthcare and well-being, green and safe transportation, Smart Cities, new business opportunities for SMEs and Start-Ups. For citizens, development of new healthcare tools and services will reduce visits to doctors and hospitals and result in shorter times of hospitalisation. There is some factual evidence that projects have led to significant increases in productivity and reduced time-to-market after the adoption of innovative tools and methodologies (e.g. SYSMODEL [49], SHIELD [50]). In some cases, demonstrators were also developed into new products. For example the project CHIRON [51] led to the creation of new products for medical image viewing on tablets, and the HIGH PROFILE [52] project facilitated the viewing of brain images on tablets leading to the creation of new business opportunities. There is also evidence that projects results have been re-used in other ARTEMIS projects, e.g. the project CESAR [53] provided the foundations for ten follow-up projects in ARTEMIS. Project outcomes have also been used in other EU programmes (e.g. the results of the project IoE-Internet of Energy were exploited in a subsequent Trans-European transport network (TEN-T) project [54]). A notable success story from ARTEMIS is in the support and development for the AUTOSAR (AUTomotive Open System ARchitecture) [35] in the automotive sector. This was originally developed as a European standard for automotive systems integration but has now been adopted world-wide by manufacturers including in the US and Japan. AUTOSAR created and established an open and standardised software architecture for automotive electronic control units (ECUs) excluding infotainment. The standard enables scalability to different vehicle and platform variants, transferability of software, the consideration of availability and safety requirements, collaboration between various partners, sustainable utilisation of natural resources, and maintainability throughout the whole product life cycle. Other key projects that have produced impact within the embedded systems sector include:

EMC2 [55] which addressed Embedded Multi-Core Systems for Mixed Criticality applications in dynamic and changeable real-time environments. The aim of this project was to strengthen Europe as a leader in highly dependable embedded systems, generating novel software architectures and a complete tool set for very adaptable, scalable and secure systems. These have been demonstrated in automotive, avionics and space, industrial manufacturing, internet of things, and cross-domain applications. The project involves 99 partners from 19 European countries with a total budget of around €100 million.

The CRYSTAL [56] (Critical sYSTem engineering AcceLeration) aimed to reconfirm European leadership in safety-critical systems, by establishing an Interoperability Specification (IOS) and a Reference Technology Platform as a European standard. The project has resulted in ready-to-use, integrated tool chains with a high technology readiness level. The project engaged a budget of €82 million to support 71 partners from 10 countries.

The ARROWHEAD [57] project defined a framework for fast development of interoperable systems. Interoperability is considered to be a significant barrier in Industry 4.0 and large-scale energy infrastructure deployments. The framework targets development time savings up to 75% across production (manufacturing, process, and energy), smart buildings and infrastructures, electro-mobility and the virtual market of energy. Two large-scale demonstrators were deployed in Barcelona in 2015, a Smart Building pilot for real-time, intelligent sensing, monitoring and control for energy use and a Smart Urban Lighting pilot

54

with sensors that monitor environmental information (e.g. light intensity), energy use and mobility.

ARTEMIS has concentrated on developing frameworks that can be shared between different communities and on tools and methods to cope with the ever increasing complexity of smart digital systems. Large projects addressing interoperability such as CESAR, CRYSTAL and ARROWHEAD have created platforms that are used by other projects. A good example of the success of ARTEMIS is the company TTTech. This SME has strategically moved into different sectors with the support of a number EU projects and notably via the EMC2 and CRYSTAL projects. The company now has 300 employees and has been involved in over 100 EU projects. A number of successful products had been developed such as TTP for aerospace and TT Ethernet. The latter is now being used by Boeing, Airbus, and by NASA as the data bus for the Orion spacecraft which is the successor to the space shuttle. The company has also been successful in pursuing applications in rail, wind power and medical applications. The continued success of the company is being supported by engagement in projects concentrated on the next generation products developing time-triggered support for multi-core chips (TT-NOC) and TT Safety.

7.1.1.6 Working Groups Established to Support the Community

The ARTEMIS-IA has set up a number of working groups. These are:

WG Strategic Research Agenda: This working group published the SRA (2011) and the SRA addendum (2013) and created an annual update of the MASP and AWP from 2008 up to 2013. The WG SRA organised yearly Summer Camps from 2008 until 2013 and held at least one two-day workshop each year.

WG Standardisation: This working group created a standardisation agenda through the CSA project “PROSE” (FP7). Meetings were also held in 2013 and 2014.

WG Education and Training: The E&T WG had the mission of addressing the long-term sustainability of innovation ecosystems.

WG Success Criteria & Metrics: This key working group created a set of metrics and organised enquiries in 2010, 2012 and 2014. The results of these assessments were published.

WG SME involvement: The Working Group SME involvement was set up by the ARTEMIS Industry Association in January 2009. A major goal of the group was to promote the setting up and leadership of ARTEMIS projects by SMEs. This acknowledges that SMEs play a key role in the capitalisation and dissemination of embedded systems technologies.

WG Tool Platform: The aim of this WG was to create a labelling criteria with an application procedure for Tool Platforms. The CESAR Tool Platform was labelled in 2012.

WG Repository: A first Repository demo was presented at the Co-Summit 2012. A second release of the Repository was launched at the Pre-Brokerage event in 2014.

WG Centres of Innovation Excellence: Three CoIE’s were set up in 2011: EICOSE, ProcessIT.EU and ES4IB. Each Centre of Innovation Excellence (CoIE) is a group of multi-country, multi-organisation, interconnected R&D actors and businesses. CoIE’s create new, self-sustaining businesses, which in turn drive employment and social responsiveness. The aim is to create a significant advantage in innovation success in a specific market via co-operation.

55

7.2. ENIAC - Main Achievements and Effectiveness of Implementation

7.2.1 ENIAC – Main Achievements

7.2.1.1 Engagement with the Community

Between 2008 and 2013 ENIAC launched 9 calls and provided funding to 63 projects, with a total eligible cost amounting to nearly €2.9 billion. In total 1384 participant entities have engaged over the period 2008-2013. ENIAC has supported 14 pilot line projects across 23 countries.

The data produced by CARSA [37] has indicated that 88% of respondents believed that the research work would not have been completed were it not for the funding received from the ENIAC JU and 78% agreed that the topics being pursued would require further support from the JU. In assessing the impact on the nanoelectronics sector, 88% also believed that the work of the ENIAC JU would create commercialisable innovative outputs. In regards to job creation, 75% believed that the projects would have a high impact. There were a number of perceived benefits to participation in the ENIAC JU. These included:

keeping up with the state-of-the-art in technology advancements

networking with other research players

improving the speed of bringing innovations to market

This data correlates to that reported in the ECSEL JU Impact Analysis Study of 2016 [58] which reported that 75% of participants rated the impact of participating in the ENIAC/ARTEMIS/ECSEL JU as highly positive and beneficial to their organisation. The main benefits listed were:

technology development

benchmarking

networking on a global level

gaining insights into future developments

It was also noted by some participants that their organisations are less focused on attaining a return on investment from a financial perspective and more on the impact based strengthening their market position through the exchange of knowledge and best practices, while collaborating with the leading actors in the industry.

56

7.2.2.2 Inclusion of SMEs

Between 2008 and 2013 SMEs represented 23% of the participating organisations in ENIAC. This is less than in ARTEMIS, however, the nanoelectronics domain is a more focused research area that is dominated by large enterprises. The share of funding to SMEs, HEIs and RES organisations was also lower for ENIAC. Only 10% of the funds went to SMEs. The assessments by JU representatives and project participants with respect to SME involvement are relatively negative [37] and public representatives highlighted the market structure being geared towards larger companies. In most cases for ENIAC large enterprises dominate and so SMEs have a supplier role in the supply chain. Although improvements during the implementation of the JUs have been identified, complicated procedures and related administrative burden are the main barrier for SME involvement.

7.2.2.3 Number of Patents and Innovation Outputs

Number of Projects

EU Public Funding

(€M)

Patents Produced

Average number of patents per 10 €M of EU funding

Scientific publications

Average number of scientific publications

per €10M of EU funding

34 444.4 209 4.7 2381 53.58

Table 4 Results of ENIAC Completed Projects Up to and Including 2015 (Source: ECSEL Annual Activity Report 2015 [41], p.33)

Table 4 shows that a total of 34 projects launched under ENIAC came to successful completion up to and including 2015 and that 209 patents were filed over this period. ENIAC projects led on average to 4.7 patent applications per €10 million of EU funding which is higher than the Horizon 2020 benchmark of 3 patent applications per €10 million of EU funding. ENIAC projects produced 2381 scientific publications, an average of 53.58 scientific publications per €10 million of EU funding. This figure is significantly less than for ARTEMIS. These figures reflect the communication practices within the industry and the emphasis on patenting in this sector. 7.2.2.4 Success in Attracting Public/Private Funding

Figure 20 Breakdown of ENIAC Funding 2008-2013 (Source: ENIAC Annual Activity Report 2013[44]) The Council Regulation dictated that the Union had to contribute up to €450 million, of which €10 million was to cover a third of the ENIAC JU running costs. In fact ENIAC only requested €5.6 million for running costs and so €4.4 million was re-dedicated to research activities. Overall, between 2008 and 2013 the EU´s contribution of €444.4 million was 15.4% of the total eligible costs as shown in

57

Figure 20. The private sector, which legally had to provide at least 50% of the total research cost, actually provided €1829.5 million which accounts for 63.4% of the total eligible costs. The private sector also provided funds through AENEAS to cover 2/3 of the ENIAC JU running costs. Over the 2008-2013 period, total national contributions amounted to €570.6 million, representing 19.8% of the total eligible costs engaged in the programme. The ENIAC Activity report of 2013 [44] showed that “key application areas” made up only 31% of the eligible costs. This was largely due to the shift to the high TRL (4-8) KET initiated pilot lines where 9 projects focused on consolidating technologies on 200mm platforms, strengthening 300mm infrastructure and enabling the future transition to the 450mm platform. The strategy was largely motivated by the desire of the European Commission to arrest the decline in Europe’s share of the semiconductor market.

7.2.2.5 Number of Projects Initiated, Success Stories and Evidence of Impact from Projects

As highlighted there is a lack of follow up and metrics collection with respect to impact for ENIAC projects. In the absence of this the panel has assessed information that is available. In 2013, the ENIAC JU performed 33 technical reviews on 31 projects with experts nominated by the national public authorities [44]. 18% were confirmed as having excellent progress with the remaining 82% returning a good progress assessment. The percentage of excellent/good reviews in the subsequent 2 years was reported [41] as 14/82% and 20/74%. The evidence shows that project participants are committed to achieving the results which they originally set out to achieve. For the 9 ENIAC projects which were completed in 2015, the activity report [44] records a number of comments recorded on impact made by the project partners;

E2SG (energy to smart grid) “the consortium has met its objectives by a demonstration of the full value chain grid applications “

EPAMO (energy efficient piezo-MEMS tuneable RF front-end antenna systems for mobile devices ) “ almost all the industrial partners have at the end of the project something which is either already on the market or optimised to be brought to the market “

GREENELEC (sustainable product manufacturing) “Products based on GreenElec results are already on the market. Ultimately the main beneficiary of the results of the project will be the society and the environment , thanks to the better treatment of the electronic waste“

The PLACES2BE project, which concludes in 2017, (“Pilot Lines for Advanced CMOS Enhanced by SOI in 2x nodes, Built in Europe) has been a notable success “PLACES2BE has been a very successful pilot lines project which gave EU companies the ability to play a significant role in the most advanced CMOS technology platforms in the 28 to 14 nm nodes.”

A good example of success for ENIAC is the company AMS. With the help of the EU funding schemes, especially ENIAC and ECSEL, it has gained the strength and technological capabilities to transform itself from a foundry with commodity products into a specialist for producing sensors and sensor systems for a variety of markets. The company develops and manufactures high performance analog semiconductors for applications which require extreme precision, accuracy, dynamic range, sensitivity, and ultra-low power consumption. The company produces sensors, sensor interfaces, power management ICs and wireless ICs for customers in the consumer, industrial, medical, mobile communications and automotive markets. With an HQ in Premstaetten near Graz, Austria it has research and development facilities in 20 design centres world-wide employing around 3,300 people in over 20 countries. It distributes products world-wide via DigiKey, Future Electronics and Mouser.

58

This transformation has been achieved through R&D work as well as via a number of acquisitions. The actual R&D quota per sales has increased over the years to 25%. Another example of success is Infineon Technologies Austria which is the world leader in power electronic discretes and modules, in part thanks to co-ordinated ENIAC projects. Infineon established its “Competence Centre for Power Electronics” in Villach Austria during the phase of the ENIAC funding period and has carried out a large number of projects together with local companies, RTOs and universities as well as with international partners. The R&D effort has been significantly increased and many new positions have been created in Villach (the main fab site), Graz and Linz. There has been continuous investment in the production capabilities of the Villach site including a development of a 300 mm fab. The R&D quota has increased from 15% to over 20% with a corresponding increase in R&D employees. A good co-operation between the large semiconductor companies, SMEs, Universities and RTOS (as e.g. Joanneum Research and AIT) has been established in the southern Austrian regions. This was possible due to an established tradition of co-operation between the organisations active in this area. Other examples are the co-operation between ASML and Carl Zeiss SMT in Germany that was supported from EU funds through ENIAC, ST Microelectronics that has a strong market position in piezoMEMS as a result of the project Lab4MEMS, and ASML which is now a world leader in lithography and sells equipment around the world. 7.2.2.6 Working Groups Established to Support the Community

ENIAC has set up one Working Group that produces the Strategic Research Agenda on behalf of the ENIAC Steering Committee. The Strategic Research Agenda has been revised every two years, with interim updates issued when needed. Both ENIAC and ARTEMIS Austria has established local organisations (ENIAC-Austria and ARTEMIS-Austria; now followed by ECSEL-Austria) with the support of the Austrian Ministry for Transport, Innovation and Technology. These local organisations have been very active in promoting both JUs in Austria, helping organisations form efficient consortia and prepare high quality proposals.

7.2.3 Effectiveness of Implementation

7.2.3.1 Programme Administration of ARTEMIS and ENIAC

Both the ARTEMIS and ENIAC JUs have issued and updated Strategic Research Agenda’s in the period 2008-2013. This was done by a specific WG that has been set up by each JU. The aim of the SRA was to create an industry-driven, long-term vision in the JUs respective areas of embedded systems and nanoelectronics. The overall aim was to align and co-ordinate research policies in Europe with the goal to match programmes and resources to different technology and policy challenges.

In the case of ARTEMIS the mission of the SRA was based on maintaining a strong technological capability in both the supply and application of embedded systems by overcoming fragmentation in the European supply base for the components and tools of embedded systems. A key aim has been to develop multi-domain, reusable components and systems that can be used across multiple application sectors using the available resources (money, people) more effectively to strengthen the European industry base in this area. There is also a focus on extending the use of digital platforms to build stronger ecosystems for accelerating the innovation and the creation of new business models.

59

The SRA has been used as a reference by the European Commission and by a number of national bodies when establishing their own research priorities and programmes.

The ENIAC SRA [59] is an extended and detailed successor of the Executive Summary presented at the European Nanoelectronics Stakeholder Forum on April 27, 2005, in Brussels. The Strategic Research Agenda was revised every two years, with interim updates issued when needed. In the case of ENIAC the mission of the SRA was to enable European industry to remain competitive in the global market, by maintaining a strong technological capability even though there are few European semiconductor manufacturers. There has thus been a concentration on development of semiconductor manufacturing capabilities requiring high investment levels. Thus the SRA has focused on consolidating technologies on 200mm platforms, strengthening 300mm infrastructure and enabling the future transition to the 450mm platforms. Notably there is increasing competition in the semiconductor market, particularly from China [6] where the government is using industrial policies backed by over one hundred billion dollars in government-directed funds to reshape the market in its favour. This has been noted by the US who in the President's Council of Advisors on Science and Technology (PCAST) report on Ensuring Long-Term U.S. Leadership in Semiconductors (January 2017)[7] highlight the need to innovate at the cutting edge to compete. The report recommends a three pillar strategy to (i) push back against innovation-inhibiting Chinese industrial policy, (ii) improve the business environment for U.S.-based semiconductor producers, and (iii) help catalyse transformative semiconductor innovation over the next decade. Delivering on this strategy will require co-operation among government, industry, and academia.

Both ARTEMIS and ENIAC have made annual calls for projects based on the defined SRAs. The analysis of the impact of the projects indicates that the projects funded as a result of these calls were of good quality. With the move to larger application based projects and pilot lines the impact of the projects has also increased particularly in bringing industry together around key topics. The quality of the funded projects reflects the rigour and suitability of the proposal evaluation and selection procedures. In the early years of the two JUs the evaluation and selection procedures were highlighted to be an issue and this was emphasised in the 1st Interim Evaluation of the two JUs. Over the final years of operation the JUs have streamlined their evaluation and selection procedures with the procedures for issuing a call for proposals and the subsequent evaluation processes for ENIAC and ARTEMIS following closely the approach used by the European Commission in the FP7 and H2020 programmes. For efficiency and to ensure quality, the overall procedure was divided into 2 stages. In the first stage the call was issued along with associated eligibility criteria (unfortunately some MS did not provide this information until the selection process was performed by the JU). In response to this consortia were formed with supporting brokerage events being provided by ARTEMIS and ENIAC. As a result project outlines were first submitted which were then evaluated by independent experts (here 50% of the experts came from industry) and feedback was provided to proposers. After this, in the second stage, a full proposal was submitted and again evaluated and ranked by independent experts. The final ranked list of proposals was provided to the PAB which made the final funding decisions with the aid of a support tool that expressed the National strategic interests. In certain calls, for both ENIAC & ARTEMIS, a single-stage application process was used, with the requirement to submit a formal project outline being removed. Both the two-stage and single-stage application processes appeared to work well, resulting in projects of high quality being selected for funding. Although there were some issues with the availability and selected priorities for Member State funding the process of contract negotiation and budget engagement were not highlighted as issues in interviews.

60

Figure 21 Survey of Coordinators on the Administrative Life-Cycle (Source: CARSA [37])

A survey of 32 coordinators performed by CARSA [37] gave the results shown in Figure 21. Here it can be seen that overall a large percentage of coordinators still felt that the application, evaluation and grant procedures were too complex. Additionally, the view is confirmed that the monitoring and reporting procedures are overly complex (here one needs to take into account that some countries request additional reporting).

Figure 22 Complexity of Administrative and Management Procedures Compared to Other

Framework Programmes (Source: CARSA Survey [37])

Coordinators were also asked about the relative complexity to engage with other framework programmes indicating that the majority felt that the ARTEMIS/ENIAC management complexity was greater as shown in Figure 22.

0.0% 3.1% 0.0% 0.0% 0.0% 0.0% 15.6%

18.8% 18.8% 18.8% 12.5% 15.6%

59.4% 53.1% 43.8% 37.5% 53.1% 43.8%

21.9% 18.8% 34.4% 40.6%

31.3% 37.5%

3.1% 6.3% 3.1% 3.1% 3.1% 3.1%

0%10%20%30%40%50%60%70%80%90%

100%

Complexity level of programme administration and management (coordinators only)

Not at all To a low level To a fair level To a high level Do not know

Yes 61.3%

No 38.7%

Administrative and management procedure more complex in

ENIAC/ARTEMIS than other FPs (coordinators only n=31)

Yes No

61

7.2.3.2 Relevance of Stakeholders

ARTEMIS has good engagement across the embedded systems community driven by end user applications. As highlighted many projects result in demonstrators of technologies at a relatively high TRL. As some applications are safety critical in this domain there is also engagement with public bodies and regulators in some cases.

Figure 23 List of Major ARTEMIS Beneficiaries by Volume of Funding (€M) and Number of Participations. (Source: CARSA study [37])

Figure 23 shows the major beneficiaries in ARTEMIS [37]. These include major aircraft companies such as EADS and THALES, medical electronics companies such as Philips, automotive drivetrain specialists AV LIST, car manufacturers such as Fiat and well-known research organisations across Europe. Notably there is also strong participation from electronics manufacturers such as Infineon and NXP who also embed their electronics into a number of applications that are addressed by ARTEMIS.

62

Figure 24 List of Major ENIAC Beneficiaries by Volume of Funding (€M) and Number of Participations. (Source: CARSA study [37])

Figure 24 shows the major beneficiaries in ENIAC. Here there is a strong participation from semiconductor manufacturers and research centres in this field. In line with the character of the JU, there is a lower participation of end users in ENIAC than in ARTEMIS. One major end user company is Philips from the medical domain where there is interest in miniaturised electronics for in-vitro applications.

7.2.3.3 Engagement of Best European Players in ARTEMIS and ENIAC

Although key European players are engaged in ARTEMIS the embedded systems area is very diverse and it is notable that the representation of major company CEOs is lacking. In ENIAC large amounts of overall funding are concentrated among a limited number of actors [37]. Key actors are Fraunhofer, Infineon, STMicroelectronics, NXP Semiconductors, CEA, TU Eindhoven, TNO and TU Delft. An analysis by CARSA [37] of both the ARTEMIS and ENIAC research organisations indicated that Fraunhofer, Infineon, STMicroelectronics, NXP Semiconductors, CEA, TU Eindhoven, TNO and TU Delft are part of the core network of both JUs showing high interactions with several industry participants among the core network. The interviews performed by CARSA [37] highlighted that 71% of beneficiaries and stakeholders believe that the JU community has brought together the best competencies in the industry across the EU. The cross-disciplinarily and trans-nationality of the players is perceived as a strength which fosters innovation across the regions. 7.2.3.4 Overall Recommendations

In summary, the ability to assess the lasting effects of the JUs is hampered by the lack of existing measures of the impact from a socio-economic perspective. This has been more rigorously addressed by ARTEMIS in the setting up of a Working Group to address KPIs and the establishment of targets. For the ENIAC JU impact metrics have not been collected but there is evidence of impact. It is clear that certain large industrial enterprises have benefited and gained leadership via the pilot line projects. Both the Annual reports for ARTEMIS and ENIAC highlight success stories but a more

63

systematic approach is needed to collecting data on impact. While great care is taken on measuring and reporting on input parameters such as funding levels, participation rates, etc., and interim measures such as detailed progress reports are recorded, little effort is put into collecting objective measures on outputs. In order to monitor impact it is not only necessary to collect data during the project but also after the project has finished for at least a year.

Table 5 Project Outputs as Reported by Coordinator and Participant Respondents to CARSA study [37]

The survey performed by CARSA [37] of coordinators and participants to ARTEMIS and ENIAC indicated a range of impacts that could be considered in future impact assessment (See Table 5). A recommendation here would be to enforce the systematic collection of metrics against KPIs such as those indicated in the table above (or a subset depending on the nature and TRL of the project) and also to monitor how well the JUs are meeting their objectives. Here for lower TRL projects it is possible that targets will not be reached as there is more risk of failure but this is to be expected if the projects are truly innovative. Additionally, respondents were asked about technology and manufacturing readiness levels of ENIAC and ARTEMIS projects. Overall this indicated that the technology and manufacturing readiness is substantially increased in both the ENIAC and ARTEMIS projects. The results suggest that ENIAC projects begin at a lower TRL and MRL than ARTEMIS projects. Moving to higher TRL levels requires significantly more funding.

64

7.2.4 ARTEMIS and ENIAC Joint Undertaking's Performance

7.2.4.1 ARTEMIS and ENIAC JU Mission and Governance

The role of both ARTEMIS and ENIAC JUs was to support Research and Innovation (R&I) specifically in embedded systems and nanoelectronics respectively, as well as for combining national and Union funding. A central element of their implementation was increasing and leveraging private and public investment in the related sectors in Europe. They were to achieve effective co-ordination and synergy of resources and funding from the 7th Framework Programme, industry and national R&D programmes, thus contributing to strengthening Europe's future growth, competitiveness and sustainable development, in their respective fields. Finally, they were to foster collaboration between all stakeholders such as industry, including small and medium-sized enterprises (SMEs), national authorities, academic and research centres, by establishing consistency in and focuses on research efforts.

7.2.4.2 Clarity of Legal Framework for ARTEMIS

The ARTEMIS JU was a tri-partite public private partnership that was established in February 2008 under Council Regulation (EC) No 74/2008 of 20 December 2007 [26] as a Community body based in Brussels. The aim of the ARTEMIS JU was to implement significant parts of the Strategic Research Agenda put forward [27] by the ARTEMIS-IA co-funded by industry, research organisations, participating Member States and the Commission's ICT programme. The administrative requirements for ARTEMIS are contained in the Council Regulations [21]. Additionally, the regulations specify operational and outcome-oriented objectives. The operational objectives highlight the need for the following: • Regular review and updating of the research agenda and a Multi-Annual Strategic Plan • The preparation and implementation of an Annual Work Programme • Calls for, evaluation and selection of proposals • Funding and oversight of selected projects With respect to the expectations from the ARTEMIS JU the overall clarity of the framework is very clear with well-defined objectives.

65

7.2.4.3 ARTEMIS Governance

Figure 25 ARTEMIS Governance Structure (Source: http://www.artemis-ia.eu/about_artemisia)

The management structure of the ARTEMIS Joint Undertaking is composed of several bodies as illustrated in Figure 25 [8]. The ARTEMIS European Technology Platform was first established in June 2004 with the aim to bring together key players in the Embedded Computing arena across the entire spectrum of industrial sectors. 17 major companies were involved. One of the core tasks was to define a common Strategic Research Agenda (SRA) to act as a reference for the Embedded Computing domain to attract investment from stakeholders. The first version of the SRA was published in March 2006. In addition, a "Mirror Group" was formed which gathered representatives from 24 Member States and Associated Countries. This group went on to become a founder of the Joint Undertaking alongside the Commission and ARTEMIS-IA.

The management bodies formed include the Governing Board, Industry and Research Committee, Public Authorities Board and the Executive Director. The Governing Board has overall responsibility for the operations of the ARTEMIS Joint Undertaking. Its role is to oversee the implementation of the JU. It consists of representatives from Industry (ARTEMIS-IA) and Public Authorities including the Commission and Member States. Voting rights are split equally: 50% for Industry and 50% for public authorities. The Industry and Research Committee represents the interests of industry and the research community through the ARTEMIS-IA. Its role is to draft the Multi-Annual Strategic Plan based on the Research Agenda. In addition, it drafts an Annual Work Programme for the activities of the JU including calls for research proposals. The Public Authorities Board consists of representatives of the ARTEMIS Member States and the European Commission. It is responsible for the decisions on the scope and budget of the calls for proposals, selection of proposals and allocation of public funds for selected proposals. A third of the voting rights are assigned to the Commission and the remaining two thirds are allocated to Member States. The 23 ARTEMIS Member States are: Austria, Belgium, Cyprus, Czech Republic, Denmark, Estonia, Finland, France, Germany, Greece, Hungary, Ireland, Italy, Latvia, Netherlands, Norway, Poland, Portugal, Romania, Slovenia, Spain, Sweden, and United Kingdom. The Executive Director is the chief executive of the Joint Undertaking whose role is to ensure its day-to-day management. The Executive Director is appointed by the Governing Board, for a period of three years and is supported by a secretariat - the ARTEMIS-JU Office - which handles the operational aspects of the JU. Governing Board: The overall responsibility for the operations of the ARTEMIS Joint Undertaking lies on the Governing Board, whose role is to oversee the implementation of the JU. It consists of

66

representatives from Industry (ARTEMIS-IA) and Public Authorities including the Commission and Member States. Voting rights are split equally: 50% for Industry and 50% for public authorities. Industry and Research Committee: The Industry and Research Committee, consisting of members appointed by ARTEMIS-IA (the ARTEMIS Industrial Association), represents the interests of industry and the research community. Its roles include drafting of the Multi-Annual Strategic Plan, based on the Research Agenda, and the preparation of an Annual Work Programme for the JU activities, including calls for research proposals. Public Authorities Board: The Public Authorities Board consists of representatives of the ARTEMIS Member States and the European Commission. It is responsible for the decisions on the scope and budget of the calls for proposals, selection of proposals and allocation of public funds for selected proposals. A third of the voting rights are assigned to the Commission and the remaining two thirds are allocated to Member States. Executive Director: The Executive Director is the chief executive of the Joint Undertaking whose role is to ensure its day-to-day management. The Executive Director is appointed by the Governing Board, for a period of three years, and is supported by a secretariat, the ARTEMIS-JU Office, handling the operational aspects of the JU.

7.2.4.4 ARTEMIS Monitoring and Control

The research agenda and a Multi-Annual Strategic Plan have been updated regularly by the Industry and Research Committee. An Annual Work Programme has been issued following consultation between the members of ARTEMIS and with input from various sources. This has been used as the basis for calls for proposals which were agreed with the Public Authorities Board. ARTEMIS has also successfully evaluated and selected proposals for funding. The PAB contributed to the integration of all national requirements and rules in the calls, and supported ARTEMIS by carrying out financial viability checks and financial audits of project participants on behalf of the Joint Undertaking. In total over the period 2008-2013 the ARTEMIS JU successfully launched 6 calls and funded 56 projects. The funded projects have been reviewed periodically and support has been provided to proposers in order to manage projects. In particular, the progress and achievements, hence quality and likely future impact, of projects is significantly affected by the quality of the project management. Over the final 2 years of the JTI very large ARTEMIS AIPP projects were introduced with large budgets and involving many partners (up to 100). Successful management of such large projects requires professional and efficient management. Stakeholder interviews have highlighted the challenges and overheads in managing such projects with a need for efficient and streamlined management processes and supporting management skills. An issue is that certain Member States do not allow funding for project management (i.e. management costs are ineligible for MS funding in some cases) which made engagement unattractive. Notably ARTEMIS has also dedicated some effort to collecting metrics on the performance of projects during execution. This has proved very useful for monitoring progress and the potential impact of projects. An issue highlighted is that there is a lack of resource for post project monitoring in order to assess the true impact and outcomes of projects.

7.2.4.5 ARTEMIS Dissemination

ARTEMIS has been very active in dissemination promoting the JTI and its projects widely to the stakeholder community, including the general public. This has exploited various communication tools such as an up-to-date, informative website, the use of social media, organisation of and/or participation in events, seminars and conferences and publications in the written press. In addition to this there is a dedicated space on the website for the dissemination of project results and ARTEMIS

67

has published documents highlighting the impact of key projects. The ARTEMIS projects have also been disseminated via the FP7 and Horizon 2020 dissemination platforms. The ARTEMIS-IA has also organised public events, forums, publications and announcements to promote the added value and impact of individual projects. Additionally, there has been a move for closer co-operation and co-ordination with other EU projects and JUs, through inclusion of stakeholders' advisory bodies, via the set-up of separate Memoranda of Understanding and via exploration of synergies with other (national and regional) programmes.

7.2.4.6 Source of Financing

The R&D funding target for ARTEMIS was €2.5 billion. The total eligible cost for the 6 calls between 2008 and 2013, however, only achieved €1.1 billion [38] falling short of the target. Although industry strongly supported ARTEMIS and has invested according to the original plan the national Member States have invested significantly less than planned over the period. In total the Member State and EU financial commitments were around €340 million and €181 million respectively over the period 2008-2013. Notably the move to larger projects addressing higher TRL developments was more attractive for industry and led to a higher funding commitment from the sector.

Table 6 Administrative Budget for ARTEMIS 2009-2014 (Source: ARTEMIS) The administrative costs for ARTEMIS are shown in Table 6. The proportion of budget allocated to administration rose in 2010 but has been managed to be within 6-8% in the period 2011-2014 commensurate with changes in the overall budget.

The first Strategic Research Agenda (SRA) was issued by the ARTEMIS-IA in 2006. This was then updated based on work by the ARTEMIS JU and issued in 2011 setting out the long-term strategy and the policy objectives of the programme taking stock of the rise of new technological opportunities and socio-economic challenges. According to the ARTEMIS Strategic Research Agenda of 2011, the general objective of the ARTEMIS JU is to overcome the fragmentation between supply of embedded systems and R&D investments. A key aim was to build greater linkages between the application sectors in order to “de-verticalise” the industry and facilitate the cross-sectoral sharing of tools and technology. Here an aim is help products and services take advantage of the new technological capabilities (such as ubiquitous wireless connectivity, cloud computing and semantic information). This was developed via the Industry and Research Committee and was supported by the Governing Board which involves representatives from Industry, the Commission and Member States. Industry had 50% of the voting rights ensuring a strong industrially driven research agenda. The PAB was responsible for funding of projects. A key aspect of the ARTEMIS strategy was to cut barriers between the applications contexts by producing multi-domain reusable results. This was to create critical mass and avoid the fragmentation of R&D effort which is seen across Europe. The ARTEMIS strategy established common technology to support the development of high value-added Embedded Systems solutions that are reusable across a wide range of application sectors and can be integrated so as to respond to the societal challenges. The ARTEMIS SRA that was issued in 2011 sets out the three following societal challenges to be addressed:

Affordable healthcare and well-being

Green, safe and supportive transportation

Smart buildings and communities of the future

2014(H1) 2013 2012 2011 2010 2009

TOTAL ADMINISTRATIVE 970.104,26 1.982.002,48 2.224.196,24 1.949.435,64 1.896.580,58 302.614,51

TOTAL BUDGET 14.519.614,84 27.827.745,50 27.898.493,88 31.700.995,12 14.755.333,43 3.725.192,32

Administrative as % of total 7% 7% 8% 6% 13% 8%

68

Over the course of the ARTEMIS programme, the scope of the strategy evolved so as to better adapt the work carried out to the emergence of new technological and societal trends. In the original ARTEMIS SRA 2006, embedded systems are mainly considered as an independent key enabling technology, while in the ARTEMIS SRA 2011 greater emphasis is placed on the pivotal role played by networked embedded systems. “In future, the emerging use of the Internet for embedded system networking provides new opportunities. Embedded Systems will be able to exploit the emerging ubiquitous network typology not only for communication but also to gain access to the knowledge of Internet based information systems. In turn, information systems will utilise Embedded Systems as sources of information to enable the Internet of Things. The availability of digital information from the physical environment is a unique opportunity for the Embedded Systems industry” (ARTEMIS SRA 2011, p.18). The change in ARTEMIS focus from Embedded Systems to Networked Embedded Systems in the SRA 2011 paved the way for a greater emphasis on Cyber-Physical Systems which have been a feature in the ARTEMIS SRAs for the ECSEL JU.

7.2.4.7 Operational Effectiveness

The operation of ARTEMIS has closely followed the legal framework that established it and has been refined over the course of operation. The ARTEMIS JU has maintained a Strategic Research Agenda that covers embedded systems and tools. It has also initiated, managed and co-ordinated research activities through open calls for proposals and via funding research projects. Following an interim review there was a change in scope of the JU over the final two years to move towards larger integrated projects addressing higher Technology Readiness Levels (TRLs). This change of scope was beneficial and led to increased industry commitment. At the same time the ARTEMIS strategy was changed to support developments of common technology that could be reused across a wide range of application sectors. More emphasis was placed on responding to societal challenges in affordable healthcare and well-being, green, safe and supportive transportation, and smart buildings and communities of the future. The flexibility provided within the legal framework to change project size and emphasis were important in ensuring the relevance of ARTEMIS to industry. Although industry has supported ARTEMIS strongly Member States have been less supportive and the leveraging expectation set out (fixed 1.8 Member State/EU factor) has not been met.

The unique ability of ARTEMIS has been to address large-scale problems by bringing together different funding sources to create funding that will support critical mass. This compares with other sources of funding such as those available nationally, in EUREKA or via FP7 that could address smaller and more bottom-up developments. Another difference was in the ability to address higher TRL developments compared with FP7. In general the funding volume in key areas supported nationally and under FP7 is not sufficient to create a “level playing field” with international competitors. Although a variety of funding sources exist programmes tend to run in parallel and are not harmonised resulting in fragmented and duplicated effort across Europe. The procedures used for issuing a call for proposals and the subsequent evaluation processes for ARTEMIS mirror those of FP7. A call is issued with documentation outlining the associated eligibility criteria; consortia are formed supported via ARTEMIS brokerage events; project outlines are submitted which are evaluated by independent experts (with the obligation that 50% of the experts are from industry); feedback is provided to proposers; who then submit a full proposal which is again evaluated and ranked by independent experts; and finally a ranked list of proposals is provided to the PAB who make funding decisions. In certain calls, a single-stage application process was used, with the requirement to submit a formal project outline being removed. Both the two-stage and single-stage application processes worked well, resulting in high quality projects being selected for funding. Although the initial process adopted led to some criticism from proposers significant efforts were made to streamline the evaluation and selection procedures and this was successful.

69

7.2.4.8 Satisfaction of Beneficiaries

In terms of bringing the community together via events and developing the Strategic Research Agenda ARTEMIS has done a good job. Interviews performed by CARSA [37] highlighted that 71% of beneficiaries and stakeholders believe that the ARTEMIS and ENIAC JUs have brought together the best competencies in the industry across the EU. The cross-disciplinarily and trans-nationality of the players is perceived as a strength which fosters innovation across the regions. Additionally, respondents highlighted that the technology readiness level and manufacturing readiness is substantially increased in projects. The added value of ARTEMIS was rated positively with particular contributions to “dialogue between researchers and industry” and “achievement of critical mass”. When considering the funding of projects beneficiaries have highlighted a number of issues. The onerous legal and administrative requirements of the JTIs is a barrier to participation for project coordinators as well as participants.

Potential proposers also indicated the difficulty in getting sufficient information on, and in understanding, the varying national rules and procedures for engagement in projects. There is also a difficulty in understanding the differences between EU funding and the individual national funding rules. Here there is a need for harmonisation and synchronisation in order to streamline the national selection processes. In some cases national and EC budgets could not always be fully exploited, due to varying degrees of oversubscription in JTI programmes by different countries. Considering proposal submission the administrative burden of submitting proposals was also considered to be onerous. The move to larger projects, while welcomed by some participants, had also led to the belief by smaller players that the big established players were being favoured and there was less chance for new actors to engage. The move towards very large projects also introduces more complex management with associated overheads. 7.2.4.9 Operational Efficiency The different bodies of the ARTEMIS JU organisation have functioned well. ARTEMIS has executed calls and the subsequent selection procedures as planned. Negotiations following the funding decisions were performed and supported professionally. In particular, the negotiations following the decision on projects in the last call were concluded in record time in order to complete them before the end of the last full year of ARTEMIS' existence. The ARTEMIS 2013 Annual Activity Report highlights a number of Key Performance Indicators. Considering the time to pay beneficiaries 95% of payments were done within the contractual time limits with a total average (including 64 late payments) of 20 days.

70

Year 2012 2013

Call 2011 2012

Days From FPP closing to End of Negotiation date (average)

198 188

Days From FPP closing to Signature of Grant Agreement

361 431*

Table 7 Average Times to Initiate a Grant (Source: ARTEMIS) The average time to grant, i.e. the period from the closing of the Full Project Proposal (FPP) phase until the signature of the grant agreement, is shown in Table 7. This is used as a standard measure to calibrate the performance of the office. It should be noted that the time to grant increased in 2013 (see *) due to higher level of complexity for the first AIPPs negotiations (ARROWHEAD and CRYSTAL) which have large number of participants. An internal procedure was subsequently put in place to improve this for future calls. There are two main reasons for delays. The first is introduced by the need to harmonise between calls with a Proposal Outline (PO) phase and the ones with no PO phase, and the second occurs after the End of Negotiation, with the need by the National Authorities to obtain the signature from coordinators for National Grant Agreements. This is an essential requirement for the Joint Undertaking Grant Agreement but this is not under the control of ARTEMIS JU.

Figure 26 Office Structure of ARTEMIS (Source: ARTEMIS) The Community body status of the ARTEMIS JU requires that it follows the EC rules for finances and staff. The ARTEMIS Office was set up to be a lean organisation as shown in Figure 26. The ARTEMIS Office staff grew to a total of 13 people in 2011 consisting of Technical Administrators (TA) and Clerical Assistants (CA) and maintained this strength until 2013. During the last half year before the transition to ECSEL, there were 12 staff members. Special attention was given to team building in order to make the office more efficient. With the low number of staff, ARTEMIS was able to make all needed financial transactions (payment of salaries, payments to project participants, etc.) according to the Financial Regulations, with all financial processes being defined, including the necessary back-ups.

71

Table 8 Operating Costs of ARTEMIS Office

The operating costs for the ARTEMIS Office are shown in Table 8. The collaboration of ARTEMIS with the Member States through the PAB allowed for a lean administrative structure for the JU with procedures that were not disruptive for national administrations. Contractual models familiar to the ARTEMIS participants were also utilised which also lessened the burden of evaluation and technical review procedures.

7.2.4.10 Performance of ENIAC

7.2.4.11 Clarity of Legal Framework for ENIAC

The “formal specifications” for ENIAC JU which was created in 2008 [9] are contained in the Council Regulation No 72/2008, Art. 2 [22]. The regulations set out the administrative requirements to ensure the probity of the operation, as well as operational and outcome-oriented objectives. ENIAC targets nanoelectronics with the aim of performing research to enhance the further integration and miniaturisation of devices and increasing their functionalities. The main operational objectives were to:

Define and implement a Research Agenda

Award funding to participants

Mobilise both public and private efforts to increase overall R&D investments

Achieve synergy and co-ordination of European R&D efforts

Promote the participation of SMEs The legal regulations also cover the accession rules and the structure of the Joint Undertaking, the rules for the implementation of R&D activities and the financial rules and the IP rules governing the ENIAC Joint Undertaking.

72

With respect to the expectations from the ENIAC JU the overall clarity of the framework is very clear with well-defined objectives.

7.2.4.12 ENIAC Governance

Figure 27 ENIAC Governance Structure (Source: http://www.eniac.eu/web/downloads/Brochure/eniac_ju_brochure2.pdf)

The ENIAC JU was set up as a public-private partnership, bringing together the European Commission and European Member and Associated States with AENEAS, the association representing the R&D actors in nanoelectronics (Corporate, SME's, research institutes and universities) in Europe. Industry and research organisations participated in the ENIAC JU through the AENEAS association. The Governance Structure of ENIAC is shown in Figure 27. The main tasks of the ENIAC JU were to co-ordinate European research in Nanoelectronics by implementing a Research Agenda, organising calls for proposals and managing the execution of research projects.

To support this the ENIAC JU had a governance structure including a:

Governing Board

Public Authority Board

an Industry & Research Committee Overall it was managed by an Executive Director. Governing Board: The overall responsibility for the operations of the ENIAC Joint Undertaking lies on the Governing Board, whose role is to oversee the implementation of the JU. It consists of representatives from Industry (AENEAS) and Public Authorities including the Commission and Member States. Voting rights are split equally: 50% for Industry and 50% for public authorities. Industry and Research Committee: The Industry and Research Committee, consisting of members appointed by AENEAS, represents the interests of industry and the research community. Its roles include drafting of the Multi-Annual Strategic Plan, based on the Research Agenda, and the preparation of an Annual Work Programme for the JU activities, including calls for research proposals.

73

Public Authorities Board: The Public Authorities Board consists of representatives of the ENIAC Member States and the European Commission. It is responsible for the decisions on the scope and budget of the calls for proposals, selection of proposals and allocation of public funds for selected proposals. A third of the voting rights are assigned to the Commission and the remaining two thirds are allocated to Member States. Executive Director: The Executive Director is the chief executive of the Joint Undertaking whose role is to ensure its day-to-day management. The Executive Director is appointed by the Governing Board, for a period of three years, and is supported by a secretariat, the ENIAC-JU Office, handling the operational aspects of the JU. 7.2.4.13 ENIAC Monitoring and Control

The quality of the project portfolio that has been funded gives an indication of the effectiveness of the proposal evaluation and selection procedures. During operation the research agenda and a Multi-Annual Strategic Plan have been updated regularly. An Annual Work Programme has also been issued which has been used to initiate calls. This was developed as a result of consultation between members of ENIAC with input from other sources. ENIAC has also successfully evaluated and selected proposals for funding according to the rules set out. Between 2008 and 2013 ENIAC launched 9 calls and provided funding to 63 projects. The funded projects have been reviewed periodically by independent experts and have been reviewed favourably. ENIAC has also provided management support for funded projects. An area where ENIAC has been remiss is in the collection of metrics to monitor the performance of projects. There is also no post project monitoring of impact. This was noted in previous reviews and it was highlighted that innovation alone is not sufficient for ensuring long-term impact. Although it was recommended that greater effort should be made to ensure that project results are exploited as fully as possible, and that exploitation should be rigorously monitored this did not happen in practice. Rather the final project reviews concentrated on the level of achievement of project deliverables, milestones and overall objectives. 7.2.4.14 ENIAC Dissemination

ENIAC has disseminated widely to the stakeholder community in nanoelectronics and also to the general public. Various communication tools have been used including a maintained website and through the use of social media. ENIAC has also organised/or participated in events, seminars and conferences. A large number of publications have also been produced. The website includes an area for the dissemination of project results and a number of documents/press releases have been published. There are also videos of presentations. The ENIAC projects have also been disseminated via the FP7 and Horizon 2020 dissemination platforms. A number of public events and forums have been organised to promote the added value and impact of individual projects. Links to other projects have also been made such as CATRENE, MEDEA+, ITEA2 and EUREKA.

7.2.4.15 Source of Financing

In the Council Regulation the European Union was committed to provide €450 million. €10 million of this was allocated to fund a third of the ENIAC JU running costs. In practice only €5.6 million was required for running costs and so €4.4 million was re-dedicated into research activities. Between 2008 and 2013 the EU provided €444.4 million (15.4% of the total eligible costs) while the private sector provided €1829.5 million 63.4% of the total eligible costs. This surpassed the legally required amount of 50% of the total research cost. Additionally, the private sector also provided funds through AENEAS to cover 2/3 of the ENIAC JU running costs. Over the 2008-2013 period, total

74

national contributions amounted to €570.6 million, representing 19.8% of the total eligible costs engaged in the programme.

Table 9 Administrative Costs for ENIAC (Source: ENIAC)

The running costs for ENIAC are shown in Table 9. These rose to 12% in 2011 but reduced in relation to the overall budget in subsequent years as the total budget nearly trebled. The Strategic Research Agenda was heavily influenced by the shift to the high TRL (4-8) KET initiated pilot lines. This was motivated by the desire of the European Commission to stop the decline in Europe’s share of the semiconductor market. To implement this shift in the SRA, 9 projects were funded focused on consolidating technologies on 200mm platforms, strengthening 300mm infrastructure and enabling the future transition to the 450mm platform. These were formulated using input from industry and the academic community to define research into production and nanoelectronics. As ENIAC became more established funding aligned itself around 5 main tracks in micro-electronics. The ENIAC Activity report of 2013 [44] showed that “key application areas” made up only 31% of the eligible costs.

The ENIAC MASP [45] identifies 8 key work areas that address socio-economic and technological challenges. The socio-economic applications address:

Automotive and Transport

Communication and digital lifestyle

Energy efficiency

Health and ageing society

Safety and Security The technological enablers are:

Design technologies

Semiconductor process and integration

Equipment, materials and manufacturing

7.2.4.16 ENIAC Operational Effectiveness

The operation of ENIAC has closely followed the legal framework that established it. The direction of ENIAC was refined during the period with the uptake of pilot line projects as a result of input from KET and also due to interim reviews. The change in emphasis towards larger projects addressing higher Technology Readiness Levels (TRLs) was initiated in the last two years of ENIAC. This led to increased industry commitment which was beyond that required legally.

It is notable that the combined investment in each pilot line is greater than can be supported via EU money alone in the area under FP7. Thus the ability to create critical mass to tackle areas with much larger investment has provided benefits particularly to larger companies engaged in projects. In ENIAC there is a strong participation from semiconductor manufacturers and research centres in the field. In general 1 Euro from the EC has resulted in 4 Euros being invested by other partners in leveraging. The KET initiated pilot lines have consolidated technologies on 200mm platforms and strengthened 300mm infrastructure. Interviewed stakeholders highlighted that they believed that

2014(H1) 2013 2012 2011 2010

TOTAL ADMINISTRATIVE 2.356.000,00 2.686.049,00 2.475.500,00 2.631.700,00 2.272.000,00

TOTAL BUDGET 76.356.000,00 38.813.865,00 42.475.500,00 22.128.517,00 24.883.654,00

Administrative as % of total 3% 7% 6% 12% 9%

75

commercialisable innovative outputs would be produced that would result in job creation. Notably there is a much lower participation of end users. In 2012, the Governing Board mandated that the Executive Director should appoint an independent observer to assess the call procedures. His report to the Governing Board confirmed the fairness and transparency of the process and identified opportunities for improvement, which were implemented by the JU. 7.2.4.17 Satisfaction of Beneficiaries

Interviews performed by CARSA [37] highlighted that 71% of beneficiaries and stakeholders believe that the ARTEMIS and ENIAC JUs have brought together the best competencies in the industry across the EU. The cross-disciplinarily and trans-nationality of the players is perceived as a strength which fosters innovation across the regions. Additionally, respondents highlighted that the technology readiness level and manufacturing readiness was substantially increased in ENIAC projects. The added value of the ENIAC JU was also rated positively although less positively than the ARTEMIS JU.

As with ARTEMIS the legal and administrative requirements for the JU were seen as a barrier, however, the ENIAC community differs from that in ARTEMIS in being dominated by a few large companies with many of the smaller parties being their suppliers. The main players hence drove the ENIAC strategic research agenda. This also resulted in a better participation in the Governing Boards of top-level management. The tri-partite nature of the JU bringing together the EC, Member States and Public Authorities inevitably leads to complex administrative procedures. Similar to ARTEMIS potential proposers highlighted the difficulty in understanding the varying national rules and procedures for engagement in projects. The administrative burden of submitting proposals was in general considered to be onerous. The move to larger projects while supporting the needs of an industry driven by larger companies resulted in a lower engagement with SMEs within ENIAC. The move towards very large projects also introduced more management overheads. 7.2.4.18 ENIAC Operational Efficiency

The ENIAC 2013 Annual Activity Report highlights that the JU could perform the project evaluation and selection in less than 100 days from Full Project Proposals submission to the PAB funding decision while maintaining a high quality selection process. After the PAB funding decision, the contractual basis must be established before the projects can start. The 2013 AAR highlights that “as per the Administrative Agreements, the NFA shall execute a National Grant Agreement with the participants from that country within 60 days after the PAB funding decision”. The overall time to grant target was thus set at 160 days to include the project evaluation and grant agreement times. No data was available for assessment of the average time to grant that was achieved although it was noted that in some cases there were delays in signing the National Grant Agreements by Member States over which the JU had no control. In one case Italy stopped signing grants altogether for a period.

76

Figure 28 a) Cumulative Payments as a Percentage of the Approved JU Grants and b) Time to

Payment (Source: ENIAC AAR 2013)

The JU made payments close to the expected schedule as the pipeline of projects was filled, but payments for projects approaching completion were delayed by the policies implemented by the National Funding Authorities. A NFA will typically perform thorough checks and financial audits before closing a project. These procedures are needed to provide financial assurance with regard to the regularity and correctness of all payments, however, they introduce several months of delay before the final payments can be executed.

Figure 29 ENIAC Office Structure (Source: ENIAC) The Community body status of the ENIAC JU requires that it follows the EC rules for finances and staff. The ENIAC office was set up to be a lean organisation as shown in Figure 29. The ENIAC Office staff grew to a total of 15 people in 2012 and maintained this strength.

77

Table 10 Operating Costs for ENIAC Office The ENIAC Annual Activity Report for 2013 highlights that out of €10.0 million initially budgeted for the ENIAC 2008-2013 running costs, €4.4 million (or 44%) could be saved and transferred to the operational budget increasing it by 1%, from €440 million to €44.4 million. The operating costs for ENIAC are shown in Table 10. The ENIAC JU took continuous actions to monitor and accelerate the execution of the contractual basis. The ENIAC JU introduced an innovative approach in defining and executing pilot line projects at higher TRLs. They tried to pre-empt roadblocks caused by the state aid rules disadvantage versus EUREKA with regard to the state aid notification threshold. The 1.8 factor between the national and the European grant did not match the levels allowed by state aid rules and the European Court of Justice Jurisdiction, departing from the conditions of the association agreement for countries in the Framework 7 programme. The ENIAC JU was not successful in adjusting any of these provisions, but this has been taken into account in the ECSEL Joint Undertaking under Horizon 2020.

78

7.2.6 Lessons Learned from Previous Evaluations

The operation of the ARTEMIS and ENIAC JUs was previously evaluated for the period 2008 to 2013 leading to the following key recommendations [12]:

The ARTEMIS and ENIAC JTI SRA and work programmes need to reflect more strongly a coherent European perspective, linking to an overarching European Electronic Components and Systems research, development and innovation strategy

The ARTEMIS and ENIAC Industrial Association should play a more active role in the definition of the overall objectives and strategy of the JTI and should engage more actively with stakeholders so as to promote and facilitate participation in project proposals, especially by SMEs, and to develop and keep up to date the Strategic Research Agenda

ARTEMIS and ENIAC JTI project reviews, including a final post-project review that should be held, should monitor more closely and rigorously the actual and planned exploitation of project results, and the measures put in place by project partners to achieve such planned exploitation

ARTEMIS projects should build, where appropriate, on previously developed ARTEMIS technology, making reference to what has been funded before and demonstrating, in addition to novelty, the appropriate re-use of previous project results combined with a suitable progression to higher TRL levels. The proportion of funding for projects targeting generic applications and services (Applications projects) should be increased.

ENIAC and CATRENE calls for, and selection of, proposals should be more closely aligned (e.g. by the use of common and/or complementary calls), with the relevant funding awarding

The evaluation also formulated the following recommendations [12] for a future ECSEL JU:

Construct a proposed new, integrated JTI, or indeed any future JTI, as a PPP body as defined in the financial regulation

Focus the JU Governing Boards on strategic issues and reduce its administrative burden in order to attract participation from high-level industry representatives

Member State participation rules, funding rates and procedures should be harmonised and synchronised wherever possible, adopting best practice as the guiding principle

The JUs should explore and develop appropriate mechanisms to create an “early warning system” to identify potential delays, or restrictions to the availability, of funding from Member States

Member States should commit to a multi-annual funding system

Take steps during the proposal evaluation and selection process to improve the match of the project portfolio to strategic European aims and to ensure optimum coverage of key areas defined in the overarching EU ECS strategy and the work plans derived from such a strategy

Specific support mechanisms for enhancing the project management processes in JTI projects should be developed and implemented. Management costs should be 100% funded by the EC for all JTI projects

JTI projects should be subject to only one (i.e. the JU) project review and reporting process

Appropriate metrics for measuring the impact and success of JTI projects should be developed and used for both current and future JTIs

The Expert Panel examined the follow-up and implementation of these recommendations and assessed the extent to which the identified shortcomings in implementation have been addressed.

79

7.2.6.1 Overarching Strategy

With respect to implementation of an overarching strategy to cover the value chain that also mapped strengths, weaknesses and needs there is still work to be performed. Although some progress has been made there is still a need for better integration between CPS and electronics. This has become a more critical issue as the ARTEMIS and ENIAC JUs have been merged into the ECSEL JU. Although applications are being pursued these are general and there is a need for more focus. Considering AENEAS, projects that address a single point in the value chain are seen to be disadvantageous and from this perspective CPS is considered to be a single point. Priorities for ENIAC and subsequent work in ECSEL have been driven by a KET report that highlights that micro-electronics is the key priority. This has led to a large amount of funding and capital expenditure on micro-electronics for pilot line projects. As this report was produced some time ago there may well be other priorities now. Notably this pilot line work is wrongly labelled as being high TRL when it is at a lower TRL level than work being pursued by ARTEMIS. There is a key need to target technologies and develop a strategy for sectors such as automotive, health, etc. Analysis of the projects put forward at an ARTEMIS brokerage event reflected that the projects were very varied addressing diverse domains indicating this lack of top-down funding strategy. The need to change from the current bottom-up approach to a top-down approach is acknowledged by both ARTEMIS and AENEAS. Overall there is a need to clearly define a mid- to long-term overarching EU research, development and innovation strategy in Electronic Components and Systems (ECS) that can be used as a key “driver” for funding decisions. The MASP provides a good first step but national and various EU programmes are, as yet, not quite aligned and there is no accepted “broad strategy”. 7.2.6.2 Role of Industrial Associations

It was recommended that the Industrial Associations should play a more active role in the definition of the overall objectives and strategy of the JTIs and should engage more actively with stakeholders to promote and facilitate participation in project proposals, especially by SMEs, and to develop and keep up-to-date the Strategic Research Agenda. It was noted from analysis that 29% of applicants to ARTEMIS are SMEs and 27% of applicants to ENIAC are SMEs. The budget picture is, however, different with ARTEMIS allocating 19% of budget to SMEs and ENIAC allocating 10% of budget to SMEs. According to EU statistics 99% of EU Enterprises are SMEs (20.7M) and 2/3 of jobs are provided by SMEs. Notably 85% of new jobs are created by SMEs. Although there are specific SME Instruments these are heavily oversubscribed (3% success rate) and do not allow SMEs to engage in major programmes. The overhead of involvement is also an issue and many SMEs prefer to engage at a regional level. Notably there is a disparity between ARTEMIS which represents the SME driven embedded systems sector and ENIAC which is dominated by large companies. There is also a bias in AENEAS towards funding what are considered to be more important larger initiatives, e.g. Silicon Saxony and Minalogic, rather than directing funding towards SMEs and start-ups. To encourage allocation of more budget to SMEs and also provide connection mechanisms between SMEs and large companies funding for smaller scale experiments (e.g. €50K - €100K) could be introduced as a new instrument.

7.2.6.3 Final Post Project Reviews

The recommendation to perform final post project reviews between 6 and 12 months after the end of a project had been taken up ARTEMIS. As a result ARTEMIS can cite successful projects with descriptions and examples of where technology had gone to market. This was backed by the creation

80

of an ARTEMIS Metrics Working Group that more rigorously follows the actual and planned exploitation of project results, and the measures to achieve exploitation. For ENIAC projects there is no evidence available of post project reviews. Here there is a need to renew efforts within ECSEL to ensure that post project reviews are performed and that suitable metrics are gathered.

7.2.6.4 ARTEMIS Projects Should Build Upon Previous Projects

It was recommended that ARTEMIS projects should build, where appropriate, on previously developed ARTEMIS technology, making reference to what has been funded before and demonstrating, in addition to novelty, the appropriate re-use of previous project results combined with a suitable progression to higher TRL levels. The panel assesses that this has happened and it is possible to see now at the end of the ARTEMIS, and moving on into ECSEL, how technologies are being built upon. An example of implementing this strategy has been the development of reference technology platforms. For instance, the CESAR reference technology platform was extended in more than 20 different projects, including not only ARTEMIS projects, but also ITEA and FP7 projects as well as national funded projects. An effort has been made to address application oriented projects (e.g. automotive, health) and it is recommended that the proportion of funding for projects targeting generic applications and services should be increased. Ideally in the future a graph of projects is needed, demonstrating how “Project A” results are used in “Project B” with an indication of the "share of reuse", i.e. an assessment of funding reuse with respect to the full budget. The links and reuse both into and from other H2020 programmes should also be considered. This will become more important as Lighthouse projects are initiated via ECSEL which will cluster activities funded by different sources.

7.2.6.5 ENIAC and CATRENE Calls Should be Aligned

The recommendation to align the ENIAC and CATRENE [24] calls and selection of proposals (e.g. by the use of common and/or complementary calls), with the relevant funding awarding bodies retaining some flexibility over the assignment of the most appropriate funding stream was partially followed. There was significant overlap between ENIAC and CATRENE which resulted in competition between the two programmes. Initially CATRENE was more attractive to Member States but the move to pilot line projects made ENIAC more attractive. Some efforts were made to enhance co-operation. ENIAC and CATRENE produced a joint “Vision, Mission and Strategy for European Micro and Nanoelectronics” with the agreement that CATRENE would support smaller projects, while ENIAC would support larger, closer to market projects. CATRENE has now finished and a new programme, PENTA has been initiated with a more focussed research agenda, primarily addressing societal challenges. It is recommended that synergies and alignment of activities between the ECSEL and PENTA are pursued with the aim of maximising the benefits from the two programmes.

7.2.6.6 Construct the JTI as a PPP body in Article 209

This has been done and is considered complete.

81

7.2.6.7 Focus JU Governing Boards on Strategic Issues and Reduce Administrative Burden

The administrative burden has been much reduced but there is still a lot of discussion on process. This has been further complicated by the merger of the ENIAC and ARTEMIS JUs into the ECSEL JU. There is a need to reduce the administrative overhead further to allow the JU Governing Boards to focus on strategic issues. An additional benefit of reducing the administration will be to attract participation from high-level industry representatives at the CEO level. By getting this level interaction, particularly in the case of the ARTEMIS community, from application domains such as automotive and aerospace the relevance of future JU activities will be enhanced.

7.2.6.8 Harmonisation of Member States Participation Rules

There has been little progress with respect to this recommendation and many Member States still have their own evaluation procedures. The Member State participation rules, funding rates and procedures should be harmonised and synchronised wherever possible, adopting best practice as the guiding principle.

7.2.6.9 JUs Should Explore and Develop Early Warning Systems to Identify Potential Delays

The recommendation that the JUs should explore and develop appropriate mechanisms to create an “early warning system” to identify potential delays, or restrictions to the availability, of funding from Member States has still not been totally addressed. Notably there were still complaints about eligibility and the time to make information available. There have also been instances of funding having to be returned to Member States due to proposals in preferred national areas not being selected. In order to bridge any financial gaps, advanced funding by the EC (on behalf of a Member State) should be allowed for projects which are considered to be mission-critical or strategic at a European level.

7.2.6.10 Member States Should Commit to a Multi-Annual Funding System

Most Member States can only provide funds according to annual budgets. It is very difficult to get commitment for more than one year in practice. To provide, stability and a longer-term focus, the adoption of a multi-annual funding system would be beneficial.

7.2.6.11 Take Steps to Cover the Strategic Aims of the JU

To avoid dominance by single technology areas there is a need to take steps (e.g. modification of evaluation criteria) during the proposal evaluation and selection process to improve the match of the project portfolio to strategic European aims. This is to ensure optimum coverage of key areas defined in the overarching EU ECS strategy. This is important in the micro-electronics area covered by ENIAC and AENEAS but is less important in the CPS software domain which is by its nature diverse. It is recommended that this is supported by an overall top-down strategy leaving room for bottom-up initiatives to meet the needs of industry.

82

7.2.6.12 Provide a Specific Support Mechanism for Enhancing Project Management Processes

The recommendation to develop and provide specific support mechanisms for enhancing project management processes in JTI projects has not been addressed. A key recommendation is that management costs should be 100% funded by the EC for all JTI projects.

7.2.6.13 JTI Projects Should be Subject to One Review and Reporting Process

This recommendation has not been addressed and the need for project participants to report to the JU and also to their Member States results is an unnecessary overhead. Efforts should be made to harmonise the reporting process, remove duplication and encourage Member States to develop trust in the level of reporting provided to the JU. This may be partially achieved via improving metrics collection and impact assessment of projects.

7.2.6.14 Appropriate Metrics for Measuring Impact and Success of JTI Projects

There is a pressing need to gather appropriate metrics from projects, particularly to support an assessment of impact. This has been addressed well by ARTEMIS with metrics gathering and the creation of a Metrics Working Group. As a result ARTEMIS can reliably point to success stories and justify the outcomes of projects. ENIAC has not formally collected metrics which makes impact assessment very difficult. Although in many cases it is difficult to give quantitative metrics for the impact of complex projects there is a need for more rigorous monitoring of the exploitation of the results. This should take place during project execution but also subsequently after the project is completed (e.g. 12 months after project end). It is recommended that some budget is reserved for post project evaluation and this should be mandatory for future projects funded under the ECSEL JU.

7.2.6.15 The ENIAC & ARTEMIS JTIs, Along with the European Technology Platform (ETP) on Smart Systems Integration (EPoSS), Should be Integrated into a Single

Organisation (an ECS JTI)

This has been done. The ECSEL JU has been running for 2 years and the merging process is still on-going. Recommendations with respect to ECSEL are contained in the Interim Evaluation Report for ECSEL [60].

83

7.3. EU Added Value

7.3.1 Added Value of ARTEMIS and ENIAC

The outcomes of the ARTEMIS and ENIAC projects prove that Member States as well as industry, can co-ordinate successfully with each other and that the European Commission and countries can co-operate together to make project calls that can bring together critical mass to address key topics. A key added value of ARTEMIS and ENIAC has been to address the fragmentation of effort across Europe and also bring together key stakeholders. To get feedback on the added value for participants in ARTEMIS and ENIAC a survey was performed by CARSA [37]. This highlighted that participants in both JUs rated the added value of participation positively. This needs to be considered against the cost of creating and running the JUs. In general the ARTEMIS JU representatives were more positive when considering the running costs versus added value achieved. Even though less positive, still 95% of the ENIAC representatives stated that the costs of running the ENIAC programme have been justified to some extent (with around 65% believing this to be “To a high extent”). A more in-depth analysis reveals no clear differences between the perceptions of public and private representatives for ENIAC, but for ARTEMIS the private representatives were clearly more positive (86% stated “To a high extent”). A number of added value aspects were considered in the survey. Overall, each of the added value aspects indicated in the survey were assessed positively (“To some extent” or “To a high extent”) for both ENIAC and ARTEMIS, with a clear tendency of higher ratings for ARTEMIS, i.e. higher shares of representatives stating “to a high extent”. The most positively evaluated aspects are related to the JUs’ contributions to “dialogue between researchers and industry” and “achievement of critical mass”. In the case of ARTEMIS respondents also identified other benefits such as development of a platform approach and the ability to focus work to produce a fast impact in a newly emerging technology area. According to the interviewed representatives, particularly from the private sector, the added value of ARTEMIS is that uniquely the projects are large-scale. The ability to bring together more than 70 partners in one project was seen as something that could not be achieved by other programmes. The representatives from the public and private sector also believed that the programme encouraged a better dialogue between researchers and industry in the sector, which enhanced cross-border and interregional co-operation. One private respondent highlighted that without the ARTEMIS JU some organisations (in particular intergovernmental and SMEs) would not be able to run some of the projects due to limited in-house R&D resources. Unlike large enterprises which have a relatively large R&D budget, SMEs cannot afford to cover the costs related to mass production. For ENIAC interviewed representatives considered that two of the most important added values of the programme were in fostering a greater dialogue between all involved actors and significantly enhancing cross-border co-operation. Both public and private representatives highlighted that the programme ensured a better dialogue among Member States and between industry and researchers. In terms of cross-border co-operation, some interviewees explained that ENIAC provided the means of extending national clusters to EU-scale ecosystems and of creating a critical mass at the ecosystem level. There were mixed views, however, regarding the role of ENIAC in supporting innovation. Both public and private representatives highlighted that ENIAC increased the available funding for research in

84

microelectronics (i.e. in particular with the introduction of the pilot lines) which proved pivotal to support a very capital intensive industry. Notably some interviewees stated that despite very successful projects, the overall impact of the programme and of other projects still remained unclear. Notably for ENIAC the added value criterion “Support/kick-start innovation in the sector” had the lowest perceived added value, particularly among public representatives, of which 35% state “To a low extent”. Other added value aspects of ENIAC, identified through survey comments from public representatives relate to internationalisation of R&D activities and the greater impact on standardisation efforts by interacting at a European level. The ability to build an efficient ecosystem and a suitable framework for co-operation were also cited as benefits. Finally, interviewed ENIAC/ARTEMIS project participants commented on the added value facilitated by the JUs, “the JU has given the grounds for projects to react to changes while they were running. Due to the harmonisation, we are able to respond to what is happening in the market as such.” With respect to pilot line projects, interviewees highlighted that it was not possible to perform such large projects at a national level. 7.3.2 Leveraging Effect

In terms of leveraging investment the ENIAC and ARTEMIS JUs have been successful in increasing the private and public investment in the two sectors. 119 projects have been funded with €630 million of EC funding leveraged with €912 million of national contributions. Industry has also contributed €2.46 billion of funding overall resulting in €4 billion being targeted at research and innovation in the two sectors. The initial ARTEMIS funding target was €2.5 billion for R&D. The total eligible cost for the 6 calls between 2008 and 2013, however, only amounted to €1.1 billion [38] falling short of the target. Although industry has strongly supported ARTEMIS and has invested according to the original plan the national Member States have invested significantly less than planned over the period. The surveyed JU representatives were asked about the leverage effect of the JUs and to give examples of investments facilitated. Surveyed ARTEMIS representatives highlighted increased European collaboration on areas such as platforms, on a pan-European ecosystem, new communication frameworks (e.g. TTP and FlexRay), reference architectures and tool chains. A more specific example related to significant progress in the development of safety-critical complex software systems for mobility was highlighted. According to the interviewed ARTEMIS representatives, domains such as safety-critical systems, automotive, manufacturing, aerospace, telecommunications, industrial software and medical technologies have benefitted from a leverage effect and facilitated investments. The most frequently mentioned projects in the interviews were CRYSTAL, ARROWHEAD, CESAR and DEWI. The overall impacts of such projects have been observed as an increase of production and growth of employment in particular domains. Interviewees also highlighted the benefits of performing large projects with many partners, which enhances the network at the European level and helps to establish international recognition. In the case of ENIAC several representatives could cite examples where investments were made related to the large-scale pilot lines; a specific example being the production of Vertical-Cavity-Surface-Emitting Lasers (VCSEL) by Philips in Ulm based on the success of the ENIAC project ViDaP. The important investments that are currently being discussed within the IPCEI on microelectronics were also highlighted. It was, however, noted by one respondent that the ENIAC contributions mostly went to a very few countries that had a nanoelectronics industry. According to interviewed public and private representatives, the ENIAC JU had a leverage effect in the following application and technology domains: sensors; heterogeneous integration; power electronics (some interviewees mentioned in particular research activities in gallium nitride technology); as well as in modelling, design, testing facilities at RTO/academic sites. Both public and private representatives said that the ENIAC pilot lines had in general shown a significant impact and managed to attract a high level of

85

investment. ENIAC projects that had a particular impact, according to public and private representatives are highlighted in Table 11.

- E450LMDAP - E450EDL - E2SG - E2COGAN - E3Car - LAB4MEMS

- ESiP - IMPROVE - LENS - EnLight - Panache

Table 11 ENIAC Projects with Perceived Impact Both ENIAC and ARTEMIS surveyed participants believed that investments to their organisation had been facilitated due to their participation in ENIAC and ARTEMIS. For ARTEMIS the percentage was 50.9% and for ENIAC it was lower at 36.2%. The majority of organisations (55.6%) highlighted national investments being facilitated, followed by Regional investments (17.78%), European investments (16.3%) and finally international investments (10.4%). ARTEMIS respondents highlighted greater levels of national investment (61.0%) than ENIAC (47.1%), but at the EU level ENIAC respondents indicated greater EU funding investment 18.9% compared to ARTEMIS (14.6%).

86

7.4. Coherence

7.4.1 ARTEMIS and ENIAC Targeted Areas

In order to assess coherence relevant initiatives that have similar goals to the ARTEMIS and ENIAC JUs were considered. Initially smaller-scale projects were funded with a degree of diversity which leads to comparisons with many smaller regional, national and EU initiatives. A key aim of the JU’s was to address fragmentation of research effort. Both ARTEMIS and ENIAC have migrated towards implementation of larger projects to create critical mass in specific areas. In the case of ARTEMIS AIPPs have been funded in key areas such as “Critical Systems Engineering Factories”, “Production and Energy Systems Automation” and “Computing platforms for embedded systems”. ARTEMIS participants also cite strong application interests in the automotive, aircraft and aerospace, health and smart grids/energy efficiency domains. In the case of ENIAC there are three technology directions [45]: miniaturisation, diversification and differentiation, heterogeneous integration, that are supported by 3 work areas: design technologies, semiconductor process and integration and equipment, materials and manufacturing. Additionally, ENIAC Work Areas have been defined covering Automotive and Transport, Communication and Digital Lifestyle, Energy Efficiency, Health and Ageing Society, Safety and Security. Considering the move to larger projects the Panel have primarily considered the larger-scale initiatives across Europe that have strong synergies with ARTEMIS and ENIAC.

7.4.2 ITEA

ITEA [23] is the EUREKA Cluster programme supporting innovative, industry-driven, pre-competitive R&D projects in the area of Software-intensive Systems & Services (SiSS). ITEA stimulates projects in an open community of large industry, SMEs, universities, research institutes and user organisations with 1500 partners in over 30 countries world-wide. Each year, ITEA issues a Call for projects starting with a two-day brokerage event. Each Call follows a two-stage procedure, in which the quality of the project proposal is evaluated and improved, finally leading to a selection of project proposals that receive the official ITEA label. ITEA projects address applications such as automotive, communications, healthcare and aerospace. There is a clear synergy with ARTEMIS and there have been efforts between ARTEMIS and ITEA to enhance the co-operation between the two programmes. A Co-operation Committee was established in order to work together towards a single vision for embedded systems in Europe. Joint events have been organised and these continue, the last event being on 10-13th May 2017 in Amsterdam, the Netherlands. The aim of this co-operation is to share results and visions for embedded systems. Even so, in response to a survey performed by CARSA [37] the synergies between ARTEMIS and ITEA were seen as limited. There were some overlaps between the research agendas which should have resulted in synergies but ITEA was very much focused on application areas (automotive, smart cities, etc.) whereas the ARTEMIS programme was more dedicated towards technology that is used across domains. The funding approach for ITEA was also fundamentally different and the survey indicated that some Member States do not participate in ITEA because of the limited funding available. Overall the survey results indicated that both public and private representatives would like to see better harmonisation between the ARTEMIS and ITEA research agendas to create a single long-term vision. It would be possible for both programmes to work more closely with each other but there is a need for creation of a common funding strategy. This is complicated because Member States are not willing to harmonise their national procedures.

87

7.4.3 EURIPIDES²

EURIPIDES² [61] is an ICT EUREKA Cluster that promotes the generation of innovative, industry-driven, pre-competitive R&D projects in the area of Smart Electronic Systems. It is an innovation hub for smart sensors, smart power modules, electronic hardware platforms and more generally electronic product integration and embedded systems for automotive, aeronautics and space, security, medical electronics, smart everywhere (cities, home, wearable) and industrial electronics. EURIPIDES² will run until 2020 with an estimated total cost of €800 million. It supports projects involving all actors along the electronic systems integration value chain, from materials, equipment and technologies, components and modules, up to embedded, enmeshed and implanted systems. Out of the 2.5 million employed in all the European electronics industry these activities involve about 1.7 million employees in Europe. An aim is that EURIPIDES² will contribute to provide 700,000 new jobs by 2020. More than 60 companies, research institutes, federations of enterprises and regional and European clusters are involved. EURIPIDES² stimulates R&D projects in an open network of large industry, SMEs, universities, research institutes and user organisations with a specific focus on innovative SMEs. As a EUREKA Cluster the network is open to participants world-wide.

There is strong complementarity with ARTEMIS in the area of embedded systems and the application domains and also with ENIAC in the materials area. The aim to address the value chain also has synergy with ECSEL which brings together ARTEMIS and ENIAC. Here there are potential overlaps and a need for closer collaboration.

7.4.4 FP7 and H2020 Smart CPS Clusters

Under FP7 and H2020 clusters of projects have been formed in the areas of Smart Cyber-Physical Systems (CPS) and also Cyber-Physical Systems of Systems [62] addressing the next generation interconnected embedded ICT systems that will make transport systems, cars, factories, hospitals, offices, homes, cities and personal devices "smarter", more intelligent, more energy-efficient and more comfortable. The focus is on reinforcing European industrial strengths as well as exploring new markets in control, monitoring and data gathering functions while meeting safety, privacy, security and near-zero power consumption as well as size, usability and adaptability constraints. A key aim is to de-verticalise technology solutions with CPS platforms that cut across the barriers between application sectors including mass consumer markets. To achieve this the aim is to bring together value chain actors from suppliers of components and customised computing systems, to system integrators and end users. Participants include suppliers and users of CPS, tool providers, suppliers of sub-systems, system integrators, auditors/certification bodies of systems and related academia and research institutes.

A number of research areas are being supported including development of new paradigms, concepts, platforms, toolboxes, CPS modelling and integration frameworks, and methods for engineering Smart, co-operative and open CPS. Projects must include a demonstration and validation phase with realistic use cases. Originally funded under FP7 under H2020 there has been more emphasis on ICT platforms for both vertical and core markets in automotive, health, smart buildings, energy, wireless communications and digital consumer products and services. The development of ecosystems has also been supported with the establishment of European networks of embedded systems design centres, to help businesses improve the quality of their products and services.

As CPS is a key topic within the ARTEMIS JU the FP7 and H2020 Smart CPS programmes have been specifically designed to be complimentary to ARTEMIS and ECSEL addressing platforms and ecosystems for a “smart everywhere” society. It was stressed in the FP7 and H2020 calls that proposals “should build on and be complementary to EU, national and regional activities such as pilot projects in ENIAC, ARTEMIS and ECSEL”. The significant difference is that the ARTEMIS and ECSEL

88

projects focus on large-scale federating projects and integrated demonstrations and pilots. There has been a particularly strong interaction between ARTEMIS and the support actions funded in the Smart CPS cluster with a number of joint clustering and communication events being organised. The support actions funded under Smart CPS have a special emphasis on SMEs and mid-cap engagement, focussing on technologies and processes, cross-sectorial platform-building, structuring of constituencies and road-mapping. Here there are direct synergies with input from the cluster into the ARTEMIS SRA [27]. There are opportunities for even tighter collaboration with ECSEL to engage with SMEs in the future.

7.4.5 CATRENE

CATRENE [24], is a EUREKA Cluster for micro and nanoelectronics, which started in 2008 and launched its last call for project proposals at the end of 2015. The aim of CATRENE was to provide Technological Leadership for a competitive European ICT industry. It was based on the ambition of Europe and European companies to deliver nano-/microelectronics solutions that respond to the needs of society with the goal of improving the economic prosperity of Europe and reinforcing the ability of industry to be at the forefront of global competition. CATRENE built upon the successful previous EUREKA programmes JESSI, MEDEA, and MEDEA+. The aim was to foster a dynamic European ecosystem with the critical mass necessary to compete at a global level in high technology industries. The CATRENE Final Report was produced in November 2016, along with a booklet on the CATRENE projects. Additionally, running projects from the last Calls continue to be supported through the Industrial Association.

Although there should be strong synergies with the aims of ENIAC the interviewed JU representatives considered that the co-operation between ENIAC and CATRENE was not optimal, and that synergies between both programmes were limited. According to both public and private representatives, there were significant overlaps between the ENIAC and CATRENE research agendas. Notably the relationship between ENIAC and CATRENE changed over time. Originally there was a preference for CATRENE by Member States as the programme ensured a greater say in the allocation of funding, however, this situation changed after the introduction of ENIAC pilot lines which led to a greater participation of Member States in ENIAC at the expense of CATRENE. Efforts were made to enhance the co-operation with the drafting of the 2010 MASP, “Vision, Mission and Strategy for European Micro and Nanoelectronics” jointly by both AENEAS and CATRENE. This joint strategy followed a “one strategy-two instruments” approach stating that CATRENE would support smaller projects, while ENIAC would support larger projects that were closer to the market. Interviews [37] have indicated that to enhance synergies between both programmes, some public representatives would have advocated a full merger of ENIAC and CATRENE. 7.4.6 Synergies with National, International and Intergovernmental Programmes

The synergy of ARTEMIS and ENIAC with a number of national initiatives was also considered. Here larger-scale initiatives have been considered that display similar goals or critical mass.

Silicon Europe [63] is the brand under which the leading micro- and nanoelectronics (MNE) clusters in Europe collaborate to represent, support and promote companies and organisations belonging to their ecosystem both on European and global level. Silicon Europe acts as intermediary between research and academia, public authorities and industry. The Silicon Europe Alliance, welcomes other European MNE clusters and unites 12 European MNE clusters with about 2,000 cluster partners.

Silicon Saxony [64] is a trade association for the semiconductor, electronic, microsystems and software industries with 300 members. To create networks the association enables forums for its

89

members and specialists. Working groups drive the development of new technologies and procedures, e.g. CPS, based on microelectronics and software.

Minalogic [65] is a global innovation cluster for digital technologies serving France’s Auvergne-Rhône-Alpes region. The cluster supports the innovators by facilitating networking, fostering collaborative R&D, and providing companies with personalised assistance throughout all phases of business growth. The products and services developed address all industries, from ICT and healthcare to energy and advanced manufacturing.

DSP Valley [66] is an independent cluster of excellence in smart electronic systems and embedded technology solutions. DSP Valley groups 100+ members: universities, research institutes and companies, from small start-ups, SMEs to large international groups. DSP Valley offers its members a networking platform that allows them to explore each other’s expertise and that stimulates innovation by exploiting complementarities. Its activities include regional and international inter cluster B2B forums, Academia-to-Business forums, a shared group booth at international exhibitions, custom matchmaking events, technical seminars, a bi-monthly newsletter and brokerage services for participation in European ICT programs. DSP Valley is headquartered in Leuven, Belgium and has a branch office in Eindhoven, The Netherlands.

NMI [67] is the Champion for the UK Electronic Systems & Technology Industry. Its mission is to help make the UK a leading location for electronic systems and technology businesses. NMI aims to support its members’ short-term priorities and its industry’s long-term needs. It covers innovation, operational excellence, investment, the skills agenda, advocacy and representation.

GAIA [68] is the Association of Electronic and Information Technologies in the Basque Country. It is a private and professional non-profit organisation, established in 1983, made up of 260 companies that offer products and services in the field of electronics, information technology and telecommunications.

High Tech NL [69] is the sector organisation for innovative Dutch high-tech companies and knowledge institutes. High Tech NL is committed to the collective interests of the sector, with a focus on long-term innovation and international collaboration. Members share their knowledge and look for ways to co-operate.

SEMI [70] is the global industry association serving the manufacturing supply chain for the micro- and nanoelectronics industries, including: Semiconductors, Photovoltaics (PV), High-Brightness LED, Flat Panel Display (FPD), Micro-electromechanical systems (MEMS), Printed and flexible electronics and related micro- and nanoelectronics.

Many of the activities of the national associations and networks are complementary to ARTEMIS and ENIAC and synergies exist in the aims and agendas being undertaken. In assessing the links to national programmes it is useful to consider the results of the two surveys performed [37] which gave similar results, with a majority of the respondents (65-70%) assessing the links to be strong. However, there were still significant negative comments highlighting that the links to national programmes differ on a case by case basis depending on the priorities of the national programme or the funding capabilities. Further, the interviews indicated that the national programmes of the Netherlands, France and Finland are strongly linked to ARTEMIS, and the ones of France, Germany and Austria are strongly linked to ENIAC. This is also reflected in the participation patterns in the two JUs.

90

7.4.7 Future Opportunities for Synergies and Recommendations

As ARTEMIS and ENIAC have now finished any assessment or recommendation needs to consider the current situation with respect to initiatives and the ECSEL JU. Previous evaluations of the JUs revealed an initial critical lack of co-ordination between ARTEMIS and ITEA, and between ENIAC and CATRENE however, a substantial improvement has been reported in both cases. The efforts for improved co-operation, e.g. by joint development of strategic vision documents and multi-annual research agendas, are also recognised in the comments from surveys and interviews carried out. Although the comments from the two assessments are fairly positive with respect to co-operation around 25% of the respondents (excluding those with no opinion) assessed the consideration of links between the programmes as low or non-existing. This indicates that there is room for improvement with respect to co-ordination between ECSEL and related initiatives during the continued implementation of the programmes. Here there are a number of opportunities for linking with the Large-Scale Platforms Initiative, the new EUREKA PENTA [71] cluster and also more generally with other JUs that have now been established.

7.4.8 Synergies with IoT Large-Scale Platforms

The Digitising European Industry [72] initiative aims to establish next generation digital platforms and re-build the underlying digital supply chain. A first batch of platform-related projects and large-scale integration, testing and experimentation pilots were funded under the H2020 Work Programme 2016/2017 [73]. The aim of the large-scale pilots is to remove cross-border and other obstacles which prevent large-scale testing, experimentation and block the full deployment of technologies into the market. This is particularly relevant for the autonomous connected vehicles and the connected smart factory areas which are key areas for ARTEMIS. Here there are also links to other areas such as Big data, cloud, HPC, autonomous systems, artificial intelligence and 3D printing.

The aims of the large-scale pilots have synergies with ARTEMIS particularly in the area to integrate converging digital innovations into sectorial platforms and full solutions, and also in testing them across national borders. There are also aims to develop Europe-wide facilities for experimentation to foster rapid development of ICT standardisation leading to new standards. Notably the large-scale pilots will leverage from co-investments with Member States and industry to develop the necessary platforms, large-scale pilots and standards much like ARTEMIS.

Under the initiative sectorial platforms and solutions are being developed for the Connected Smart Factory and also for Connected and Automated Driving with the aim to set-up a cross-border testing facility pooling investments across Europe and connecting various stakeholders (AI-experts, automotive OEMs, communication service providers). Additionally, a platform on Co-operative Intelligent Transport Systems [74] has been funded which will build on large-scale pilot deployment, testing and experimenting facilities available across Member States with the aim to stimulate EU-wide interoperability and continuity of services.

Integration platforms are also being pursued that address cross-sector challenges such as Leadership in IoT building on existing open service platforms, such as FIWARE [75], Industrial Data Platforms [76] open data platforms and 5G demonstrations.

Here there is a need to ensure that there is an awareness of the platforms and solutions being developed and promoted in the large-scale pilots within ECSEL to ensure successful industry take-up and avoid fragmented development efforts.

91

7.4.9 Synergies with PENTA

PENTA [71] is a new EUREKA initiative launched in 2016 to replace CATRENE with the aim to catalyse research, development and innovation in the areas of micro and nanoelectronics enabled systems and applications where there is shared high national and industrial interest. Micro and nanoelectronics were identified as a Key Enabling Technology for Europe. The focus of the projects is on applications in automotive, healthcare and production technologies. PENTA will bring together complementary capabilities into focussed research and development programmes. The aim is to operate across the electronics value chain and accelerate the development of products and services intended to support European leadership in key system and application market sectors. PENTA was set up to meet the needs of industry, covering all facets of development from research (TRL 2) to pilot production or service launch (TRL8). The programme addresses micro and nanoelectronics based components and systems. An aim is to be able to quickly and flexibly set up projects by identifying opportunities, quickly assessing national governmental support and operating with a short approval process. A fundamental premise of the PENTA Cluster is openness and inclusivity. This is reflected in the Cluster’s governance, offering equal opportunities to all interested participants, and “inclusivity” via creation of a “market place”. A special focus is placed on SME involvement.

As a follow-on to CATRENE, PENTA has a natural synergy with the AENEAS activities under ECSEL. Unlike ENIAC PENTA will focus on smaller, faster and flexible projects thereby providing an opportunity to quickly take advantage of rapidly developing markets and their related value chains. PENTA is expected to benefit from the best practices and lessons learnt from ENIAC and EUREKA projects. PENTA has a much more focussed research agenda, mainly dedicated to societal challenges. This should ease the co-operation with ECSEL, and increase the number of synergies in comparison to those obtained between ENIAC and CATRENE.

7.4.10 Synergies with Other PPPs

Already there are activities to explore synergies between PPPs. For example the PPPs addressing Robotics, the EIP on Active and Healthy Ageing, Big data and IoT could potentially work together for exploiting joint outcomes in domains such as smart hospitals and healthy living. Here there is also an opportunity to explore synergies with health related projects being supported by ECSEL.

Similarly there are opportunities for collaboration with other PPPs. There are strong links between ECSEL and Factories of the Future [77], and considering the sectors being addressed and the cross-domain goals of ECSEL there are other opportunities for collaboration with SPIRE [78], Clean Sky’s 2 [79] …, etc.

7.4.11 ECSEL Lighthouse Projects

With the integration of ARTEMIS and ENIAC JUs into the ECSEL JU [16] the funding of fewer larger-scale projects has continued. Under ECSEL a new Lighthouse project initiative has been launched with a budget of €45 million. Lighthouse projects will build on well-identified market-pull related to societal needs that have a strong pan-European dimension. The aim is to offer visionary solutions creating ecosystems along relevant value and supply chains, working towards clustering of projects. Lighthouse projects are also expected to have a strategic IP management policy, establish a standardisation strategy and address the relevant non-technical aspects (legislative, regulatory, social).

92

The two initial ECSEL Lighthouse projects will be in Smart production (Industrie 4E) and Mobility (Mobility4E). For smart production the aim is to use technology for sensing along with data analytics to cover the whole life cycle. The Lighthouse project on production will co-operate with I4MS and Factories of the Future and ECSEL is already engaged with Industrie 4.0 [14] via its member companies. The concept of Lighthouse projects goes beyond the ECSEL community and there will be a need to involve experts from outside the “ECSEL world” to leverage key enabling technologies. Although several companies involved in ECSEL are also involved in ERTRAC for smart mobility, for the Mobility4E Lighthouse project, in order to ensure the acceptance of autonomous cars, there is a need to promote public trust via the media, e.g. television, and consider the social implications on employment. There is also a need to engage with standardisation bodies, insurance companies, etc., to remove barriers and also with entities that can give visionary views to accelerate the demand side. Overall the initiation of Lighthouse projects is expected to result in much wider collaboration across JUs and initiatives and go beyond this, clustering together projects providing input from a diverse range actors.

93

7.5. Relevance

7.5.1 Relevance of Initial Tasks Entrusted to ARTEMIS and ENIAC

The ARTEMIS and ENIAC JUs had the overriding aim to contribute towards the development and implementation of strategic research programmes in the areas of nanoelectronics and embedded systems, respectively. Each JU had a number of defined tasks that it must undertake. These were to:

Define and implement a Research Agenda

Award funding to participants

Mobilise both public and private efforts to increase overall R&D investments

Achieve synergy and co-ordination of European R&D efforts

Promote the participation of SMEs

It is clear that the JU approach has been successful in bringing together EU, national and industrial actors to create a Strategic Research Agenda around common goals. Both ARTEMIS and ENIAC have established and built-up communities made up of key industry players across the value chain linked with research communities at a European level. This is a key outcome for both ARTEMIS and ENIAC and has led to better co-ordination of research activities. In the case of ARTEMIS Centres of Innovation Excellence have also been created to bring together actors. Public and private funding has also been mobilised by the initiatives and it is notable that industry has been a very strong supporter in both cases making major commitments to funding research activities and indeed surpassing the commitments required. The Member State funding has been more difficult to manage due to changing priorities and the need to co-ordinate across Member States. The need to meet different reporting requirements imposed by different Member States has also been an issue for successful proposers. The creation of the JUs has proved that a tri-partite funding scheme can work and result in significant co-ordination of funding to tackle major issues. Notably the move towards larger projects has indicated the real worth of the tri-partite funding scheme as these very large projects would not have been possible considering the funds provided by conventional EU programmes at a national level. These major projects have allowed ARTEMIS to make major advances in certain technology areas, such as that of tools and platforms that are reusable across applications. There has also been re-use of project results with some projects building upon results from previous projects with cross-industry transfer between applications.

The objectives of ARTEMIS and ENIAC correspond to the needs of their industrial communities. In the case of ARTEMIS embedded systems are used in products throughout the entire supply chain bringing together semiconductor, pure software and sub-system suppliers. In the case of ENIAC there has been a major shift to support pilot lines to support the position of Europe in the area of nanoelectronics and manufacturing of semiconductors. Both the strategies followed have attempted to leverage as much as possible the existing strengths in Europe. The flexibility to change strategic objectives, adapting to the prevailing European economic and social landscape, and to act proactively to change the landscape have been key features of the two JUs over the last two years of operation with the introduction of much larger-scale projects. Another key task defined for the JUs was to support SME engagement. As a result of actions performed there has been good engagement. The embedded systems domain is characterised by SMEs and within ARTEMIS there has been a good level of SME participation (29%). For ENIAC the

94

participation has also been healthy at 27%. The share of budget allocated to SMEs for ARTEMIS has also been positive being 19%. In the case of ENIAC this has been less positive with only 10% of funds being allocated to SMEs. Research organisations (HEIs & RES) also received a comparatively higher share of funding in ARTEMIS than in ENIAC. Surveys by CARSA [37] indicated that participants assessed the efforts to involve SMEs more positively for ARTEMIS than for ENIAC. It was noted that the market structure for the semiconductor industry is geared towards larger companies.

7.5.2 Contribution of ARTEMIS and ENIAC to FP7 Objectives

The broad objectives of FP7 can be grouped into four categories: Co-operation, Ideas, People and Capacities. For each type of objective, there is a specific programme corresponding to the main areas of EU research policy. All specific programmes work together to promote and encourage the creation of European poles of (scientific) excellence. Under the area of co-operation research activities are carried out between universities, industry, research centres and public authorities at a trans-national level with the aim to gain or consolidate leadership in 10 key scientific and technology areas. The ten identified themes reflect the most important fields of knowledge and technology where research excellence is particularly important to improve Europe’s ability to address its social, economic, public health, environmental and industrial challenges of the future. The Strategic Research Agendas developed by ARTEMIS and ENIAC were used to ensure the relevance of these themes thus they have had an influence on the FP7 programme. Other aims within FP7 were to promote collaborative research and also co-ordinate national research programmes to create European Excellence.

It is clear that ARTEMIS and ENIAC have led to significant trans-national collaborations across Europe. In total 1420 entities (ARTEMIS) and 1384 entities (ENIAC) have been engaged in collaborative research with 1000’s of full time researchers being engaged on projects with additional further direct and indirect employment being a catalyst for the two communities. This is backed by interview responses to the CARSA survey [37] which highlighted that 71% of beneficiaries and stakeholders believe that the JU community has brought together the best competencies in the industry across the EU. In particular the cross-disciplinarily and trans-nationality of the players involved is perceived as a strength which fosters innovation across the regions. In terms of added value respondents highlighted the JUs’ contributions to “dialogue between researchers and industry” and “achievement of critical mass” as being the two most important aspects.

7.5.3 Coherence with Other Policy Interventions

Both JUs made considerable efforts to align their activities with other programmes which have similar goals, e.g. ITEA, CATRENE. The EC also encouraged engagement of projects, such as in the area of Smart CPS with ARTEMIS and a number of joint events have been held bringing the two communities together. Despite these efforts to reinforce synergies analysis of the projects undertaken and their coherence with other EC and National programmes still shows overlaps and a lack of linkage.

For instance in a survey of participants performed by CARSA [37] the synergies between ARTEMIS and ITEA were seen as limited. Although there were some overlaps between the research agendas which should have resulted in synergies ITEA was very much focused on application areas (automotive, smart cities, etc.) whereas the ARTEMIS programme was more dedicated towards technology that can be used across domains. Similarly there should have been strong synergies between ENIAC and CATRENE as there were significant overlaps between the ENIAC and CATRENE research agendas. In practice surveyed participants highlighted that the synergies between both programmes were limited.

95

Overall the level of interaction has not been optimal and there is a need for more work. The move towards pilot lines in the case of ENIAC and large application driven projects in ARTEMIS has helped in this respect. This provides a clear distinction between the JUs that are able to address major challenges at a European level compared with smaller initiatives that can address important topics. There is still a need, however, to define a top-down overarching European Strategy that can then be used to help align activities with other national programmes. Here there are opportunities in the future to align with the PENTA EUREKA initiative and also with other JUs and initiatives that are application oriented, e.g. Factories of the Future, Robotics, the EIP on Active and Healthy Ageing, Big data and IoT. The adoption of the Lighthouse projects within ECSEL clustering activities into umbrella domains, e.g. smart factories and smart mobility, is also seen as a very good opportunity for bringing projects together.

7.5.4 Openness, Transparency and Efficiency

The ARTEMIS and ENIAC JU’s provide concrete examples of the European Union's efforts towards strengthening its competitiveness through scientific excellence, innovation and industry led research. Although strongly driven by industrial stakeholders, publicly available strategic research agendas and roadmaps are used in programming with close consultation via Partnership Boards between Commission services (DG RTD/DG CNECT) and the industrial associations to ensure relevant needs and innovation trends are reflected in the programme. In addition, the Programme Committee has input from Member State representatives who formally support the work programme on the basis of a vote. Thus, national administrations have a major say on the contents of the work programme.

Additionally, there has been a move to closer co-operation and co-ordination with other JUs, through inclusion of stakeholders' advisory bodies, via the set-up of separate Memoranda of Understanding and via exploration of synergies with other (national and regional) programmes.

The JUs have been implemented effectively and have done a good job in performing the tasks demanded from them. This is reflected by the engagement from the communities addressed and also by the number and quality of the projects performed. Engagement in projects has been “open to all” and the number of participants (2804) in both programmes indicates good inclusivity. The levels of SME participation ARTEMIS (29%) and ENIAC (27%) have also been positive.

Both JUs have disseminated widely to the stakeholder community and to the general public via their respective websites, press releases and through the use of social media. ARTEMIS and ENIAC have also participated in a considerable number of events, seminars and conferences to disseminate project results. A number of public events and forums have also been organised to promote the added value and impact of individual projects.

7.5.5 Added Value of ARTEMIS and ENIAC

The ARTEMIS and ENIAC JUs have been successful in increasing the private and public investment in the two sectors. As a result 119 projects have been funded with €630 million of EC funding leveraged with €912 million of national contributions. Notably industry has also contributed €2.46 billion of funding which goes beyond the original commitment required. Overall this has resulted in €4 billion of funding being targeted at research and innovation in the two sectors. The financial commitment by Member States and industry increased as the JUs adopted an approach to funding larger-scale projects. There has been good engagement in ARTEMIS across the embedded systems community driven by end user applications that have resulted in technology demonstrators at a relatively high TRL. Key

96

European players include major aircraft companies such as EADS and THALES, medical electronics companies such as Philips, automotive drivetrain specialists AV LIST, car manufacturers such as Fiat and well-known research organisations across Europe. There has also been strong participation from downstream players such as electronics manufacturers (e.g. Infineon, NXP) who embed their electronics into applications. The areas of safety-critical systems, automotive, manufacturing, aerospace, telecommunications, industrial software and medical technologies have benefitted from the funding and this has facilitated further investments. In a survey 51% of ARTEMIS participants [37] highlighted that investments to their organisation have been as a result of their participation in the projects funded by the JU. Large industry driven projects, e.g. CRYSTAL, EMC2, have had a high impact. Respondents to ARTEMIS-IA questionnaires in 2014 [48], indicated that ARTEMIS had had a significant and pivotal business impact on reducing development costs (53% of respondents), had contributed to reduced time-to-market (41%) and had developed a new generation of products (38%). A key enabler for this had been the funding of demonstrators. In particular the ARTEMIS JU has been successful in bringing organisations together to develop prototypes and demonstrators. In the ARTEMIS-IA survey performed in 2014, 95% of the respondents indicated that a key aim was to build application prototypes [48]. ENIAC has been characterised by strong participation from semiconductor manufacturers and research centres with little involvement of end users which is in line with the character of the field and thus the programme. One major end user company is Philips from the medical domain where there is interest in miniaturised electronics for in-vitro applications. The KET initiated pilot lines have focussed work on consolidating technologies on 200mm platforms and strengthening 300mm infrastructure. This work has been successfully exploited by key players involved in the projects. In assessing the impact on the nanoelectronics sector, the CARSA Survey [37] highlighted that 88% of the participants of ENIAC believed that the work would create commercialisable innovative outputs. It was also believed by 75% of surveyed participants that the projects would result in job creation. Several representatives could cite examples where investments were made related to the large-scale pilot lines; a specific example being the production of Vertical-Cavity-Surface-Emitting Lasers (VCSEL) by Philips in Ulm based on the success of the ENIAC project ViDaP. A key outcome that was noted for both JUs was that the technology and manufacturing readiness levels have been substantially increased by the work in the ENIAC and ARTEMIS projects. Notably participants in the CARSA Survey [37] for both the ARTEMIS and ENIAC JUs rated the added value aspects of the JUs positively. The most positively evaluated aspects were related to the JUs’ contributions to “dialogue between researchers and industry” and “achievement of critical mass” which relate well to the goals of FP7.

7.5.6 Success of ARTEMIS and ENIAC in FP7 and Shortcomings Addressed in Moving Forward to H2020

ARTEMIS and ENIAC have been successful in engaging their respective communities and in bringing together tri-partite funding to create critical mass to address “big issues” that could not have been addressed via EU or national funding alone. The strong funding contribution from the industrial sector is a good indicator of the relevance of the research topics pursued. The JUs have also been very successful in engendering trans-national collaboration on key topics. The end user involvement has been very strong, particularly in the case of ARTEMIS, and operation at higher TRLs has led to an industry focus bringing together the necessary actors to address specific applications. In the case of ARTEMIS many of the projects envisage providing R&D results with commercial impact within a 3 to 5 years’ time in the automotive, avionics, space, factory automation, healthcare and Internet of Things application areas. In the case of ENIAC the pilot lines have consolidated technologies on 200mm platforms, strengthened 300mm infrastructure and enabled the future transition to the 450mm platform. Already the work is being exploited by key companies in both the ARTEMIS and ENIAC

97

domains. This can be seen as a key achievement in supporting European Excellence which was an aim of FP7. Factually the lack of assessment of impact post completion of projects leads to a lack of hard evidence on the impact of projects although plenty of evidence could be cited for both ARTEMIS and ENIAC. In the case of ARTEMIS the metrics gathering and surveys performed provides more concrete information on impact compared to ENIAC where no metrics gathering or surveys have been performed. There is some factual evidence that projects have led to significant increases in productivity and reduced time-to-market after the adoption of innovative tools and methodologies. It can be seen from the number of patents filed (ARTEMIS 28, ENIAC 209) that new innovations are being created and the 119 industrially relevant projects performed clearly have benefited industry. Notably ENIAC projects led on average to 4.7 patent applications per €10 million of EU funding which is higher than the Horizon 2020 benchmark of 3 patent applications per €10 million of EU funding. In terms of research outputs a large number of publications (ARTEMIS 1460, ENIAC 2381) have been produced which have been disseminated widely to the community. Both ARTEMIS and ENIAC can cite success stories. In the case of ARTEMIS support and development for the world-wide AUTOSAR automotive standard and companies such as TTTech that has expanded and strategically moved into different sectors with the support of a number EU projects (notably via the EMC2 and CRYSTAL projects). In the case of ENIAC the companies AMS, Infineon Technologies Austria, ASML and Carl Zeiss SMT who have developed specific world-leading competences. The JUs have been successful in increasing European collaboration which is a feature of both FP7 and Horizon 2020. In the case of ARTEMIS there has been increased European collaboration on areas such as platforms, on a pan-European ecosystem, new communication frameworks (e.g. TTP and FlexRay), reference architectures and tool chains. For ENIAC there has been increased collaboration on nanoelectronics and semiconductors particularly in the manufacturing area via the pilot line projects. It is notable that both ENIAC and ARTEMIS surveyed participants believed that investments to their organisation had been facilitated due to their participation in ENIAC and ARTEMIS which indicates that the projects are also stimulating further R&D investment in their respective sectors. There is also evidence that projects results have been re-used in other projects both within the JUs and in other EU programmes. Overall, participants of ARTEMIS and ENIAC were positive about the added value of participation in the JUs with participants in ARTEMIS being more positive. The most important benefits were in improving the dialogue between researchers and industry and in achievement of critical mass. ARTEMIS respondents also cited the platform approach and the ability to focus work to get a fast impact in newly emerging technology areas whilst ENIAC respondents also cited better dialogue among Member States and the ability to extending national clusters to EU-scale ecosystems to create a critical mass at the ecosystem level. The key aim of Horizon 2020 is to secure European Competitiveness. In order to compete with strong world-wide competition there is a need to address the current fragmentation of activities across Europe and create critical mass. This was achieved to some degree through the move to larger-scale projects in both ARTEMIS and ENIAC over the final 2 years of operation. The coherence of ARTEMIS and ENIAC with other national initiatives despite obvious synergies in terms of strategic agenda has not been optimal and there is a clearly a need for more work. In order to address European Competiveness there is a need to clearly define a long-term, top-down research, development and innovation strategy in Electronic Components and Systems (ECS) bringing the ARTEMIS and ENIAC communities together. Already this has been initiated by the establishment of the ECSEL JU but it will take time to formulate a cohesive Electronic Components and Systems (ECS) strategic research agenda and effectively bring the two communities together. The funding strategy for projects should

98

also leave room for bottom-up initiatives. The overall strategy should be developed with top-level input from major companies at CEO level and should concentrate on the integration of CPS and electronics with a focus on application domains such as automotive, health, etc., looking for synergies with national initiatives and other JUs to ensure coherence.

99

8. CONCLUSIONS

This report has considered the following three criteria with respect to evaluation of the ARTEMIS and ENIAC JUs by analysis of documentation and interviews:

• Effectiveness: progress made towards meeting the objectives set. • Efficiency: extent to which the JUs were managed and operated efficiently. • Research Quality: extent to which the JUs enabled world-class research that helped Europe

to establish a leadership position globally, and how it engaged with a wider constituency to open the research to the broader society.

The intended role and objective of both ARTEMIS and ENIAC JUs was to support Research and Innovation (R&I) in the Information and Communication Technologies (ICT). The adoption of a tri-partite funding scheme aimed to combine resources and funding from the 7th Framework Programme, industry, national R&D programmes and intergovernmental R&D schemes to tackle problems that could not be addressed by single funding sources alone and create significant impacts. A key goal of setting up the two JUs was to increase and leverage private and public investment in the related sectors in Europe to strengthen Europe's future growth, competitiveness and sustainable development, in their respective fields by enabling greater coherence of R&D across Europe.

Considering this first goal overall the ENIAC and ARTEMIS JUs have been successful in increasing the private and public investment in the two sectors. 119 projects have been funded with €630 million of EC funding leveraged with €912 million of national contributions. Industry has welcomed the support and also contributed €2.46 billion of funding overall resulting in €4 billion being targeted at research and innovation in the two sectors. It is estimated that each Euro contributed by the EC has resulted in 6.4 Euros of research and innovation activity in Europe (source ECSEL programme Impact document). The concentration of ENIAC on large-scale pilot projects has resulted in a high commitment from Member States in particular, with a few large industrial partners benefitting from development of new manufacturing processes. This has kept Europe at the forefront in key microelectronics and nanoelectronics areas. With respect to ARTEMIS the embedded systems area is driven more by applications and here the concentration has been on development of software and hardware that can be used across domains. This diversity of applications has made it more difficult for Member States to commit to funding ARTEMIS. However, the adoption of larger projects at higher TRL levels in the later calls renewed Member State interest and more importantly led to a significant increase in commitment from Industry with large projects, e.g. CRYSTAL, EMC2, being funded. Going forward with the integration of ENIAC and ARTEMIS into the ECSEL JU strategic measures are needed that fully support the Electronic Components and Systems domain.

A second key goal was to foster collaboration between all stakeholders such as industry, including small and medium-sized enterprises (SMEs), national authorities, academic and research centres, by providing a focus for research efforts. Analysis of the projects undertaken and their coherence with other EC and National programmes still shows overlaps and a lack of linkage. It is clear that both JUs have made considerable efforts to align their activities with other programmes which have similar goals, e.g. CATRENE. The EC has also encouraged engagement of projects, such as in the area of Smart CPS with ARTEMIS and ECSEL and a number of joint events have been held bringing the two communities together. Although synergies clearly exist surveys of the JU project participants still show that the level of interaction is not optimal and there is a need for more work. Notably a previous recommendation from an interim review of the two JUs highlighted the need for a top-down strategy rather than a bottom-up strategy. The move towards pilot lines in the case of ENIAC and large application driven projects in ARTEMIS has helped in this respect but it is clear that there is a pressing need to more clearly define an overarching European Strategy that can then be used to

100

help align with other national programmes. Here there are opportunities in the future to align with the PENTA EUREKA initiative and also with other JUs and initiatives that are application oriented, e.g. Factories of the Future, Robotics, the EIP on Active and Healthy Ageing, Big data and IoT. The adoption of the Lighthouse projects within ECSEL which will cluster activities into umbrella domains, e.g. smart factories and smart mobility, is seen as a very good opportunity for bringing projects together.

A key aim has been to engage with the SME community which accounts for 99% of companies across Europe (20.7M), 2/3 of European jobs and 85% of new jobs. Positively it was noted from analysis that 29% of ARTEMIS and 27% of ENIAC applicants are SMEs, however, with respect to budget ENIAC allocates a lower budget to SMEs (10%), preferring to fund pilot line initiatives driven by large industry, compared to ARTEMIS that allocated 19% budget to SMEs (close to the Horizon 2020 target of 20%). Notably ENIAC is dominated by a few larger companies and the embedded systems domain supported by ARTEMIS is characterised by a high level of SME companies. Looking to ECSEL it is recommended that to encourage easy access and provide connection mechanisms to larger companies some funding should be targeted at SMEs supporting smaller scale experiments (e.g. €50K - €100K).

In terms of operations as the ARTEMIS and ENIAC JTIs were two of the first JUs to be established in

2008 it was to be expected that there would be some initial problems. It should also be noted that

the ARTEMIS and ENIAC JTIs were unique in the adoption of a tri-partite funding strategy. This

requires considerably more effort to co-ordinate Member States which leads to a number of

administration complexities. In order to justify funding Member States played an active role in the

governance of the two JUs, to define strategy, funding priorities and in the selection of projects.

Some Member States also wish to perform their own project monitoring in addition to the

monitoring by the JU to justify the funding allocated. With the involvement of many Member States

across Europe, with elections changing governments in a number of these every year and with

national funding priorities changing every year there is a significant challenge to co-ordinate, reach

consensus and guarantee long-term funding. The creation of Multi-Annual Strategic Plans for the JUs

has addressed this to some extent but gaining commitment for multi-annual funding has still not

been possible except in a few instances. There is also a need to develop trust to decrease the

additional administrative reporting burden on project leaders imposed by some Member States.

Overall the Member State participation rules, funding rates and procedures should be harmonised

and synchronised wherever possible. The administrative overhead is also a barrier to engagement of

CEOs from large companies active in the application domains, e.g. aerospace, automotive.

Participation of high-level industry representatives would add credibility to the strategies formed at a

European level and also to lobby for Member State funding.

A key justification for the JUs and the funding allocated to them by the EC, Member States and Industry is the impact that the projects have had. The two JUs have adopted different approaches to impact assessment. At a high level ARTEMIS set a challenging goal targeting the use of ARTEMIS technology outcomes in 20% of the embedded systems in the world and the ENIAC JU set a challenging target of creating 250,000 jobs in Europe. The attainment of this impact is hard to measure as it will take time for outcomes to be transferred into products and create jobs. It should also be noted that some projects have yet to finish.

The ARTEMIS JU has addressed impact in a more rigorous manner by setting up a Working Group specifically to address monitoring of KPIs. Targets were also set such that projects deliver cross-domain re-use and interoperability for different product categories and application domains, or promise a reduction of system design costs and development cycles for both hardware and software. Many projects expect commercial impact within a 3 to 5 years’ time. A key aim is to address software

101

design tools and environments and general purpose architectures that facilitate integration and re-use across sectors. These are important in the automotive, aerospace, factory automation, industrial processes, smart buildings, energy production and medical/healthcare sectors. Shorter development cycles and increased interoperability are important to achieve a shorter time-to-market for new products and services in the CPS domain the internet of things market. A number of notable success stories could be cited such as the AUTOSAR standard which was supported by ARTEMIS projects and is now used world-wide in the automotive sector. The EMC2, CRYSTAL and ARROWHEAD projects have addressed pan-European industry issues developing frameworks that can be shared between different communities, platforms for interoperability and tools and methods to cope with the ever increasing complexity of smart digital systems. Exploitation of outputs, company growth and world leadership can be seen in examples such as TTTech that provides core safety-critical data bus technologies to Airbus, Boeing and NASA. In the case of ENIAC, however, there has been no formal follow up of impact metrics, however, evidence of impact can be cited. This includes AMS that has used ENIAC funding to transform itself from a foundry with commodity products into a specialist for producing sensors and sensor systems, Infineon Technologies Austria which is the world leader in power electronic discretes and modules with development of a 300 mm fab., ST Microelectronics that has a strong market position in piezoMEMS and ASML which is now a world leader in lithography and sells equipment around the world. In order to consider impact the panel considered the following criteria:

• Engagement with the community • Inclusion of SMEs • Number of patents and innovation outputs • Success in attracting public/private funding • Number of projects initiated, success stories and evidence of impact from projects • Working Groups established to support the Community

This identified a number of positive impacts including good engagement with the community, positive SME participation, filing of a number of patents (ARTEMIS 28, ENIAC 209), dissemination via publications (ARTEMIS 1460, ENIAC 2381), good success in achieving industrial funding in particular, the funding of 119 industry relevant projects and the establishment of a vibrant set of Working Groups in the case of ARTEMIS. Additionally, the participation across Europe of 1420 entities (ARTEMIS) and 1384 entities (ENIAC) is also seen to be catalyst for the two communities.

In summary, it is clear that the JU approach has been successful in bringing together EU, national and industrial actors to pursue common goals. Although a global strategy across Europe is the ultimate goal there is still some work needed to achieve this in practice. The tri-partite nature of funding has the advantage of reducing fragmentation across Europe but this is at the expense of complex administration and is subject to changing priorities. Here more work needs to be done to synchronise national activities, harmonise participation rules, funding rates and procedures. This simplification of administration will make future tri-partite JUs more attractive and encourage engagement at CEO level from large companies thus adding credibility to the Strategic Research Agenda and instilling confidence in Member States.

The main critical issue identified is the ability to assess the lasting effects of the JUs. This is hampered by the lack of existing measures of the impact from a socio-economic perspective. While great care is

102

taken on measuring and reporting on input parameters such as funding levels, participation rates, etc. and interim measures such as detailed progress reports are recorded, there are few objective measures on outputs. In order to provide justification for the EC, Member States and Industry to contribute to future JUs it is important to address this issue and follow up projects after completion to gather concrete and quantifiable evidence of impact.

103

104

European Commission Final Evaluation of the ARTEMIS and ENIAC Joint Undertaking (2008-2013) Operating

under FP7 – Final Report

Luxembourg, Publications Office of the European Union

2017 – 105 pages

ISBN 978-92-79-69634-3

doi:10.2759/271765

105

KK

-04

-17

-55

1-E

N-N

doi:10.2759/271765 ISBN 978-92-79-69634-3

Documents

Final Evaluation of the ARTEMIS and ENIAC Joint Undertaking … · 2017-10-11 · systems market to grow from $3.43 Billion in 2016, to $4.71 Billion by 2021. The Unmanned Aerial