Insights in the evaluation process of a Research & Innovation Action (RIA)

11th June 2015, Grant Development Team, Luris

How to score with your H2020 collaborative

proposal

Goal of this meeting

Show you how to write high scoring H2020 collaborative proposals (RIA) that lead to solid research projects and a lasting infrastructure

Topics:Expectations from the EUDos & don’ts for the proposalThe evaluator

Highlighted: Ethics, data management & genderFirst hand experience: Dr. Antoaneta Dimitrova

Research and Innovation Action

Definition according to ECAction primarily consisting of activities aiming to establish new knowledge and/or to explore the feasibility of a new or improved technology, product, process, service or solution.For this purpose they may include basic and applied research, technology development and integration, testing and validation on a small-scale prototype in a laboratory or simulated environmentProjects may contain closely connected but limited demonstration or pilot activities aiming to show technical feasibility in a near to operational environment

RIAs are always collaborative projects

What does your proposal need? *And where does it go wrong…

1. Complete* "If I do not know what they mean, I just leave it blank.”

2. Compliant (fulfilling all criteria)* Not knowing unwritten rules: Number of partners, countries, diversity in type of organizations

3. CompetitiveTo win you need a very high score on each of the 3 evaluation criteria (3x5=15 pts)* Neglected sections Impact and Implementation. Too much time is spent on section Excellence.

What does it take to be competitive?

CCC Be in

the top 3

≥ 13.5 ptsCC Be compliant and

reach the threshold

≥ 10 pts

C Avoid (stupid) mistakes (font size, # of pages, budget range, # of partners)

not eligible ≤ 9.5 pts

How to become compliant? (1)

The Model Grant Agreement (MGA) defines the rules

Partner or subcontractor & managing IP: in MGA

Partner or not?Art 8-14

Subcontracting or not?Art 10 & 13

IPArt 24-28 & 30-31

Rules for budget:Art 5 & 6

Making the proposal meet EU standards on:Management

Art 17-23 & 41Open access

Art 29Gender

Art 33Ethics

Art 34Data protection

Art 39

These are minimum requirements!

How to become compliant ? (2)

The evaluation criteria: (RIA project)

1. Excellence2. Impact3. Implementation

The EU is looking for:

A solution to their problemCreating economic benefitProgress through innovation The best people to perform the projectValue for money

From criteria to the proposal templateSection

1. Excellence relevant to the topics addressed by the call

1.1 Objectives

–

1.2 Relation to the work programme

–

1.3 Concept and approach

1.4 Ambition

2. Impact

2.1 Expected impacts

2.2 Measures to maximise impact

a) Dissemination and exploitation of results

b) Communication activities

3. Implementation

3.1 Work plan — Work packages, deliverables and milestones

3.2 Management structure and procedures

3.3 Consortium as a whole

3.4 Resources to be committed

4. Members of the consortium

4.1. Participants (applicants)

5. Ethics and Security

idea, incl . science

expected results

execution project & management

5 pts

5 pts

5 pts

1. Excellence

2. Impact

3. Implementation

How to become competitive ?Section

1. Excellence relevant to the topics addressed by the call

1.1 Objectives

–

1.2 Relation to the work programme

–

1.3 Concept and approach

1.4 Ambition

2. Impact

2.1 Expected impacts

2.2 Measures to maximise impact

a) Dissemination and exploitation of results

b) Communication activities

3. Implementation

3.1 Work plan — Work packages, deliverables and milestones

3.2 Management structure and procedures

3.3 Consortium as a whole

3.4 Resources to be committed

4. Members of the consortium

4.1. Participants (applicants)

5. Ethics and Security

5 pts

5 pts

5 pts

Equally invest in all sections!2/3 of your total score

is based on factors other than science!

Getting started

The Participant Portal:

Information on the rules & regulations, call, topic, and submission service: http://ec.europa.eu/research/participants/portal/desktop/en/home.html

Example templates and evaluation guidelines: bottom of webpage http://ec.europa.eu/research/participants/portal/desktop/en/funding/reference_docs.html

Register in Participant Portal Download call specific template R&I proposal in Submission Service: p14 onwards part B Technical Annex Section 1-3

Never use old templates or from other calls!

Participant Portal

call

topic

general rules &

regulations

call

topic

specific rules &

regulations

start submissio

n

Participant Portal

Submission Service start page

2x

navigation

Submission Service download template

navigation

Section

1. Excellence relevant to the topics addressed by the call

1.1 Objectives

–

1.2 Relation to the work programme

–

1.3 Concept and approach

1.4 Ambition

2. Impact

2.1 Expected impacts

2.2 Measures to maximise impact

a) Dissemination and exploitation of results

b) Communication activities

3. Implementation

3.1 Work plan — Work packages, deliverables and milestones

3.2 Management structure and procedures

3.3 Consortium as a whole

3.4 Resources to be committed

4. Members of the consortium

4.1. Participants (applicants)

5. Ethics and Security

From call to proposal

One simple rule:help the evaluator

Take the template guidelines very literally. The evaluator expects to find everything at the designated place.

(tip: read the entire template before you start dividing the work )

All sections are related. Make cross-references & connections. Repetition can be functional to stress important things, but avoid redundancy.Use consistent names / words / partner abbreviations throughout.Keep it short and concise.

17

The template

The starting point: call topicWASTE-2-2014: A systems approach for the reduction, recycling and reuse of food wasteSpecific challenge: …………………….Technologies for the collection, sorting/grading, stabilisation and valorisation of food waste, by-products and packaging material need improvement or development. The aim is to optimise the performance of the whole food system, including packaging, catering and consumers, and achieve a secure and sustainable food supply, also for the poor. Scope: ……………A comprehensive methodology for evaluating food waste in all its components should be developed addressing quality, safety, sustainability, legislation and costs. Inter-disciplinary research methods include practical, close-to-market approaches for characterising possible new foods and feeds and identifying the risks and benefits related to the new production processes. A database/inventory should be developed of recyclable materials, valuable molecules, substances and materials originating from waste and by-products, also in view of future life cycle assessments (LCAs). Solid involvement of social sciences and humanities and civil society is a prerequisite to better understanding the socio-economic, cultural and environmental dimension of food waste and promoting change in the business and consumer environment for social innovation, while the use of ICT tools is expected to accelerate this. “………..proposals are encouraged to include third country participants, especially those established in China. The Commission considers that proposals requesting a contribution from the EU in the range of EUR 9 million would allow this specific challenge to be addressed appropriately. Expected impact: • A significant contribution to achieving the European policy target of reducing food waste by

50% by 2030, including at the consumer level. • A reduction in waste management costs, and in environmental impacts, including emission of

greenhouse gases. Type of action: Research and innovation actions

partners

work package

commercialpartner

dissemination

China partner

partners

deliverable

deliverables

budget

objectives

aim

Section

1. Excellence relevant to the topics addressed by the call

1.1 Objectives

–

1.2 Relation to the work programme

–

1.3 Concept and approach

1.4 Ambition

2. Impact

2.1 Expected impacts

2.2 Measures to maximise impact

a) Dissemination and exploitation of results

b) Communication activities

3. Implementation

3.1 Work plan — Work packages, deliverables and milestones

3.2 Management structure and procedures

3.3 Consortium as a whole

3.4 Resources to be committed

4. Members of the consortium

4.1. Participants (applicants)

5. Ethics and Security

The proposal

1. Excellence

1.1 Objectives

Analyze the topic’s Expected impact -> Your objectives should correspond with EU objectivesLink! mention words expected impact in your objectivesMain aim & 3-6 objectives, one sentence eachSMART (specific, measurable, achievable/assignable, realistic, time-related)No long introduction, how you got to this will be described in 1.3 & 1.4

1. Excellence

Example to do:In three years build and disseminate an open access sound library that will form part of an integrated framework of data on ambient noise with over 10 thousands of individual recordings to support marine science and ocean discovery.

Example don’t: Build an open access sound library.

1. Excellence

1.2 Relation to the work programme

Show maximum fit with the specific challenge and scope.

Tip: itemize specific challenge and scope, and describe fit per itemTip: use wording from the call or clearly connect your own wording to their wording

Also fit your proposal in the broader framework of challenges and scope using keywords of:

the introduction of the Work Programme the text of the call above your topic

23

1.3 Concept and approach (1)

Introduce the concept (what) & approach (how) clearly Frame in research landscape:

Description of current state of fieldConnect to running or to be funded nat./ EU/ int. projects

Tip: for EU projects search on Cordis: http://cordis.europa.eu/projects/home_en.html

Highlighted in template & instructions for evaluators:

‘Trans-disciplinary considerations/ discuss cross-cutting issues’

‘Gender issues and gendered innovations’

1. Excellence

1.3 Concept and approach (2)

Link!Connect concept diagram (1.3) to Pert & Gantt (3.1)Approach -> elaborate in WP descriptions (3.1a)

Tip: clearly link using consistent names & terminologyImportant: not of research exclusively, but for all activities (demonstration, piloting, testing, dissemination etc.)

Summary of progress beyond state of the art -> elaborate in 1.4 AmbitionExpected results -> link to 1.1 Objectives

1. Excellence

1) Description of the conceptual relations between the various research questions (objectives), and the ‘umbrella’ hypothesis (main aim).

Example: Main hypothesis, to be approached via research questions 1, 2, and 3. Results of 1 feed into 2 and 3. The PI writes the synthesis.

3) Practical aspects of the various relations: GANTT: time-frame, feedback sessions, deliverables, milestonesPERT: components inter-relations

Example: 3 work packages; wp1 starts (and ends) first, followed after 1 year by wp 2 and 3. Wp 4 (the synthesis) starts near to the point of completion of wp 2 and 3.

The difference between Excellence (1) and Implementation (3)

1. Excellence

Evaluators comments

'The proposal only marginally addresses the concept of the call topic (focus on one system)’.‘Because large parts of the proposal are outside the remit of the call, the overall methodology and work plan is not appropriate.’‘The project outline would have benefitted from a clarification and more systematic implementation of the research hypotheses within the substantial WPs.’'The concept and overall objective is only broadly described, but how it will be implemented is not explained in sufficient detail. Technologies to be used are barely mentioned.'

1. Excellence

1.4 Ambition (= FP7: Progress beyond the state-of-the-art)Instructions for evaluators:‘Extent that proposed work is ambitious, has innovation potential, and is beyond the state-of-the-art’

Tip: Make a bullet list of all innovative aspects of the proposal. Describe in detail per item: a) the current state-of-the art, b) what constitutes beyond the state-of the art, c) and justify how this is achieved within the project.What will be enabled that is not possible right now and HOW will this be achieved?

1. Excellence

1.4 Ambition

Example to do:For the first time it will be possible to address the fragmentation of sources and tools currently available. In addition, as described below, the platform to be developed will offer new tools to analyze xxx, which was hitherto with xxx and xxx (current tools) impossible, because…

Example don’t: The project is innovative, because this has not been done before

1. Excellence

Evaluators comments'The “state of the art” is too succinct and does not provide enough explanation to ascertain the added value.’

‘The claimed innovation aspects of the proposed effort are not clearly demonstrated due to “high level” descriptions of what will be accomplished and how. There is no clear explanation of new techniques that will be developed and how such developments go beyond the state- of-the-art.’

‘The current state-of-the-art has been well described and what constitutes beyond the state-of-the-art has been clearly identified.’

FET-Open!

‘The work has a limited long-term vision and does not go far beyond the state of the art. Whilst the idea of xxx will bring us one step further towards xxx, it cannot be considered a major breakthrough.’

Section

1. Excellence relevant to the topics addressed by the call

1.1 Objectives

–

1.2 Relation to the work programme

–

1.3 Concept and approach

1.4 Ambition

2. Impact

2.1 Expected impacts

2.2 Measures to maximise impact

a) Dissemination and exploitation of results

b) Communication activities

3. Implementation

3.1 Work plan — Work packages, deliverables and milestones

3.2 Management structure and procedures

3.3 Consortium as a whole

3.4 Resources to be committed

4. Members of the consortium

4.1. Participants (applicants)

5. Ethics and Security

The proposal

2. Impact

2.1 Expected impacts

Impacts as described in the topic (but then in your own words)Impact key words from the introduction of work programme and of call above topicConsider societal impact, economic impact, environmental impact, etc. (Grand Challenges)

Contribution to Europe and world (market) (‘European excellence’. Not only academic!!!)

Be realistic. Describe conditions and potential barriers (‘if’)

2. Impact

2.2 Measures to maximise impact (1)

Separate WP on Dissemination & Exploitation (Allocate around 10% of PMs/€)

Get your stakeholders involved from the start, (preferably) as partners

SMEs (exra credit), industry, NGOs, international/EU organisations

GOs, branch organisations and private foundations are also important -> dissemination channelIf not partners give stakeholders power to influence the project -> Advisory BoardIdentify your end users, organise their feedbackNo bla bla. Specify & quantify activities & stakeholders!

2. Impact

2.2 Measures to maximise impact (2)

Dissemination plan (FAQ 8 Guidance for evaluators ) Exploitation plan (e.g. exploitable results table)Management of Intellectual Property RightsData management plan (including protection and open access)

Link!Describe approach here and connect to execution in

3.2 Management structure and proceduresTable 3.1.a. WP Management and WP Dissemination & ExploitationDeliverables (of WPs and in Table 3.1.c)

54

2. Impact

Evaluators comments'Dissemination is rather standard. The strategy for dissemination does not clearly show how the end-users will benefit from the project.''Dissemination strongly relies on classical tools like "developing of advertisement material" or "launch and maintenance of project website". 'The proposal did not lay out a convincing description of effective ways to access and engage those targets.' 'Impacts might be reduced by the limited connection of the Consortium with the REDD network and with other policy networks.''The project team should consider how end users will access the resources’‘There is little evidence of enhancement of competitiveness of companies, at least directly.’

2. Impact

Evaluators comments‘A good plan is provided on specific measures to disseminate and exploit project results across the relevant interested parties.’‘The steps to approach the concerns of general public are also well elaborated (raise awareness, ensure transparancy and demonstrate ways to overcome challenges).’'Target groups are well defined’. 'The impact of the proposal on European excellence is well substantiated’.‘The licensing scheme and exploitation routes are appropriate for the scope of the proposal. Management of IPR is appropriately addressed.

Section

1. Excellence relevant to the topics addressed by the call

1.1 Objectives

–

1.2 Relation to the work programme

–

1.3 Concept and approach

1.4 Ambition

2. Impact

2.1 Expected impacts

2.2 Measures to maximise impact

a) Dissemination and exploitation of results

b) Communication activities

3. Implementation

3.1 Work plan — Work packages, deliverables and milestones

3.2 Management structure and procedures

3.3 Consortium as a whole

3.4 Resources to be committed

4. Members of the consortium

4.1. Participants (applicants)

5. Ethics and Security

The proposal

3. Implementation

3.1 Work plan – work packages (WP), deliverables, milestones (1)

38

39

Gantt chart

Pert diagram

WP2: Definition of a conceptual and methodological

resilience framework WP1: Screening of theoretical

background

and empirical evidence

of resilience

WP3: Participatory operationalization and adaptation

of the framework in representative case studies

WP4 : Testing

transferability

to complex systems WP5: Strategies

to enhance resilience

at different scales

WP6: Awareness raising, dissemination and knowledge transfer

WP7: Project management

http://www.wikihow.com/Create-a-PERT-Chart-Using-Microsoft-Office-2007

3. Implementation

3.1 Work plan – work packages (WP), deliverables, milestones

Useful numbers:2 M€ -> ±5 WPs, 6 M€ -> ±9 WPs Deliverables 2-4 per WPMilestones 1-6 per projectIdeal WP length 3 pagesTotal section 1-3 50 pages (max=70 pages, not a target!)

3. Implementation

3.2 Management structure & procedures

WP Management/Coordination (WP 1 or last)Allocate 7% (never more!) of person months/budget

Management diagram Innovation management!!!

tip: lead by business partner/policy partner or societal partner depending on the topic

Risk table / contingency plan (Table 3.2b.)tip: include management risks, such as conflict and partner

lossConsortium Agreement and Managing IP

NB: foresee any “legacy” and infrastructure from the project, in terms of formalised entities that might take up the management of a network created during the project

3. Implementation

3.3 Consortium as a whole

Explanation on why this is the dream team, no overlap in specific expertises for this topic, but complementaryHighlight expertise & experience of individual partners in relation to the research objectives, but also for management, exploitation and disseminationTeam covers the width of the call:

science -> society, value chain, target groupsgeographical balance, include underperforming countriestypes of organisations (SMEs!)

3. Implementation

3.4 Budget

Evenly balanced (€ and person months) across partners or explainNever >35% to 1 countryWP MGT (7%) and WP Dissemination & Exploitation (around 10%)Eligible direct costs: Model Grant AgreementOutline budget whole consortium: ask advise LURISDetails plus internal budget: ask project office/ IM

3. Implementation

Evaluators comments'The proposal does not contain a formal structure for ensuring the continued involvement of participants and the information flow between them.''No instrument is planned for ensuring participation of stakeholders into decision-making processes and for allowing end-user feedback during the project.''The allocation of resources is not appropriate. 128 person months seems too much to spend on the assessment of xxx (WP3).''The panel further would have welcomed the inclusion of an external scientific advisory board.’‘The management and coordination budget is relatively high.’‘The consortium is complementary but unbalanced and there is a lack of explanation of the unbalanced workload.’‘Industry is sufficiently targeted in the exploitation and dissemination plan. This is good. However no partner in the consortium represents IT industry or is an SME.’

4. Members of the consortium

Show that you have all the necessary expertise, including management, dissemination, exploitation, etc and infrastructureShow previous experience in project tasks as well as with working togetherBe aware of gender balance and/or issuesGeographical balance (new member states)

tip: send out template and example to partners -> consistent lay-out

Cover page

Use the cover page as business card of the consortium

Tip: Analyze the competition

Photo by: globaldesignw

orkshop.com

Do you have knowledge of a competing consortium? Emphasize the differences / complementarities in the abstract

Will more than one proposal be granted?Adjust the budget (2 proposals, total 10 M€ then 5 M€ each?)

The summary

Show:Aim of the projectSpecific objectives Why now?Why do you have the dream team?What are the expected results for the EU and why are they innovative? What will the impact in the EU be after the project has finished. And a few years after that!

(repeat this in the first two pages of the proposal!)

Finally

The acronym

One word, or couple of words without spacesPreferably relevant to the projectDo not use registered trademarksCheck in googleMake it pronounceable, alliteration is a plus

Goal of this meeting

Show you how to write high scoring H2020 collaborative proposals (RIA) that lead to solid research projects and a lasting infrastructure

Topics:Expectations from the EUDos & don’ts for the proposalThe evaluator

Highlighted: Ethics, data management & genderFirst hand experience: Dr. Antoaneta Dimitrova

Who are my evaluators?

Independent expertsnot only scientific expertise but also experience with:

commerce and business

specific technologies found in the topics of the draft work programmes

working multidisciplinary with social sciences and humanities

innovation

ethics

gender

communication

project management

HORIZON 2020

57

HORIZON 2020

57

Evaluation Process

Individual Evaluation

Report

Individual Evaluation

Report Individual Evaluation

Report

Consensus group

Consensus Report

Individual Evaluation

Report

Individual Evaluation

Report

Expert Expert Expert ExpertExpert Minimum 3 experts

Individual evaluation

Consensus

Proposal Eligible proposal

HORIZON 2020

58

HORIZON 2020

58

Evaluation criteria

For the first stage of a two-stage procedure, only the aspects of the criteria in yellow are evaluated

Clarity and pertinence of the objectives

Soundness of the concept, including trans-disciplinary considerations, where relevant

Extent that proposed work is ambitious, has innovation potential, and is beyond the state of the art (e.g. ground-breaking objectives, novel concepts and approaches)

Credibility of the proposed approach

Exc

elle

nce

The expected impacts listed in the work programme under the relevant topic

Enhancing innovation capacity and integration of new knowledge

Strengthening the competitiveness and growth of companies by developing innovations meeting the needs of European and global markets; and, where relevant, by delivering such innovations to the markets

Any other environmental and socially important impacts (not already covered above)

Effectiveness of the proposed measures to exploit and disseminate the project results (including management of IPR), to communicate the project, and to manage research data where relevant

Impact

Research and Innovation/Innovation/SME instrument

Coherence and effectiveness of the work plan, including appropriateness of the allocation of tasks and resources

Complementarity of the participants within the consortium (when relevant)

Appropriateness of the management structures and procedures, including risk and innovation management

Imple

menta

tion

HORIZON 2020

59

HORIZON 2020

59

Proposals with identical total scores For each group of proposals with identical total scores, the panel

considers first proposals that address topics that are not already covered by more highly-ranked proposals

The panel then orders them according to:

• first, their score for Excellence,

• and second, their score for Impact

[for Innovation actions and SME instrument, first their score for Impact and second for Excellence]

If there are ties, the panel takes into account the following factors:

• First, the size of the budget allocated to SMEs

• Second, the gender balance of personnel carrying out the research and/or innovation activities

If there are still ties, the panel agrees further factors to consider:

• e.g. synergies between projects or contribution to the objectives of the call or of Horizon 2020

The same method is then applied to proposals that address topics that are already covered by more highly-ranked proposals

Tools to help yourself focus on the evaluation

Guidance for evaluators of Horizon 2020 proposals

General briefing for evaluators

Sign up as an evaluator

Review proposals or projects (1-3 days)Remote or in BrusselsFee 450 E / day + travel expenses + subsistence expenses

More info: http://www.leidenuniv.nl/luris-intra/Horizon2020.-.call.for.experts.html

Making a online profile to sign up takes 20 min:http://ec.europa.eu/research/participants/portal/desktop/en/experts/index.html

Goal of this meeting

Show you how to write high scoring H2020 collaborative proposals (RIA) that lead to solid research projects and a lasting infrastructure

Topics:Expectations from the EUDos & don’ts for the proposalThe evaluator

Highlighted: Ethics, data management & genderFirst hand experience: Dr. Antoaneta Dimitrova

Poortgebouw Noord

Rijnsburgerweg 10

2333 AA Leiden

+31 (0)71 5 26 55 73

luris@luris.nl

www.luris.nl

Management assistant:José Zweekhorst j.b.a.zweekhorst@luris.nl 071-526 5573

Grant Development Advisors: Anke Klerkx j.h.e.m.klerkx@luris.nl071-526 5661Anna Groeninx a.k.groeninx@luris.nl 071-526 5567Jorrit Kelder j.m.kelder@luris.nl 071-526 5569

Grant development support

Questions?