Proposal evaluation process in FP7

Moldova – Research Horizon29 January 2013

Kristin Kraav

Evaluation process

Individual evaluation

Commission pre-actions

Consensus meetings

Panel meeting

CommissionFollow-up

Commission pre-actions

Individual evaluation

Commission pre-actions

Consensus meetings

Panel meeting

CommissionFollow-up

Commission pre-actions• Appointment of call coordinator and coordinators

for the topics• Selection of the independent experts for the Call• FP7 experts database (https://cordis.europa.eu/emmfp7)

• personal head-hunt• Proposals eligibility check• eligibility criteria may be budget, number of partners,

duration … • Formation of the expert panels for each topic of

the call• Assignment of proposals to chosen experts

Individual assessment

Individual evaluation

Commission pre-actions

Consensus meetings

Panel meeting

CommissionFollow-up

Individual assessment• Remote evaluation period ~4-6 weeks• Everything happens in online system RIVET• experts can access only assigned proposals

• Max 10 proposals per expert, min 3 experts per proposal

• Experts have to make their decisions based on:• proposal• Call background documents• evaluation criteria

• Result – Individual Assessment Report• experts personal opinion• basis for the discussions in the next stage

The evaluation criteria• Three evaluation criteria (Cooperation and

Capacities – People and Ideas differ significantly):– Scientific and/or Technological excellence– Quality and efficiency of the implementation and

the management – The potential impact through the development,

dissemination and use of project results

Scoring the proposals• Each proposal is evaluated against all criteria

and scored between 0 and 5• Threshold per criteria – 3• Overall threshold per proposal – 10 • This means – if any of the criteria under 3

points or the total scoring of the proposal under 10 points, the proposal will be placed to the list of rejected proposals for reason under the threshold

Interpretation of the scores• 0 – The proposal fails to address the criterion under examination or

cannot be judged due to missing or incomplete information.• 1 – Poor. The criterion is addressed in an inadequate manner, or

there are serious inherent weaknesses.• 2 – Fair. While the proposal broadly addresses the criterion, there are

significant weaknesses.• 3 – Good. The proposal addresses the criterion well, although

improvements would be necessary.• 4 – Very good. The proposal addresses the criterion very well,

although certain improvements are still possible.• 5 – Excellent. The proposal successfully addresses all relevant aspects

of the criterion in question. Any shortcomings are minor.

Slide 13

Consensus meetings

Individual evaluation

Commission pre-actions

Consensus meetings

Panel meeting

CommissionFollow-up

Panel of experts

Consensus meetings• 1 week for face-to-face meetings, 12 hour per day• Consensus meeting• 1.5 hour meeting per proposal gathering all experts

who evaluated it, moderated by EC staff-member• basically restart of evaluation• has to end with consensus on scores and comments

for each of the criteria → Consensus Report• On the bases of discussions rapporteur (one of

the evaluators) drafts the first version of Consensus Report

Consensus meetings → ESR• First draft of CR turns into Evaluation Summary

Report through:• co-operation of all experts involved• comments from Call coordinator and topic coordinator• help of language editor

• ~4-5 day parallel process for all proposals submitted under the topic

• If the CM does not end with consensus:• additional experts from the panel will evaluate it• new CM is called

• If still no consensus → total restart of the process with new panel of experts

Panel meeting

Individual evaluation

Commission pre-actions

Consensus meetings

Panel meeting

CommissionFollow-up

Panel meeting → proposed ranking list• Drawing up the ranking list of the topic• Comparison of scores and ESR• Final marks and comments for each proposal• Suggestions on order of priority, clustering,

amendments, etc. • Simple process if the ranking is clear• Tricky process if many proposals have equal score

and all of them can not be funded• Priority order of criteria in case of equal scores• If all scores equal, experts have to decide which score

is stronger and support it with relevant arguments

Commission follow-up

Individual evaluation

Commission pre-actions

Consensus meetings

Panel meeting

CommissionFollow-up

Commission follow-up• Draw up final ranking lists

• decisions on proposals selected for funding• decisions on rejected proposals

• Information and data to the Programme Committee

• PC formal agreement needed for financing proposals with budget over 600 000€

• Independent Observers’ report• Invitation to Contract Negotiation and Evaluation Summary Reports sent

to coordinators• Contract negotiations

Evaluation process

Individual evaluation

Commission pre-actions

Consensus meetings

Panel meeting

CommissionFollow-up

Evaluation process – lessons learned

Individual evaluation

Commission pre-actions

Consensus meetings

Panel meeting

CommissionFollow-up

Process of evaluation• Evaluation is a deconstruction of proposals• Comparison of different elements of proposal against

each other• overall aim → steps to achieve it (activities) → expertise

(partners) for managing the process → allocated resources• Everything should be written as clearly and shortly as

possible – still all required parts of Part B have to be there

• “A picture is worth more than 100 words” – tables, graphs, schemes help immensely

• Do not expect evaluators to assume things

Remember:

• Evaluators are humans , they• come from different professional and ethnical backgrounds;• have different beliefs of right and wrong;• speak different languages;• usually are under time pressure

• Respect the evaluators:• follow the structure• edit your proposal to eliminate typos and other mistakes• make the proposal legible: font and font size, structured text, test

that all graphics can be read in b&w

Conclusions• Fit to the call : read carefully the call topic

• Be outstanding on the Scientific and Technology point of view but do not underscore the other criteria

• Be credible

• Demonstrate an EU added value

• Respect the rules : read the Work Programme and Guide for Applicants