The Evaluation Process, Tips from an Evaluator’s Point of View

Leonardo Piccinetti

EFB

OUTLINE

• Evaluation principles• The experts, who are they ?• Role of Commission staff• FP7 evaluation process• FP7 Evaluation Criteria

Basic principles • Excellence. Projects selected for funding must demonstrate a high

quality in the context of the topics and criteria set out in the calls.• Transparency. Funding decisions are based on clearly described

rules and procedures, and applicants should receive adequate feedback on the outcome of the evaluation of their proposals.

• Impartiality. All proposals submitted to a call are treated equally. They are evaluated impartially on their merits, irrespective of their origin or the identity of the applicants.

Basic principles • Confidentiality. All proposals and related data, knowledge and documents

communicated to the Commission are treated in confidence.• Efficiency and speed. Evaluation, award and grant preparation should be as

rapid as possible, commensurate with maintaining the quality of the evaluation, and respecting the legal framework.

• Ethical and security considerations: Any proposal which contravenes fundamental ethical principles, or which fails to comply with the relevant security procedures may be excluded at any time from the process of evaluation, selection and award

The experts, who are they ?

•The Commission draws on a wide pool of evaluators (database) in all scientific fields

–c. 50,000 in FP6–Experts/evaluators from ICPC are very welcome! •Calls for “candidates”–Call for applications from individuals; and from institutions–Applications via CORDIS (database of experts)

•A mass-emailing of FP6 experts was sent–A simple tick-box will ensure registration for FP7

•Commission invites individuals on a call-by-call basis–Not self-selection!

•Expertise, and experience are paramount–Geography, gender and “rotation” also considered

Independent experts• Expert evaluators are at the heart of the FP7 system• Expert provides independent, impartial and objective

advice to the Commissionrepresents neither the employer, nor the country!

• Significant funding decisions will be made on the basis of expert advice

• The integrity of the process is crucial – Experts have to read the Code of Conduct annexed to the

appointment letter…and follow it!

• Experts agree to terms and conditions of an “appointment letter”

• Typically, an individual will review 6-8 proposals “remotely”….

• …then spend a couple of days in Brussels• Some will participate in “hearings” with the consortia • Travel and subsistence reimbursed

– Plus €450 honorarium per day• Experts sign confidentiality and conflict of interest

declaration • Names published after the evaluations

Independent experts

Actors Confidentiality– The content of proposals, or the evaluation results, can’t be

discussed with anyone• The sole exception: in the presence of the EC moderator with experts

who are evaluating the same proposal in a consensus meeting group or final panel

– Is not possible to distribute any documents related to the evaluation of a proposal, or take any documents from the evaluation building

• Note: The Commission publishes names annually, but as a group – no link between expert and proposal

Conflicts of interest (2)• Types of COI set out in appointment letter

– Check the exact wording!• Disqualifying COI

– Involved in preparation of proposal– Stands to benefit directly– Close family relationship– Director/trustee/partner– Employee of a partner in a proposal– Member of Advisory Group– Any other situation that compromises impartiality

• Potential COI– Employed within the last 3 years by a partner in a proposal– Involved in research collaboration with proposers in the previous 3 years– Any other situation that casts doubt…or that could reasonably appear to

do so…

Role of Commission staff

• Check the eligibility of the proposals

• Oversee work of experts

• Moderate discussions

• Organise the panel and its work

• Ensure coherence and consistency

Evaluation

• Peer-Review System• Two-stage evaluation procedure• Remote evaluation• Evaluation on a non-anonymous basis• Unless otherwise specified in call for proposal• Register as an Evaluator• https://cordis.europa.eu/emmfp7/

Role of Commission staff• Commission staff may advise on:

Background on previously supported or on-going projects Relevant supplementary information (directives,

regulations, policies, etc)Evaluation rulesKey points within the Work Programme,

e.g. issues related to “Relevance”• Commission staff may not introduce:

New elements (cannot fill in “gaps” in proposals)Interpretations

Writing an FP7 proposal is NOT just a creative process for A NICE IDEA

• It requires to SHOW scientific, technological and depth knowledge of the subject

• You must present references, legislatures, previous work and experience

Be precise and to the point• Use drawings to show your

methodology at once• Use bold types in phrases that you

would like to emphasize• Present cohesion and interaction

among work packages

Overview of the Evaluation Overview of the Evaluation ProcessProcess

Full Proposal

Proposalforms

Evaluators

Eligibility

Evaluators Evaluators Final rankinglist

PanelSubmission ConsensusIndividual

reading

Proposals insuggestedpriority order

Rejection list

Finalisation

CriteriaCriteria Criteria

COMMISSION COMMISSION

“rem

ote”

may

be

“rem

ote”

Role of experts

FP7 evaluation process

IAR 1

IAR 2

IAR 3

CONSENSUS MEETING

RECOMMENDANTION

FOR FUNDING OR NOT

CONSENSUS REPORT BY THE RAPORTER

PANNEL MEETING – RESULTS ARE SEND TO APPLICANTEVERY IAR IS 3 TO 4 HOURS

0 - the proposal fails to address the issue under examination or can not be judged against the criterion due to missing or incomplete information

1 - poor2 - fair

3 - good4 - very good5 - excellent

RELEVANCE

POTENTIAL IMPACT

QUALITY OF COORDINATION

QUALITY OF MANAGEMENT

QUALITY OF THE CONSORTIUM

1. CONCEPT – CAPABILITIES OF THE PARTNERS

2. ACTIVITIES – AND CAPACITY TO DO THE TASK

MOBILISATION OF RESOURCES

3. THE TARGET GROUPS THAT BENEFIT

The experts:• Evaluate the proposal individually

– without discussing with the other evaluators• Check whether the proposal is ‘in scope’• Complete an Individual Evaluation Report (IER) form giving

comments and scores on all criteria• Sign and date the form

– IERs should be checked and, if necessary, returned with a request to further justify the score given

– Scores must be in line with comments

Process Evaluation - Individual reading(Will be done remotely)

• Built on the basis of the individual evaluations• The aim is agreement on scores and comments• Usually involves a discussion

– 1st part may be carried out remotely • “Outlying” opinions need to be explored

– Not just a simple averaging exercise

– It is quite normal for individual views to change • Moderated by a Commission staff-member

– helps the group reach a conclusion– provides information if necessary– does not contribute opinions

Process Evaluation - Consensus (1)

• A rapporteur is appointed, who is responsible for drafting the consensus report (CR)– includes consensus marks and comments

• The quality of the CR is paramount– It is not often changed at the panel review stage

• The aim is:– a clear assessment of the proposal, with justification– clear feedback on weaknesses & strengths

• To be avoided:– scores that don’t correspond with the comments– recommendations in view of resubmission

Process Evaluation - Consensus (2)

• EC ask some evaluators in each sub-activity to examine and compare the CRs of every proposal that passes all thresholds

• Key function is to ensure consistency• The Panel will recommend for a sub-activity a priority order

including final marks and comments for each proposal Evaluation Summary Reports (ESR) Any new scores (if necessary) … should be carefully justified

• Ranking of proposals with identical consensus scores• Prioritise certain criteria?• Consider overall balance?• Budget?

• Clear guidance for contract negotiation

The Panel Review

Process Commission Follow-up• Evaluation summary reports sent to applicants

– “initial information letter”– Redress procedure

• Draw up final ranking lists• Information to the Programme Committee• Contract negotiation• Formal consultation of Programme Committee (when required)• Commission decisions • Survey of evaluators• Independent Observers’ reports

New for FP7

The evaluation criteria • Criteria adapted to each funding scheme and

each thematic area – specified in the work programme

• Three main criteria:– S&T Quality (relevant to the topic of the call)

• Concept, objective, work-plan– Implementation

• Individual participants and consortium as a whole• Allocation of resources

– Impact• Contribution to expected impacts listed in work

programme• Plans for dissemination/exploitation

Process FP7 Evaluation CriteriaApplicable to ALL funding schemes

1. S/T quality(in relation to the

topics addressed by the call)

2. Implementation 3. Impact

• Sound concept, and quality of objectives

• Appropriateness of the management structure and procedures

• Quality and relevant experience of the individual participants

• Contribution, at the European and / or international level, to the expected impacts listed in the work programme under the relevant topic / activity

29 Guidance for evaluators – Call FP7-ENV-2008-1

Collaborative projects

Support to research projects carried out by consortia with participants from different countries, aiming at developing new knowledge, new technology, products, demonstration activities or common resources for research.

The size, scope and internal organisation of projects can vary from field to field and from topic to topic.Projects can range from small or medium-scale focused research actions to large-scale integrating projects for achieving a defined objectiveProjects may also be targeted to special groups such as SMEs, Specific International Co-operation Actions, etc.

Funding schemesFunding schemes

Template of the IER for your information(double click on the object)

Microsoft Word Document

Process

Process FP7 Evaluation CriteriaCollaborative projects

1. S/T quality(relevant to the topics addressed by the call)

2. Implementation 3. Impact

• Prgress beyond the state-of-the-art

• Quality and effectiveness of the S/T methodology and associated work plan

• Quality of the consortium as a whole (incl. complementarity, balance)

• Appropriate allocation and justification of the resources to be committed (budget, staff, equipment)

•Appropriateness of measures for the dissemination and/or exploitation of projects results, and management of intellectual property.

31Guidance for evaluators – Call FP7-ENV-2008-1

Coordination & support actions

Support to activities aimed at coordinating or supporting research activities and policies (networking, exchanges, trans-national access to research infrastructures, studies, conferences, etc).

same scope and objectives as in FP6same scope and objectives as in FP6

Funding schemesFunding schemes

Template of the IERs for your information(double click on the object)

Coordination Action - IER

Support Action - IER

Process

Coordination & support actions1. S/T quality

(in relation to the topics addressed by the call)

2. Implementation 3. Impact

CA

• Contribution to the co-ordination of high quality research• Quality and effectiveness of the co-ordination mechanisms, and associated work plan

• Quality of the consortium as a whole (including complementarity and balance)

• Appropriate allocation and justification of the resources to be committed (budget, staff, equipment)

• Appropriateness of measures for spreading excellence, exploiting results, and dissemination knowledge, through engagement with stakeholders, and the public at large.

SA

• Quality and effectiveness of the support action mechanisms, and associated work plan

RELEVANCE

Examines if the objectives of the work programme are met

The proposer must read in between lines

It must be clear what the commission wants from the call

Good practice: attend info day, inside information

Covering the objectives u get a 4, additional objectives might give u a 5

TIPS:

• Do not copy paste the objectives from the work programme

• U must copy and specialize them to the concept of the proposal

• FIND THE KEY WORDS (ie networking, knowledge transfer) relating the key word to the objective

• TRY TO QUANTIFY OBJECTIVES ( i.E. Not just networking – but networking of three clusters)

• Always make a direct reference under each objective with the related work packages of how this objective will be achieved

Proposal relevance tips

• Identification of impact from Call• Definition of strategic goals for our proposal

which clearly fulfill the expected.• Definition of main actions/activities clearly

related to the strategic goals• Work Packages for actions/activities.

QUALITY OF CONSORTIUM

Examines the experience of the consortium in relation to the objectives and the work need to be done, complementarities between partners, suitability of work undertaken, and geographical coverage

TIPS

Always include a description in the beginning of consortium description indicating the role of each partner and their added value to the project.

It is preferred to have inside the consortium mix Member States, New member states and Associated states indicating transfer of good practices to new member and associated states.

Under each partner description indicating the excellences (previous projects, research activities and experiences) and of the partner related to the proposal followed by the CVs of key persons involved and their role in the project

These excellences should be summarized into an excellence key areas of the consortium in relation to the work need to be done

Indicate the logic behind consortium geographical and context selection (why these partners are selected)

The coordinator

• There is no limitation related to the size, nature, legal status and the years of existence for the proposed coordinator.

• In the management forms (A1, A2 and A3) the turnover is presented. The is a indirect but very important effect to the decision of the Commission.

QUALITY OF COORDINATION

Examines coherency of the proposal. If there is a methodology that defines the interrelationships between the work packages and the work tasks

The plan must convince the evaluators that is a bottom down approach from the implementation plan to the work packages

TIPS

• Start with the relation of the objectives to the work need.

• Define your methodology of work mentioning entities (not yet work packages)

• Define the interrelationships of the WPs and the work tasks inside the work packages

• Your methodology should have a clear description of impact creation in the target groups and European Added Value through the actions of the dissemination plan

• Do not include management packages, just coordination packages this would be explained in another section.

• YOU MUST ALWAYS START WITH THE METHODOLOGY OF WORK AND THEN WRITE THE WORKPACKAGES

• Work packages should mirror the work in the implementation plan with CLEAR milestones and derivables

• In general you could exceed the number of pages asked from the commission

General tips

• Every text with more than three paragraphs must begin with a 2-3 lines abstract with a clear reference to what the remaining text presents.

• In most cases the opinion of the evaluator is coming from this abstract.

• Phrases 8 words, paragraphs max 3-4 lines.

QUALITY OF MANAGEMENT

Examines the experience and management mechanisms of the management unit

The evaluator examines the capacity of the coordinator to handle the tasks

They like to have a strong centralized management and not very complex structures

They like to see decision making mechanisms from steering committees

TIPS

Always start indicating the central management control indicating the experience of the coordinator to undertake the tasks in similar projects

Illustrate decision making mechanism (simple but partner inclusive)

Show contingency plan with risks (i.e. what happens if partner withdraws)

Show IPR management

Show quality assurance mechanisms

Show knowledge management with in partners.

For each of the above relate them with management instruments (web sites, consortium agreements, meetings, etc)

MOBILISATION OF RESOURCES

Examines the coherence of the financial plan to handle the tasks, relating to the allocation of work in PM within work packages and among partners.

The financial plan must be delivered with the proposal although there are no predefined forms from that (except A3, deliverable lists)

TIPS

ALWAYS state the CVs of key persons involved with the project and their role

Illustrate other resources for each organization that are allocated to the project in the partnership description (ie equipment, previous work done etc)

Subcontracting is always examined in detail (why and who will do it?)

Avoid large differences in work allocation between partners

If a partner gets more than 30% of the budget (management and coordination) is negative

Management no more than 7%

POTENTIAL IMPACT

Examines the impact (results) to target group defined, and the European Added value and mainly your dissemination plan

Relates to the previous work done in European and National Level

TIPS

Always start with the clear definition of the proposal target groups indicating the potential impact in quantitative terms if possible

Use structured bulleted writing and not abstract descriptions

The dissemination plan should provide evidence of European added value in quantitative terms (not general instruments i.e. web page but exclusive actions such as connection with specific initiatives, work groups etc)

Always illustrate impact in relation to previous work done – specific national and European programmes indicating the approach mechanisms (even it is obvious)

The main evaluation criteria is the dissemination plan. The above issues should be described as work to be done in the dissemination work package

ABSTRACT

Should start with objectives followed by the consortium logic

Then should describe the implementation plan with the expected results

and finish with the impact through the dissemination plan

Self-assessment process

• Most of failed proposals are taking a low mark in 1-2 evaluation criteria (relevance is the most usual).

• The pre-evaluation of the proposal by internal or external executives (definitely not involved in the proposal preparation process) based on the published evaluation criteria.

Conclusion

• Become evaluator is the best way to learning to write proposals

• Understanding how EC works • Networking • Well paid

Thank you!

Leonardo Piccinetti