Consigli per la presentazione delle proposte: il punto di ...€¦ · Parma, 23 Settembre 2008 Consigli per la presentazione delle proposte: il punto di vista di un valutatore Luca

Parma, 23 Settembre 2008

Consigli per la presentazione delle proposte:il punto di vista di un valutatore

Luca Fontanesi

DIPROVAL, Sezione di Allevamenti ZootecniciUniversità di Bologna


What an applicant should known

1) The experts, who are they ?

2) The experts, what they do ?The evaluation process

3) The experts, they are human beingsDynamics of the evaluation process


The experts, who are they ? (1)

• The Commission draws on a wide pool of evaluators – c. 50,000 in FP6

• Calls for “candidates” are to be published mid-December– Call for applications from individuals; and from institutions– Applications via CORDIS

• A mass-emailing of FP6 experts was sent– A simple tick-box will ensure registration for FP7

• Commission invites individuals on a call-by-call basis– Not self-selection!

• Expertise, and experience are paramount– Geography, gender and “rotation” also considered


The experts, who are they ? (2)

• Experts agree to terms and conditions of an “appointment letter”

• Typically, an individual will review 6-8 proposals “remotely”….

• …then spend a couple of days in Brussels

• Travel and subsistence reimbursed– Plus €450 honorarium per day

• Experts sign confidentiality and conflict of interest declaration

• Names published after the evaluations


The experts, what they do ? The evaluation process (1)

• Process and principles• Confidentiality and Conflicts of interest• The evaluation criteria• The procedure in practice

– Individual evaluation (remote)– Consensus meeting (in Brussels)– Final panel meeting (in Brussels)


The evaluation process (2)

1)Appointment letter

2)Conflicts of Interest (COI)

3)Remote evaluation

4)Consensus meeting in Brussels



Conflicts of interest (3.1)

• COI definitions are spelled out in the appointment letter

• You must immediately inform the Commission if you become aware of a conflict of interest (COI)

• If there is a “disqualifying” COI for a certain proposal you cannot evaluate it

• Experts with a “potential” COI:– The Commission will consider such circumstances

on a case by case and will take the appropriate decision



Conflicts of interest (3.2)

• Types of COI set out in the appointment letter– Check the exact wording!

• Disqualifying COI – Involved in preparation of proposal in the same Call– Stands to benefit directly– Close family relationship with an applicant– Director/trustee/partner of an applicant

organisation– Employee (but, possible exception�)– Member of an Advisory Group– Any other situation that compromises impartiality

• Potential COI– Employed by one of the applicants in last 3 years– Research collaboration with one of the applicants in

previous 3 years– Any other situation that casts doubt� or that could

reasonably appear to do so�


Evaluating a proposal

• Three guiding principles:• Objectivity

– each proposal is evaluated as it is written (no assumptions)

• Accuracy – The experts should make their judgment against the

two-three official evaluation criteria, and nothing else• Consistency

– The experts should apply the same standard of judgment to each proposal




Evaluators

Eligibility

Evaluators Evaluators Above-thresholdlist

Final PanelSubmissionto stage 1 ConsensusIndividual

reading

Proposals in 2 lists

(above and below

threshold)

Below-threshold list

Finalisationstage 1

2 Criteria 2 Criteria

COMMISSION COMMISSION

�remote

�

Role of experts

Submissionto stage 2

FullProposal

Brusse

ls

Proposals forms

20 pages proposal

Stage 1



Evaluators

Eligibility

Evaluators Evaluators Score based list

Final PanelSubmissionto stage 2 ConsensusIndividual

reading

Proposals with the

final score

Finalisationstage 2

3 Criteria 3 Criteria

COMMISSION COMMISSION

�remote

�

Role of experts

Negotiation

Brusse

ls

Proposals forms

Full proposal

Stage 2 or only one Stage


Proposal Xcopy 1

Proposal Xcopy 2

Proposal Xcopy 3

IERexpert 1

IER expert 2

IER expert 3

Consensus meeting

CR 3 experts

�Remote� �In Brussels�


For each proposal


Evaluation Process - Basic PrinciplesBasic Principles

TRANSPARENCY

EXCELLENCEEXCELLENCE

FAIRNESS & FAIRNESS &

IMPARTIALITYIMPARTIALITY

CONFIDENTIALITYCONFIDENTIALITY ETHICAL & SECURITY ETHICAL & SECURITY

CONSIDERATIONSCONSIDERATIONSEFFICIENCY & EFFICIENCY &

SPEEDSPEED

Guide for applicants (annex 2)

�Evaluation Rules�




• Criteria adapted to each funding scheme and each thematic area– specified in the work programme

• Two main criteria for stage 1:

– S&T Quality (relevant to the topic of the call)• Concept, objective

– Impact• Contribution to expected impacts listed in work programme

• Three main criteria for stage 2 (or single stage):

– S&T Quality (relevant to the topic of the call)• Concept, objective

– Implementation• Individual participants and consortium as a whole• Allocation of resources

– Impact• Contribution to expected impacts listed in work programme

The Criteria




• Each criterion is scored from 0 to 5 points– half-points allowed– whole score range should be considered

• Thresholds apply to individual criteria…– Default threshold for Stage-1 is 3– Scores must be equal or above thresholds for

each evaluated criterion for a proposal to pass the evaluation


Proposal scoring


• 0: The proposal fails to address the criterion under examination or cannot be judged due to missing or incomplete information

• 1: Very poor. The criterion is addressed in a cursory and unsatisfactory manner.

• 2: Poor. There are serious inherent weaknesses in relation to the criterion in question.

• 3: Fair. While the proposal broadly addresses the criterion, there are significant weaknesses that would need correcting.

• 4: Good. The proposal addresses the criterion well, although certain improvements are possible.

• 5: Excellent. The proposal successfully addresses all relevant aspects of the criterion in question. Any shortcomings are minor.


Interpretation of the scores


• Normally these have already been carried out by the Commission:

– Receipt before deadline • Firm deadlines

– Minimum number of partners• As set out in work programme and the call

– Completeness of proposal • Presence of all requested forms

– “Out of scope”• A proposal will only be deemed ineligible in clear cut case

– Other criteria may apply on a call-by-call basis• Eg. Maximum EU contribution


Eligibility checks


The evaluators first carry out individual readings remotelyThe evaluators will:

• Evaluate the proposals individually – without discussing with the other evaluators

• Check whether the proposal is �in scope�• Complete online (RIvET) an Individual Evaluation

Report (IER) form for each proposal, giving scores and comments on all relevant criteria (2 in Stage 1)– Scores must be consistent with the comments

Remote individual reading



• To discuss the individual evaluations/opinions• Find consensus on final scores and comments for each proposal

and produce Consensus Reports• “Outlying” opinions need to be explored

– Not just a simple averaging exercise• Moderated by a Commission staff-member who:

– appoints a rapporteur for each proposal– does not participate in the discussion– does not contribute her/his opinions– may help the group to reach a conclusion– may provide additional information if necessary

Consensus meeting in Brussels (1)



• The rapporteur is responsible for drafting the Consensus report (CR) for the assigned proposal– includes consensus scores and comments

• The quality of the CR is paramount– clear analysis and assessment of the proposal, with

justification for the given scores– clear feedback on weaknesses & strengths

• To be avoided:– comments inconsistent/not in line with the scores– recommendations for resubmission

Consensus meeting in Brussels (2)



The evaluation process consists of 2 steps

1 Remote individual reading The experts will receive the proposals by postal mail and will be kindly requested to read and evaluate them against the relevant set of criteria.

2 Consensus meeting The experts are invited to come to Brussels to discuss with other evaluators the proposals of your consensus group. For each proposal agreement should be reached on the final consensus score for each criterion and on the relevant comments. A Consensus Report (CR) will be produced and signed for each proposal.

NB: Some evaluators will be asked to represent their single panel in the Final panel which will take place at the very end of the evaluation session.



• To assess that equal treatment and consistency have been applied to all the proposals

• Resolve any cases where a minority view is recorded in the Consensus report

• Propose revised scores and comments (in exceptional cases)!Any new score should be properly justified

• To approve the final list of above-threshold and below-threshold proposals

• The Evaluation Summary Reports (ESR) will be prepared based on the final Consensus reports

The Final Panel



How will the evaluation run?

• Provided the Commission has received the signed copies of the expert’s Appointment Letter, the expert can start evaluating.

• The expert will have to read the proposals and to evaluate them according to the set of criteria. Knowledge of the workprogramme and of the specific topics is also required.

• The experts will use an electronic tool called RIVET. This programme will allow to fill in the Individual Evaluation Report (IER).



Overview of the evaluation process

Remote reading

Consensus + Final panelmeeting

25 Sep � 8 Oct 2007

Dea

dlin

e

11/09/2007

15-19 Oct 2007

08/10/2007C

ompl

etin

g IE

Rs

on R

IvE

T



Dynamics of the evaluation process (1)

The experts follow the evaluation rules

The experts are usually scientists

The experts have their own opinions

The experts are human beings


Dynamics of the evaluation process (2)

Usually the best proposal can pass the evaluation process


Grazie per l�attenzione

Documents

Consigli per la presentazione delle proposte: il punto di ...€¦ · Parma, 23 Settembre 2008 Consigli per la presentazione delle proposte: il punto di vista di un valutatore Luca