Information Society TechnologiesInformation Society Technologiesin the 6th Framework Programmein the 6th Framework Programme

Evaluation of ProposalsEvaluation of ProposalsDr. Evangelos Ouzounis Dr. Evangelos Ouzounis

Directorate CDirectorate CDG Information SocietyDG Information SocietyEuropean CommissionEuropean Commission

DefinitionsDefinitions

The evaluation processThe evaluation process

Evaluator responsibilitiesEvaluator responsibilities

Summary Summary

Proposals can Proposals can onlyonly be submitted in response to publicly be submitted in response to publicly announced calls for proposalsannounced calls for proposals

Always detailed at http://www.cordis.lu/istAlways detailed at http://www.cordis.lu/ist

All proposals * are presented by consortia of at least three All proposals * are presented by consortia of at least three mutually-independent organisations from different Member mutually-independent organisations from different Member or Associated Statesor Associated States

We offer a number of partner search facilities, run conferences and We offer a number of partner search facilities, run conferences and proposers´days, launch calls for expressions of interest, to assist you to proposers´days, launch calls for expressions of interest, to assist you to find partners for your proposalfind partners for your proposal

Register with your National Contact Point, who handles partner search Register with your National Contact Point, who handles partner search enquires from other countriesenquires from other countries

* except Specific support actions* except Specific support actions

Basic principles of FP6Basic principles of FP6

Proposals are submitted electronically via the InternetProposals are submitted electronically via the Internet Online or offline preparationOnline or offline preparation But always online submissionBut always online submission

Proposals are evaluated by independent expertsProposals are evaluated by independent experts Pre-defined evaluation criteria are used, scored 1-5Pre-defined evaluation criteria are used, scored 1-5

All criteria have threshold scores which must be reachedAll criteria have threshold scores which must be reached

All proposers receive an Evaluation Summary ReportAll proposers receive an Evaluation Summary Report

Funding follows successful evaluation, selection and detailed Funding follows successful evaluation, selection and detailed contract negotiationscontract negotiations

Basic principles of FP6Basic principles of FP6

PanelPanel – A group of evaluators responsible for the evaluation of a subset of A group of evaluators responsible for the evaluation of a subset of

proposals within the Strategic objectiveproposals within the Strategic objective

Panel coordinatorPanel coordinator – The Commission official in charge of a panelThe Commission official in charge of a panel

The subset of proposals in the Panel may be defined:The subset of proposals in the Panel may be defined:

as a particular instrument or instruments within the S.O.as a particular instrument or instruments within the S.O.

by a technical sub-category within the Strategic objective by a technical sub-category within the Strategic objective

by a combination of the twoby a combination of the two

Definitions - PanelsDefinitions - Panels

IST proposals are submitted in writing, in two parts:IST proposals are submitted in writing, in two parts:

Part A (Forms):Part A (Forms):

A1 - Proposal summary (one form per proposal)A1 - Proposal summary (one form per proposal)

A2 - Details of participants (one form per participant)A2 - Details of participants (one form per participant)

A3 - Financial breakdown (one form per proposal)A3 - Financial breakdown (one form per proposal)

Part BPart B

A text document (with supporting tables), written to a pre-A text document (with supporting tables), written to a pre-determined structure, describing the proposed projectdetermined structure, describing the proposed project

The structure of Part B varies per type of instrumentThe structure of Part B varies per type of instrument

Definitions - Proposal partsDefinitions - Proposal parts

Each proposal is evaluated on a number of pre-determined Each proposal is evaluated on a number of pre-determined criteria, which differ according to instrument typecriteria, which differ according to instrument type

Each criterion is scored 0-5Each criterion is scored 0-5

Each criterion has a “threshold” - a score which a proposal Each criterion has a “threshold” - a score which a proposal must reach to be considered for fundingmust reach to be considered for funding

A total score is also calculated for each proposal, by simple A total score is also calculated for each proposal, by simple addition of its five/six criterion scores (no weighting scheme)addition of its five/six criterion scores (no weighting scheme)

A threshold also applies to the total score. (And it is higher A threshold also applies to the total score. (And it is higher than the sum of the individual thresholds !)than the sum of the individual thresholds !)

These criteria, and their threshold scores, are detailed on the These criteria, and their threshold scores, are detailed on the forms used by the evaluatorsforms used by the evaluators

Definitions - Evaluation criteriaDefinitions - Evaluation criteria

Evaluation criteriaEvaluation criteria IP NOE STREP CA SSAIP NOE STREP CA SSA

RelevanceRelevance XX XX XX XX XX

Potential impactPotential impact XX XX XX XX XX

Scientific & tech excellenceScientific & tech excellence XX XXDegree of integration & the JPADegree of integration & the JPA XXQuality of the coordinationQuality of the coordination XXQuality of the support actionQuality of the support action XX

Quality of the consortiumQuality of the consortium XX XX XXExcellence of participantsExcellence of participants XX

Quality of managementQuality of management XX XX XX XXOrganisation and managementOrganisation and management XX

Mobilisation of resourcesMobilisation of resources XX XX XX XX

Roles and definitions - Evaluation CriteriaRoles and definitions - Evaluation Criteria

Criteria are scored 0-5Criteria are scored 0-5

5 = Excellent5 = Excellent

4 = Very good4 = Very good

3 = Good3 = Good

2 = Fair2 = Fair

1 = Poor1 = Poor

0 = 0 = The proposal fails to address the issue under The proposal fails to address the issue under examination or cannot be judged against the criterion due to examination or cannot be judged against the criterion due to missing or incomplete informationmissing or incomplete information

Half marks may be given if necessaryHalf marks may be given if necessary

Definitions - ScoresDefinitions - Scores

Evaluators bear in mind “horizontal issues”Evaluators bear in mind “horizontal issues” – gender issues related to the subject of the proposalgender issues related to the subject of the proposal– ethical and safety aspectsethical and safety aspects– synergies with education at all levelssynergies with education at all levels

““Third country” participation - comment on its significance to Third country” participation - comment on its significance to the project and the degree of integrationthe project and the degree of integration

– Other European states (ERA)Other European states (ERA)– Developing countries (INCO target list)Developing countries (INCO target list)– Other developed countriesOther developed countries

These horizontal issues are not scored, only commented. These horizontal issues are not scored, only commented.

Definitions - Horizontal IssuesDefinitions - Horizontal Issues

Independent external observers monitor evaluations, to Independent external observers monitor evaluations, to ensure they are properly carried out and to make ensure they are properly carried out and to make recommendations for future evaluationsrecommendations for future evaluations

ObserversObservers::wear identifying badgeswear identifying badgeshave full access to proposals, information and meetingshave full access to proposals, information and meetingsask any questions ask any questions are bound by the same confidentiality rules as evaluatorsare bound by the same confidentiality rules as evaluatorsreport their conclusions and recommendations to the Director report their conclusions and recommendations to the Director

General and the IST CommitteeGeneral and the IST Committeedo not influence the evaluation result of any proposal !do not influence the evaluation result of any proposal !

Roles and definitions - ObserversRoles and definitions - Observers

On arrival at the CommissionOn arrival at the Commission

Proposal data (the content of the Part A forms) is entered in Proposal data (the content of the Part A forms) is entered in the database, and all the pages (Part A and Part B) are the database, and all the pages (Part A and Part B) are scannedscanned

The assignment of the proposal to a Strategic objective is The assignment of the proposal to a Strategic objective is confirmedconfirmed

If necessary, a cross-objective evaluation is planned If necessary, a cross-objective evaluation is planned

Evaluation Process - Proposal assignmentEvaluation Process - Proposal assignment

The Strategic objective coordinator checks that each of the The Strategic objective coordinator checks that each of the proposals assigned to his/her SO ....proposals assigned to his/her SO ....

– is complete with a Part A and a Part Bis complete with a Part A and a Part B– arrived before the deadlinearrived before the deadline– is in scope for the callis in scope for the call– is composed of an eligible consortiumis composed of an eligible consortium

Completeness pre-checkCompleteness pre-check– If If allall of Part A or of Part A or allall of Part B are missing, the proposal will not go to of Part B are missing, the proposal will not go to

evaluationevaluation– If only If only somesome information is missing, the proposal will go to information is missing, the proposal will go to

evaluation. If the evaluators find they cannot score a particular evaluation. If the evaluators find they cannot score a particular criterion because of missing or incomplete information, they should criterion because of missing or incomplete information, they should score 0 on that criterionscore 0 on that criterion

Evaluation Process - Eligibility checkEvaluation Process - Eligibility check

Arrival before deadlineArrival before deadline– Proposals arrived after the deadline will not go to evaluationProposals arrived after the deadline will not go to evaluation– In cases of uncertainty, the proposal will go to evaluation. If it is subsequently found to In cases of uncertainty, the proposal will go to evaluation. If it is subsequently found to

have arrived late, its evaluation result will be declared null and voidhave arrived late, its evaluation result will be declared null and void

Proposal in scope of callProposal in scope of call– A proposal clearly out of scope of the call will not go to evaluationA proposal clearly out of scope of the call will not go to evaluation– In cases of uncertainty, the proposal will go to evaluation. If the evaluators find the In cases of uncertainty, the proposal will go to evaluation. If the evaluators find the

centre of gravity of a proposal is in fact not within the call, they should score 1 on the centre of gravity of a proposal is in fact not within the call, they should score 1 on the criterion “Relevance”criterion “Relevance”

Consortium compositionConsortium composition– Proposals not meeting eligible consortium requirements do not go to evaluation Proposals not meeting eligible consortium requirements do not go to evaluation

IP, NOE, STREP, CAIP, NOE, STREP, CA – At least three independent legal entities:At least three independent legal entities:

• two established in different Member/Candidate Statestwo established in different Member/Candidate States

• one established in another Member/Candidate States or in an Associated State or their Overseas territoriesone established in another Member/Candidate States or in an Associated State or their Overseas territories

SSASSA

No specific requirementNo specific requirement

Evaluation Process - Eligibility check (2) Evaluation Process - Eligibility check (2)

Evaluators are assigned to each proposalEvaluators are assigned to each proposal– The Panel coordinator assigns proposals to evaluators based on the The Panel coordinator assigns proposals to evaluators based on the

evaluator’s skills and experience, and avoiding conflict of interestevaluator’s skills and experience, and avoiding conflict of interest

– Three evaluators are planned to read each STREP, CA or SSA Three evaluators are planned to read each STREP, CA or SSA proposal, five evaluators to read each IP or NOE proposalproposal, five evaluators to read each IP or NOE proposal

The evaluators first carry out “individual” readingsThe evaluators first carry out “individual” readings– He/she evaluates the proposal individually, without discussion with He/she evaluates the proposal individually, without discussion with

the other evaluators who are also reading itthe other evaluators who are also reading it

– He/she completes an Individual Assessment Report (IAR form) He/she completes an Individual Assessment Report (IAR form) giving scores and comments on all criteriagiving scores and comments on all criteria

Evaluation Process - Individual readingEvaluation Process - Individual reading

Form differs according to instrument typeForm differs according to instrument type

Form lists each of the five/six criteria applicable to the instrumentForm lists each of the five/six criteria applicable to the instrument For each criterion it also gives supporting “sub-criteria”, which interpret For each criterion it also gives supporting “sub-criteria”, which interpret

the criterion but which are not themselves markedthe criterion but which are not themselves marked Contains a box for overall remarks and total score (calculated by the Contains a box for overall remarks and total score (calculated by the

arithmetic sum of all the criteria scores)arithmetic sum of all the criteria scores)

For IPs and NOEs, it Includes a box for questions to be asked at a For IPs and NOEs, it Includes a box for questions to be asked at a possible proposal hearingpossible proposal hearing

Includes a box for comment on the “horizontal issues” if relevant Includes a box for comment on the “horizontal issues” if relevant (gender, ethics, links to education….)(gender, ethics, links to education….)

Contains a flag box for “Ethical issues”Contains a flag box for “Ethical issues”

Evaluation Process - Individual Assessment ReportEvaluation Process - Individual Assessment Report

After all the individual readings are complete for a particular After all the individual readings are complete for a particular proposal, the evaluators who read that proposal meet in a proposal, the evaluators who read that proposal meet in a “Consensus group”, to agree scores and comments on all criteria “Consensus group”, to agree scores and comments on all criteria for the proposalfor the proposal

– The group consists of the five evaluators (IPs and NOEs) or three The group consists of the five evaluators (IPs and NOEs) or three evaluators (STREPs, CAs or SSAs), with a Commission Moderatorevaluators (STREPs, CAs or SSAs), with a Commission Moderator

– The evaluators continue in discussion until a consensus is reached, The evaluators continue in discussion until a consensus is reached, i.e. a decision to which all agreei.e. a decision to which all agree

– In case of deadlock the Moderator may bring in extra evaluators, or In case of deadlock the Moderator may bring in extra evaluators, or may finally accept a majority viewmay finally accept a majority view

– The Commission Moderator chairs the discussion, but he doesn’t The Commission Moderator chairs the discussion, but he doesn’t contribute his opinion to it !contribute his opinion to it !

– The group completes the Consensus Report (CR) and Consensus The group completes the Consensus Report (CR) and Consensus Meeting Minute (CMM) formsMeeting Minute (CMM) forms

Evaluation Process - Consensus groupEvaluation Process - Consensus group

Consensus Report (CR form)Consensus Report (CR form)– Form differs according to instrument typeForm differs according to instrument type– Form lists each of the five/six criteria applicable to the instrument, Form lists each of the five/six criteria applicable to the instrument,

without “sub-criteria”, plus space for overall remarks and total scorewithout “sub-criteria”, plus space for overall remarks and total score– (For IPs and NOEs) Includes a box for questions to be asked at a (For IPs and NOEs) Includes a box for questions to be asked at a

proposal hearing - To be completed if the proposal is above all proposal hearing - To be completed if the proposal is above all thresholdsthresholds

– Contains a flag box for “Ethical issues”Contains a flag box for “Ethical issues”– Can contain comment on horizontal issues under appropriate criteriaCan contain comment on horizontal issues under appropriate criteria– A hard copy of the form is signed by all the members of the A hard copy of the form is signed by all the members of the

Consensus group and the Commission ModeratorConsensus group and the Commission Moderator– If the “Ethical issues” box is flagged by even a single expert in the If the “Ethical issues” box is flagged by even a single expert in the

Consensus group, an Ethical Issues Report (EIR form) is completedConsensus group, an Ethical Issues Report (EIR form) is completed

Evaluation Process - Consensus groupEvaluation Process - Consensus group

When the Consensus group discussions are completed for all When the Consensus group discussions are completed for all the proposals in a panel, all the evaluators assemble in a the proposals in a panel, all the evaluators assemble in a meeting with two objectivesmeeting with two objectives

finalize the evaluation of the proposals and finalize the evaluation of the proposals and write the Panel reportwrite the Panel report

– The meeting is chaired by the Panel coordinator, and a Panel The meeting is chaired by the Panel coordinator, and a Panel recorder is appointed* to assist with report writing recorder is appointed* to assist with report writing

– The panel reviews each of the Consensus reports, so that the The panel reviews each of the Consensus reports, so that the knowledge and experience of all is applied to each proposalknowledge and experience of all is applied to each proposal

– The panel may have a new opinion on the scores and comments for The panel may have a new opinion on the scores and comments for any proposalany proposal

– The original CR forms are not changed, the new opinion is reflected The original CR forms are not changed, the new opinion is reflected in the Evaluation Summary Report of the proposalin the Evaluation Summary Report of the proposal

* either an evaluator on the panel or a specialist recorder* either an evaluator on the panel or a specialist recorder

Evaluation Process - Panel meetingEvaluation Process - Panel meeting

For Old Instruments (STREP, CA and SSA) For Old Instruments (STREP, CA and SSA) – the ESR is now final the ESR is now final

For New Instruments (IPs and NOEs) the scores agreed in the For New Instruments (IPs and NOEs) the scores agreed in the panel discussion determine the next steppanel discussion determine the next step

– If one or more of the scores is below threshold, the ESR is now finalIf one or more of the scores is below threshold, the ESR is now final

– If the scores are above threshold, the ESR is draft. The proposers If the scores are above threshold, the ESR is draft. The proposers will be invited to a hearing on their proposalwill be invited to a hearing on their proposal

– After the hearing, a final ESR will be agreed for these proposalsAfter the hearing, a final ESR will be agreed for these proposals

– Evaluators involved in these hearings will be briefed in more detail Evaluators involved in these hearings will be briefed in more detail on the hearing procedure at that timeon the hearing procedure at that time

Evaluation Process - Panel meetingEvaluation Process - Panel meeting

The ESR sent to the proposal coordinator will follow certain The ESR sent to the proposal coordinator will follow certain content guidelinescontent guidelines

– If the proposal passed all thresholds, the ESR shows scores and If the proposal passed all thresholds, the ESR shows scores and comments on all criteria, and a total score. The overall comment comments on all criteria, and a total score. The overall comment must include any recommendations for negotiation must include any recommendations for negotiation

– If the proposal failed one or more thresholds, the ESR nevertheless If the proposal failed one or more thresholds, the ESR nevertheless shows scores and comments on all criteria, and a total score. But shows scores and comments on all criteria, and a total score. But the overall comment only identifies the failed threshold(s)the overall comment only identifies the failed threshold(s)

– If the proposal was ineligible, the Commission will prepare an ESR If the proposal was ineligible, the Commission will prepare an ESR showing no scores, only an overall comment identifying the reason showing no scores, only an overall comment identifying the reason for ineligibilityfor ineligibility

Evaluation Process - Sending of ESREvaluation Process - Sending of ESR

After completing its ESRs, the panel prepares a Panel reportAfter completing its ESRs, the panel prepares a Panel report

It first “prioritises” all its above-threshold proposals…..It first “prioritises” all its above-threshold proposals…..– The Commission prepares lists* of all the above-threshold proposals, The Commission prepares lists* of all the above-threshold proposals,

ordered by overall scoreordered by overall score– The panel considers any proposals with tied scores, and imposes a The panel considers any proposals with tied scores, and imposes a

priority between thempriority between them

then it writes a detailed reportthen it writes a detailed report– Under the supervision of the Panel Coordinator and Panel recorder, Under the supervision of the Panel Coordinator and Panel recorder,

the panel writes its final Panel report in a predetermined formatthe panel writes its final Panel report in a predetermined format– This Panel report contains the prioritised lists, with supporting This Panel report contains the prioritised lists, with supporting

commentscomments

*different instruments are listed separately*different instruments are listed separately

Evaluation Process - Panel reportEvaluation Process - Panel report

The Strategic objective coordinator consolidates the Panel The Strategic objective coordinator consolidates the Panel reports into a reports into a Strategic objective reportStrategic objective report, including all the , including all the ESRs in annex ESRs in annex

The Strategic objective reports are brought together by The Strategic objective reports are brought together by Commission staff into an overall Commission staff into an overall Evaluation reportEvaluation report, with , with supporting statistics and summary of the procedure, for supporting statistics and summary of the procedure, for consultation with the IST Committee and other Commission consultation with the IST Committee and other Commission services.services.

Based on the opinions of the evaluators and the results of Based on the opinions of the evaluators and the results of the consultation an the consultation an Implementation planImplementation plan is agreed is agreed

Proposals selected for implementation commence contract Proposals selected for implementation commence contract negotiationsnegotiations

After the evaluationAfter the evaluation

Commission officials are present to Commission officials are present to – ensure that the evaluation is carried out fairly and according to the ensure that the evaluation is carried out fairly and according to the

official procedure, official procedure,

– all voices are heard and that the evaluators’ conclusions are all voices are heard and that the evaluators’ conclusions are accurately recordedaccurately recorded

Commission officials do not attempt to influence the results Commission officials do not attempt to influence the results of the evaluation of any proposalof the evaluation of any proposal

Even if asked they may not express any opinion on the Even if asked they may not express any opinion on the merits or otherwise of any proposal, or of any proposermerits or otherwise of any proposal, or of any proposer

Commission responsibilitiesCommission responsibilities

Evaluator respect the rules regarding conflicts of interestEvaluator respect the rules regarding conflicts of interest

– Evaluators should declare their known conflicts of interest before the Evaluators should declare their known conflicts of interest before the start of the evaluationstart of the evaluation

– If evaluators spot a new conflict of interest during the evaluation (for If evaluators spot a new conflict of interest during the evaluation (for example during the reading of a proposal), they declare this to the example during the reading of a proposal), they declare this to the Panel coordinator immediately. The Panel Co-ordinator takes the Panel coordinator immediately. The Panel Co-ordinator takes the appropriate actionappropriate action

– An evaluator with conflicts of interests will not participate in the An evaluator with conflicts of interests will not participate in the evaluation of that or competing proposalsevaluation of that or competing proposals

Evaluator responsibilities Evaluator responsibilities

EvaluatorsEvaluators give a fair and clear opinion on each proposal give a fair and clear opinion on each proposal and a prioritisation among themand a prioritisation among them

Evaluators are: Evaluators are: – IndependentIndependent (evaluators represent their selves, not their employer, (evaluators represent their selves, not their employer,

not their country…..)not their country…..)

– AccurateAccurate (use the official evaluation criteria only) (use the official evaluation criteria only)

– ConsistentConsistent (apply the same standard of judgement to all proposals (apply the same standard of judgement to all proposals))

– IncommunicadoIncommunicado (external contacts on evaluation are not permitted (external contacts on evaluation are not permitted during or after during or after the evaluationthe evaluation))

– AnonymousAnonymous

– Individual roles or opinions of evaluators are NEVER revealed Not Individual roles or opinions of evaluators are NEVER revealed Not by other evaluators, not by the Commission by other evaluators, not by the Commission

Evaluator responsibilities (2)Evaluator responsibilities (2)

Fatal mistakesFatal mistakes Incomplete proposal (Must have a Part A and Part B)Incomplete proposal (Must have a Part A and Part B) Ineligible consortiumIneligible consortium Out of scope of call (activity Out of scope of call (activity oror instrument) instrument) Focusing on a purely national issue, or one with no benefit to the EUFocusing on a purely national issue, or one with no benefit to the EU Late submissionLate submission

HandicapsHandicaps Wrong instrument chosenWrong instrument chosen Not conforming to required Part B contents, lengthNot conforming to required Part B contents, length Unfocused project, diverse aimsUnfocused project, diverse aims Lack of scientific and technological excellenceLack of scientific and technological excellence

Proposal evaluation in ISTProposal evaluation in IST