Guideline: Research Proposals Evaluation · Office of the Vice-President for Research and Community Services Guideline: ... proposal based on the technical evaluation criteria (Table

Mekelle University

Office of the Vice-President for Research and Community Services

Guideline:

Research Proposals Evaluation

April 2016

Mekelle

1

IntroductionMekelle University researchers have been undertaking researches at different scales using funds

sourced from University's research recurrent budget and other external fund sources. Selection of

proposals has through times varied according to specific conditions and requirements of funds.

However, since varying criteria for evaluation has been a reason for repeated complaints and

grievances among different colleges and institutes, the office of VPRCS has decided to develop

permanent criteria (with provisions for specific timely modifications). The evaluation criteria are

developed based on concepts and principles stipulated in the Mekelle University Research and

Development Policy document. These principles include objectivity, enhanced impact,

ethicality, servitude to the public etc.

Generally, research projects in Mekelle University fall within three different categories namely:

large scale projects, Medium Scale Projects, Small Scale Projects (which include Young

and Female Researcher Funds). These categories are developed based on currently existing

research needs and staff involvement. Therefore, in the future, if a new level of categorization

arises, new criteria, based on the principles of the research and policy guidelines might be

developed. Separate guidelines are developed for each of the three categories. The need for

having separate guidelines for large-scale, young and female proposals is because the different

categories have different objectives and require different these have different objectives (stated

in the document), we need to have different guidelines that accommodate the differences in

objectives. For instance while affirmative action for involvement of women researchers can and

should be a measurable criteria for large scale, we cannot make it a criteria for the female-

researcher fund.

Moreover, any given guideline is made to have enough provisions for modifying according to

newly emerging issues. For instance, while we can use the GTP-II issues to serve as a guide for

next five years, we cannot have a constant thematic area for all calls in the future, . Therefore,

the thematic area criteria is constant, but specifics will depend on the thematic area of

importance in the time. Thematic areas might also vary with different funders, objectives of calls

etc. Therefore, guidelines, while constant interns of the principles of the criteria, will be open for

modification.

2

Generally, the criteria are developed based on the following concepts or principles:

1. Team of experts based evaluation: Evaluation is made by a team of multi-disciplinary

professions selected and agreed by colleges and institutes who participate in proposal

selection.

2. Automatic exclusion criteria : Proposals that do not fulfill certain criteria are

automatically excluded. These exclusion criteria, which are different for the different

categories are clearly listed

3. Public defense: Shortlisted proposals (number depends on availability of fund) will be

invited to public defense

4. Provision of feedback of comments to applicants Evaluation committee will provide

generic comments to all proposals

Objectives

These set of guidelines is prepared for the following purposes

1. To minimize bias or subjectivity in the evaluation of research proposals, enhancing

effective competition among applicants

2. Ensuring that high quality research proposals that promise the highest possible quality

and quantity of outputs are funded

3. Ensuring that proposals relevant to thematic area and priorities of a certain call are

selected

Working definition of Project types

1. Large-scale projects

Large scale research projects are those multi-disciplinary and inter-disciplinary research projects

with a budget requirement minimum 250,000 Birr and that will take at least three years to

complete. The objectives of large-scale projects is to bring usable outputs or solutions for

specified real life problems, while enhancing the research experience of junior and senior

3

researchers. Thematic areas for large-scale projects shall be defined by the fund source. Large

scale projects are evaluated and administered at the University (center level)

2. Medium scale research projects

Medium scale research projects are those multi-disciplinary and inter-disciplinary research

projects with a budget minimum of 150,000- Ethiopian Birr and that will take up to two years to

complete. The objectives of medium scale projects is to developed the research experience of

junior researchers in solving identified problems that require urgent solutions. Moreover, this

research category is aimed at developing the experience of junior researchers in publishing in

high impact factor journals. Evaluation procedure for medium scale projects will be in the same

way as to the large-scale projects.

3. Small-Scale research Projects

Small-scale research projects are research projects that require minimum of 60,00 Ethiopian Birr

and that will take maximum of one year to complete. The objective of small-scale research

projects is to provide young researchers (academic rank below assistant professor) with

opportunities to practice and develop their research skills while working on a real life problem.

Small-scale research fund are aimed young researchers to start writing papers, make scientific

presentations and generally join the research community. It shall consist of a minimum of three

members. Teams are encouraged to fulfill the senior-junior composition for sharing and learning

experiences. Applications for small scale project fund are evaluated at college level (both for

technical evaluation and presentations), with the supervision of members from the Directorate

office for Research and Graduate Studies. Small-scale research project fund shall be

administered at department level.

Small-scale research projects can be of three types.

1. General small scale projects- These are small scale projects where the applicant

(Principal Investigator) can be any academic staff member. Senior staff (assistant

professor and above) are not encouraged to compete for these funds as they are expected

to draw funds from external sources. However, they can apply as co-PI only to work with

MSc students.

4

2. Young researcher small scale projects- these are small scale projects where the

applicant (PI) can be any academic member with an academic rank below assistant

professor. The aim of this project is to encourage young researchers who show promise in

improving their research experience and publication profile.

3. Female researcher- These are small scale projects where the applicant (PI) is a female

staff member with an academic rank of below assistant professor. The female researcher

projects are aimed at alleviating the very low participation of young female researchers in

research and publication, thereby empowering female researchers. Moreover, this fund

aims to provide young female researchers with opportunities to practice and develop

their research skills while working on a real life problem.

5

1. Large Scale Projects

1. 1. Submission Procedures All proposals should be prepared based on the Mekelle University proposal submission format

and should be prepared as follows. Proposals shall not exceed twenty pages and shall include the

following as annexes

a) CV's of all proponents (maximum of three pages per person). CV's should clearly show

the relevant publications, research and project management experiences of members.

b) A one page description of the roles and responsibilities of each of the proposed co-

applicants.

c) A one page description of the proposed post graduate student research.

1.2.ConstitutionofteamsLarge scale proposals should include at least four team members from at least three inter-

college/institute professionals, including two seniors (Assistant professor and above) and two

young or junior researchers (below assistant professors) and at least two post graduate students

who will be attached to the project for their postgraduate research. .

1.3.EvaluationcommitteeThe office of the Director for Research and Postgraduate Studies will invite relevant colleges/

institutes representing the following broad categories to assign technical experts for constituting

an evaluation team. The following procedures will be followed in the constitution of evaluation

team

1. Team members will be constituted by professionals at a level of assistant professor

and above and shall represent the following six broad areas of specializations

a. College of Social Sciences and Languages (CSSL)

b. College of Law and Governance (CLG)

c. College of Business and Economics (CBE)

d. College of Health Sciences (CHS)

e. College of Veterinary Medicine (CVM)

f. College of Dryland Agriculture and Natural Resources (CoDANR)

6

g. College of Natural and Computational Sciences (CNCS)

h. Ethiopian Institute of Technology - Mekelle (EIT-M)

i. Mekelle Institute of Technology

2. The evaluation team shall work together to evaluate each submitted proposals.

Relevant professionals might give explanations, about technicalities of a proposal, to

those not directly versed in the subject matter of a proposal in question. Finally each

and every member of the evaluation committed will give independent marks to each

of the proposals.

3. The evaluation committee shall be chaired by a member of the Research and Post-

Graduate Directorate (RGD) Office. Another member from the RGD office shall act

as a secretary. Both the chair and the secretary shall also give independent marks to

proposals.

4. Final marks for each proposal will be averages of total marks given by each of the

above mentioned nine members both for technical evaluation (Table 1) and public

defence evaluation (Table 2).

1.4.Evaluationprocedureandcriteria

1.4.1.AutomaticexclusioncriteriaProposals will first be screened according the following criteria. Those proposals with the

following limitations shall be excluded from further evaluation

1. Proposals that do not fulfil the formatting and submission requirement mentioned in

section 1.

2. Proposals that fail to fulfil the minimum technical feasibility requirement

3. Proposals whose PI's or member have been funded by any of the Mekelle University

research fund schemes (two years ago, the latest), but fail to show publications (from

these funds) in a reputable peer reviewed journal. Applicants should attach the

publication mentioning the registration number and fund from which such a

publication was produced

7

1.4.2.SpecificevaluationcriteriaThe nine members of the evaluation committee shall give independent marks to each submitted

proposal based on the technical evaluation criteria (Table 1). Proposals from each thematic area

will then be ranked according to their average marks given by each member of the evaluation

committee. Certain number of top ranked proposals (number shall depend on availability of

fund) will be invited for public defence.

During public defence (which will take 15 minutes presentation and ten minutes question and

answers), the evaluation committee will mark each proposal according to the public defence

evaluation criteria given on Table 1.2. Only the PI can do the public presentation. There will be

mandatory questions from the evaluation committee and other members of the audience may also

ask questions.

Finally, the results from the technical evaluation (Table 1.1) and those from the public defence

(Table 1.2) shall be converted into 70% and 30% of the final mark respectively. Certain number

of top ranked proposals (the number shall depend on availability of funds) will then be selected

for funding. Whatever number of proposals is possible to fund, according to the available budget,

any proposal that does not get a minimum 50% of the overall evaluation shall not be considered

for funding.

Evaluation committee will prepare important feedback (generic) to both accepted and rejected

proponents which will be directly communicated directly to the PI's.

Table 1.1. Detailed evaluation criteria and their relative weights

1

Technical feasibility and soundness of Rationale/conceptual framework,

objectives and methods * 20

Originality 8

Objectives and Rationale 5

Methods 7

2

Alignment and relevance to a specific thematic area (will depend on the

year, call etc) 5

8

3

Contribution and relevance to achieving climate resilient green economy,

reducing vulnerability to climate change, and contribution to climate

change adaptation and mitigation 4

4

Contributions and relevance for achieving good governance and social

equity 4

5

Contribution and relevance to improving food security for rural and urban

communities 4

6

Contributions, relevance and opportunities for involving and benefiting

civil society organizations, Industry and the public 4

7

Contribution and relevance towards enhancing the use of emerging

technologies (Bio-technology, ICT, nano-technology) 5

8

Prospects and opportunities for producing marketable output and fostering

entrepreneurship and commercialization 4

9

Prospects and contributions towards generation of new knowledge,

technology, ideas and build new capacity (attitude, skill knowledge) 5

10

Technical (subject matter), project management and publication experience

of PI and co-PI's (to be evaluated from submitted CV's) 8

Technical (subject matter) 2

Project management skills and experiences 3

Publications (numbers relative to opportunities, relevance of publications) 3

11

Prospects and opportunities for improving urban and rural employment

specially to disadvantages communities (landless youth, women,

pastoralists minorities etc), equitable development 4

12

Prospects and plan for producing scientific publications in high impact

factor peer reviewed international journals, policy and media briefs 5

13

Prospects and opportunities for involving graduate and undergraduate

student researchers 3

14

Opportunities and plans for involving post and undergraduate student

researchers 5

15 Opportunities and plans for using and upgrading Mekelle University's 5

9

research infrastructure and equipment

16 Language, formatting and presentation quality of the proposal 5

17 Matching of technical, financial and time plans 5

18

Affirmative action for involvement of female researchers (5 for >50%, 3

for 30%-50%, 2 for <30%, and 0 for 0%) 5%

Total (100%) 100

* Proposals that fail to fulfil the minimum 15% technical feasibility will be automatically

excluded from further evaluation

Table1.2. Evaluation criteria for public defense

SN Criteria Mark (%)

1 Explanation 10%

Articulatin of the problem 5

Articulation and explanation of proposed solution 5

2 Effort and devotion 15%

Confidence 5

Reflected commitment 5

Reflected effort in reviewing all related knowldege 5

3 Technical competence 15%

Grasp of technical concepts and ideas 5

Understanding local dimensions 5

Response to questions 5

4 Presentation 10%

Presentation style 3Time management 2

language 5

10

2.Medium‐ScaleProjects

2.1.SubmissionProceduresAll proposals should be prepared based on the Mekelle University proposal submission format


following as annexes. Submissions will be made on the University e-research portal and will be

approved by the College Research and Community Services Council.

a) CV's of all proponents (maximum of three pages per person). CV's should clearly show

the relevant publications, research and project management experiences of members.

b) A one page description of the roles and responsibilities of each of the proposed co-

applicants.

c) A one page description of the proposed post graduate student research-

2.2.ConstitutionofteamsMedium scale proposals should include at least four team members from at least three inter-

college/institute professionals, including two seniors (Assistant professor and above) and two

young or junior researchers (below assistant professors) and at least two post graduate students

who will be attached to the project for their postgraduate research.

2.3.Evaluationcommitteethe evaluation of medium scale projects will be done in the same manner as that of large-scle

projects




Proposals will first be screened according the following criteria. Those proposals with the



section 1.


11

3. Proposals whose PI's have been funded by any of the Mekelle University research

fund schemes (two years ago, the latest), but fail to show publications (from these

funds) in a reputable peer reviewed journal. Applicants should attach the publication

mentioning the registration number and fund from which such a publication was

produced

2.4.3SpecificevaluationcriteriaThe members of the evaluation committee, chaired by the college/institute CRCS head will give

independent marks to each of the proposals according to criteria in Table 2.1. Finally shortlisted

proposals with technical evaluation marks above 50% will be invited to a college level public

defence.

During the public defence, only PI's shall present a 20 minute presentation of their proposals

which will be followed by ten minute question and answer session. During the public defence,

every member of the evaluation committee shall give independent marks based on criteria listed

in Table 2.2

Finally, the results from the technical evaluation (Table 2.1) and those from the public defence

(Table 2.2) shall be converted into 70% and 30% of the final mark respectively. Certain top

ranked proposals (number will depend on the amount of fund) be selected for funding. Final

evaluation results of each of the proposals, shall be sent in an official letter from the office of

CRCS to the Office of Research and Post Graduate Director. Minutes of the evaluation

committee shall be attached to this letter.

Finally, based on results obtained from all colleges and institutes, the office of Research and

Postgraduate Directorate, shall decided on a list of proposals to be funded.

Table 2.1. Detailed evaluation criteria and their relative weights

1



Originality 8


Methods 7

12

2 Alignment and relevance to a thematic area specified in a call 15

3

Relevance to any of the national policies for instance, currently to GTP-II

(climate change adaptation and mitigation, enhancement of use of

emerging technologies and achievement of good governance) 5

4


communities 5

5



6

Prospects and contributions towards generation of new knowledge,

technology, ideas and build new capacity (attitude, skill knowledge) 10

7

Technical (subject matter), project management and publication experience

of PI and co-PI's (to be evaluated from submitted CV's) 8

Technical (subject matter) 2

Project management skills and experiences 3

Publications (numbers relative to opportunities, relevance of publications) 3

8

Prospects and plan for producing scientific publications in high impact

factor peer reviewed international journals, policy and media briefs 5

9

Opportunities and plans for involving post and undergraduate student

researchers 5

10

Opportunities and plans for using and upgrading Mekelle University's

research infrastructure and equipment 5



13

Affirmative action for involvement of female researchers (5 for >50%, 3

for 30%-50%, 2 for <30%, and 0 for 0%) 5

Total (100%) 100



Table 2.2. Evaluation criteria for public defense


13

1 Explanation 10%

Articulatin of the problem 5



Confidence 5


Reflected effort in reviewing all related knowldege 5





4 Presentation 10%


language 5

14

3.Small‐ScaleProjects

3.1.SubmissionProceduresAll proposals should be prepared based on the Mekelle University proposal submission format


following as annexes

a) A one page description of the roles and responsibilities of each of the proposed co-

applicants, including part of the project objectives which will be dealt by the postgraduate

student.

3.2.ConstitutionofteamsThe small-scale researches shall be of three types namely 1). General small scale-where the

applicant can be any academic staff, 2) Young researcher small-scale fund- is reserved for young

academic staff (below assistant professor) and 3) Female researcher small-scale fund- which is

reserved for only young female researchers (below assistant professors). The required team

composition for each proposal types is as follows

1. General small scale- one PI, with at least two co-PI's (one of whom a senior or junior

mentor) and an additional one postgraduate student

2. Young researcher small scale- a young PI (below assistant professor) with at least two

co-PI's (one of whom a senior or junior mentor) and an additional one postgraduate

student

3. Female researcher small scale- a young female PI (below assistant professor) with at least

two co-PI's (one of whom a senior or junior mentor) and an additional one postgraduate

student. Except for the junior or senior mentor, all the co-PI's and the postgraduate

students should be females.

15

3.3.Evaluationcommittee

3.3.1.EvaluationcommitteeforGeneralsmall‐scaleprojectsThe college/ Institute Research and Community Services Council heads, shall appoint one

evaluator (Assit. Prof. and above) from each of the departments. The team of evaluators chaired

by the college/institute CRCS head, shall work together to evaluate each submitted proposals.

Relevant professionals might give explanations, about technicalities of a proposal, to those not

directly versed in the subject matter of a proposal in question

3.3.2.Evaluationcommitteeforyoungresearchersmall‐scaleprojectsThese are similar to the general small scale projects in terms of the evaluation criteria, however,

since this specific program is aimed at supporting young researchers who are showing promise in

their research endeavors, there will no be quotas to each college, rather applications will be

evaluated at center level using the procedures followed for large-scale project evaluation

(Section 1.3)

3.2.3.Evaluationcommitteeforfemaleresearchersmall‐scaleprojectsThese are similar to the young researcher small sclae projects, but only focused on supporting

young female researchers who are showing promise in their research endeavors. The evaluation

of these projects will be similar to the procedures used for large-scale projects (Section 1.3)





section 1.


3. Proposals whose PI's have been funded by any of the Mekelle University research

fund schemes (two years ago, the latest), but fail to show publications (from these

funds) in a reputable peer reviewed journal. Applicants should attach the publication

16

mentioning the registration number and fund from which such a publication was

produced

3.4.Specificevaluationcriteria

3.4.1.Generalsmall‐ScaleandYoungresearchersmallscaleprojectsThe members of the evaluation committee, chaired by the will give independent marks to each of

the proposals according to criteria in Table 5. Finally shortlisted proposals with technical

evaluation marks above 50% will be invited to a college level public defence.

During the public defence, only PI's shall present a 20 minute presentation of their proposals

which will be followed by ten minute question and answer session. During the public defence,

every member of the evaluation committee shall give independent marks based on criteria listed

in Table 6

Finally, the results from the technical evaluation and those from the public defence shall be

converted into 70% and 30% of the final mark respectively.

The number of proposals to be funded from each college in this research category will depend on

the amount of research budget available for each college or institute. Therefore, the evaluation

committee will decide on how many of the top ranked proposals should be funded.

Final list of funded proposals, shall be sent in an official letter from the office of CRCS to the

Office of Research and Post Graduate Director. Minutes of the evaluation committee shall be

attached to this letter.

Table 3.1. Detailed evaluation criteria for general small scale and young researcher small scale

project proposals

Criteria Mark%

1



Originality 5


17

Methods 10


3


communities 5

4

Contribution and relevance to any of the national GTP-II issues (climate

change, emerging technologies and good-governance) 5

5



6

Prospects and contributions towards building new capacity (attitude, skill

knowledge) e.g. the use of a software, tool, equipment etc 15

7

Prospects and plan for producing scientific publications, conference

proceedings and other forms of publications 10

8 Opportunities and plans for involving ndergraduate student researchers 5

9





12

Affirmative action for involvement of female researchers (5 for >50%, 3 for

30%-50%, 2 for <30%, and 0 for 0%) 5

Total (100%) 100



Table 3.2. Public defence evaluation criteria


1 Explanation 10%

Articulation of the problem 5



18

Confidence 5


Reflected effort in reviewing all related knowledge 5





4 Presentation 10%


language 5

3.4.2.Specificevaluationcriteriaforfemaleresearcherssmall‐scaleproposalsSince the objectives and the targeted groups of this scheme are different from the above

mentioned categories, there will be different set of specific criteria that will be used for

evaluation of female researcher small-scale criteria. These criteria are given in Table 3.3. for

technical evaluation and Table 3.4 for public defense.

Table 3.3. Detailed technical evaluation criteria for female researchers small-scale proposals

Criteria Mark%

1



Originality 5


Methods 10


3


communities 5

4

Contribution and relevance to any of the national GTP-II issues (climate

change, emerging technologies and good-governance) 5

19

5



6

Prospects and contributions towards building new capacity (attitude, skill

knowledge) e.g. the use of a software, tool, equipment etc 15

7

Prospects and plan for producing scientific publications, conference

proceedings and other forms of publications 10

8 Opportunities and plans for involving undergraduate student researchers 5

9





12 Prospectus and opportunities for gender empowerment 5

Total (100%) 100



Table 3.4. . Evaluation criteria for public defense


1 Explanation 10%

Articulation of the problem 5



Confidence 5


Reflected effort in reviewing all related knowledge 5





4 Presentation 10%


20

language 5

Documents

Guideline: Research Proposals Evaluation · Office of the Vice-President for Research and Community Services Guideline: ... proposal based on the technical evaluation criteria (Table