Proposal EvaluationPrasit Palittapongarnpim, M.D.Mahidol University

Proposal Is a description of R&D activities,

supposedly, in advance. Is a management tool for resources

allocation. Can be used as TOR between

researchers and sponsors. Hardly be accurate.

Sponsors’ purposes of evaluation Are the proposal goals fit to the funding goals? Is the methodology appropriate? Are the researchers capable to finish the

projects? Is the hosting institute ready and supportive

enough? Is the budget realistic? In some particular cases

cost-effectiveness may be considered? Other considerations

Ethics Safety Capacity building

Proposal Evaluation Methodology depends on The goals of the funding programs Funding strategy The number of applicants How many stages will proposals be

prepared? Levels of intellectual rights needed to be

protected

What are the goals of R&D MFU? Is MFU a Provincial University National University Regional University International University

Are the proposal goals fit to the funding goals and scope? What is the required output?

Proof of concepts Information Papers Patents Materials Products. Capacity building Policy translation (usually with papers) Industrial translation Proof of concepts Hypothesis formulations

What is the scope?

Possible Goal settings and Funding Strategies for various categories of R&D Descriptive information seeking (e.g., incidence,

prevalence): Top down/Commission Hypothesis-driven research (e.g., basic, target

discovery, etc.): Researcher initiatives/fully competitive

Candidate discovery (e.g., drug or vaccine candidates): Researcher initiatives/ scoped competition.

Product development (Preclinical and clinical): Major: Top down/commission with stage-gate

protocols Minor: Bottom up/competitive

Translation to policy(e.g., cost-effectiveness study, QOL): Top down/ Scoped competition

Translation to industry(e.g., Industrial technology assistance): Customer-based/commission and competition

มู�ลค่�าของ outputs จากโค่รงการเป็�นสิ่��งที่��น�กวิ�จ�ยค่าด แต่�มู�ป็�จจ�ยอ��นๆเก��ยวิข!องมูาก มู�กจะเก�ดข#$นต่�อเมู��อหน�วิยงานลงมู�อจ�ดการ

มู�ลค่�าแลกเป็ล��ยน (exchange value) เช่�น ขายสิ่�ที่ธิ�บั�ต่ร เก�ดธิ)รก�จ/รายได!

มู�ลค่�าใช่!สิ่อย (use value) เช่�น การให!บัร�การมู�ลค่�าสิ่�ญญะ (sign value) เช่�น เก�ดการ

ป็ระช่าสิ่�มูพั�นธิ.หน�วิยงาน

How many kinds of goals are needed in the proposals? Long term goals (visions) Outputs Operational objectives

In any cases it should be stated and clear Outputs and operational objectives

should be measurable and promised. Avoid overpromising

How are the proposals selected? Open competitive Scoped competitive

By areas of research By eligibility of applicants

Self-competitive Commission Concept- full -revised

Intellectual Rights in Proposal Contents. How much should be disclosed? How much can be protected during

submission? How confidential is the reviewing

processes?

Sponsors’ purposes of evaluation Are the proposal goals fit to the funding goals? Is the methodology appropriate? Are the researchers capable to finish the

projects? Is the hosting institute ready and supportive

enough? Is the budget realistic? In some particular cases

cost-effectiveness may be considered? Other considerations

Ethics Safety Capacity building

Evaluation of Methodology Flawed or unclear methodology is usually

amendable. Competitive granting may not allow that at the times of funding decisions.

Some (S&T) projects required un-established methodology, which require strong technical capacity of researchers.

Researchers may be more familiar with suboptimal technology. Decision is required to force researchers to better but unfamiliar methods or stick with the familiar method.

Methodology Evaluations Do we start from a right place? Is the

material good and most advanced? Literature review is NOT a part of project

methodology! Try to go beyond “me too”.

Is the pathway to the goal feasible and optimal?

“Optimal” may means Fastest Conclusive vs. Supportive vs. Suggestive Including risk management

Secrets in Methodology Secret recipe. Vague strategy

Common Pitfalls in Methodology Writing Wrong strategy/methods Inferior strategy/methods Vague strategy/methods Methods that researchers have not

enough experiences Too sketchy

Common Pitfalls in Methodology Evaluations MOST COMMON: Reviewers do not understand or

misunderstand the methods Diagrams usually help. Groups discussions are usually very helpful.

Reviewers have own methodology preferences Reviewers subsequently use it in a different

project but finish it before the applicant!? Should funder help improve the methods? YES. Is it fair? Depending on the goal and process of

funding. It is nor fair for competitive funding.

The purposes of evaluation Are the proposal goals fit to the funding

goals? Is the methodology appropriate? Are the researchers capable to finish the

projects? Is the hosting institute ready and

supportive enough? Budgets. Other considerations

Ethics Safety

Supporting Facility Space Equipment Special facility

Clinical cohorts Sample banking systems BSL3 Animal lab Pilot Production PD-PK facility Genomics facility High-performance computing

Research Budget Personnel Equipment Chemicals and consumables Travels Miscellaneous (+Contingency) Overhead/Utilities

Ethics Absolutisms, e.g.

Most major religions Relativisms, e.g.

Utilitarianism

Biosafety The purpose is to manage the risks at

the acceptable level, by the COMBINATION OF 3 methods Safe experimental protocols (Standard

operational protocols) Personal protection device Physical containment

All the textbooks are guidelines. Nothing can replace RISK-BENEFIT ANALYSIS.

Product Development A sequential multidisciplinary process, usually

need several institutes and good management. Have definite goals but can be of multiple

pathways Highly quality oriented (GLP (good laboratory

practice), GMP (..manufacturing..), GCP (..clinical..), GAP (..agricultural..))

Usually with strict experimental protocol with limited innovation.

Cannot be done very well in most university settings. Requiring good organization, discipline and infrastructure.

There are at least two ways of product development Backward engineering Evolutionary approach

Creating diversified candidates Selection

Figure 2 Positioning pharmacogenomics and pharmacogenetics in the drug discovery and development process.

Diagnostics Vaccines Therapeutics Health

promotives Medical/

Health Services

Biomedical/genomicsKnowledge

Genomics/High-throughput

Technology

Clinical/EpidemiologyKnowledge

Biomedical Informatics/ Robotics/

Nanotechnology

DevelopmentInfrastructure

ProductDevelopment

Suprastructure:Policy commitment/

Government Resource Allocation

International Infrastructure

International Input

HRD

Managing Product Developments

Product development needs Development team specialized at each step.

Foreign teams can be used in some steps. Infrastructures usually go with the development

teams. Product development manager/management team Sponsor/financers

Financing Product Development Product development are costly. Financer

is the game controller. Financer can be Private companies Government Institutes Philanthropic organizations International organizations

Product development are costly, one can do very few at a time. Which one would you do? How can you end up with the conclusion? Is it generally agreed?

Concept of Execution the goals : Focus on Wildly Important Goals :(WIGs)

“WIGs: The Goals must be achieved. Failure to achieve this goals renders any of the other achievements inconsequential.”

4 Disciplines of execution

Discipline 3Keep a Compelling Scoreboard

Discipline 1 Focus on the Wildly Important Goals (1-3 goals)

Discipline 2Action on the Lead measures

Discipline 4 Create a cadence of Accountability

The enemy of the best are the goods. There are many good products that can

be developed, but usually one can only do one.

The average cost of developing a drug is about $800,000,000 USD. The cost for clinical trials are at least a few to several dozen millions USD.

A poor clinical trial is not better than doing nothing, e.g., BCG vaccine.

When to stop: Exit Strategy Since we should devote all resources to

a few products, it is extremely important to stop the work immediately when appropriate or in the case of failure.

Tool: Stage-Gate Protocol. Development is divided into stages, each

which clear end-point (Gate Review Criteria).

Allowance to pass the gate is judged by “gate keeper”.

Stage and Gate

37

37

Discovery Stage 1 Stage 2 Stage 3 Stage 4 Stage 5

R&D

idea

idea

Gate 1

Gate 2

Gate 3

Gate 4

Gate 5

Testing LaunchBusiness plan

Feasibility

study

R&D FunnelStage gate

Technology Readiness Levels Provide a Common Language

http://trl.sandia.gov

TRL1 Basic Research “Experimental data revealing useful information about the basic principles observed”

TRL2 Applied Research “Model that explains the underlying science and how it could be applied to solve a particular application’s problem”

TRL3 Research Result “Experimental or analytical demonstration that shows that the key elements of an approach are likely to be feasible”

TRL4 Research Demonstration (Lab demo)“Experiment in a Laboratory”

TRL5 Research Prototype (Demo Unit) “Looks like a Product, Hand-built by PhDs, Breaks a Lot”

TRL6 Engineering Prototype (Alpha Unit)“Research Prototype that is Rugged and Repeatable”

TRL7 Flight / Field Prototype (Beta Unit) “Engineering Prototype that is Reliable and Manufacturable”

TRL8 WR / Hi-Rel (Production Unit)“Field Prototype that has cost “wrung out” (if applicable), and has completed qualification”

TRL9 Proven Product “Product that has been used successfully in a system before and is being adapted for use in a similar application.

Funding and Outputs of Product Development

Product development succeeds only with adequate Infrastructures (animal testing lab, pilot production,

quality assurance/regulation, etc.) Personnel (engineers, chemists, etc.) Funding (at least opportunity to get funding all the

ways of development) Material supplies (esp. the ones need to be imported

or without commercial suppliers) Usually need a full time manager.

Outputs are usually visible only when development finishes. But can be visible during development by Stage-Gate management processes (Meeting or not meeting Gate Review Criteria).

Take home message. Proposal writing is an integral part of

R&D process. Most Proposal fails to get funded Do not be discouraged.