A Novel Evaluation System of Public Research Institute in Japan A Novel Evaluation System of Public Research Institute in Japan International Workshop

A Novel Evaluation System A Novel Evaluation System of Public Research Instituteof Public Research Institute

in Japan in Japan

International Workshop on the Evaluation of Publicly Funded Research

Berlin, 26-27 September 2005

Naoto KobayashiNational Institute of Advanced Industrial Science and

Technology (AIST)

CONTENTSCONTENTS

1.1.Missions and Principles of AISTMissions and Principles of AIST

2.2.Evaluation Policy in AISTEvaluation Policy in AIST

3.3.Novel Evaluation System from the Novel Evaluation System from the Viewpoint of OutcomeViewpoint of Outcome

4.4.Evaluation in the Future for the Evaluation in the Future for the Publicly Funded Research Publicly Funded Research InstitutesInstitutes

3

Interdisciplinary Researches to enable to plan long-range governmental policies attributing to the current and future needs of society

Basic Researchesto develop and maintain high standards of scientific and engineering research

Advanced Researchesto explore broad spectra of

research fields and to integrate multidisciplinary

subjects

Standard & Measurements

Type-II Basic Research

　 & Full Research

　 The mission of AIST is to generate innovative technologies for the new industries and to contribute to the economic development of our country and to the lives of people and , thus, finally to the sustainable global world!!

Missions of AIST: for all mankind and global consciousness to our future

Missions of AIST Missions of AIST in association with the Ministry of in association with the Ministry of Economy, Economy,

Trade and Industry (METI)Trade and Industry (METI)

Environment

Energy

Resources

Geology

Life ScienceInformation

Material

MachineNanotechnology

Manufacturing

CommunicationElectronics

Oceanography

4

AuditorAuditor

FellowFellow

Advisory BoardAdvisory Board

Technology Information Dept.Technology Information Dept.

Collaboration Dept.Collaboration Dept.

Intellectual Property Dept.Intellectual Property Dept.

International Affairs Dept.International Affairs Dept.

General Administration Dept.General Administration Dept.

Human Resource Dept.Human Resource Dept.

Financial Affairs Dept.Financial Affairs Dept.

Research Facilities Dept.Research Facilities Dept.

〔 Sites 〕　 AIST Hokkaido　 AIST Tohoku　 AIST Tsukuba　 AIST Tokyo 　 AIST Chubu　 AIST Kansai　 AIST Chugoku　 AIST Shikoku　 AIST Kyushu


AIST Innovations(TLO)


PresidentSenior Vice-President

Vice-President


Vice-President

Evaluation Dept.Evaluation Dept.

Safety & Environmental Protection Dept.


Tsukuba Advanced Computing CenterTsukuba Advanced Computing Center

International Patent Organism DepositoryInternational Patent Organism Depository

PlanningHeadquarter

PlanningHeadquarter

General AdministrationHeadquarter


Research Center (28)

Research Institute (21)

Auditors’ OfficeAuditors’ Office

Research Coordinator


EmeritusAdvisor

EmeritusAdvisor

EmeritusCouncilor

EmeritusCouncilor

AIST Innovation Center for Start-upsAIST Innovation Center for Start-ups

Public Relations Dept.Public Relations Dept.

Res

earc

h ad

min

istr

atio

n de

part

men

tsR

esea

rch

adm

inis

trat

ion

depa

rtm

ents

Res

earc

h su

ppor

t de

part

men

tsR

esea

rch

supp

ort

depa

rtm

ents

Research Initiative (5)

Geo-information CenterGeo-information Center

Metrology Management CenterMetrology Management Center June, 2005

Researchers: ~2,500Administrative staff:

~720Budget: ~900M$

5

AIST Chugoku

AIST Shikoku

AIST Kansai

AIST Chubu

AIST TohokuAIST Hokkaido

AIST Kyushu

AIST Tsukuba

産総研

Tokyo Headquarters

AIST Tokyo Waterfront

old AIST 15 Research Institutes 8 Institutes in Tsukuba 7 Institutes in other regions

new AIST Merged into 1 Institute 9 Research Basis

Research Bases of AISTResearch Bases of AIST

In 2001

6

　 Concept of Full Concept of Full Research in AISTResearch in AIST 　　

AIST defines Type-II Basic Research, which integrates multiple disciplines and creates methods for the use of integrated knowledge; traditional basic research is defined as Type-I Basic Research, which is the pursuit and discovery of novel laws, principles and theories that govern natural phenomena. AIST places its highest priority on the pursuit of complete research, "Full Research", ranging from Type-I Basic Research to the development of “products” by conducting intensive Type-II Basic Research.

Type-I Basic Research Type-II Basic Research

Contribution to the Society

Coherent and concurrent research based on scenario

Full Research

Unknown

PhenomenaUnknown

PhenomenaEconomic

& Social

Needs

Economic

& Social

Needs

Law

Integration of Knowledge

Finding of New Knowledge

Application

Development

Principle

Theory

Theorem

President Dr. YoshikawaDr. Yoshikawa

7

Sustainable Development by Full Research

AISTAIST SocietySocietyPublic resources

Outcomes

Full Research

People’sInterest(applause)

dream nightmare reality

Type-1 basic research

Type-2 basic

research

Industrial developme

nt

AIST

Universities Industries

Innovation!!

8

R&D Evaluation Guideline (1997, 2001, and 2005 ）：：・・ appropriate and fair evaluation→ flexible, competitive, and open research environment appropriate and fair evaluation→ flexible, competitive, and open research environment ・・ encouraging evaluationencouraging evaluation →→ original and promising R&D, researchers with original and promising R&D, researchers with great ambitions andgreat ambitions and abilities abilities , and significant policy and program, and significant policy and program ・・ announcement of results of evaluation→ accountability to the nationannouncement of results of evaluation→ accountability to the nation ・・ strict evaluationstrict evaluation→ concentrated and effective allocation of resources→ concentrated and effective allocation of resources

⇒ ⇒ internationally high level of R&D which contributes to society and economy, and to development of new discipline

R&D Evaluation Guideline (1997, 2001, and 2005 ）：：・・ appropriate and fair evaluation→ flexible, competitive, and open research environment appropriate and fair evaluation→ flexible, competitive, and open research environment ・・ encouraging evaluationencouraging evaluation →→ original and promising R&D, researchers with original and promising R&D, researchers with great ambitions andgreat ambitions and abilities abilities , and significant policy and program, and significant policy and program ・・ announcement of results of evaluation→ accountability to the nationannouncement of results of evaluation→ accountability to the nation ・・ strict evaluationstrict evaluation→ concentrated and effective allocation of resources→ concentrated and effective allocation of resources

⇒ ⇒ internationally high level of R&D which contributes to society and economy, and to development of new discipline

Council for Science and Technology PolicyCouncil for Science and Technology Policy

Independent Administrative Institution ActIndependent Administrative Institution Act 20012001Independent Administrative Institution ActIndependent Administrative Institution Act 20012001

・・ setting Evaluation Department setting Evaluation Department ・・ annual evaluationannual evaluation 　　　　　　 20012001 ～～20042004

・・ setting Evaluation Department setting Evaluation Department ・・ annual evaluationannual evaluation 　　　　　　 20012001 ～～20042004

Introduction of evaluation system in AISTIntroduction of evaluation system in AIST

　How to evaluate R&D in AISTHow to evaluate R&D in AISTas an Independent Administrative Institutionas an Independent Administrative Institutionpromoting R&Dpromoting R&D　・　・ auto-evaluationauto-evaluation　　・・ evaluation interval　・ outcome 　　　　　　　　　　

　How to evaluate R&D in AISTHow to evaluate R&D in AISTas an Independent Administrative Institutionas an Independent Administrative Institutionpromoting R&Dpromoting R&D　・　・ auto-evaluationauto-evaluation　　・・ evaluation interval　・ outcome 　　　　　　　　　　

AIST Evaluation System Advisory Committee 20032003 ～～ 55AIST Evaluation System Advisory Committee 20032003 ～～ 55

Evaluation from the viewpoint of Outcome

Evaluation from the viewpoint of Outcome

9

Medium-Term Plan

Medium-Term Target Annual Report

Evaluation Department

Evaluation of Each Unit’s

Research Units (Center, Institute and Initiative)

Advisory Board

Com

men

t

Comment for Management

Evaluation of Activity

Results

of

Evalu

ation

Evaluation Committee for IAI

in METIReport

PresidentReflection on Reflection on Future Future ManagementManagement

Evaluation Committees(Peers, Specialists or Intellectuals,

Internal Reviewers)

Res

earc

h R

esou

rces

Evaluation System of AISTEvaluation System of AIST

10

　

To evaluate research activities is to encourage researchers with great ambitions and abilities and to assist research organizations which contribute to the development of economy and the quality of our lives.

The top management of AIST requires the recognition of check and balance function in the evaluation of research activities, to consider the allocation of resources for research and to be equal to the occasion such as the reorganization.

AIST is also responsible for the accountability for public resources by showing performances and achievements to the society.

To evaluate research activities is to encourage researchers with great ambitions and abilities and to assist research organizations which contribute to the development of economy and the quality of our lives.

The top management of AIST requires the recognition of check and balance function in the evaluation of research activities, to consider the allocation of resources for research and to be equal to the occasion such as the reorganization.

AIST is also responsible for the accountability for public resources by showing performances and achievements to the society.

Significance of Evaluationof Evaluation

11

Toward Outcome

Every a few years

Based on Output

Every year

The 1st Term (FY2001-2004)

The 2nd Term (FY2005-2009)

　 Activation and Increasing in Efficiency　 Making sure the research strategy　 Reference to the Managements

Purpose:

Novel Evaluation System of Research Units


・ Reconsideration of roadmap・ Output Indexes to outcomes・ Management procedure

Roadmap towardoutcomes

Research UnitsResearch Units

Startup Interim Summarized RI and RC

FollowingRC

Evaluation IntervalsEvaluation Intervals

Change

12

Transfer

Startups

New Industry

New Market

Market ShareSampleNovel Idea Output

Research in AIST

Companies 、 New StartupsProduction ・ Marketing

Pattern of Outcome Creation(Product Type)

Acceptance in Market & Society

Market

Outcome

Solar Panel

Liquid Crystal Display

Collaboration/ Technology Transfer(Prototype ・ Improvement ）

13

DB Construction

New Industry

New Market

Research in AIST

　 System Development 　　　 System Improvement ・ Data Accumulation 　　　

Acceptance in Domestic and International Society

α 　 Version ， β Version Supply

DB Open to the Public

Extension of External Access

System Adjust and Implementation inStartups and/or Companies

System Adjust and Implementation in AIST

Collaborative Work

Pattern of Outcome Creation(Database Type)

Transfer

Market ShareMarket

Outcome

SampleNovel Idea Output

Acceptance in Market & Society

Startups

Organic Compounds

Spectrum DB

14

①① ：： Evaluation of Realized OutcomesEvaluation of Realized Outcomes②② ：： Evaluation of Scenario/RoadmapEvaluation of Scenario/Roadmap③③ ：： Evaluation of Outputs to contribute to Evaluation of Outputs to contribute to OutcomesOutcomes④④ ：： Evaluation of Management to contribute to Evaluation of Management to contribute to OutcomesOutcomes

①① ：： Evaluation of Realized OutcomesEvaluation of Realized Outcomes②② ：： Evaluation of Scenario/RoadmapEvaluation of Scenario/Roadmap③③ ：： Evaluation of Outputs to contribute to Evaluation of Outputs to contribute to OutcomesOutcomes④④ ：： Evaluation of Management to contribute to Evaluation of Management to contribute to OutcomesOutcomes

Outputs in the Outputs in the PastPast

Outputs in the Outputs in the PastPast

Expected Outcomes in the

Future

Expected Outcomes in the

Future

Present ResearchPresent ResearchPastPast FutureFuture

① ① ：： Realized Realized OutcomesOutcomes① ① ：： Realized Realized OutcomesOutcomes

②② ：： RoadmapsRoadmaps③③ ：： OutputsOutputs④④ ：：ManagementManagement

②② ：： RoadmapsRoadmaps③③ ：： OutputsOutputs④④ ：：ManagementManagement

Evaluation is performed through Items Evaluation is performed through Items ②② ～～ ④④

Evaluation from the Viewpoint of the Evaluation from the Viewpoint of the OutcomeOutcome

15

Evaluationfrom the viewpoint of outcome

to reflect the future

Evaluationfrom the viewpoint of outcome

to reflect the future

Element of Evaluation to evaluate the process of R&D

Roadmap

Output

Management

16

external reviewers:external reviewers:Specialists or IntellectualsSpecialists or Intellectuals


＊＊ Roadmap EvaluationRoadmap Evaluation ＊＊ Output EvaluationOutput Evaluation ＊＊ Management EvaluationManagement Evaluation


　　　　 Evaluation: Rating with commentsEvaluation: Rating with comments　　　　 Evaluation: Rating with commentsEvaluation: Rating with comments

Evaluation DepartmentEvaluation DepartmentEvaluation DepartmentEvaluation Department

Strategy of AISTStrategy of AISTStrategy of AISTStrategy of AIST

President of AISTPresident of AISTPresident of AISTPresident of AIST

Research UnitsResearch UnitsResearch UnitsResearch Units

internal reviewers:internal reviewers:Principal ReviewersPrincipal Reviewers


Procedure of EvaluationProcedure of Evaluation

NationNation Evaluation CommitteeEvaluation CommitteeResource Allo

cation

Reorganization

Comments

＊＊ Roadmap for outcomesRoadmap for outcomes ＊＊ Output indexesOutput indexes ＊＊ ManagementManagement　　


Roadmap EvaluationRoadmap Evaluation

17

Evaluation of Roadmapto make sure the research

strategy

Evaluation of Roadmapto make sure the research

strategyEvaluation Index To know

OutcomeSocial and economical

effects

Milestone Progress plan

Technical element Core technique

Benchmark Comparison with others

18

Example of Roadmap for OutcomesExample of Roadmap for Outcomes:　 Ultra-Low-Loss power devices 　Example of Roadmap for OutcomesExample of Roadmap for Outcomes:　 Ultra-Low-Loss power devices 　

Subjects

(Targets)

（（ Power Electronics Research CenterPower Electronics Research Center ））

2002 2002 　　　　　　 20042004 　　　　　　 2006 2006 　　　　　　 20082008 　　　　　　　　　　 20102010 　　　　　　　　 20122012 　　　　　　　　　　　　 20202020

Substrate / Wafer

Epitaxy

Processing

PackagingFET DevicesModules

ApplicationsInstrumentsInstrumentsInstrumentsInstruments

Few 10μm/h

3 inch Wafer

Lamp BallastsLamp Ballasts Factory

Motor drive

Factory

Motor drive

Electric Power Distribution SystemElectric Power Distribution System

Dispersed

Power supply

Dispersed

Power supply Power Transmission

System

Power Transmission

System

General Inverter

General Inverter

（ＦＥＴ）（ＦＥＴ）（ＩＧＢ（ＩＧＢＴ）Ｔ）

Technology transferTechnology transferTechnology transferTechnology transfer600V10A600V10A600V100A600V100A 2.5kV100A2.5kV100A

Low-EMI

Bipolar DevicesBipolar Devices

The 1st Term The 2nd Term

R&D in AISTR&D in AISTR&D in AISTR&D in AISTlow-Ron

4 inch Wafer 6 inch Wafer

Air-Con.Air-Con.

Multi-Wafer-Epi.

Self-AlignmentBipolar-Tech.

High-reliable SiO2 Low-Temp Process

T T jj 　　>250℃>250℃ Wire-less

Heatproof Capacitor-less

19














cation

Reorganization

Comments



Output EvaluationOutput Evaluation

20

Evaluation of Output Indexes

to know the level of research unit

Evaluation of Output Indexes

to know the level of research unitPapers Press releases

Patents Prizes

StandardsPrototypal products

Activities at academic party

etc.

21














cation

Reorganization

Comments



Management EvaluationManagement Evaluation

22

Evaluation of ManagementEvaluation of Management

Leadership Project formation

Budget Intellectual property

Human resource Technology transfer

Collaboration Start-ups etc.

Concept of Full Research : Policy statement Concept of Full Research : Policy statement

23














cation

Reorganization

Comments



Nation Evaluation Committee

24

Toward Outcome

Every a few Years

Based on Output

Every year

The 1st Term (FY2001-2004)

The 2nd Term (FY2005-2009)

　 Activation and Increasing in Efficiency　 Making sure the research strategy　 Reference to the Managements

Purpose:



・ Reconsideration of roadmap・ Output Indexes to outcomes・ Management procedure

Roadmap towardoutcomes

Research UnitsResearch Units

Startup Interim Summarized RI and RC

FollowingRC

Evaluation IntervalsEvaluation Intervals

Change

25

Mid-term Objectives by the minister (e.g. METI) Mid-term Objectives

by the minister (e.g. METI)

Mid-term Plans by each Institute Mid-term Plans by each Institute

Check of Performances & AchievementsCheck of Performances & Achievements

Evaluation Committee in each ministry Evaluation Committee in each ministry

Evaluation Committee in Ministry of

Internal Affairs and CommunicationsEvaluation Committee in Ministry of

Internal Affairs and Communications

3-5 years

26

AuditorAuditor

FellowFellow

Advisory BoardAdvisory Board

Technology Information Dept.Technology Information Dept.

Collaboration Dept.Collaboration Dept.

Intellectual Property Dept.Intellectual Property Dept.

International Affairs Dept.International Affairs Dept.

General Administration Dept.General Administration Dept.

Human Resource Dept.Human Resource Dept.

Financial Affairs Dept.Financial Affairs Dept.

Research Facilities Dept.Research Facilities Dept.






Vice-President


Vice-President

Evaluation Dept.Evaluation Dept.



Tsukuba Advanced Computing CenterTsukuba Advanced Computing Center

International Patent Organism DepositoryInternational Patent Organism Depository

PlanningHeadquarter

PlanningHeadquarter



Research Center (28)

Research Institute (21)

Auditors’ OfficeAuditors’ Office



EmeritusAdvisor

EmeritusAdvisor

EmeritusCouncilor

EmeritusCouncilor

AIST Innovation Center for Start-upsAIST Innovation Center for Start-ups

Public Relations Dept.Public Relations Dept.

Res

earc

h ad

min

istr

atio

n de

part

men

tsR

esea

rch

adm

inis

trat

ion

depa

rtm

ents

Res

earc

h su

ppor

t de

part

men

tsR

esea

rch

supp

ort

depa

rtm

ents

Research Initiative (5)

Geo-information CenterGeo-information Center

Metrology Management CenterMetrology Management Center June, 2005

Researchers: ~2,500Administrative staff:

~720Budget: ~900M$

Improvement in services

Increase in efficiency

Activation of operation

・ Setting of “Proposal Box” for accepting opinions to improve administrative work

・ List-up and sorting of the importance of work・ Re-arrangement of the staffs so as to maximize the total efficiency of the work

・ Quantitative target-setup・ Preparation of “Work Flow Manual”

Aim: Support for research activities of AISTMore effective execution of administrative and supportive work

Aim: Support for research activities of AISTMore effective execution of administrative and supportive work

Evaluation of Research Administration Departments

and Research Support Departments

Evaluation of Research Administration Departments

and Research Support Departments

28

Strategy

Implementation Evaluation

PlanDo

See

EvaluationPlanning

Management

ResearchInput Output

Report

AdvicePlan

Action

Do

Check

Strategy of EvaluationStrategy of Evaluation

Evaluation of R&D Policy

Proposal

Evaluation of R&D in the Research Institute

What is Innovation?What is Innovation?

◆Adequate Evolution of the Economy, Society 　 and Environment based on Novel Science & Technology 　　　　　　　

The trend to intensify the interface of Science and Industrial Innovation is prevailing at universities and public research industries in the world.Especially it is necessary to make breakthroughs by technological fusion in the interdisciplinary fields (ICT, Nanotechnology, Biotechnology etc.).

　 R&D Industrial Society

Innovation!!Industrial Transformation is necessary toward the Sustainabl

e Global Society.

Innovation on the Basis of Novel R&D is necessary for the Industrial Transformation.

Adequate Evaluation System of R&D which contributing to the Innovation is important, especially for the Publicly Funded Research Institutes.

Evaluation from the Viewpoint of the Outcome is one of the attractive tools for the novel evaluation.

Conclusion: Evaluation for InnovationConclusion: Evaluation for Innovation

31

• Evaluation is for Encouraging with Love.• Evaluation is for Communication.• Evaluation is for Strategy.• Evaluation is for Creation.• Evaluation is to be fed back.• Evaluation reflects the Future.

Notes:

• Evaluation is like a Compass for Voyage.

32

Thank you !

33

Appendix

Definition of independent administrative institution

Definition of independent administrative institution

【 Performance 】- to be highly public- need not to be performed by government directly- hard to be achieved by private organization independently

① effective and with high quality ② independent ③ open

【 Performance 】- to be highly public- need not to be performed by government directly- hard to be achieved by private organization independently

① effective and with high quality ② independent ③ open

35

① To perform effectively and qualitatively

① To perform effectively and qualitatively

- to set mid-term objectives and to receive periodical evaluation by third

party- to entrust some service to private organization or to abolish institutions- to manage as a private corporation- to organize minimum size

- to set mid-term objectives and to receive periodical evaluation by third

party- to entrust some service to private organization or to abolish institutions- to manage as a private corporation- to organize minimum size

36

② To perform independently ② To perform independently

- to concentrate authority to president - to exclude the excess involvement of minister- to have an appropriate personnel including executives from private corporation

- to concentrate authority to president - to exclude the excess involvement of minister- to have an appropriate personnel including executives from private corporation

③ To perform openly ③ To perform openly

- to open all performances and achievements to the public - to open all performances and achievements to the public

37

Evaluation CommitteeEvaluation Committee

Specialistse.g. Professor, or Head of other IAI

Intellectualse.g. Head of enterprise, or Journalist

【 outside of AIST 】

【 inside of AIST 】Principal Reviewers

etc.

38

Evaluation Interval

in the 2nd Research Phase

Evaluation Interval

in the 2nd Research Phase

研究ラボ・研究センターの創出・時限的

Research Initiative

・ to be Research Center

・ time limit A new Research Center

Middle Evaluation

Advise for Roadmap

研究部門・技術ポテンシャルの発展・継続的

Research Institution・ bottom up・ no time limit Advise for

Roadmap

Terminal EvaluationMiddle Evaluation

研究センター・産総研の看板・時限的

Research Center・ top down・ time limit Advise for

Roadmap

Terminal EvaluationMiddle Evaluation Middle EvaluationStart up

39

～ Organizational shift in accordance with society needs at the times ～

　　　　　　R

eorganization by Unification

　

　

Merging research fields and Administrative needs

19801960 1970

Industry Type Research Areas

Concept for Reorganization

Catching up advanced nations

Aiming atFront Runner

Aiming at COE

Recovery from World War II

Machinery

Microorganic Industry

Fiber & Polymer

Electrical Electronics

Production Science

2 Regional Institutes

Mining ResourcesResources & Pollution

Weighing & Testing National Research Laboratory of Metrology

7 Regional Institutes

Nat’l Inst.for Advanced Interdisciplinary Research

Mechanical Engineering Laboratory

Geological Survey of Japan

Fermentation

Alcohol

Textile

Industrial Arts

Geological Survey

Chemical Technology

Tokyo National Research Laboratory

1903

1900

1882

1891

National Institute of Materials and Chemical Research

Electrotechnical Laboratory

National Institute for Resources and Environment

7 Regional National Industrial Research Institutes

Weights and Measures Training Institute

1948

Establishment of Former AIST

National Institute of Bioscience and Human-Technology

1990 2000

Na

tion

al In

stitu

te o

f A

dv

anc

ed

Ind

us

trial S

cie

nc

e an

d T

ech

no

log

yHistory of AISTHistory of AIST

Documents

A Novel Evaluation System of Public Research Institute in Japan A Novel Evaluation System of Public Research Institute in Japan International Workshop