Evaluation of the National Natural Science Foundation of China

Erik ArnoldAmerican Evaluation Association

Anaheim, CA3 November 2011

2

Some milestones

• Cultural Revolution: S&T system effectively destroyed• 1978 National Science Conference: “modernisation of

S&T is the key of the four modernisations”; legitimacy but little money

• 1985-92 reforming the science system• CAS agency 1982-6; NSFC; marketisation of technology and

business; decentralisation of strategy to research performers; reform of HR system to introduce merit-based pay

• 1986, NSFC established, based on DFG/NSF model• 1992-8 integrating S&T to the economy, developing

industry, universities and institutes• 1998-2005 building a national innovation system

• Especially splitting off the industrially-focused parts of CAS, supporting SMEs, increased attention to human resource development and basic research

3

Medium- and Long Term Plan for National Science and Technology Development, 2006-2020• Goal: to make China an innovation-driven economy by

2020• High-priority clusters

• Technologies for water, energy and environmental protection• IT, advanced materials and manufacturing• Biotechnologies and their applications• Space and marine technology• Basic sciences and frontier technology - Raise basic research

to 15% of GERD by 2020• 16 mission-driven megaprojects

4

Eight thrusts

• A boost for investment in R&D• Tax incentives for investment in STI• Government procurement policy to promote innovation• Innovation based on assimilating imported advanced

technology• Capacity-building in generating and protecting IPRs,

standards• Building national infrastructure and platforms for STI• Cultivate and utilise talents for STI• Support endogenous innovation via financial measures

5

Key role of NSFC in basic research – the only bottom-up funder

1999

2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

02,0004,0006,0008,000

10,00012,00014,00016,00018,00020,000 National Major Science Engineer-

ing Construction Project

Knowledge Innovation Program

973 Program

985 Program

211 Program

Special Research Fund for the Doctoral Program

State Key Laboratory Program

NSFC Fund

Million RMB

6

NSFC main tasks

1. The combination of curiosity- and demand- driven research as a ‘dual driving force’. The first NSFC General Assembly stressed that its basic research funding was aimed to support economic development

2. Promoting the balanced, coordinated and sustainable development of academic disciplines in China

3. Emphasis on fostering talents 4. Facilitating international exchange and cooperation

in basic research

7

Evaluation objectives

1. Provide an independent assessment of the overall performance of NSFC’s funding and management during the past 25 years, with a truly global perspective

2. Present key findings, lessons learned and recommendations to improve the NSFC’s funding and management performance as well as to achieve excellence in management

3. Develop a set of forward-looking guiding ideas, based on an international perspective, supporting NSFC’s strategic role within the NIS of China

8

Method

• Review by an International Evaluation Committee• Academic chair• Thirteen researchers, including one evaluation professional

• Interviews with NSFC and stakeholders• Extensive background report prepared by the National

Centre for Science and Technology Evaluation (NCSTE)• Bibliometrics• Surveys• Interviews/focus groups• Document review

9

International Evaluation Committee• Prof Richard N ZARE (Chair),

Stanford University, Chemistry• Prof HAN Qide (Vice Chair), Vice

Chairman, National People’s Congress, Medicine

• Prof Ernst-Ludwig WINNACKER (Vice Chair), Human Frontier Science Programme , Biochemistry

• Prof Erik ARNOLD (Rapporteur), Technopolis Group; University of Twente, Research and Innovation Policy

• Prof LU Yonglong (Rapporteur), Research Center for Eco-Environmental Sciences, Chinese Academy of Sciences, Environmental Science and Management

• Prof XUE Lan (Rapporteur), Tsinghua University, Science and Technology Policy and Management

• Prof Akito ARIMA, Chairperson, Japan Science Foundation, Nuclear Physics

• Dr Richard A ANTHES, University Corporation for Atmospheric Research, Atmospheric Science

• Prof Anthony K CHEETHAM, University of Cambridge, Materials Science

• Prof MA Zhiming, Academy of Mathematics and Systems Science, Chinese Academy of Science, Mathematics

• Prof Andrew (FA) SMITH, University of Adelaide, Agriculture, Food and Wine

• Prof Jeannette M WING, Carnegie Mellon University, Computer and Information Sciences

• Prof XU Zhihong, Peking University, Life Sciences

10

Stupendous growth in GERD (RMB billions)

1990

1991

1992

1993

1994

1995

1996

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

0

100

200

300

400

500

600

Development

Applied Research

Basic Research

11

BERD = 73% of GERD, but there is a low proportion of basic research in GERD

Japan 07 Korea 08 USA 07 China 08 Canada 080%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%Other

Development

Applied

Basic

12

Building infrastructure, low labour cost component

China 08 Japan 08 Germany 08 Korea 08 France 080%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%Instruments, equipment

Land, buildings

Other current

Labour

13

Shifting the system towards research universities

14

A key innovation of NSFC was German-style peer review

15

Growth in projects and institutions funded across the research system

1986 1990 1995 2000 2005 20090

5,000

10,000

15,000

20,000

25,000Number of NSFC-funded institutions

Number of PI of NSFC-funded projects

16

NSFC moved from general to more focused instruments

1986 1991 1996 2001 2006 20090%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100% OtherInternational (Regional) Cooper-ation and Exchange fundNational Science Fund for Talent Training in Basic ScienceJoint Research Fund for Overseas Chinese (HK and Macao) ScholarsNational DYS fundScience Fund for Creative Research GroupScience fund for Less Developed RegionYoung Scientists FundJoint FundMajor Research PlanMajor ProgramKey ProgramGeneral Program

17

Age distribution of Principal Investigators

≦25 26-30

31-35

36-40

41-45

46-50

51-55

56-60

61-65

66-70

≧710

5

10

15

20

25

30

1986-1990

1996-2000

2006-2009

Age

%

18

Discipline development

Type of discipline

NSFC as Main Funding Sourcein 1980s/1990s (No. of Cases)

NSFC as Main Funding SourceToday

(No. of cases)Sole Source Main Source One of

Multiple Sources

Sole Source Main Source One of Multiple Sources

Traditional or strong disciplines

6 40 1 1 9 37

Emerging disciplines and interdisciplinary research

4 32 4 0 5 35

Weak disciplines 1 18 0 0 6 13

Total 11 90 5 1 20 85

19

Women and ethnic minorities

Year Female Applicants Minority Applicants

Number of applications

Share in total

Number of projects funded

Share in total Number of

applicationsShare in

total

Number of projects funded

Share in total

2008 11776 23.9% 1880 21.1% 1410 2.7% 312 3.5%

2009 14485 25.2% 2231 22.2% 2195 3.8% 363 3.6%

2010 17061 26.2% 3002 23.0% 2431 3.7% 470 3.6%

20

Staff workload

2001 2002 2003 2004 2005 2006 2007 2008 20090

100

200

300

400

500

600

0

5,000

10,000

15,000

20,000

25,000

30,000

35,000Average number of applications managed per person

Average number of funded projects managed per per-son

Average amount of funding man-aged per person

1,000 RMB

21

Administrative efficiency is very highCountry Organisation Year Administrative Cost as

% of BudgetCanada CIHR 2009 6%

SSHRC 2009 3.3%

China NSFC 2009 2%

Denmark Basic Research Council 2009 4.4%

Germany DFG 2009 2.5%

Sweden Swedish Science Council 2009 9%

UK EPSRC 2009 3.5%

MRC 2009 3.5%

USA NSF 2009 3%

22

Publications in the WoS19

8619

8719

8819

8919

9019

9119

9219

9319

9419

9519

9619

9719

9819

9920

0020

0120

0220

0320

0420

0520

0620

0720

0820

09

0

50,000

100,000

150,000

200,000

250,000

300,000

350,000 USARussiaJapanIndiaGermanyFranceChinaBrazilUK

23

Relative impacts of publications relative to the World

1986

1987

1988

1989

1990

1991

1992

1993

1994

1995

1996

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

USA

Russia

Japan

India

Germany

France

China

Brazil

UK

World average

24

Copublications

Sweden

Netherlands

Russia

Italy

Taiwan

South Korea

Singapore

France

Australia

Canada

UK

Germany

Japan

USA

0 5000 10000 15000 20000 25000 30000 35000 40000

2004 to 20081999 to 2003

Source: Thomson-Reuters

25

Growth, copublications 1999/2003 to 2004/9

Italy

Russia

Germany

Japan

Taiwan

UK

France

Netherlands

USA

Sweden

Australia

Singapore

Canada

South Korea

0% 50% 100% 150% 200% 250% 300%

26

UK/China co-publications grow faster than publications

27

From the UK perspective, there’s a quality penalty to pay for cooperation (mean impact factors 2000-2005)

Source: RCUK

28

Surprisingly insular travel pattern

29

What the IEC said

• Increase the share of basic research in GERD• NSFC needs more staff and resources• Bigger grants, more calls for proposals• Strengthen panels: interdisciplinarity; involve more

foreigners• Ensure assessment is, and looks, confidential and

‘squeaky clean’• More international connections; International Advisory

Board• More flexible use of funding• More high-risk research

30

What the IEC didn’t talk about

• Outcomes and impacts of NSFC funding (!)• Discipline development• The rich tradition of consultation in programming• Linking bottom-up and top-down funding approaches

• Programme 1 versus Programme 2, in Swedish terminology• The systemic role of NSFC in developing the research

and innovation community• NSFC’s role in wider policy development and

implementation

31

Issues in the Chinese NIS

• Learning to use all the investments• Raising quality while growing the system• Doing novel research in a top-down culture• Squeezing out ‘influence’ – is this still an issue?• Increasing basic research• Science-industry links, absorptive capacity • Autarchy vs globalisation and learning from abroad

32

Thank you

technopolis |group| has offices in Amsterdam, Ankara, Brighton, Brussels, Frankfurt/Main, Paris, Stockholm, Tallinn and Vienna

http://www.technopolis-group.com/resources/downloads/reports/nsfc_evaluation_report.pdf