1

Holistic and timely monitoring of a Japanese science and technology innovation

system through an annual panel survey of experts and researchers

Masatsura Igami (igami@nistep.go.jp)

National Institute of Science and Technology Policy,

Ministry of Education, Culture, Sports, Science, and Technology,

Kasumigaseki, Tokyo 100-0013, Japan

ABSTRACT

The National Institute of Science and Technology Policy (NISTEP) expert survey on Japanese

science, technology and innovation (STI) system is an annual panel survey administered to Japanese

experts and researchers at universities, public research institutions, and private firms. It intends to

track the status of STI in Japan through 57 questions related to Japanese STI system. The survey

provides a holistic view of the STI system in timely manner and qualitative information such as

diversity in basic research and usability of research funds, which is generally difficult to gauge based

on research and development statistics. Owing to the originality of the data, various governmental

councils and committees have referenced the survey’s results in their official documents, including

planning the fifth Science and Technology Basic Plan (STBP). This paper provides an overview of

the survey’s methodology and design, and discusses changes that have occurred in Japanese STI

during the fourth STBP’s implementation period between 2011–2015. Finally, directions for the

survey’s future development are discussed.

2

1. Introduction

Japan’s science and technology policies are promoted based on a Science and Technology Basic

Plan (STBP) that is formulated every five years. The fourth STBP (2011–2015) focused on the

period between FY2011 and FY2015, and served as the basis upon which a variety of policies was

implemented. The fifth STBP was launched in 2016. The effective implementation of the

plan-do-check-act cycle in science, technology, and innovation (STI) policies requires evidence

regarding how Japanese STI has changed over time.

Monitoring the status of STI is a challenging undertaking. First, its status does not change

overnight, thereby necessitating continuous monitoring in order to identify changes (i.e., continuity).

Second, assessing the national STI system requires a comprehensive point of view (i.e.,

comprehensive understanding). Third, there are often instances wherein quantitative data

documenting STI’s status are unavailable (i.e., measurability). Fourth, the timely monitoring of the

impact of various policies is needed (i.e., timeliness).

To determine the status of Japanese STI while taking continuity, comprehensive understanding,

measurability, and timeliness into consideration, the National Institute of Science and Technology

Policy (NISTEP) conducted a panel survey that targeted Japan’s leading researchers and experts,

whose views were assessed by means of a set of questions intended to gauge their levels of

satisfaction regarding issues related to STI. This survey can be likened to a health monitoring system,

wherein the status of Japanese STI is the subject being monitored, and the sensor used to monitor

this subject is the recognition of the country’s leading researchers and experts (i.e., combined

wisdom).

A well-known and similar survey is the Bank of Japan’s Tankan survey (also known as the

Short-Term Economic Survey of Enterprises in Japan), which examines how companies perceive

their business performance, status, and prospects in relation to economic conditions. The survey’s

results are critical, and greatly impact the stock market and policies related to it. The Cabinet

Office’s Economy Watchers Survey is similar. Hence, the methodology adopted in the present

study’s survey is widely used (i.e., the qualitative assessment of phenomena through cooperation

with individuals capable of observing those phenomena).

This paper provides an overview of NISTEP Teiten survey (also known as the NISTEP Expert

Survey on Japanese S&T and Innovation System), and discusses changes in the Japanese STI system

during the fourth STBP. Potential directions are then discussed for the future development of the

NISTEP Teiten survey.

3

2. Overview of NISTEP Teiten surveys conducted during the fourth STBP

2-1. Questions

The NISTEP Teiten surveys conducted during the fourth STBP attempted to determine the status

of Japanese STI and its situational changes according to 57 questions asked annually, in addition to

questions specific to certain years. The 57 questions can be categorized into either of research

personnel; research environment; industry-academia-government collaboration; basic research; and

innovation policy (see Figure 1). The survey was designed so that respondents would answer each

question based on a six-point scale ranging from “insufficient” to “sufficient.” Responses in the

previous year were fed back to respondents. When respondents changed their answer from the

previous year, we asked them to provide a short description indicating why they changed their

position.

Figure 1: Composition of the 57 NISTEP Teiten survey questions

2-2. Respondents

The survey’s respondents comprised two groups. The first included approximately 1,000

individuals who were presidents of universities or public research institutions (PRIs), principal

investigators of large funding programs, or researchers nominated by department heads at

universities or PRIs. To obtain an intergenerational perspective, we asked each department head to

nominate three candidates: a professor, associate professor, and assistant professor (or their

equivalents). Additionally, respondents were nominated to monitor how conditions differed

according to university size and department field.

The second group, the innovation overview group, comprised approximately 500 industry experts

(e.g., members of science and technology policy-related councils or subcommittees, executives in

charge of research and development at private firms, representatives of small or medium-sized

enterprises), individuals who bridge research and development with innovation, and those involved

in science and technology think tanks or mass media.

4

Based on the responses of the aforementioned researchers and experts, how did the status of

Japanese STI change between 2011–2015? The section 3 focuses on items wherein there were

significant increases or decreases in satisfaction when compared to the 2011 NISTEP Teiten survey.

It should be noted that the yearly response rates for these surveys were extremely high, averaging

86%.

2-3. Examples of questions and aggregate results

Figure 2 shows an example of a question and their aggregations. This example asks about the

status of the baseline funding for executing research and development at universities and PRIs

(Q1–18).

Figure 2(a) includes aggregation results for a given question broken down according to attribute;

the values at the top and bottom are for 2011 and 2015, respectively. The results show that the

satisfaction indices (i.e., values denoting satisfaction on a scale of 1–10) declined for many attributes,

especially in the public research institute. In Figure 2(b), respondents indicated why their levels of

satisfaction changed between years. By analyzing their responses, it is possible to understand the

specific contexts in which respondents’ attitudes changed.

The NISTEP Teiten survey report contains an analysis of all 57 questions. More detailed

aggregations according to attribute, as well as all open-ended responses, have been published in the

form of supplementary materials1.

Figure 2: Examples of questions and their aggregate results

(a) Aggregate results according to attribute

Note: Categorization of university group was done by scientific publication share in Japan. Group 1: 5% or more; Group 2: 1%

or more and less than 5%; Group 3: 0.5% or more and less than 1%; Group 4: 0.05% or more and less than 0.5%.

1 http://www.nistep.go.jp/research/scisip/nistep-teiten-data (in Japanese)

Index

change

Index

change

-0.47

(-0.1)

Natural

sciences

-1.06

(-0.46)

-1.57

(-0.77)Engineering

-0.53

(0.07)

Agricultural

sciences

-0.22

(-0.08)

G1-0.97

(-0.15)

Medical

sciences

-0.24

(-0.06)

G2-0.55

(-0.2)

G3-0.44

(-0.23)

G4-0.17

(0.07)

Q1-18: Status of the baseline funding for executing research and development at universities and

PRIs

University

Fie

ld o

f d

epar

tmen

t

Public research

institute

Un

iver

sity

gro

up

3 4 5 6

Index Index

2 2

Su

ffic

ien

t

Insu

ffic

ien

t

Su

ffic

ien

t

Insu

ffic

ien

t

3Attributes Attributes

4 5 62.7(748) 3.0(108)

4.0(122) 3.1(248)

1.7(83)

2.9(148) 2.5(236)

2.2(237)

2.2(160)

3.7(203)

2.6(713) 2.9(105)

3.8(117) 2.9(236)

1.5(78)

2.6(134) 2.3(221)

2.1(230)

2.1(153)

3.5(196)

2.5(720) 2.6(105)

3.4(117) 2.8(235)

1.4(82)

2.3(134) 2.3(223)

2.0(235)

2.1(153)

3.5(196)

2.4(725) 2.4(101)

3.2(114) 2.5(239)

1.6(75)

2.1(134) 2.3(227)

1.9(227)

2.0(158)

3.4(206)

2.3(689) 2.0(96)

2.4(109) 2.6(218)

1.5(75)

2.0(121) 2.3(219)

1.7(221)

1.7(143)

3.5(204)

5

(b) Reasons for change in opinion

Reasons for increased satisfaction Reasons for decreased satisfaction

Increased funding made available by the president or

dean of university

Initiatives by the university’s executive directors.

Changes in circumstances owing to a respondent

transferring to a different institution.

President of university forced to decrease spending,

thereby resulting in less funds being allocated to

departments and academic staff for research

purposes.

Management expenses grants decreased considerably

over time, necessitating budget cuts.

Sufficient money for research cannot be procured

through facility maintenance funds alone.

Insufficient funding for conducting experiment based

research activities.

Increased cost of electricity and consumption taxes

have forced researchers to bear an increasing

proportion of expenses related to the use of common

facilities.

Funds are being depleted to cover just printing and

postage costs.

3. Situational changes of Japanese STI system during the fourth STBP

3-1. Questions for which respondents felt circumstances had improved

Table 1 shows the top 10 questions in terms of positive change with respect to satisfaction when

compared to the 2011 NISTEP Teiten survey. The fourth STBP identified important issues that

must be tackled. The integrated promotion of science and technology/innovation policies intended

to resolve such issues is a fundamental objective of the basic plan.

Increases or positive trends were observed in the satisfaction indices for questions related to

innovation policy (e.g., cooperation beyond the boundaries of natural sciences to address technical

issues in resolving important problems [Q3–4]; unified public-private efforts to implement Japanese

technology and systems overseas [Q3–12]; concentration on core competence in government-led

research and development [Q3-3]; implementation of strategies and national projects to resolve

important issues through cooperation between industry, academia, and government [Q3–2];

introduction or relaxation of regulations and/or enhancement or establishment of systems [Q3–7]).

With respect to reasons for higher levels of satisfaction when compared to previous years,

respondents mentioned specific national research and development programs (e.g., “the

Cross-ministerial Strategic Innovation Promotion Program (SIP)”, “the Impulsing Paradigm Change

through Disruptive Technologies Program (ImPACT)”, and “the Center of Innovation (COI)

Program”). Concerning the introduction or relaxation of regulations, specific initiatives were

mentioned (e.g., the New Regenerative Medicine Act, Revision of the Pharmaceutical Affairs Act,

and the easing of regulations related to fuel-cell vehicles).

From these results, it is apparent that respondents felt that some progress was made in resolving

6

important issues during the fourth STBP. However, the satisfaction indices’ absolute values

indicate that there were many questions for which a strong feeling of insufficiency remained, thereby

suggesting that further improvement is needed.

Table 1: Top 10 questions in terms of positive change with respect to satisfaction between 2011–2015

Note 1: Darker shades in the “index change” column indicate greater change in satisfaction. The top and bottom figures in each

row indicate the degree of change in satisfaction between 2011–2015 and 2014–2015, respectively.

Note 2: The satisfaction index values are on a scale of zero (insufficient) to 10 (sufficient). Values of 5.5 or greater are

considered “unproblematic” ( ), 4.5–5.4 “not generally problematic” ( ), 3.5–4.4 “insufficient” ( ), 2.5–3.4

“generally insufficient” ( ), and less than 2.5 “extremely insufficient” ( ).

Note 3: KAKENHI is the largest competitive fund for academic research in Japan.

3-2. Questions for which respondents felt circumstances had worsened

Although progress was achieved in some areas during the fourth STBP, there were also questions

indicative of growing concerns when compared to the 2011 NISTEP Teiten survey (see Table 2).

Questions that exhibited the largest satisfaction index declines were those regarding baseline funding

for executing research and development at universities and PRIs (Q1–18). Based on the “Basic

Policies for Economic and Fiscal Policy Management and Structural Reform 2006”, the management

expenses grants of the national university corporations were reduced by 1%, and have been

consistently reduced for about ten years. Respondents expressed that the reduction of the

management expenses grants had large impact of research and development at university.

RankQuestion

No.Category Question Index change

Index

value 2015

1 Q1-19Research

environment

Usability of research expenses in Grants-in-Aid for Scientific Research

(KAKENHI)

0.79

(0.13)5.4

2 Q1-22Research

environment

Fostering and securing of specialized personnel to handle operations

necessary for the smooth execution of research activities (URAs) (in

universities and PRIs)

0.35

(0.09)2.4

3 Q3-04Innovation

policy

Cooperation beyond the boundaries of the natural sciences to address

technical issues for resolving important issues

0.34

(0.07)3.6

4 Q3-12Innovation

policy

Unified public-private efforts to implement Japanese technology and systems

to overseas

0.32

(0.04)2.8

5 Q3-03Innovation

policy

Concentration on core competence in government-led research and

development

0.30

(0.10)3.9

6 Q3-02Innovation

policy

Implementation of strategies and national projects to resolve important issues

through cooperation between industry, academia, and government

0.24

(0.03)3.6

7 Q1-13Research

personnelNumber of foreign researchers (in universities and PRIs)

0.23

(0.09)2.8

8 Q1-20Research

environmentEffectiveness of the multi-year funds for executing of R&D

0.23

(0.04)7.3

9 Q3-07Innovation

policy

Introduction or relaxation of regulations and/or enhancement or establishment

of systems

0.16

(-0.04)2.8

10 Q2-02Industry-academia-

government

collaboration

Interests for the needs of private companies (technical issues, etc.) (in

universities and PRIs)

0.15

(0.03)4.8

7

Table 2: Top 10 questions in terms of negative change with respect to satisfaction between 2011–2015

Note 1: Darker shades in the “index change” column indicate greater change in satisfaction. The top and bottom figures in each

row indicate the degree of change in satisfaction between 2011–2015 and 2014–2015, respectively.

Note 2: The satisfaction index values are on a scale of zero (insufficient) to 10 (sufficient). Values of 5.5 or greater are

considered “unproblematic” ( ), 4.5–5.4 “not generally problematic” ( ), 3.5–4.4 “insufficient” ( ), 2.5–3.4

“generally insufficient” ( ), and less than 2.5 “extremely insufficient” ( ).

There was also growing concern regarding a lack of diversity and/or originality in basic research.

Detailed questions were conducted in the 2014 NISTEP Teiten survey to examine changes in

research activities and behaviors among researchers at universities and PRIs (see Figure 3 and Figure

4). According to the results, respondents believed that there was growth in research for (a) the

direct purposes of resolving social issues and creating economic value, (b) to fulfill organizational

goals (e.g., regional or social contributions), and (c) to achieve integration between different

disciplines. In examining the behavior of researchers, there was a growing awareness that they

were producing output other than scientific publications, such as patents and prototypes. Given that

the resolution of important issues was emphasized in the fourth STBP, these changes can be

considered the result of various policies that were promoted under the plan.

The results likewise show that respondents were increasingly aware that research pursuing

temporary trends was growing, whereas exploratory research to find new research themes and

pioneering research toward producing new research fields were in decline—albeit at a minimal rate.

With respect to the behavior of researchers, respondents were aware that most researchers were

strongly inclined to (a) produce short-term results, (b) conduct research with a high probability of

RankQuestion

No.Category Question Index change

Index

value 2015

1 Q1-18Research

environment

Baseline funding for executing research and development at universities and

PRIs

-0.62

(-0.19)2.3

2 Q1-06Research

personnel

Whether or not individuals with the required capabilities are applying to

doctoral programs

-0.57

(-0.17)3.0

3 Q1-24Research

environment

Research facilities and equipment for innovative and advanced R&D; and the

training of high-quality human resources (in universities and PRIs)

-0.49

(-0.07)4.4

4 Q2-22 Basic researchDegree of diversity in Japanese basic research as the source of future

innovation

-0.43

(-0.14)3.0

5 Q2-23 Basic researchDegree of originality in Japanese basic research as the source of future

innovation

-0.40

(-0.16)3.0

6 Q2-17Research

environment

Amount of indirect funding related to the government's public appeal-based

research funds (competitive research funds, etc.) (in universities and PRIs)

-0.36

(-0.07)4.0

7 Q1-16Research

personnel

Multi-faceted evaluation of researchers, rather than a single indicator related to

scientific publications (in universities and PRIs)

-0.35

(-0.03)4.5

8 Q1-21Research

environment

Efforts to secure researchers' time allocated to R&D activities (in universities

and PRIs)

-0.31

(-0.06)2.2

9 Q2-19Research

environment

Circumstances of intellectual and research information infrastructure in Japan

(in universities and PRIs)

-0.30

(-0.03)4.2

10 Q2-16Research

environmentGovernment's S&T budget, taking account of current S&T situation in Japan

-0.28

(-0.16)2.7

8

success, and (c) emphasize the number of scientific publications as research output (in response to

evaluation). In contrast, there was a decline in terms of researchers addressing themes emphasizing

long-term research strategies. None of these changes is desirable with respect to ensuring research

diversity.

Generally speaking, it is clear from the 2011–15 NISTEP Teiten survey results that there is

growing concern regarding fundamental research activities at universities and PRIs.

Figure 3: Changes in research activities at universities and PRIs over the past ten years categorized

according to respondent group and content

Figure 4: Changes in research activities at universities and PRIs over the past ten years categorized

according to respondent group and researchers’ behaviors

Note: Respondents were asked to indicate the degree of change compared to 2005 by selecting “decreased greatly,” “decreased,”

“no change,” “increased,” and “increased greatly.” The results obtained after indexation were as follows: “decreased

greatly” (-10 points), “decreased” (-5 points), “no change” (0 points), “increased” (5 points), “increased greatly” (10

points).

-0.9

-0.8

0.2

1.0

2.6

2.9

3.0

3.0

-1.4

-1.0

0.0

-0.6

2.0

2.4

2.3

2.4

-5.0 -4.0 -3.0 -2.0 -1.0 0.0 1.0 2.0 3.0 4.0 5.0

Innovation overview group University and PRI group

Dec

reas

ing

Incr

easi

ng

Research for the direct purposes of resolving social issues and creating

economic value

Research to fulfill organizational goals (e.g., regional or social

contributions)

Research to achieve integration between different disciplines

Research in pursuit of temporary trends

Research on basic technologies required by industry (material testing,

etc.)

Research that grasps fragmented knowledge from a comprehensive

viewpoint

Pioneering research toward producing new research fields

Exploratory research to find new research themes

-4.7

1.8

2.8

3.0

3.7

4.6

-4.5

1.9

2.7

2.0

3.3

4.2

-5.0 -4.0 -3.0 -2.0 -1.0 0.0 1.0 2.0 3.0 4.0 5.0

Innovation overview group University and PRI group

Incr

easi

ng

Dec

reas

ing

Researchers who are strongly inclined to produce short-term results

Researchers who conduct research with a high probability of success

Researchers who place emphasis on the number of scientific

publications papers as research output (in response to evaluation)

Researchers who announce results on a piecemeal basis (in response to

evaluation)

Researchers who produce research output other than scientific

publications (e.g., patents and prototypes)

Researchers who address research themes with emphasizing long-term

research strategies

9

4. In-depth analyses of selected questions

4-1. Monitoring the breadth of the effects of policy measures implementation

Breaking down the NISTEP Teiten survey results according to attribute widens an understanding

of the breadth of the effects of policy measures implementation. Figure 5 shows the aggregate

results by attribute for the usability of research expenses in Grants-in-Aid for Scientific Research

(KAKENHI), the item for which improvement of the satisfaction indices were highest.

As the results show, satisfaction indices increased for each attribute. The stated reasons for this

change included smoother annual carryover, and increased convenience owing to the introduction of

multi-year funds. Grants-in-Aid for Scientific Research (KAKENHI) is the largest competitive

fund in Japan; hence, alterations to it influence many of the country’s researchers. However, it

requires elaborative efforts.

Figure 5: Usability of Grants-in-Aid for Scientific Research (KAKENHI)

Note: Categorization of university group was done by scientific publication share in Japan. Group 1: 5% or more; Group 2: 1%

or more and less than 5%; Group 3: 0.5% or more and less than 1%; Group 4: 0.05% or more and less than 0.5%.

An in-depth analysis of a question regarding university research administrators (URAs), an item

with the second-highest satisfaction index rise, was conducted. URA universities were compared to

other universities in terms of whether differences existed in circumstances related to Q1–22 (i.e., the

fostering and securing of specialized personnel to handle operations necessary for the smooth

execution of research activities (URAs)).

For this analysis, URA universities were defined as those that adopted the 2011–2014 Initiative for

‘the Developing a System for Fostering and Recruiting URAs’ or the 2013–2023 Initiative for ‘the

Strengthening of Research Universities,’ as well as members of the Association of Research

Administrators. Of the 36 universities that fell into these categories, 32 were targeted in the

Index

change

Index

change

0.85

(0.13)

Natural

sciences

0.92

(0.09)

0.44

(0.13)Engineering

0.5

(0.09)

Agricultural

sciences

1.15

(0.09)

G10.91

(0.04)

Medical

sciences

1.12

(0.21)

G20.78

(0.11)

G30.85

(0.08)

G40.9

(0.26)

Un

iver

sity

gro

up

Eas

y t

o u

se

6

No

t ea

sy t

o u

se

Eas

y t

o u

se No

t ea

sy t

o u

se

Attributes

Index Index

2 3 4 5 6 2

Q1-19: Usability of research expenses in Grants-in-Aid for Scientific Research (KAKENHI)

3 4 5

University

Fie

ld o

f d

epar

tmen

t

Public research

institute

Attributes

4.5(731) 5.0(108)

4.7(106) 5.1(239)

4.1(78)

4.7(149) 3.8(233)

4.3(235)

4.8(156)

4.5(191)

4.9(699) 5.7(103)

4.8(101) 5.4(228)

4.6(76)

5.3(133) 4.0(219)

4.7(228)

5.1(149)

4.8(189)

5.2(708) 5.9(103)

4.9(103) 5.4(230)

5.0(80)

5.6(134) 4.5(220)

5.0(232)

5.3(150)

5.0(190)

5.3(714) 5.8(101)

5.0(103) 5.5(233)

5.2(74)

5.6(133) 4.7(223)

5.0(226)

5.6(156)

5.1(199)

5.4(678) 5.9(96)

5.1(98) 5.6(214)

5.3(72)

5.7(121) 4.9(216)

5.1(219)

5.6(141)

5.4(197)

10

NISTEP Teiten survey2.

Figure 6(a) shows the difference between URA universities and other universities regarding

changes in satisfaction indices with respect to fostering and securing of URAs (Q1–22). The two

sets of universities showed similar satisfaction indices in 2011; however, the satisfaction index for

URA universities increased each year, such that by 2015 it was 0.79 points higher than in 2011. In

contrast, the satisfaction index for the other universities barely changed.

Furthermore, the change in the satisfaction index varied according to respondents’ duties. Figure

6(b) contains the aggregate results for respondents whose duties were management related. Among

URA universities, the satisfaction index was 1.9 for the 2011 NISTEP Teiten survey. Nevertheless,

between 2011–2013 the index increased sharply, reaching 4.4 in 2015. Given that between

2011–2013 these universities adopted ‘the Initiative for Developing a System for Fostering and

Recruiting URAs’ and/or ‘the Initiative for the Strengthening of Research Universities,’ the

commencement and adoption of these programs were reflected in the form of changes in the

satisfaction index.

The satisfaction index for respondents whose duties were not managerial (see Figure 6(c)) rose

between 2011–15, albeit markedly less than for their manager counterparts. This finding suggests

that although universities are progressing in fostering and securing URAs, these advances are not

having a substantial impact within institutions. Indeed, in this question, one of the reasons cited for

a decline in satisfaction level was “the benefits have not yet been felt at a department level.”

2 Data from the following websites (each accessed on February 27, 2016) were used to prepare the list of URA universities:

Initiative for Developing a System for Fostering/securing URAs (http://www.mext.go.jp/a_menu/jinzai/ura/detail/1315871.htm),

Initiative for the Strengthening of Research Universities (http://www.mext.go.jp/a_menu/kagaku/sokushinhi/1338460.htm),

Association for Research Administrators (http://www.rman.jp/aboutus/memberlist.html).

11

Figure 6: Comparison between URA universities and other universities

(a) All respondents

(b) Respondents whose primary duties are management related

(c) Respondents whose primary duties are not management related

1.9

2.7

1.92.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

2011 2012 2013 2014 2015

Sat

isfa

ctio

n i

nd

ex a

bo

ut

fost

erin

g a

nd

secu

rin

g U

RA

s (T

ota

l)

Year of survey

URA universities

Other universities

1.9

4.4

2.32.5

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

2011 2012 2013 2014 2015

Sat

isfa

ctio

n i

nd

ex a

bo

ut

fost

erin

g a

nd

secu

rin

g U

RA

s (R

esp

on

den

ts w

ho

se

du

ties

wer

e m

anag

emen

t re

late

d)

Year of survey

URA universities

Other universities

1.9

2.5

1.91.9

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

2011 2012 2013 2014 2015

Sat

isfa

ctio

n i

nd

ex a

bo

ut

fost

erin

g a

nd

secu

rin

g U

RA

s (R

esp

on

den

ts w

ho

se

du

ties

wer

e n

ot

man

ager

ial)

Year of survey

URA universities

Other universities

12

4-2. Monitoring attitudinal differences between actors

Analyzing the aggregate results by attribute makes it possible to ascertain attitudinal differences

between actors. Respondents at the private firm, university, and government all expressed

dissatisfaction regarding the utilization of intellectual property by private firms obtained from

research and development conducted at universities and PRIs (see Figure 7). Individuals in the

innovation overview group were particularly dissatisfied: their satisfaction index was 0.8 and 1.2

points lower than university and PRI respondents, respectively.

Respondents from private firms indicated that the quality of universities’ granted patent was low,

and furthermore that universities were unable to fund international applications pursuant to the

Patent Cooperation Treaty; consequently, the most of patent applications were domestic rather than

international. In contrast, university respondents pointed out that there was significant

apprehension and an unwillingness to take risks in product development at private firms, which

prevented knowledge transfer from universities to private firms.

Figure 7: Example of a question showing attitudinal differences between actors

Note1: Categorization of university group was done by scientific publication share in Japan. Group 1: 5% or more; Group 2: 1%

or more and less than 5%; Group 3: 0.5% or more and less than 1%; Group 4: 0.05% or more and less than 0.5%.

Note2: Innovation overview group comprised approximately 500 industry experts (e.g., members of science and technology

policy-related councils or subcommittees, executives in charge of research and development at private firms,

representatives of small or medium-sized enterprises), individuals who bridge research and development with innovation,

and those involved in science and technology think tanks or mass media.

4-3. Complementary analysis along with quantitative data

Analyzing the NISTEP Teiten survey results in conjunction with qualitative data facilitates an

understanding of the underlying causes of changes in the quantitative data.

Respondents from universities and PRIs were of the strong opinion that individuals with the

required capabilities were not applying to doctoral programs (see Figure 8). In examining changes

since the 2011 NISTEP Teiten survey, the satisfaction indices for all attributes either declined or

Index

change

Index

change

-0.09

(0.05)

Natural

sciences

0.06

(0.14)

-0.21

(0.13)Engineering

0.03

(0.18)

-0.05

(-0.02)

Agricultural

sciences

0.03

(0.05)

G1-0.02

(0.1)

Medical

sciences

-0.27

(-0.1)

G2-0.02

(0.12)

G30.15

(0.07)

G4-0.35

(-0.04)

Attributes Attributes

Q2-8: Utilization of intellectual property by private firms obtained from research and development

conducted at universities and PRIs

University

Fie

ld o

f d

epar

tmen

t

Public research

institute

Innovation

overview group

Un

iver

sity

gro

up

Su

ffic

ien

t

Insu

ffic

ien

t

Su

ffic

ien

t

Insu

ffic

ien

t

3 4 5 622 3 4 5 6

Index Index

3.7(536) 3.6(62)

4.2(106) 4.1(179)

2.8(414) 3.8(54)

4.1(103) 3.3(172)

3.5(170)

3.6(119)

3.8(144)

3.7(513) 3.5(61)

4.0(104) 4.0(172)

2.8(392) 3.6(46)

4.2(91) 3.2(165)

3.4(159)

3.7(110)

3.7(153)

3.6(520) 3.4(61)

3.9(105) 4.0(171)

2.8(371) 3.5(53)

3.9(95) 3.0(165)

3.3(163)

3.7(110)

3.5(150)

3.6(527) 3.6(59)

3.8(104) 3.9(174)

2.8(384) 3.8(49)

4.0(92) 3.1(168)

3.4(164)

3.7(117)

3.5(154)

3.6(495) 3.7(54)

4.0(101) 4.1(156)

2.8(371) 3.8(51)

4.1(81) 3.0(161)

3.5(156)

3.8(104)

3.5(154)

13

exhibited a declining trend. With respect to individual university groups, the satisfaction indices

for groups 1 and 4 declined by more than 0.6 points; as for various university departments, the

satisfaction indices for natural sciences, agricultural sciences, and medical sciences declined by

nearly 0.7 points. Engineering showed a smaller decline when compared to other departments,

although its dissatisfaction level was already relatively high in 2011.

Figure 8: Aggregate results by attribute for the quality of applicants to doctoral programs

Note: Categorization of university group was done by scientific publication share in Japan. Group 1: 5% or more; Group 2: 1%

or more and less than 5%; Group 3: 0.5% or more and less than 1%; Group 4: 0.05% or more and less than 0.5%.

Figure 9: Number of students enrolled in doctoral programs

(a) Change in the number of enrollees by course (b) Change in proportion of working enrollees

Note: “Other” includes courses that do not fall under the umbrella of engineering, agriculture, health, the humanities, or

social/physical sciences.

Source: National Institute of Science and Technology Policy (2015). Japanese science and technology indicators 2015. Research

Material No. 229.

Index

change

Index

change

-0.56

(-0.18)

Natural

sciences

-0.68

(-0.29)

-0.73

(-0.06)Engineering

-0.26

(0.02)

Agricultural

sciences

-0.7

(-0.28)

G1-0.65

(-0.19)

Medical

sciences

-0.67

(-0.16)

G2-0.5

(-0.14)

G3-0.45

(-0.19)

G4-0.66

(-0.23)

3 4 5 6

Index

Q1-6: Whether or not individuals with the required capabilities are applying to doctoral programs

University

Fie

ld o

f d

epar

tmen

t

Public research

institute

Un

iver

sity

gro

up

Ap

ply

ing

3 4 5 6

No

t-ap

ply

ing No

t-A

pp

lyin

g

Index

2A

pp

lyin

g

Attributes Attributes2

3.5(723) 3.6(104)

4.2(77) 3.0(243)

3.2(80)

3.7(148) 3.7(224)

3.3(238)

3.4(152)

3.7(185)

3.2(691) 3.3(97)

3.9(76) 2.8(229)

3.3(78)

3.3(132) 3.3(216)

3.2(228)

3.2(146)

3.3(185)

3.2(698) 3.1(101)

3.7(83) 2.8(228)

3.0(81)

3.1(134) 3.2(215)

3.0(229)

3.1(148)

3.4(185)

3.1(703) 3.2(96)

3.5(82) 2.7(228)

2.8(75)

3.2(133) 3.2(223)

2.9(222)

3.2(153)

3.3(195)

2.9(673) 2.9(94)

3.5(77) 2.7(214)

2.5(73)

3.0(120) 3.0(214)

2.8(220)

3.0(140)

3.0(193)

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

1.8

2.0

1981 84 87 90 93 96 99 02 05 08 11 2014

Others

Social

science

Humanities

Medical

sciences

Agricultural

sciences

Engineering

Natural

sciences

The

num

ber

of

enro

llm

ent

10,000 people

FY

21.7%

37.7%

0%

10%

20%

30%

40%

50%

0.0

0.5

1.0

1.5

2.0

2003 10 2014

Other

students

Working

students

Percentage

of working

students

10,000 people

FY

14

Figure 9 shows the change in the number of students entering doctoral programs according to the

Ministry of Education, Culture, Sports, Science, and Technology’s Basic School Survey. After

peaking in 2003, the number of enrollees entered a declining trajectory; by 2014, the number had

fallen to a level nearly identical to the late 1990s (see Figure 9(a)). The temporary rise that

occurred in 2010 could be attributable to the 2008 financial crisis.

The proportion of working students enrolled in doctoral programs rose from 21.7% in 2003 to

37.7% in 2014. Whereas the number of working students has remained relatively flat since the late

2000s, the number of non-working students has declined. As for why satisfaction dropped with

respect to the question regarding the quality of doctoral program applicants (Q1–6), respondents

cited the absence of reliable financial support or feasible career paths. Making such statements

suggest that the result of NISTEP Teiten survey likely reflects the declining number of enrollees

among non-working students (Figure 9(b)).

5. Conclusion and future works

It can be deduced that the NISTEP Teiten surveys are effective in comprehensively monitoring the

STI system’s status. The data provided by these surveys are useful in formulating science and

technology policies, and have been referenced in official documents by various governmental

councils and committees. Likewise, the survey’s results were referenced in science and technology

white papers, in planning the fifth STBP, and in media such as newspapers. The fifth STBP

outlines initiatives to strengthen fundamental capacities, particularly with respect to fostering and

promotion of young researchers, as well as reforming and strengthening the functions of universities.

When policymakers were establishing the direction of these policies, they referred to the NISTEP

Teiten survey results as one of evidence. Hence, it is clear that the NISTEP Teiten survey provide

useful information that others cannot. Looking forward, it should be possible to build data that are

even more useful in policy formation and evaluation by implementing surveys and related analyses

wherein the following three points are emphasized.

5-1. Understanding the relationships between questions

As mentioned previously, the NISTEP Teiten survey results can be employed in various ways;

nevertheless, they are often used in a partial manner. The survey’s 57 questions should be

examined in an interrelated fashion. For instance, in attempting to improve upon one item, its

relationship to others ought to be considered. This requires a comprehensive understanding of the

relationships between all questions.

5-2. Further analysis of the open-ended responses

The NISTEP Teiten surveys include open-ended questions regarding to various STI issues. The

total responses to these questions between 2011–2015 exceeded 2.5 million Japanese characters.

15

Through text mining or machine learning, it is possible to extract words used in specific contexts.

Currently, this is a manual and time-consuming process. Hence, a methodology based on

computerized text mining or machine learning should be developed in order to identify new issues,

as well as those that have not been discussed previously.

5-3. Analysis of the extent to which the effects of policies spread

There are some questions for which no significant changes were observed in the satisfaction

indices between 2011–2015 that means the effects of the policies were not felt by researchers,

perhaps because of a lack to implement relevant ones, or to do so on a limited scale. Accordingly,

continuous monitoring is necessary that goes beyond the STBP’s five-year framework, since a

sufficient amount of time is needed for conditions to improve. However, the NISTEP Teiten survey

includes identical respondents each year, and their ages obviously increase. Consequently, the use

of respondent groups and surveying methods that facilitate long-term chronological monitoring

while simultaneously maintaining respondent continuity should be examined.

5-4. Determining question content and identifying target respondents

The NISTEP Teiten surveys are effective in assessing the state of STI in Japan. Nevertheless, the

results depend largely on the content of questions and to whom they are directed. For instance,

some individuals argue that because the NISTEP Teiten surveys comprise many questions related to

research environment, that the results tend to emphasize dissatisfaction among researchers. It

should be possible to solve this problem by analyzing attitudinal differences between actors and

setting questions that examine researchers’ own efforts.

This paper presented a method for qualitatively determining the status of STI through a panel

survey administered to researchers and experts. Of course, determining the status of STI based on

qualitative data alone is insufficient. Hence, utilizing qualitative data from the NISTEP Teiten

surveys in conjunction with various research and development statistics would promote a fuller

understanding of STI systems.

Currently, preparation for the new NISTEP Teiten survey, which will be conducted during fifth

STBP, is undergoing. Fifth STBP identified numerical targets/indicators that should be monitored

during the basic plan; therefore, we believe that original evidence that NISTEP Teiten survey

provides is getting more important in order to understand a context of changes of the indicators.

To author’s knowledge, the NISTEP Teiten surveys are unique in their focus on monitoring the

status of STI systems, we hope that our experience will be useful for the development of new STI

indicators and widens our collective knowledge for the measurement of STI activities.

16

References

National Institute of Science and Technology Policy (2016). Analytical report for 2015 NISTEP expert survey on

Japanese S&T and innovation system (2015 NISTEP TEITEN survey). NISTEP Report No. 167 (in Japanese).

National Institute of Science and Technology Policy (2015). Japanese science and technology indicators 2015.

Research Material No. 229.