51
How to consider the effects of research and innovation investments Per M. Koch Head of the Science Project

Koch taftie-measuring the effects of research

Embed Size (px)

DESCRIPTION

Presentation for TAFTIE on the use of R&D and innovation statistics in policy development.

Citation preview

Page 1: Koch taftie-measuring the effects of research

How to consider the effects of research and innovation investments

Per M. KochHead of the Science Project

Page 2: Koch taftie-measuring the effects of research

2 Norwegian Ministry of Education and Research

Why do policy makers look for numbers?

• To understand the world and how research functions in it

• To develop a story that can be used to convince others to invest in R&D

• We are number fetishists– The prestige of the natural

sciences– The power of macro

economics and the ministries of finance

Page 3: Koch taftie-measuring the effects of research

3 Norwegian Ministry of Education and Research

I am going to talk about:

• The potential and limitations of evaluating the effects of– Innovation and research policy measures

on the aggregate national level– Individual policy instruments

• The fact that the complexity of social systems makes these very challenging tasks

Page 4: Koch taftie-measuring the effects of research

4 Norwegian Ministry of Education and Research

Data sources

Page 5: Koch taftie-measuring the effects of research

5 Norwegian Ministry of Education and Research

The Frascati Manual

• June 1963: OECD NESTI group (National Experts on Science and Technology Indicators) met at Villa Falconieri in Frascati. Italy

• The document sets forth the fundamental definitions (type of activity: basic research, applied research, experimental development; research personnel: researchers, technicians, auxiliary personnel). It primarily deals with measuring the resources devoted to R&D – expenditure and personnel – in the performing sectors: higher education, government, business enterprise, private non-profit organizations.

Page 6: Koch taftie-measuring the effects of research

6 Norwegian Ministry of Education and Research

The Oslo Manual• OECD Oslo 1992:

Guidelines for collecting and interpreting technological innovation data.

• Basis for the Eurostat Community Innovation Survey (CIS)

• Revision 2005 3rd edition:– Marketing– Organizational

innovation– Innovation linkages

Page 7: Koch taftie-measuring the effects of research

7 Norwegian Ministry of Education and Research

Canberra Manual • NESTI Canberra, 1994: The Manual on the Measurement of Human Resources devoted to S&T (the "Canberra Manual")

• The OECD and Eurostat, the European Commission

• Definitions of human resources devoted to science and technology in terms of qualification (levels and fields of study) and occupation

• The International Standard Classification of Education (ISCED)

• The International Standard Classification of Occupation (ISCO)

Page 8: Koch taftie-measuring the effects of research

8 Norwegian Ministry of Education and Research

Other data • Tax authorities• Social security agencies• Higher education

institutions• Accounting data

• Register-based statistical systems using standardized identifiers.– Problems due to

privacy considerations

Page 9: Koch taftie-measuring the effects of research

9 Norwegian Ministry of Education and Research

Beyond evidence based policy development

Page 10: Koch taftie-measuring the effects of research

10 Norwegian Ministry of Education and Research

Evidence Based Medicine

• Evidence-based medicine (EBM) aims to apply the best available evidence gained from the scientific method to medical decision making. It seeks to assess the quality of evidence of the risks and benefits of treatments (including lack of treatment).(Wikipedia)

• Using techniques from:– Science– Engineering– Statistics– meta-analysis of medical literature– risk-benefit analysis– randomized controlled trials

Page 11: Koch taftie-measuring the effects of research

11 Norwegian Ministry of Education and Research

Evidence-based policy

• “…helps people make well informed decisions about policies, programmes and projects by putting the best available evidence from research at the heart of policy development and implementation” (P Davies, 1999)

• Vs. “opinion-based policy, which relies heavily on either the selective use of evidence … or on the untested views of individuals or groups, often inspired by ideological standpoints, prejudices, or speculative conjecture.” (P Davies 2004)

Page 12: Koch taftie-measuring the effects of research

12 Norwegian Ministry of Education and Research

The myth of evidence based policy-development• “…policy making must be soundly based on

evidence of what works” Professional Policy Making for the Twenty-First Century (UK Cabinet Office, 1999)

• But how to you prove what works?– Any social and economic phenomena are a

result of interaction between a limitless number of factors

– Society is constantly changing, a moving target

• Any interpretation of the reality underlying policy-development will be only that: an interpretation

• Researchers are as fallible as other experts, anchored in their own belief systems and prejudices

Page 13: Koch taftie-measuring the effects of research

13 Norwegian Ministry of Education and Research

There are many beliefs and narratives

• Different understanding of:– What society is and how it

works– How research interacts with

society– Common terms (e.g.

“innovation”, “research”)– What the role of science is– What the best theoretical

and methodological foundation for policy development is

• Different cultures have different narratives they use to make sense of their surroundings.

Page 14: Koch taftie-measuring the effects of research

14 Norwegian Ministry of Education and Research

Linear vs. systemic views of innovation

Page 15: Koch taftie-measuring the effects of research

15 Norwegian Ministry of Education and Research

The linear model works perfectly in a policy setting Research and

technological development in universities, RTOs and companies gives birth to an idea and relevant new knowledge

Companies make use of these ideas in the development of new products and processes

The company brings the new product to the market

Page 16: Koch taftie-measuring the effects of research

16 Norwegian Ministry of Education and Research

Why?• It is a good story:

– The unselfish scientist striving for the common good

– It is simple: Give us the money and we will give you the results

– By black boxing the whole process it communicates easily. • You don’t need to know

how the computer works to make use of it

• It works well with many of the traditional macroeconomists

Page 17: Koch taftie-measuring the effects of research

17 Norwegian Ministry of Education and Research

But this model leads to one specific interpretation of data

• A pipe-line model: 1. Research > findings > products > profits2. Science > findings > publications > scientific progress

• Main focus on input indicators – (money, doctorates)

• To the extent there is output data:– Publications and citations– Reported number of innovations in companies– Patents and licenses

Page 18: Koch taftie-measuring the effects of research

18 Norwegian Ministry of Education and Research

Policy consequences of the linear model

• Input indicators as policy objectives:– 3% R&D of GDP (Barcelona)– Number of PhDs

• Focus on technology push rather than market pull

• Oversimplified idea of best practice as regards innovation (“More research!”)

Page 19: Koch taftie-measuring the effects of research

19 Norwegian Ministry of Education and Research

The correlation problem

R&D-investments

Economic growth

It is possible to study the correlation between R&D and growth…

Page 20: Koch taftie-measuring the effects of research

20 Norwegian Ministry of Education and Research

The correlation problem

R&D-investments Research

Economic growth

…but are we really studying the effects of research alone?

Page 21: Koch taftie-measuring the effects of research

21 Norwegian Ministry of Education and Research

The correlation problem

R&D-investments

Research Science Development Learning Innovation Mobility Machinery Technology Education Literature Patents Licenses Design Marketing Branding Culture Regulations Globalization Experience Creativity

Economic growth

There is nothing wrong in having increased investments in R&D as a policy objective, but it cannot be used as a proxy for innovation

“Total factor productivity?”

Page 22: Koch taftie-measuring the effects of research

22 Norwegian Ministry of Education and Research

We are missing a large part of the picture!

• Major problem: – there is no direct

correlation between national R&D investments, productivity and economic growth.

• The Norwegian paradox: seemingly low R&D investments, but record high productivity and economic growth

• The Swedish paradox: very high R&D investments, lower productivity and economic growth

Page 23: Koch taftie-measuring the effects of research

23 Norwegian Ministry of Education and Research

Rogers

Klein

Havelock/Przybylinski

The linear model masks the complexity of the role of research in social systems

Page 24: Koch taftie-measuring the effects of research

24 Norwegian Ministry of Education and Research

In-house learning based innovation

Following the OECD systemic approach: Research is given a new and broader role as a learning tool

market pull

Knowledge about customers and competitors

The company brings new or altered products, processes or services on to the market

Market competencesTacit knowledge

Acquired technology

Literature

Conferences and fairs

Recruits

Acquired R&D

In-house R&D

Page 25: Koch taftie-measuring the effects of research

25 Norwegian Ministry of Education and Research

Company or public institution

Learning NetworksInnovation

Customers and users

Suppliers

Policy-institutions

FinancialInstitutions

R&DInstitutions

Consultants

Public policy Cultural framework

International framework Industrial structure

Understanding competence flows in the innovation system

Page 26: Koch taftie-measuring the effects of research

26 Norwegian Ministry of Education and Research

Challenges beyond the pipe-line

Page 27: Koch taftie-measuring the effects of research

27 Norwegian Ministry of Education and Research

Indirect use of research

• Research is embedded in technology and machinery. We need to measure the effect of research in the use of advanced technology in low tech industries.

• Research is embedded in students and employees. We need to measure the effect research based education has on the learning capabilities of men and women.

Page 28: Koch taftie-measuring the effects of research

28 Norwegian Ministry of Education and Research

The role of competence flows

• Education• User-driven innovation• Customer/supplier

relationships• “Open innovation” and

industry collaboration• KIBS• Global companies as

gateways to the world

Page 29: Koch taftie-measuring the effects of research

29 Norwegian Ministry of Education and Research

We need to get a better understanding of the role of services

• The largest part of the economy

• Interacts with all of industry

• A very heterogeneous sector

• Often left out of statistics

Until now: Industrial policies have been focused on manufacturing, even if 70 percent are employed in services

Page 30: Koch taftie-measuring the effects of research

30 Norwegian Ministry of Education and Research

Understanding the effects of public sector innovation

• Innovation in the private sector is understood as an investment, in the public sector as an expense

• Innovation in the public sector and the effect on industry

• The effect of social welfare on risk taking and company behavior

• We have no output indicators!

Page 31: Koch taftie-measuring the effects of research

31 Norwegian Ministry of Education and Research

The Copenhagen Manual• New Nordic project aimed at developing a

statistics manual for public sector innovation– Danish Agency for Science, Technology and

Innovation, Denmark– Danish Centre for Studies in Research and

Research Policy, CFA, – NIFU STEP, Norway – RANNIS, Iceland – Innovation Norway, Norway– The Research Council of Norway– DAMVAD, Denmark – Statistics Finland – Statistics Norway – Statistics Denmark – Statistics Sweden

In cooperation with: OECD NESTI NESTA UK

Page 32: Koch taftie-measuring the effects of research

32 Norwegian Ministry of Education and Research

Understanding socio-cultural framework conditions

• Stable macro-economic framework conditions– Disciplined fiscal policy– Competition policy encouraging

innovation– Low taxes– An open economy

• Socio-cultural framework conditions– Egalitarian culture with high social

mobility– High wages for blue collar work gives

impetus towards innovation (robots, internet banking)

– High educational levels brings flexibility and labor mobility

– An efficient public sector helps industry– A trustworthy welfare system reduces

risk– Political and social stability gives trust

Page 33: Koch taftie-measuring the effects of research

33 Norwegian Ministry of Education and Research

-1,00000 0,00000 1,00000

Market learning

10,00

20,00

30,00

40,00

Denmark

Germany Finland

Sw edenUK

Norw ay

Slovenia

Spain

Hungary

France

Austria

Be lgium

Czech R

Sw itzerlan

Turkey

GreeceIta ly

Netherland

Poland

Portugal

Slovak R

Ire land

Europe: market learningGDP/ capita

Source:Fourth European Working Conditions SurveyOECD

Understanding socio-cultural framework conditions

• European Working Conditions Survey (Translearn)The correlation between GDP and employer autonomy

Åge Mariussen

Page 34: Koch taftie-measuring the effects of research

34 Norwegian Ministry of Education and Research

We must understand the heterogeneity of innovation systems

• Industrial structure– Large vs. small companies– Low tech vs. high tech– Traditional industries vs. services

• The knowledge system– What kind of universities?

• Research? Polytechnics? Humboldtian?– Other types of research institutions

• Institutes, government labs, academies• Globalization and integration in the world economy• Political stability and social welfare

Page 35: Koch taftie-measuring the effects of research

35 Norwegian Ministry of Education and Research

European innovation scoreboard

• A praiseworthy effort to enlarge the scope of innovation indicators

• But the selection of indicators is slanted towards R&D intensive activities

• The same for the weighing of the indicators underpinning the composite indicator

Norway

Page 36: Koch taftie-measuring the effects of research

36 Norwegian Ministry of Education and Research

EIS 2008

This figure reflects a specific view of innovation as much as it does the facts on the ground

Page 37: Koch taftie-measuring the effects of research

37 Norwegian Ministry of Education and Research

Numbers must be accompanied by stories

• Data is of no value unless they are given in a meaningful context.

• Still: The EU keeps the Innovation Scoreboard but plans scrapping the Trend Chart on Innovation, the tool that interprets the statistics in a national context.

Page 38: Koch taftie-measuring the effects of research

38 Norwegian Ministry of Education and Research

Measuring policy measures

Page 39: Koch taftie-measuring the effects of research

39 Norwegian Ministry of Education and Research

From needs to ex-post impact assessments

• Needs: Problems to be solved• Objective: Outcomes intended to be

achieved• Inputs: Resources mobilized• Processes: Procedures and activities

employed• Outputs: Products and accomplishments• Outcomes: Changes caused by intervention

– Results: Immediate changes for addressees

– Impacts: Longer term socio-economic consequences

OECD DSTI/STP/TIP(2009)18 Henri Delanghe et al

Old focus

New focus

Page 40: Koch taftie-measuring the effects of research

40 Norwegian Ministry of Education and Research

The use of evaluations

• Traditional qualitative evaluations are important– The generate insight into

• Research quality• The organization of

research• Collaboration with other

research institutions– Say little about the effects on

society– Say little about how the

research agenda is shaped through interaction between research and society

Page 41: Koch taftie-measuring the effects of research

41 Norwegian Ministry of Education and Research

Using quantitative measurements for industry oriented policy measures1. The direct effects on the companies participating

1. Economic additionality (Company profits over time)

2. Behavioral additionality (learning, absorptive capacity, ability to change)

2. The direct effects on other companies and organizations1. Spill-over effects on suppliers

(innovation and profits)2. Spill-over effects on customers

(innovation, productivity and profits)

3. Indirect societal effects1. Profitable companies will expand and buy more goods and

services from others2. Their products and services increase productivity in other

parts of society3. Their competences are used by others

(through for instance mobility of personnel)

Incre

ased

com

ple

xity

an

d d

ifficu

lty

Page 42: Koch taftie-measuring the effects of research

42 Norwegian Ministry of Education and Research

Intended and unintended consequences

• We develop policy instruments for a lot of reasons:– To solve a problem– To gain political visibility and influence– Because the Finns are doing it– To make the minister look good

• Policy instruments have a dynamic of their own– Based on the competences, interests

and initiatives of the people involved(They may use the instrument for a new purpose)

– They may have unexpected positive and negative effects elsewhere in the system

• Policy measures outside the realm of research and innovation may have a positive impact on innovation– Social security

Formal objective

Informal objective

Unintended effect

If you want to ascertain the success of a measure, you have to take all of these into consideration

Policy instrument

Page 43: Koch taftie-measuring the effects of research

43 Norwegian Ministry of Education and Research

The role of risk

• Market failure: “When private rates of return, due to high risk, uncertainty or ‘externalities’, may be sufficiently low to deter private investment in areas of crucial long-term national interest.”

• Public measures must allow for higher risk.

• Still: Policy makers ask for high rate of returns!

Page 44: Koch taftie-measuring the effects of research

44 Norwegian Ministry of Education and Research

RCN’s user-oriented R&D programs

• To measure the effects of the program participations in the

• Research done by Møreforskning in collaboration with the Research Council of Norway since the middle of the 1990s.

• Led by professor Arild Hervik.• Based on data from the

databases of the RCN and interviews with the companies

Page 45: Koch taftie-measuring the effects of research

45 Norwegian Ministry of Education and Research

To make sure that the right projects are supported

• Good projects that would have been implemented anyway should not get support

• Low quality projects with no chance of success should be kept out

• You should support good projects where RCN support is vital for its implementation (Input additionality)

Page 46: Koch taftie-measuring the effects of research

46 Norwegian Ministry of Education and Research

Project selection aimed at high social effects (The Jaffe Model)

Page 47: Koch taftie-measuring the effects of research

47 Norwegian Ministry of Education and Research

Understanding the social dynamics of policy instrument development

• An evaluation of the success of a policy instruments must also include an analysis of its social and political context

• I.e.: we can not only mechanistically measure input and outcomes. We must look into the box of policy development.

• Interim evaluations

Page 48: Koch taftie-measuring the effects of research

48 Norwegian Ministry of Education and Research

Traditional view of evidence based policy learning

Independent researchers or experts

Civil servants

Politicians

The wall of disinterested objectivity

Page 49: Koch taftie-measuring the effects of research

49 Norwegian Ministry of Education and Research

An interactive learning arena

Independent researchers

(experts)

Civil servants(experts)

Politicians(experts)

Stake-holders

(experts)

They are all experts in their field, some in technology, others in policy

Page 50: Koch taftie-measuring the effects of research

50 Norwegian Ministry of Education and Research

The ultimate challenge

Analysis of the national innovation system

+ policy instrument evaluations

policy mix reviews

– Green papers– White papers– OECD and EU

innovation policy country reports

VINNOVA’s Portfolio Approach

In order to evaluate public initiatives aimed at promoting innovation, it is necessary to focus on the combined impacts of public efforts.

Page 51: Koch taftie-measuring the effects of research

51 Norwegian Ministry of Education and Research

Conclusions

• There is no simple, mathematical model that can be used to measure all effects of research and innovation.

• We need to combine methodologies, both quantitative and qualitative.

• We are all experts here, and this can not be left to the economists or statisticians

• We have to build common learning arenas, for measuring and for developing an understanding the context of innovation