Innovation surveys: design, implementation, lessons learnt Micheline Goedhuys

Innovation surveys: design, implementation, lessons learnt

Micheline Goedhuys

June 11 2008 DEIP, Amman June, 10-12 2008 2

Why do we need to measure innovation?

Scarcity of data in general, on innovation in particular

Lack of policy tools for benchmarking

Insufficient monitoring and evaluation of policies

Nature of innovation calls for firm-level information

Mine


Structure of session:

1. Conceptual background

2. Experiences with innovation surveys

3. Methodological aspects

4. Use of innovation survey data

5. Opportunities and limitations to innovation data collection: key issues



linear view that science, research and discovery underlie innovation (science push)

innovation measured by science indicators: R&D engineers patenting bibliometrics, publications, citation indices

surveys (USA, 1960s) collecting R&D, patent data;



End 1980s, 1990s ‘activity approach’: investigating the ‘black box’ innovation results from interaction firm-market,

learning, feedback (chain-link model of Kline and Rosenberg 1986)

need for indicators capturing non-R&D activities and incremental change

development of surveys asking firms about their innovation process



Harmonisation of survey efforts in the

‘Oslo Manual’, 1992, 1997, 2005 basis for Community Innovation Surveys innovation is measured as :

an activity (R&D, industrial design, acquisition of machinery, external technology, training) and

an output (introduction of product or process innovations)

Features: new-to-the-firm, significant improvements


2. Experiences : CIS

CIS-1: 1990-92; first regional effort to collect innovation data; 13 European countries,

CIS-2: 1994-1996; 17 countries CIS-3: 1998-2000; more firms, more questions,

services, organisational change, 29 countries CIS-light: 2000-2002, limited set of questions, 18

countries CIS-4: 2002-2004: 29 countries, organisational

innovation and effect CIS-2006: 2004-2006; 29 countries; no data

available yet CIS-2008: 2006-2008


2. Experiences : Latin America

Need of information to monitor the impacts of economic reforms (trade liberalisation, privatisation, deregulation, FDI,etc).

Source: Crespi, 2007

ARGENTINA CHILE COLOMBIA MÉXICO VENEZUE

LA

Survey Number

I I I I I

Reference period

1992-1996 1994-1995 1993-1996 1994-1996 1994-1996

Collection period

1997 1995 1997 1997 1997

Agency responsible for survey

INDEC-SECYT

INE-SETPI COLCIENCIA

-DNP INEGI-

CONACYT OCEI



Specific nature of innovation in Latin American countries: Importance of incremental innovation;

organisational and marketing innovation; Importance of innovation embodied in machinery

and equipment (dissemination) Less private and more informal R&D Fragmented flows of information



Need for changes to the survey instrument : Bogotá Manual to complement OSLO Manual. from innovations to firm-level innovative

activities and technology efforts human resources, capabilities enlarged data need on organisational, delivery

and design innovations lack of centralised agency, different questionnaires

and sampling methodologies



Second wave of Innovation surveys:10 countries; 2000-2001; Argentina, Brazil, Chile,

Cuba, Ecuador, Mexico, Panama, Peru, Trinidad & Tobago and Uruguay

more of uniformity but without common questionnaire and sampling methodologies

revision of Bogotá Manual and Annex to Oslo Manual (2005).



Third wave of Innovation surveys: 5 countries; 2003-2005; Argentina, Brazil, Chile,

Colombia and Uruguay A lot of exit and a core group of countries with

“consolidated” routines (but still with institutional problems and financial issues).

ECLAC-RICYT-OAS network (2006) to create a harmonized “core” questionnaire (plus access to micro-data)


2. Experiences : Asia, Africa

In Southeast Asia: Malaysia (3), Taiwan (1), Singapore (1), Thailand

(2), China, India…

In Africa: South Africa (2) Planning to conduct an innovation survey in 20+

countries (NEPAD survey)


2. Agenda:

ongoing debate to design innovation surveys to the context of developing countries

concept of innovation : organisational, packaging, delivery, design innovations, waste management techniques, …

trade off between country/regional design and benchmarking options

increasing policy relevance inclusion of services and resource-based sectors


3. Methodological aspects : questionnaire content

Basic information: name, location, industry, ownership, year established…

Firm performance: sales, employment, … Innovation activities: Investment, Training, intra-

mural and external R&D, …and expenditures Innovation outputs (product/process/organisational) Sources of information for innovation Cooperation for innovation Government policy or incentives affecting

innovation


3. Methodological aspects : questionnaire content

Objectives, goals or reasons for innovating

Impact of innovations on firm performance

Obstacles to innovation



Organisation: national statistics agency, MOST, universities, consultants

Reference period: 2 or 3 years (mostly 3) Participation: voluntary, compulsary (in Latin

America)


0 1 2 3

AustraliaCanada

European-17 (CIS)HungaryMexico

New ZealandPoland

Slovak RepublicSouth KoreaSwitzerland

Turkey

ArgentinaBrazilChile

ColombiaMalaysiaRomania

RussiaSingapore

SloveniaSouth Africa

TaiwanThailandUruguay

Venezuela

NSA MOST/GOVT UNIV/INST CONSULT


0

1

2

3

4

5Nu

mbe

r of Y

ears



Survey modalities: postal, PTEF follow up, personal interview, telephone interview, online questionnaire, CATI

Sector coverage: initially manufacturing, increasingly services, resource based industries

Firm size: cutoff points: 5, 10, 20 or 50 workers


4. Use of innovation surveys

by academics and researchers Innovation and firm performance Identify determinants/constraints to innovation Innovation strategies Regional and country studies Industry studies Innovation patterns over time Developing innovation indicators: measurement

issues


4. Use of innovation surveys

for policy making:

Indicators for benchmarking Mapping innovation ; innovation in new sectors Assessing trends Monitoring specific policy instruments


Uses 20 Indicators Cross-country comparisons, industry comparisons changes over time consensus on policy action

uses CIS based indicators % SMEs with in-house innovative activities % SMEs that collaborate on innovation total innovation expenditures as % sales % new-to-market products/sales % new-to-firm products/sales

4. Example: European innovation scoreboard


5. Opportunities and limitations

Innovation surveys have become key research inputs of modern innovation studies (Crespi, 2007)

the use of CIS data in academic research

0

5

10

15

20

25

30

35

40

45

50

1 2 3 4 5 6 7 8 9 10 11 12 13

years since reference year

am

ou

nt

of

pu

blicati

on

s

cis1

cis2

cis3



Heterogeneity across questionnaires and methodologies remains and is even on the rise due to broadening concept of innovation, scope, …

Lots of country studies, little cross-country comparisons in developing countries

This limits the use of survey data as benchmarking tool (e.g. Crespi, 2007)



On the questionnaire: Need for the development of harmonized

guidelines with a core set of questions Optional policy-relevant questions can be added

for policy monitoring Methodology for country benchmarking:

Preferably common sampling methodology: size cut-off point, industry coverage, …

Compulsory common (length of) reference period, and participation mode



Dissemination of non-aggregated micro-data is crucial

Assessing trends: need for panel data Need of involvement of stakeholders from the start Need for strong coordination mechanism


Useful links:

For a download of the CIS-4 questionnaire:

http://www.berr.gov.uk/files/file9688.pdf Oslo Manual:

http://epp.eurostat.ec.europa.eu/cache/ITY_PUBLIC/OSLO/EN/OSLO-EN.PDF Bogotá manual:

http://www.ricyt.edu.ar/interior//difusion/pubs/bogota/bogota_eng.pdf NEPAD study: http://www.nepadst.org/doclibrary/pdfs/innopolicy_aug2004.pdf

Documents

Innovation surveys: design, implementation, lessons learnt Micheline Goedhuys