www.uis.unesco.org

Innovation Surveys:Advice from the Oslo Manual

National training workshopAmman, Jordan

18-20 October 2010

www.uis.unesco.org

Ch 8 OM - Innovation Survey Procedures

Guidelines on central elements for the collection and analysis of innovation data;

Following these guidelines will generally lead to comparable results over time and across countries;

Particular circumstances may require other methodology comparability should be in mind.

www.uis.unesco.org

Populations

The target population:The target population:

• Innovation surveys should refer to innovation activities in the business enterprise sector (goods-producing and services industries);

• It should include, at a minimum, all statistical units with at least ten employees.

www.uis.unesco.org

Populations

The frame population:The frame population:

• The units from which a survey sample or census is drawn form the frame population;

• It is based on the last year of the observation period for surveys;

• Ideal frame = up-to-date official business register established for statistical purposes NSOs;

• If the register forms the basis for several surveys (innovation, the R&D, general business), the information collected in the innovation survey can be restricted to issues specific to innovation.

www.uis.unesco.org

Survey methods

Mandatory Mandatory surveys increase response rates;

Census or sample survey?Census or sample survey?

• Sample surveys should be representative of the basic characteristics of the target population (industry, size, region) a stratified sample is necessary;

• Census, though costly, might be unavoidable in some cases (legal requirement, small frame population, inclusion of all units in the frame with a certain number of employees).

www.uis.unesco.org

Survey methods

Domains Domains (sub-populations) are subsets of the sampling strata;

• Potential sub-populations: industry groupings, size classes, regions, units that engage in R&D and innovation-active;

• Guidelines:» Statistical units and classifications should be the same;

» Methods used to calculate results for subsets should be consistent with those used for results from the main sample.

» Deviations in data treatment or differences in the quality of the results from the domains should be documented.

www.uis.unesco.org

Survey methods

Sampling techniques:Sampling techniques:

• Stratified sample surveys (reliable results): based on the size and principal activity of the units;

• Sampling fractions should not be the same for all strata: the sampling fraction of a stratum should be higher for more heterogeneous strata and for smaller strata.

Cross-sections: Cross-sections: standard approach - a new random sample is drawn from a given population for each innovation survey;

Alternative/supplementary approach: panel datapanel data.

www.uis.unesco.org

Survey methods

Suitable respondents:Suitable respondents:

• A variety of methods can be used to conduct innovation surveys, including postal surveys and personal interviews;

• Choosing the unit’s most suitable respondent is particularly important in innovation surveys (questions are very specialised and can be answered by only a few people in the unit);

• It is highly recommended to make a special effort to It is highly recommended to make a special effort to identify respondents by name before data collection identify respondents by name before data collection starts.starts.

www.uis.unesco.org

Survey methods

The questionnaire:The questionnaire:• Pre-test each questionnaire before fieldwork;

• Keep it as simple and short as possible;

• Pay attention to the order of the questions;

• Questions on a number of qualitative indicators can use either a binary or an ordinal scale;

• In the case of international innovation surveys, special attention should be given to the translation and design of the questionnaire;

• Short-form questionnaires for units with little/no innovation activity previously reported.

www.uis.unesco.org

Survey methods

Combination of Innovation and R&D surveys:Combination of Innovation and R&D surveys: Reduction in the overall response burden of the reporting

units; Scope for analysing the relations between R&D and

innovation activities at the unit level; Efficient method of increasing the frequency of innovation

surveys; Country experiences indicate that it is possible to obtain

reliable results for R&D expenditures; Longer questionnaire; Units not familiar with the concepts of R&D and innovation

may confuse them; The frames for the two surveys are generally different.

www.uis.unesco.org

Survey methods

Guidelines for conducting combined surveys:Guidelines for conducting combined surveys:

• The questionnaire should have two distinct sections;

• Individual sections for R&D and innovation should be smaller than in separate surveys;

• Comparisons of results from combined surveys with those from separate innovation surveys should be done with care, and surveying methods should be reported;

• Samples to carry out such surveys should be extracted from a common business register.

www.uis.unesco.org

Estimation of results

Weighting methods:Weighting methods:

• The simplest one is weighting by the inverse of the sampling fractions of the sampling units, corrected by the unit non-response;

• If a stratified sampling technique with different sampling fractions is used, weights should be calculated individually for each;

• Commonly based on the number of enterprises in a stratum.

• In international and other comparisons, be sure that the same weighting method is used.

www.uis.unesco.org

Estimation of results

Non-response:Non-response:

• Unit non-response: a reporting unit does not reply at all;

• Item non-response: refers to the response rate to a specific question and is equal to the percentage of blank or missing answers among the reporting units;

» Disregarding missing values and applying simple weighting procedures based only on the responses received implicitly assumes that non-respondents are distributed in the same way as respondents biased resultsbiased results;

» Possibility: imputation methodsimputation methods to estimate missing values on the basis of additional information.

www.uis.unesco.org

Presentation of results

Descriptive analysis:Descriptive analysis:• Description of the

statistical units in terms of their innovative or non-innovative activities without drawing any conclusions about the underlying survey or target population;

• No generalisation of the results;

• Unit non-response rate is of minor importance.

Inferential analysis:Inferential analysis:

• Drawing of conclusions about the target population;

• The results should give a representative estimation of the situation;

• Weighted results;

• Unit non-response rate is very important.

www.uis.unesco.org

Presentation of results

Errors:Errors:

• Random errors due to the random process used to select the units;

• Systematic errors containing all non-random errors (bias);

» To get at least an idea of the variance for the results, it is recommended to calculate both (average) values for innovation indicators and also their coefficients of variation and/or confidence intervals;

» Results presentation should contain: metadata (including information on data collection procedure), sampling methods, procedures for dealing with non-response and quality indicators.

www.uis.unesco.org

Frequency of data collection

Innovation should be conducted every two years;

Where this is not economically feasible, a frequency of three or four years may be chosen;

Surveys must specify an observation period for questions on innovation;

• The length of the observation period for innovation surveys should not exceed three years nor be less than one year.

www.uis.unesco.org

Annex A - 5. Methodological issues for developing country contexts

Information system specificities:

• Relative weakness of statistical systems

• Absence of linkages between surveys and data sets; lack of official business registers information from other surveys cannot be used;

• Involvement of NSOs;

• When lacking, basic variables about firms’ performance can be included in the innovation survey - to enable further analysis.

www.uis.unesco.org

Annex A - 5. Methodological issues for developing country contexts

General methodological considerations:

• Questionnaire design: » Sections can be separated to allow different persons in the

firm to reply them;

» Guidance / definitions;

» Language and the translation of technical terms;

• Survey application:» In-person;

» Trained personnel.

www.uis.unesco.org

Annex A - 5. Methodological issues for developing country contexts

General methodological considerations:

• Frequency:» Every three to four years (e.g., timed to CIS rounds);

» Try to update a minimum set of variables every year;

• The purpose of surveys needs to be clearly stated and the questions clearly formulated;

• An adequate legislative base for the collection of innovation statistics can help ensure the success of such an exercise;

• The results should be published and distributed The results should be published and distributed widely.widely.

www.uis.unesco.org

Thank you!

http://www.uis.unesco.org

m.schaaper@uis.unesco.org

CIS: http://www.oecd.org/dataoecd/37/39/37489901.pdf