1
1
Assessing the impact of combined
data collection strategies and of
response rate on the measurement of
innovation activity in Norway
Lars Wilhelmsen, Statistics Norway
Nordisk Statistikermøte, Bergen 14-17. August 2013
2
Background for the project
• Norway scores highly on several core economic indicators:
– GDP per capita
– Trade balance
– Unemployment
• But below the EU average on certain R&D and innovation comparisons
based on the CIS
3
Internationally: Enterprises with product and/or
process innovation activity, CIS 2010
0
10
20
30
40
50
60
Latv
ia
Ungarn
Rom
ania
Pole
n
Slo
vaki
a
Bulg
aria
Lita
uen
Malta
Spania
Norg
e
EU
27
Nederland
Kro
atia
Fra
nkr
ike
Slo
venia
Tsje
kkia
Italia
Danm
ark
Irla
nd
Øste
rrik
e
Sverige
Kypro
s
Fin
land
Luxem
burg
Belg
ia
Estla
nd
Port
ugal
Tyskl
and
Prosent avalle foretak
0
10
20
30
40
50
60
70
Perc
en
t o
f p
op
ula
tio
n
4
Background for the project
• Norway scores highly on several core economic indicators:
– GDP per capita
– Trade balance
– Unemployment
• But below the EU average on certain R&D and innovation comparisons
based on the CIS
• And our share of innovators have been declining
5
Norway: Innovation surveys, 2004-2010
0
5
10
15
20
25
30
35
2004 2005 2006 2007 2008 2009 2010
Perc
en
t o
f en
terp
rices i
n t
he p
op
ula
tio
n
Product and/or process (PP-) innovators
PP-innovators with in-house R&D
6
Background for the project
• Norway scores highly on several core economic indicators:
– GDP per capita & GDP per capita growth
– Trade balance
– Unemployment
• But below the EU average on certain R&D and innovation comparisons
based on the CIS
• And our share of innovators have been declining
• This may be seen as a paradox if we expect a positive relationship
between innovation scores and economic success
• It has been questioned by policy actors, businesses and media whether
the Norwegian scores on common innovation indicators are too low and
does not reflect reality
About the Norwegian Innovation survey
• Modeled closely on the standardized Community Innovation
Survey (CIS) questionnaire developed by Eurostat
• The CIS and the Business Enterprise R&D surveys have
been combined in Norway since 2001
– First part: Questions on R&D
– Second part: Questions on innovation
• Most countries have separate surveys
– (incl. Sweden, Finland and Iceland)
– while some have combined surveys (incl. Denmark)
• Mandatory survey with enforcement
– High response rate (>95%)
Combined vs. separate surveys in general
• Timeliness of results
• NSO survey cost
• Response burden for the enterprises
• Data linkages and data coherency
• Weighing different user needs
– International benchmarking
– Researchers
– Policymakers
– Policy implementing agencies
– National accounts
9
Methodological exercise: A separate CIS
• We wanted to explore two separate (but related) issues
– Combined R&D and innovation survey vs. a separate CIS Does the presence of detailed R&D questions impact the reporting of innovation?
– The importance of response rate; a mandatory vs. a voluntary survey Overall, response rates are on the rise (most countries in the 75-85% range)
But there are countries where they are much lower (down to ~25%)
Other countries have seen a drop in innovation rates after making their survey
mandatory
There is some doubt about the comparability of data with and without non-
response adjustment
• Thus, we sent an innovation-only survey to two distinct extra samples
– One with and one without a mandatory response requirement (and enforcement)
– In addition to the regular survey, so three samples from the same population
• Final response rates: – 96% (combined, mandatory), 95% (separate, mandatory), 41% (separate, voluntary)
10
Results: Product and/or process innovation,
combined vs. separate survey
0 %
5 %
10 %
15 %
20 %
25 %
30 %
35 %
40 %
Combined Survey Mandatory Innovation Survey
Sample
Pe
rce
nt
of
po
pu
lati
on
11
Possible explanations for the findings
• Different type of respondents? – We know surveys that ask for lots of ”hard numbers” are more likely to be
answered by accounting or similar.
– We know some respondents see the CIS portion as an appendix to the R&D survey when they are combined
• Lower threshold for reporting an activity as innovative? – Asking for a too detailed breakdown of certain R&D activities may
suppress reporting of other R&D activity that does not fit the available categories Interestingly, a significantly higher share of enterprises report having
performed R&D in the separate innovation survey.
– A similar mechanism may be also in play for non-technological innovation when removing the R&D part of the survey
– R&D-focused respondents may answer innovation questions based on R&D outcomes in the enterprise rather than from an overall perspective.
• Respondents finding the survey more relevant and applicable to them, and thus answering more diligently
12
Product and/or process innovation,
mandatory vs. voluntary survey
0 %
5 %
10 %
15 %
20 %
25 %
30 %
35 %
40 %
45 %
50 %
Mandatory Innovation Survey Voluntary Innovation Survey
Sample
Perc
en
t o
f p
op
ula
tio
n
13
Possible explanations for the findings
• Readily apparent self-selection in the voluntary sample
• We did not have the possibility of doing a non-response
survey
• Internationally (some) countries with a very low response
rate and a high share of innovators, further increase this
after performing non response adjustments
– Hypothesis: Non response adjustment on our voluntary sample
would yield higher results than our mandatory sample Simplification, different respondents and the possibility of interacting
with the interviewer may further strengthen the trends observed when
removing the detailed R&D questions
In which case we would get deceptively high innovation rates with
deceptively low sampling errors
Follow-up interviews with respondents
• Organized as semi-structured qualitative interviews
– Thoughts on response burden Separate surveys perceived as a higher response burden
– But only if receiving both
Most government mandated reporting seen as an unnecessary chore – R&D perceived as more difficult and less “fun” to answer
– Some respondents eager to report on innovation
On reflection, little opposition to separating the surveys
– Factors influencing the results The concept of R&D is confusing
– Seen as a somewhat alien “white lab coat” activity
– Also limits the understanding of what constitutes a reportable innovation
Chance that different people would respond to the surveys if separated – Several respondents thinks this will lead to better reporting
The shared questionnaire is seen as being too long and convoluted – “Invites” cases of “substantial none-response”, just to get it over with
– Questions on innovation, being located after R&D, may suffer the most
14
15
Lessons learned
• Context matters when attempting to measure “hard to quantify social phenomena” – A combined survey seems to work well for industries where a traditional
linear innovation model with R&D-driven innovations is applicable.
– A separate survey may give more correct results for industries, in particular in services, with low R&D intensity, or where innovations are developed in an incremental/iterative process
• Respondents do NOT have a clear and accurate understanding about what constitutes an innovation
• Norwegian innovation rates are probably somewhat underreported compared to many other CIS-implementing countries – But we don’t know exactly by how much
– And we don’t know how these findings would impact a comparison against any particular country
• Does not say anything about the observed declining trend in the Norwegian innovation rates
• It is still unclear which results are “better” or most useful
16
So, now what?
• Doing a full scale separate innovation survey for 2013
– Gauging the “correct” level of innovation compared to the regular
combined CIS 2012
• Will revisit the shared vs. combined debate with core
users once these results are available
• There are both benefits and drawbacks with combined
surveys
– Impossible to fulfill all users’ needs all the time
• Some compromises will have to be made either way
– Limited room to increase total cost
– Unwillingness to increase overall response burden
Thank you for your attention!
17