Brian MacAulay "Evaluando el impacto de la Innovación"

Innobasque - Evaluating the Impact of InnovationBilbao, 22nd March 2011

UK Innovation Index: Measuring Innovation that mattersBrian MacAulay – Director Innovation Index, NESTA

What should an Innovation Index set out to do?

1

2

3

Measure the level of investment in innovation

Link the impact of innovation to GDP

Measure innovation in UK businesses

Underlying questions

Is investment in innovation in the UK rising or falling?

Is the UK investing as much in innovation as other developed countries?

How does this translate into economic growth and what does it do for the productivity gap?

How innovative are businesses, and how does this compare across sectors?

4 Assess wider conditions for innovation in the UK

How good a place is the UK to innovate compared to other developed economies?

Goals

Innovation and Economic Growth – NESTA’s approach

There are four factors that we can focus on to promote innovation across the economy

Rationale

• Innovation is the most important driver of economic growth

• Minority of high-growth, innovative businesses vital to the economy

• Demand from government to help foster economic growth

Markets

Capital

BIS’s Annual Innovation Report

The Innovation Index has been officially recognised in BIS’s Annual Innovation Report

The measurement of innovation expenditures- Investment in Intangible Asset Survey

1

2

3

Measure wider spending on knowledge assets beyond R&D

Measure level of in-house and external expenditures

Enables estimate of depreciation rates for intangible assets

What was done Key Findings Comparison to Pilot data

• Questions asked for detailed data on the 6 categories of intangible assets: R&D, Design, Training, Software, Branding and Organisational development

• Respondents asked to report the level of expenditure conducted by own staff (including associated costs such as office facilities etc)

• Respondents asked to report average life-span of benefits from each asset type

• 50% of UK firms invest in non-R&D assets compared to 8% for R&D

• Level of investment in 2009 was £39bn* with largest component is software (£11.3bn)

• Services invest significantly more software, training and branding than manufacturing firms

• 55% of spending in-house compared to 45% purchased externally

• Only in Branding are external purchases significantly higher than in-house (70%)

• R&D has longest average benefit of 4.6 years

• Average of others is 3.2 years• Estimated depreciation rates of 23%

for R&D and 40% for other intangibles

• Data for Training, Software, R&D and Branding are comparable with Pilot findings

• Issues remain for Design and Organisational development possibly due to sampling, and recessionary effects or CHS assumptions in pilot

• Pilot uses CHS depreciation rates which are based on a small literature

• IAS finds Branding and Training relatively short lived (same as CHS)

What is new in survey

* £39bn is total weighted expenditure from IAS

The UK invested £137bn in innovation in 2008

21.6

27.1

3.8

23.3

16.0

Total 137.4

15.0

30.7

Investment in innovation, ₤bn, 2008

Training & skills development

“Go-to-market”

Design

R&D

Other (copyright, etc)

Software development

Organisational innovation

13.8% of private sector

output

The Index also provides historical innovation investment data going back to 1990

Source: Innovation Index

0%

1%

2%

3%

4%

5%

6%

7%

8%

9%

10%

11%

12%

13%

14%

15%

16%

Training

Org. Development

Design

Software

Go to Market

R&D

% of business output

Software

A wider view of innovation investments casts new light on the so-called “Innovation Gap”

Germany

US

Finland

Italy

Canada

UK (2009)

France

Business R&D as a share of GDP, 2008

Sources: R&D: OECD, except UK: ONS/Imperial,

2.8%

2.7%

2.0%

1.9%

1.3%

1.1%

1.3%

1.0%

Japan

A wider view of innovation investments casts new light on the so-called “Innovation Gap”

Germany

US

Finland

Italy

Canada

UK (2009)

France

Business R&D as a share of GDP, 2008

Sources: R&D: OECD, except UK: ONS/Imperial,

2.8%

2.7%

2.0%

1.9%

1.3%

1.1%

1.3%

1.0%

Japan

Investment in innovation as a share of Market Sector Gross Value Added 2006*

Italy

Germany

UK (2008)

France

Finland

Canada (2005)

US

13.8%

12.0%

9.8%

9.0%

7.9%

7.1%

5.0%

Non Farm Business Sector

Whole Economy

Non Financial Business Sector

Market Sector

Market Sector

Market Sector

Market Sector

11.1% Whole EconomyJapan (2005)

Software developmentIncludes Training & skills development; Organisational Improvement, Market research & advertising Hidden innovation

Traditional innovation Includes R&D, Design, Mineral exploration & copyright development

The measurement of innovation should be linked to the measurement of economic growth

Closing the UK’s productivity gap with other developed countries is an important goal of economic policy

Productivity (GDP per hour worked, current PPP), UK=100

100

110

120

130

140

1991

US

UK

France

The productivity gap remains significant

089293 94 95 96 97 98 992000 01 02 03 04 05 06 07

90

Source: ONS

Germany

09

0 1 2 3

Innovation contributes around two-thirds of productivity growth

Contribution to productivity growth, %, 2000 - 2008

7%

30%

23%

40%

Innovation investment TFP

Labour qualityCapital

investment

2.24

0.90

0.51

0.67

0.16

Total

Total factor productivity (wider benefits of

innovations)

Investment in innovation

Capital investment

Labour quality

Source: Innovation Index

Index provides breakdown of sector Labour Productivity growth

-4.00 -3.00 -2.00 -1.00 0.00 1.00 2.00 3.00 4.00 5.00

Series1

Average Annual Labour Productivity Growth 2000 - 2007

LPG

Average labour productivity growth (%pa)

Agriculture, Fishing & Mining

Manufacturing

Utilities

Construction

Retail, Hotels & Transport

Financial Services

Business Services

Index provides breakdown of sector Labour Productivity growth – importance of ICT

-4.00 -3.00 -2.00 -1.00 0.00 1.00 2.00 3.00 4.00 5.00

Series1


LPG



Manufacturing

Utilities

Construction


Financial Services

Business Services

Index provides breakdown of sector Labour Productivity growth – importance of ICT

-4.00 -3.00 -2.00 -1.00 0.00 1.00 2.00 3.00 4.00 5.00

Series1Series1


LPGICT



Manufacturing

Utilities

Construction


Financial Services

Business Services ICT is important to financial services

But not so important for manufacturing

Index provides breakdown of sector Labour Productivity growth – Intangible capital

-4.00 -3.00 -2.00 -1.00 0.00 1.00 2.00 3.00 4.00 5.00

Series1




Manufacturing

Utilities

Construction


Financial Services

Business Services


-4.00 -3.00 -2.00 -1.00 0.00 1.00 2.00 3.00 4.00 5.00

Series1Series1




Manufacturing

Utilities

Construction


Financial Services

Business Services

The Index reinforces the view that intangibles are important for service sector productivity

But, they are also important in driving productivity in manufacturing – servitisation?


-4.00 -3.00 -2.00 -1.00 0.00 1.00 2.00 3.00 4.00 5.00

Series1Series1




Manufacturing

Utilities

Construction


Financial Services

Business ServicesThe dampened effect in financial services as labour hours rise greatly

Innovation is important in the key sectors – Business services, finance and manufacturing

-4.00 -3.00 -2.00 -1.00 0.00 1.00 2.00 3.00 4.00 5.00

Series1Series1




Manufacturing

Utilities

Construction


Financial Services

Business Services

Innovation is important in the key sectors – Business services, finance and manufacturing

-4.00 -3.00 -2.00 -1.00 0.00 1.00 2.00 3.00 4.00 5.00

Series1 Series1Series1




Manufacturing

Utilities

Construction


Financial Services

Business Services

In financial services, although productivity is lower, innovation through TFP plays the large part in driving productivity growth – 90%.

Manufacturing driven by within industry intangible investments

MEASURING INNOVATION IN THE PUBLIC SECTOR

We developed a survey tool aligned with this framework, and also based on a development of NESTA’s existing private sector index tool:

► Our first priority was to accurately reflect how innovation happens in the public sector.

► Our second priority was to enable future comparability of innovation across the private and public sectors, in the UK and internationally.

• In order to achieve this, we sought to maximise the use of the existing private sector survey tool, and align additional questions with existing relevant international innovation survey instruments where possible.

• We conducted user (cognitive) testing with 7 interviewees (4 Local Govt, 3 NHS) and refined the survey

► We analysed the survey findings based on the framework of public sector innovation.

Background to the pilot survey

Part of NESTA’s ‘Innovation Index’ Programme of work, to develop a new national innovation index.

1. The ambition of the Public Sector Innovation Index is that it will form a part of the national commentary on economic growth.

2. The index includes commentary and indices relating to the impact of innovation on UK economic growth, the conditions for innovation, private sector innovation and public sector innovation. Our work forms an important part of the latter.

3. Our project is a pilot approach to measuring innovation across the public sector using a survey approach.• Based on a survey of two parts of the public sector: the health sector

and local authorities• Objective 1: to help accurately understand the drivers of innovation,

the levels of innovation, and the impact of innovation• Objective 2: to enable comparability across sectors. The survey tool

aligns with NESTA’s Private Sector Index

Participation levels

Sector Health Local Government

Number of organisations in England

388 353

Number interviewed

64 (16%) 111 (31%)

Local Government Breakdown

Total in England

Surveyed

Non-Metropolitan districts 201 63 (31%)

Unitary authorities 56 13 (23%)

Metropolitan districts 36 11 (31%)

London boroughs 33 9 (27%)

Two-tier ‘shire’ counties 27 15 (56%)

Total 353 111 (31%)

Health Sector Breakdown

Eas

t M

idla

nds

Eas

t o

f E

ngla

nd

Lon

don

Nor

th E

ast

Nor

th

Wes

t

Sou

th

Cen

tral

Sou

th E

ast

Sou

th

Wes

t

Wes

t M

idla

nds

York

shir

e &

Hum

ber

Surveyed Total in England

Acute trusts 3 3 2 4 9 2 1 4 4 4 36 (22%) 167

PCTs (and Care trusts) 3 0 0 0 3 2 1 2 4 2 17 (11%) 151

Mental health trusts 3 0 1 0 1 1 0 1 0 1 8 (14%) 58

Ambulance trusts 0 1 0 0 0 0 1 1 0 0 3 (25%) 12

Total 9 4 3 4 13 5 3 8 8 7 64 (16%) 388

Foundation trusts 1 1 2 4 6 2 1 4 3 1 25 (19%) 131

The survey tool can be used by policy-makers and delivery organisations in the following ways:

1. As a dashboard – a lead indicator of performance improvement

2. To provide a body of research evidence to help improve innovation in the public sector

3. To identify where innovation is happening well (without the lag associated with performance monitoring) – to support identification and diffusion of innovation

4. As an organisational awareness and development tool

Our view is that a survey tool has value (in the achievement of the above) and there is demand for its use. Its future potential can best be realised by integrating the lessons from this pilot study and developing it in to a highly engaging online tool, available for use across the public sector

The outputs of the survey provide results that can inform different levels of the public sector

Approach

We have based the survey on the framework of public sector innovation shown below - developed from NESTA’s Innovation Index research to-date

Impact on Performance

Innovation Activity

Innovation Capability

Wider Sector Conditions for Innovation

Impact - describes the impact of innovation activity on an organisation’s performance in terms of impact on outcomes, service and efficiency measures, as well as the context for change:

► Improvement in output KPIs

► Improvement in service evaluation

► Improvement in efficiency

► Improvement context

Wider Sector Conditions for Innovation - describes how well the system in which an organisation operates helps or hinders innovation. There are 4 key innovation ‘levers’ that we investigate:

► Incentives

► Autonomy

► Leadership and culture

► Enablers

Innovation Activity – describes the pipelines of ideas flowing through an organisation and the effectiveness of the associated key innovation activities:

► Accessing new ideas

► Selecting and developing ideas

► Implementing ideas

► Diffusing what works

Innovation Capability – describes the key underpinning organisational capabilities that can sustainably influence innovation activity:

► Management of innovation

► Leadership and culture

► Organisational enablers of innovation


Innovation Activity Innovation Capability


Framework of Innovation inPublic Sector Organisations

The framework acknowledges two key differences between the public and the private sectors:

1. Differences in how value is defined. Innovation in the public sector is assessed through impact on a range of social value as well as economic value indicators.

2. Differences in the systems in which organisations operate. The framework reflects that public sector organisations operate in a range of different systems and assesses the impact of the system conditions on innovation in organisations.

Innovation Activity

Accessing new ideas

Implementing ideas

Selecting and developing ideas

Diffusing what works

This part of the framework provides a view of the current innovation activities of your organisation.

This provides a view of the activities most likely to impact future performance in the short - medium term (1-3 yrs).

Innovation activity can be considered as pipelines of new ideas flowing through your organisation as they are accessed and identified, selected and developed, implemented and diffused.

To gain a clear picture of the overall pipeline, an assessment of this activity should be based on quantitative indicators where possible, but will need to be augmented by qualitative assessment.

Description

The process of accessing and identifying a sufficient number of different types of new ideas from a range of sources

► Volume and types of ideas

► Novelty of ideas

► Sourcing: front line staff , consumers, senior management, research, competitors, sector scanning, delivery partners/collaborators, suppliers, intermediaries

Description

The process of selecting the best new ideas for development; allocating resources and working collaboratively during development

► Selecting the best ideas

► Allocating resources(skills and investment)

► Developing the ideas as a multi-disciplinary team

► Piloting/testing activity

Description

The process of converting developed and tested ideas to fully implemented solutions; allocating appropriate resources

► Embedding and scaling

► Training and investing

► Measuring benefits

► Securing benefits (including intangibles)

Description

The process of sharing and disseminating successful ideas within and outside the organisation

► Disseminating

► Sharing

Innovation Activity

Your organisation’s Innovation Activity is represented by the chart on the top left hand side of your scorecard. Innovation Activity has been categorised into four main process indicators described below.

For each of these four process indicators, a red X shows the corresponding index scores for your organisation, based on our analysis of your survey responses - relative to participating peer organisations. Comparisons can be made between your organisation and the average index scores, as well as the minimum, maximum and spread of responses from similar organisations.



This part of the framework provides a view of the impact innovation is having on your organisation’s current performance.

These indicators are the ones that research suggests that senior management and staff pay most attention to when managing innovation.

Depending on how your organisation views innovation, your innovation activities might fit alongside other improvement initiatives to improve performance. It is important to understand how your organisation views these collective contributions.

It is useful to augment survey findings with organisational performance improvement data.

Improvement in output KPIs

Improvement in serviceevaluation

Improvement in efficiency

Improvementcontext

Description► Improvements in service

evaluation/feedback from consumers over the last 1-3 years

Description► Improvements in output KPIs

over the last 1-3 years► Impact of these on outcomes

Description► Improvements in key

efficiency/productivity indicators over the last year

Description► Understanding of the context for

improvement in impact (degree of challenge)

The impact innovation is having on your organisation’s performance, based on our analysis of your survey responses, is represented by the chart on the top right hand side of your scorecard. Impact on Performance has been broken down into four main impact indicators, as described below.

For each of the four impact indicators, a red X shows the corresponding index scores for your organisation, based on our analysis of your survey responses - relative to participating peer organisations. Comparisons can be made between your organisation and the average index scores, as well as the minimum, maximum and spread of responses from similar organisations.


Description

The behaviours and conditions required for innovation to flourish

► Vision and spirit of senior managers

► Prioritisation of innovation

► Attitudes to risk taking and learning

► Attentiveness to views of users, front-line staff and middle-managers

► Space and capacity for creative thinking

► Term of office for leadership

Description

The critical enablers of innovation activity within the control of the organisation

► Management information

► Connectedness

► Incentives and rewards

► Profile/forums/events

► ICT Infrastructure

► Access to supportand skills (including quality of staff)

This part of the framework provides a view of the Innovation Capability of your organisation.

It contains the key underpinning capabilities that can sustainably influence innovation activity and performance in the medium-longer term (3 years+).

The ability to develop these capabilities is within the control of your organisation.

The assessment of these capabilities is mostly qualitative in nature and can be supported by other data such as staff surveys.

Description

The quality of organisation and planning for innovation activities

► Innovation objectives linked to performance priorities

► Investment intensity

► Innovation governance

► Professional engagement

► Risk management

Leadership & Culture

Management of Innovation

Organisational Enablers of Innovation


Your organisation’s Innovation Capability is represented by the chart on the bottom left hand side of your scorecard. Innovation Capability has been divided into three main capability indicators, as described below.

For each of these three capability indicators, a red line shows the corresponding index scores for your organisation, based on our analysis of your survey responses - relative to participating peer organisations. Comparisons can be made between your organisation and the average index scores, as well as the minimum, maximum and spread of responses from similar organisations.

This part of the framework provides a view of how well the system in which your organisation operates helps it to innovate.

The system contains policy levers that can help or hinder innovation.

These policy levers are outside the control of your organisation but within the control of policy-makers or other influencing bodies.

The framework allows your organisation to provide a view on the effectiveness of the use of these levers to policy-makers.


EnablersLeadership& culture

Incentives Autonomy

DescriptionEffectiveness and alignment of a system of incentives► Demand► Competition► Performance targets► Performance transparency► Accountability to consumers► Recognition & reward► Regulation

DescriptionThe responsibility and freedom to innovate► Responsibility for innovation► Flexibility to shape local strategy► Budget flexibility► Freedom to use rules and guidance► Legislative basis

DescriptionAccess to critical enablers of innovation► Access to transparent comparable

performance data► Access to best practice information

across public and private sectors► Access to innovation funds and

support► Award schemes► Learning from inspections/audits► Adequate IT systems► Access to shared structures

and tools► Peer review processes► Measurement of innovation

DescriptionThe behaviours and conditions required for innovation to flourish► Vision and spirit of innovation► Innovation linked to strategy► Attitudes to risk taking and learning► Attentiveness to views of users,

front-line staff and middle-managers► Attitudes to collaboration/working across

organisational boundaries► Focus on short/medium/long term goals► Quality of new initiatives


Your organisation’s view regarding how well the Wider Sector Conditions help or hinder your ability to innovate - are represented by the chart on the bottom right hand side of your scorecard. These conditions have been divided into four main category indicators, as described below.

For each of these four category indicators, a red line shows the corresponding index scores for the views of your organisation, based on our analysis of your survey responses - relative to participating peer organisations. Comparisons can be made between your organisation and the average index scores, as well as the minimum, maximum and spread of responses from similar organisations.

We are testing findings► With key

policy representatives from DH, CLG, Cabinet Office and BIS

► With survey participants – who will receive a scorecard (shown)

► We will ask survey participants for feedback

Innovation Index Scorecard

These charts show the results f rom the pilot innovation survey that you kindly took part in. We compare your results with all other Acute Trusts participating in the survey (in this case, the number of Acute Trusts participating is 36).For each part of the scorecard, your organisation’s index scores are shown in red compared to your peers participating in the survey.

1. Standard deviation: a measure of the spread of most responses

Mean MaxStandard deviation1 MinMean MaxStandard deviation1 Min

0%

20%

40%

60%

80%

100%Leadership and Culture

Management of Innovation

Enablers of Innovation


Mean Min Max

0%

20%

40%

60%

80%

100%Incentives

Leadership and Culture

Autonomy

Enablers

Wider Sector Conditions

Mean Min Max

0.00

0.20

0.40

0.60

0.80

1.00

Accessing new ideas

Selecting and developing ideas

Implementing ideas Diffusing what works

Innovation Activity

0.00

0.20

0.40

0.60

0.80

1.00

Improvement in output KPIs

Improvement in service

evaluation

Improvement in efficiency

Improvement context


A survey-based innovation index, if developed well, can be useful enabler of innovation in the public sector – in 4 ways

1. As a dashboard – a lead indicator of performance improvement2. To provide a body of research evidence to help improve innovation in the public

sector3. To identify where innovation is happening well (without the lag associated with

performance monitoring) – to support identification and diffusion of innovation4. As an organisational awareness and development tool

Our pilot study has provided useful lessons to improve the tool - through development in to an online version

► Limitations of a telephone-based survey. These include particular limitations of the ability to assess performance impact through a telephone interview approach

► The need for flexibility regarding number of participants, per organisation► How to galvanise participation► The use of the tool in promoting innovation and raising organisational capability

We are consulting on future use of such a tool. In particular:A. Potential for application across different levels: nationally coordinated

surveys; sector-led approaches; a tool for organisations’ useB. Functional requirements of the tool: benchmarking results, research and

organisational developmentC. Whether you agree that a web-based tool should be developed to meet

the above requirements

Should the survey tool be developed further and made available for use across the public sector?

Documents

Brian MacAulay "Evaluando el impacto de la Innovación"