Performance Metrics for the Open Innovation Model - David Scarlatti

T802

MSc in Technology Management

Performance metrics for the Open Innovation model

David Scarlatti

X7922828

February 2010

T802

MSc in Technology Management

Performance metrics for the Open Innovation model

David Scarlatti

X7922828

February 2010

This dissertation is submitted in partial fulfilment of the requirements for the MSc in

Technology Management

David Scarlatti - X792282

Abstract

Innovation is seen as a key element of an organisation’s competitiveness. Members of

senior management teams and managers in charge of Research and Development have

struggled for years to find the most appropriate way to measure innovation. While best

practices for measuring innovation are still under development, a new model for

innovation has emerged, known as Open Innovation. This new model is expanding quickly

and is causing researchers to question traditionally accepted principles. For those who are

embracing the new model, a key question arises: Can we measure innovation in the same

way that we have until now? Open Innovation proponents seem to have paid little

attention to this topic to date. The aim of this research is to produce recommendations

concerning metrics for evaluating the effectiveness of an ‘Open Innovation’ approach to

research. A major objective is to explore the use of performance metrics in research

pertaining to Open Innovation. The validity of existing innovation performance metrics is

examined using the principles of Open Innovation and by applying defined criteria of

applicability. I conclude that while most metrics currently in place remain useful, they need

some adaptation. In many cases, additional supporting metrics, especially those related to

external resources and activities, must be added. A “Decalogue”–outlining the changes

needed in the current set of metrics–is proposed as a useful tool for managers in charge of

evaluating this new approach to innovation. The corollary of the Decalogue is that one

should avoid using a current innovation performance measurement system for the new

Open Innovation model without adapting the relevant metrics.


iii

Table of Contents

Figures ........................................................................................................................................... v

Tables ........................................................................................................................................... vi

Glossary ........................................................................................................................................vii

Chapter 1 Introduction............................................................................................................. 1

1.1 Background on the problem/issue .............................................................................. 1

1.2 Justification for the research ....................................................................................... 1

1.3 Aim and objectives....................................................................................................... 2

1.4 Scope of the research .................................................................................................. 2

1.5 Outline of the dissertation........................................................................................... 3

Chapter 2 Research definition.................................................................................................. 5

2.1 The practical problem/issue ........................................................................................ 5

2.2 Existing relevant knowledge ...................................................................................... 11

2.3 Research questions.................................................................................................... 20

Chapter 3 Methodology ......................................................................................................... 21

3.1 Methods and techniques selected............................................................................. 21

3.2 Justification ................................................................................................................ 21

3.3 Research procedures ................................................................................................. 22

3.4 Ethical considerations................................................................................................ 22

Chapter 4 Analysis and interpretation ................................................................................... 23

4.1 Summary of data collected........................................................................................ 23

4.2 Data Analysis.............................................................................................................. 29

4.3 Interpretation in relation to the research questions................................................. 33

4.4 Interpretation in relation to the aim ......................................................................... 35

Chapter 5 Conclusions............................................................................................................ 37

5.1 Conclusions regarding the research questions.......................................................... 37

5.2 Conclusions regarding the research aim.................................................................... 37


iv

5.3 Further work .............................................................................................................. 38

5.4 Implications of this research...................................................................................... 38

References................................................................................................................................... 40


v

Figures

Figure 1‐1 Scope of this research.................................................................................................. 3

Figure 2‐1 Problem Definition Flow .............................................................................................. 5

Figure 2‐2 The Closed Innovation Model (Chesbrough 2003) ...................................................... 6

Figure 2‐3 The Open Innovation Model (Chesbrough 2003) ........................................................ 7

Figure 2‐4 Reasons for measuring Innovation ............................................................................ 13

Figure 2‐5 Challenges measuring Innovation.............................................................................. 14

Figure 2‐6 Approaches facilitating Innovation Measurement .................................................... 18

Figure 4‐1 The measurement and control process (KerssensvanDrongelen, Cook 1997).......... 23

Figure 4‐2 Balance in Open Innovation (Lichtenthaler 2008) ..................................................... 32


vi

Tables

Table 2‐1 Contrasting Principles of Closed and Open Innovation (Chesbrough 2003)................. 8

Table 2‐3 Contrasting Principles between Closed and Open Innovation (Chesbrough 2003) and

measurement questions ............................................................................................................. 10

Table 4‐1 The dimensions of R&D performance (Ojanen, Vuola 2003)...................................... 25

Table 4‐2 Grouped Innovation metrics ....................................................................................... 26

Table 4‐3 Innovation metrics for individuals............................................................................... 29

Table 4‐4 Innovation metrics regarding R&D value .................................................................... 30


vii

Glossary

Business process At its most generic, any set of activities performed by a business that is initiated by an event; transforms information, materials or business commitments; and produces an output. Value chains and large‐scale business processes produce outputs that are valued by customers. Other processes generate outputs that are valued by other processes.

Effectiveness The degree to which the process’s stated objectives have been met.

Firm A term equivalent to ‘organisation’ in Open Innovation literature.

Innovation The implementation of a new or significantly improved product (good or service) or process; a new marketing method; or a new organisational method used in business practices, workplace organisation or external relations.(OECD 2005)

IP Intellectual Property. Any product of someone's intellect that has commercial value, especially copyrighted material, patents, and trademarks.

IPR Intellectual Property Rights. A general term for the assignment of property rights through patents, copyrights and trademarks. These property rights allow the holder to exercise a monopoly on the use of the item for a specified period of time.

KPI Key Performance Indicator. A metric that is especially important for measuring the performance of a process.

Measure In science, the process of obtaining the magnitude of a quantity (e.g., length or mass), relative to a unit of measurement (e.g., meter or kilogram). In management, a way to find out whether a goal is being met.

Metric A measure for something; a means of deriving a quantitative measurement or approximation for an otherwise qualitative phenomenon.

PMS Performance Management System. A system for evaluation against a set of determined criteria with respect to the economy, efficiency and effectiveness with which an organisation executes a particular activity. Organisations may set regular objectives for particular metrics, against which the organisation’s performance is evaluated.

R&D Research & Development. An organisation’s activity undertaken with the intent of discovering or developing new products or services. R&D includes pursuing enhanced versions of (or qualities in) existing products or services, or it may include discovering or developing new or more efficient processes of production.


1

Chapter 1 Introduction

1.1 Background on the problem/issue Today, most organisations rely on the innovation process to remain competitive and to grow in

a healthy manner (Kumpe, Bolwijn 1994). Such reliance makes innovation and its

measurement an extremely important topic. The same attention should be paid to this area of

business practise as is paid to other business processes, such as production or marketing

(Kerssens‐van Drongelen, Bilderbeek 1999, Andrew, Haanaes 2008).

If innovation is important in general, it is especially important in technology fields. Technology

is an enabler of innovation, and innovation confers a competitive advantage in technology

industries (Cumming 1998).

Open Innovation was first introduced by Chesbrough (Chesbrough 2003a), who referred to a

new model for innovation at the firm level. He observed that some companies started to

compete and subsequently found success in the marketplace using approaches to innovation

that had never been used before. The Open Innovation model contrasts with the traditional

“closed” model because it removes any strong barrier delimiting or shielding the internal

research and its development process.

Those in charge of funding, evaluating or executing innovation must use a measurement

system. Metrics, targets and results are required to make sound decisions about the process.

Measuring innovation is not an easy task; ongoing research continues in its quest to identify

good practices and metrics (Pearson, Nixon & Kerssens‐van Drongelen 2000).

Furthermore, for the novel Open Innovation model, much research is still required for the

purpose of properly defining the model (Chesbrough 2006b) and the appropriate ways of

measuring its effectiveness.

1.2 Justification for the research This research investigates the innovation measurement process for an organisation that

embraces the Open Innovation model. Many candidate organisations for such an investigation

exist, including companies along the lines of Eli Lilly, IBM, Procter & Gamble and Dow

Chemical.

Having a good set of measures (i.e., key performance indicators) can help organisations to

make the right choices about investments and organisational changes. Companies also rely on


2

their measurement system to demonstrate that positive results are happening as a

consequence of the actions taken.

Open Innovation is a very new model and, as such, there is still a great opportunity to learn

and explore its measurement metrics and its role as a driver for decision making.

Considering that a single error in product development investment could have an economic

impact several times greater than the research effort described here, it is worth investing in

research on Open Innovation.

Additionally, I am personally interested in the Open Innovation model because I believe that it

could have a major impact on the way businesses will operate in the coming years.

1.3 Aim and objectives

Aim

The aim of this project is to produce recommendations for metrics designed to evaluate the

effectiveness of an ‘Open Innovation’ approach to research.

Objectives

The objectives for this project are as follows:

1. To survey current innovation effectiveness metrics to create a catalogue of known

options.

2. To identify the basic criteria for the applicability of metrics based both on the Open

Innovation principles and on the examples available in the literature.

3. To correlate the deliverables of Objective 1 and Objective 2 to generate a proposal

about the use of effectiveness metrics for using Open Innovation in research.

1.4 Scope of the research Innovation can occur in very different ways. The research presented here focuses on

innovation in the research department or research unit of an organisation. Moreover, the

results presented here have a clear bias towards technology‐oriented organisations, which

makes perfect sense because this research falls under the “Technology Management” domain.

However, this research still may be of interest for those involved in measuring innovation in

other contexts, but such individuals must translate and amend concepts from the research and


3

development discipline in technology to the appropriate business role and market. Figure 1‐1

shows the scope of this research in terms of the intersection of the three concepts.

INNOVATIONTECHNOLOGY

R&D

SCOPE

Figure 1‐1 Scope of this research

From a historical point of view, Open Innovation is a very young model, so this research is not

limited to any specific period of time. Geographically, it is true that most of the available

information comes from Europe and the USA, so there is an implicit limitation to these areas.

However, this limitation arises only because these are the geographic areas where Open

Innovation is undergoing active development.

1.5 Outline of the dissertation This dissertation is organised into five chapters.

Chapter one serves as an introduction to the problem that originated the research, and it

defines the aim and three objectives of the research.

Chapter two explains, in detail, the problem that this research tries to solve, and it presents

existing knowledge relevant to the problem and its possible solutions. From the initial

problem, three specific research questions are outlined.

Chapter three focuses on the methodology used for this research and justifies the selection of

the method.


4

Chapter four presents the information collected, and it presents the interpretation of that data

as they relate to the research questions and the aim.

Chapter five serves as wrap‐up and includes the conclusion, as well as some considerations for

future work.


5

Chapter 2 Research definition

2.1 The practical problem/issue The objective of this research is to produce recommendations for the implementation of an

Open Innovation performance measurement system.

The Open Innovation model represents a drastic change from the traditional view of the

business rules related to inventions, research and product development. However, because a

lot of literature is already available on the subject, this research will not study the intricacies of

Open Innovation per se. Instead, this paper will focus on the problem of its measurement,

which has far less coverage in the available literature.

This section follows on others’ results ((McGrath, Romeri 1994), (Chiesa et al. 2008), (Bremser,

Barsky 2004), (Schumann, Ransley & Prestwood 1995), (Kerssens‐van Drongelen, Bilderbeek

1999, Davila, Epstein & Shelton 2005), (Mark Rogers 1998); see section 2.2 “Existing relevant

knowledge” ) that demonstrate the following:

1. Measuring innovation is widely accepted as a good practice

2. Measuring innovation is not an easy task

Previous research has also tried to demonstrate the following principle:

3. The Open Innovation model, because of its novelty and completely new principles,

requires more research into how it should be measured.

For a clear understanding of the problem, this section flows in the manner described in Figure

2‐1.

Why measure innovation?

Challenges measuring innovation

Current approaches for measuring innovation

What about measuring Open Innovation?

What is Open Innovation?

Figure 2‐1 Problem Definition Flow


6

Fully studying the problem of measuring Open Innovation is important to gain an

understanding of the differences between this new model and the traditional one.

Traditionally, innovation has been used by companies to create barriers to entry in their

markets. Research and development departments would maintain operational secrecy.

Talented personnel were hired by firms and retained to avoid the transfer of knowledge to

competitors. Strong intellectual property clauses in contracts protected the company from

partners, customers or employees using the company’s knowledge for their own benefit.

Open Innovation was first introduced by Chesbrough (Chesbrough 2003a), who referred to a

new model for innovation at the company level. The concept refers to a fresh approach,

focused on removing the barriers that delimit or defend the internal research and its

development process. The process is in contrast with the old, “closed” model. The author

observed how some companies started to compete and subsequently found success in the

marketplace using this new approach.

The closed model is usually represented with a clear boundary isolating the internal and the

external world, as shown in Figure 2‐2.

Figure 2‐2 The Closed Innovation Model (Chesbrough 2003)


7

The Open Innovation model presents a porous boundary for the organisation, a boundary that

allows knowledge to enter and exit at practically any stage of the traditional R&D cycle, as

shown in Figure 2‐3:

Figure 2‐3 The Open Innovation Model (Chesbrough 2003)

The model above does not assume that benefits only arise from one’s own knowledge; rather,

the model recognises that one can also benefit from several other factors:

• The available knowledge of others

• The ability of other organisations to use a business’s knowledge

This new model offers principles that contrast with those of the well‐established, traditional

closed model. Table 2‐1 summarises these contrasting principles according to Chesbrough

(2003a).


8

Closed Innovation Principles Open Innovation Principles

The smart people in our field work for us.

Not all of the smart people work for us, so we

must find and tap into the knowledge and

expertise of bright individuals outside our

company.

To profit from R&D, we must discover,

develop and ship it ourselves.

External R&D can create significant value;

internal R&D is needed to claim some portion of

that value.

If we discover it ourselves, we will get it

to market first.

We don't have to originate the research to profit

from it.

If we are the first to commercialise an

innovation, we will win.

Building a better business model is better than

getting to the market first.

If we create the most and best ideas in

the Industry, we will win.

If we make the best use of internal and external

ideas, we will win.

We should control our intellectual

property (IP) so that our competitors

don't profit from our ideas.

We should profit from others' use of our IP, and

we should buy others' IP whenever it advances

our own business model.

Table 2‐1 Contrasting Principles of Closed and Open Innovation (Chesbrough 2003)

We will revisit Table 2‐1 later in the report to determine how these principles impact the

measurement of innovation.

Chesbrough predicted that even the more traditional sectors and companies will evolve from

the old, closed model to an Open Innovation model (Chesbrough 2003a). The concept can

already be found today in the IT industry (e.g., IBM, Linux, etc.)(Andrews 2003), biotech

companies (e.g., Eli Lilly, Pfizer, etc.)(Linder, Jarvenpaa & Davenport 2003) and in less

advanced technology‐related companies (e.g., Procter & Gamble) (Huston, Sakkab 2006,

Sarkar, Costa 2008).


9

The differences between this model and the closed model are so important that, even at the

strategic level, that the new model cannot be explained by dominant business theory; basic

concepts such as entry‐level barriers or the competitive advantage of resource ownership are

questioned (Chesbrough, Appleyard 2007). The new Open Innovation models escape from the

classic Porter's vision of business strategy and subsequent schools. At the same time, the new

model’s survival may depend on these classic theories. Appleyard calls for a new "Open

Strategy" concept that gains the advantages of the "emerging anomalies" and still creates a

sustainable business. The open strategy urges a balance between openness and value capture.

Appleyard links sustainability with value capture and community engagement following some

existing business models for open‐source software.

Open Innovation represents an important shift in the innovation world, and it is a sufficiently

new concept to merit substantial attention from researchers.

During the past year, the European Innovation Scoreboard (EIS) was reviewed to better serve

the European Community as a true measurement of innovation at the EU level. However, the

report indicates that “To reflect and measure new forms of innovation in future editions of the

EIS, we must develop and incorporate new indicators measuring Open Innovation…”

(Hollanders, van Cruysen 2008).

In 2008, NESTA, the National Endowment for Science Technology and the Arts, published a

report as part of the Innovation Index project and proposed measures of firm‐level Innovation.

The report highlights interest from participants for measuring “Open Innovation”–“… a

respondent in the aerospace and defence sector expressed interest in a future measure of

Open Innovation…”. However, the authors report “It is likely, though, that the output of Open

Innovation can be measured using the same metrics as for traditional innovation” (Adams et

al. 2008). The argument is valid as long as one accepts that measuring innovation can be

limited to measuring the outputs of the process. We have found, however, a multitude of

metrics related to the input of the process as well as the way in which it is performed.

Although the need to measure Open Innovation is broadly recognised, there is a lack of

research on the applicability of existing methods and metrics for this new model. Because

Open Innovation principles are so different from previous models, it is expected that at least

some adaptation of the existing methods and metrics would be needed.


10

Table 2‐2 introduces a number of questions relevant to the measurement of Open Innovation:

Closed Innovation Principles

Open Innovation Principles Measurement‐related Questions

The smart people in our field work for us.

Not all of the smart people work for us, so we must find and tap into the knowledge and expertise of bright individuals outside our company.

Which metrics related to human resources should we use?

How do we measure knowledge outside of the firm?

Can we use metrics to motivate external people?

To profit from R&D, we must discover, develop and ship it ourselves.

External R&D can create significant value; internal R&D is needed to claim some portion of that value.

How do we measure R&D value?

How do we measure the portion of each R&D?

How do you measure access to external R&D?

If we discover it ourselves, we will get it to market first.

We don't have to originate the research to profit from it.

What measures of the discovery phase are relevant in Open Innovation?

If we are the first to commercialise an innovation, we will win.

Building a better business model is better than getting to market first.

Are speed measures relevant?

What substitutes for speed measures are of interest?

If we create the most and best ideas in the industry, we will win.

If we make the best use of internal and external ideas, we will win.

What uses we can make of these ideas? How do we mix internal and external ideas?

How can we measure the use of others’ ideas?

We should control our intellectual property (IP) so that our competitors don't profit from our ideas.

We should profit from others' use of our IP, and we should buy others' IP whenever it advances our own business model.

How do we trade with IP?

What do we do about patent metrics?

How do we measure licensing?

Table 2‐2 Contrasting Principles between Closed and Open Innovation (Chesbrough 2003) and measurement questions

The questions shown in Table 2‐2 point to a general research question: Can Open Innovation

be measured in the same way as traditional, closed Innovation?


11

The research presented here analyses this issue and, based on any contrast found between the

new and old models, elaborates on recommendations for the implementation of metrics for

Open Innovation.

2.2 Existing relevant knowledge This section summarises knowledge from existing published research.

Innovation measurement

Existing literature helps one to understand the motivation to measure innovation. There is

agreement among past and current literature on the need to measure the research and

development process and, taking a wider viewpoint, to measure innovation.

One needs measurement to improve things. The measurement process allows researchers to

set up a reference base, to determine whether changes have had an impact (positive or

negative) and to determine the size of the impacts (e.g., “Do you execute more projects on

time using this project planning tool?”). In the research and development process, there was a

time when improvements were made more easily, in large steps, but once performance

approached its limit, companies needed measurement metrics to carefully track new

improvements (McGrath, Romeri 1994). The performance measurement for R&D was

accepted later than such measurements were accepted for other processes (Chiesa et al.

2008). Nevertheless, there was a clear, strong case for implementing such measures in R&D

(Bremser, Barsky 2004).

The introduction of quality management systems and certifications (such as ISO9001) has been

another precursor of measurement in research and development departments (Schumann,

Ransley & Prestwood 1995).

Modern techniques for people management, such as variable compensation linked to

objectives, also called for measurement. To successfully tighten the compensation packages of

research and development staff, there must be agreement on measurable objectives. (Davila,

Epstein & Shelton 2005). Davila, Epstein & Shelton (2005) define innovation and present two

different strategies a company can follow to influence the innovation process: Play to Win and

Play Not to Lose. The book offers tips for innovation strategy definition and for structuring a

company for innovation. More relevant to this research is the chapter “How to measure

Innovation”, which introduces soft metrics versus hard financial metrics. These metrics are

linked to an incentives programme to support innovation.


12

Support for these different academic trends exists among current senior management in this

field. Among respondents to a 2008 Boston Consulting Group survey on measuring innovation,

74% agreed with the need for and relevance of such measurements (Andrew, Haanaes 2008).

The Boston Consulting Group report also highlighted the reality of innovation measurement in

companies (2008 survey). There is agreement on the need to measure innovation, but such

measurements are rarely performed. Moreover, there is a generalised dissatisfaction with the

results of efforts made to measure innovation. The 2008 report describes the cost and

competitive implications of innovation measurement. In addition, some of the most innovative

companies have some of the most rigorous measurement systems, but the report fails to

outline its selection process for identifying the most innovative companies.

Most companies are not happy with their measurement system because they system uses only

a few metrics and does not have a tie‐in to reward and incentive systems. Innovation output is

more accurately measured than the process or the input. The most commonly measured

components are presented and can be compared with Damanpour and Wischnevsky's (2006)

proposal. Similar measures are time to market, effectiveness and efficiency, time to volume,

and lifecycle performance. Some components added by this report are profitability (similar to

the R&D index from (McGrath, Romeri 1994)), idea generation and the selection and health of

portfolio management.

The most commonly used metrics relate to funding/revenue and time performance. The report

shows the gap between the number of available theories and their implementation at various

companies. Research into measurement frameworks is required to facilitate methodology

adoption by companies.

Even governments in different parts of the world agree on the need to measure innovation:

• “The centrality of the need to advance innovation measurement cannot be

understated” (Innovation Measurement. Tracking the State of Innovation in the

American Economy, The Advisory Committee on Measuring Innovation in the 21st

Century Economy).

• “It has been long understood that the generation, exploitation and diffusion of

knowledge are fundamental to economic growth, development and the well being of

nations. Central to this is the need for better measures of innovation” (OECD 2005).


13

It is possible to group the reasons for measuring Innovation into four main categories (see

Figure 2‐4).

Figure 2‐4 Reasons for measuring Innovation

The literature indicates a general agreement regarding the difficulties involved in measuring

innovation (Chiesa et al. 2008) (McGrath, Romeri 1994). Some authors have recognised

peculiarities in the R&D process (KerssensvanDrongelen, Cook 1997), and there is agreement

that this complexity requires the use of several combined metrics (Schumann, Ransley &

Prestwood 1995) (Mark Rogers 1998). Again, the research presented here found that senior

management agreed on this: “… innovation performance measurement belongs in the ‘too

hard’ basket…” (Adams et al. 2008).

KerssensvanDrongelen and Cook (1997) explain how the changing role of R&D, from a support

process to a driver of competitive advantage, calls for the proper performance measurements.

The authors review the R&D performance measurement literature and position it in the

context of control theory. Furthermore, they identify four main problems experienced when

measuring R&D performance: R&D contribution isolation, time lag, correct norms and

acceptance. They also discuss two main issues, the purpose of measurement and contingency

factors. The purpose of measurement is then broken down into two groups: “motivating” and


14

“diagnosing”. Contingency identifies several issues: the type of R&D, company size,

aggregation level, etc. KerssensvanDrongelen and Cook (1997) introduce an interesting

categorisation of metrics for their research and align it with the four categories found in the

Balanced Scorecard. The authors further define the system requirements, design parameters

and principles required for a performance measurement system. Interestingly, because the

type of R&D is seen as a key concept for designing the system, one can view Open Innovation

as a type of R&D.

The main challenges can be grouped into four categories (see Figure 2‐5).

Figure 2‐5 Challenges measuring Innovation

Most approaches proposed for innovation measurement focus on a few sets of dimensions

and metrics (Brown, Svenson 1988, Driva, Pawar & Menon 2000, Godener, Soderquist 2004,

Hauser 1998, Pawar, Driva 1999, Werner, Souder 1997b, Werner, Souder 1997a, Pappas,

Remer 1985). Only a few have tried to define a complete framework or Performance

Management System (PMS) (Chiesa et al. 2008), frequently based on the Balanced Scorecard

(BSC) (Bremser, Barsky 2004).

Chiesa et al. (2008) state that, in addition to the R&D function, the use of a Performance

Measurement System (PMS) is a critical success factor. They explore some of the reasons why

it can be a difficult task, but although they do recognise some approaches, they see most of


15

them as being limited to a set of dimensions and metrics. The authors further reference a few

approaches to a full PMS system, based on the Balanced Scorecard of Kaplan and Norton. They

complement this approach by proposing a framework applicable to any kind of R&D as part of

technology innovation. They define the steps necessary to apply the framework, and they offer

a practical example for a Clinical Research Organisation (CRO). The full set of PMS elements

includes a set of dimensions and metrics, together with a structure and a process for operating

the system. The framework takes into account contextual factors to define the elements,

namely, strategy, entities, types of activity, PMS objectives, and available resources.

Chiesa et al. outline the relationships among each factor in the framework, but do not always

specify or clarify these relationships. The steps given in the example are not generalised

enough to learn how each factor can influence each element in the PMS. The study is a good

example, but it is not a complete guide.

Bremser and Barsky (2004) focus on the framework aspects of the measurement systems and

work to integrate the Stage‐Gate and BSC approaches. The main contribution of the paper is

the evaluation of the Balanced Scorecard in an R&D process. Bremser and Barsky (2004) map

the functions of the performance measurement system using the characteristics of the BSC

and demonstrate its suitability. Some examples of metrics that are useful for this research are

reported.

Some organisations have developed their own frameworks. The European Foundation for

Quality Management (EFQM), published in 2005, is an adaptation of the Excellence Model for

innovation (EFQM 2004). The Oslo Manual proposes a conceptual framework for collecting

quantitative data (OECD 2005).

Categorisation or classification of innovation has been used to simplify the process of

measuring it (Damanpour, Wischnevsky 2006). Damanpour sets innovation as a prerequisite

for competitive advantage and growth, and the authors highlight how previous research on

the determinants, processes and consequences of innovation has produced inconsistent

results. Damanpour states that current approaches to explain inconsistencies based on

innovation types (product vs. process, technical vs. administrative, radical vs. incremental) fail,

and proposes a model based on organisational types (Innovation Generating Organisations vs.

Innovation Adopting Organisations). However, Damanpour restricts his model to the study of

one of the aforementioned types, namely technical innovation. Damanpour does not mention

applicability to administrative innovations.


16

Damanpour explains that Innovation Generating Organisations deal with creative processes

and Innovation Adopting Organisations deal with a problem‐solving process. These different

approaches are highlighted as the root cause of the differences between the two approaches.

Innovation Adopting Organisations are supposed to use what Innovation Generation

Organisations deliver, which is closely related to the use of technology both as a product and

as an application, as shown in Open University’s course T840. What happens if a product for

one organisation becomes an application for another, following a hierarchy of components?

Damanpour applied the model to explain how innovation is influenced by other factors such as

an organisation’s size or age and how radical it is in terms of innovation. Interestingly for this

research, Damanpour also describes the implications of these factors on the measurement of

innovation. Damanpour focused only on the outcome measurement, discarding input

measurement (R&D expenditure) or intermediate outcomes (Patens), in direct disagreement

with Katila (2008), who sees patents as an outcome measure.

Damanpour reviewed an interesting number of measures, each with reference to the rates and

speed of innovation (e.g., speed of development, earliness of adoption, percent adopters over

a period of time, timeliness, speed of implementation, extent of implementation, rate of

adoption, and innovation impact). The authors also explained the relation between Innovation

Generating Organisations and Innovation Adopting Organisations. The aforementioned lists,

however, must be revisited from the point of view of Open Innovation to assess whether these

factors are still relevant.

Damanpour introduces organisations in which there is coexistence between different units

that are adopting and generating innovation for the Innovation Integration Unit (IIU), a

managing organisation focused on strategic integration. It seems from these findings that an

Innovation Integration Unit would need different metrics for performance measurement.

At the other extreme of generic frameworks, there are very specific proposals such as the use

of patents in radical technology innovation (Katila 2008) or the definition of the R&D

effectiveness index for product development (McGrath, Romeri 1994).

Katila proposes the use of patents and patent citations for measuring innovation. However,

she proposes this strategy for a particular kind of innovation, namely, radical innovation. More

specifically, she focuses on technological radical innovation and excludes the other three kinds

of radical innovation: industry, organisation and user radical innovations.


17

Katila sometimes refers to measuring the ‘radicality’ of innovation, a notion that is well suited

for measuring innovation because it is assumed that more radical innovations are also better

innovations. However, this may not be the case for all organisations. Katila occasionally implies

that incremental innovation can also be measured with patents, but this thought is not

explicitly mentioned. Some limitations are recognised, and Katila points out that probably the

best use of innovation measurements is for comparison within an industry (i.e., benchmarking

use). One of the limitations mentioned is that measurements that are “…only applicable to a

fraction of the total output” can be greater than expected because the size of the fraction is

not mentioned and is likely small for some industries. Moreover, technological innovations are

not always patented. Either there are other options available for their protection (trade

secrets) or there is no protection at all (standards). Specifically for Open Innovation, patents

are an option, but are not always preferred.

Conversely, some technological innovations may be part of a group of other inventions. For

example, a new cell phone can include hundreds (if not thousands) of patents behind it. The

correction of some limitations is palliated using patent citation, based on citation only in

subsequent patents. This endogamy is a weakness of the system, but it clearly measures only

the output of the innovation process.

It seems that patents and patent citations could be used as a metric but that other metrics will

also be needed if the organisation wants to balance incremental and radical innovation. The

need for other metrics also applies if the organisation wants to develop other kinds of

innovation (industrial, organisational, user, etc.). This is true for measuring inputs and the

process itself and is not limited only to the measurement of outputs.

McGrath focuses on product development and indicates that it is much more difficult to

measure than other business processes. A company should criticise the percentage of R&D

spending as a measure and should advocate for measuring the effectiveness of R&D (instead

of the level of investment). McGrath proposes a very specific measure: the R&D Effectiveness

Index. The R&D Effectiveness Index focuses on the economic output of the process and is

based on the strong correlation between growth and profitability, as demonstrated through a

benchmarking study.


18

Figure 2‐6 summarises the existing approaches:

Figure 2‐6 Approaches facilitating Innovation Measurement

Open Innovation

The father of Open Innovation, Henry Chesbrough, defined the concept in his seminal works in

2003 (Chesbrough 2003a), (Chesbrough 2003b). Chesbrough presents the "Open Innovation”

(OI) model in opposition to the traditional closed approach. He shows how many well‐

established, giant firms have been challenged by newcomers that bypass the supposed "entry

barrier," thanks to this new approach to innovation. Chesbrough states that the Open

Innovation model will be adopted steadily in almost all industries by all companies, and he

defines three areas to show how the adoption of the model is happening in each of them:

funding, generating innovation and commercialising innovation.

By that time, others had already started researching other ways of sourcing innovation (Linder,

Jarvenpaa & Davenport 2003), but Chesbrough blended both sourcing and exploiting. Since

then, the model initially adopted by technology firms (e.g., IBM, Intel, Lucent, Xerox and

others) (Andrews 2003) has been adopted by more and more companies in very different


19

sectors (e.g., Procter & Gamble, General Motors, Eli Lilly Colgate‐Palmolive and others)

(Gwynne 2007).

Different ways have been identified to use the new model. Such areas for implementation

include corporate venture capital, internal spin‐off programmes, external licensing

programmes, "use it or lose it" initiatives, programmes to license technologies, portals to

outplace technologies and other portals to solicit input technologies (Andrews 2003). Andrews

pinpoints the two sides of the Open Innovation approach, namely the access to external

knowledge versus the monetisation of internal knowledge. Open Innovation practitioners

identified by Andrews include IBM, Intel, Lucent, Xerox, Procter & Gamble, Merck, Pfizer and

Millennium. Andrews proposes the following metrics:

• Identify the sources of the most important ideas in your industry over the past five

years.

• Identify how many came from inside your company, how many came from inside other

current industry participants, and how many came from outsiders.

• Identify what percentage of your own pipeline of future projects come from outside

versus from within. How does that ratio compare with the first ratio?

Using this approach, relations with customers, competitors and suppliers may be modified. For

example, areas for modification can include the following: customer integration, supplier

integration, competitive alliances, customer of customer integration, integration of second‐tier

suppliers, cross‐industry innovation, university‐industry cooperation and globalisation of

innovation (Enkel, Gassmann 2007)

A new concept has even been created to enable the model: knowledge brokers (Gwynne

2007), (Sousa 2008). Gwynne introduces the idea of the “Knowledge Broker” as a catalyst in

the Open Innovation model. Examples he mentions include NineSigma, yet2.com or

InnoCentive. Gwynne points to one very specific problem of Open Innovation, namely, how to

integrate the technology coming from outside the organisation. He worked with early adopters

of the Open Innovation model including Procter & Gamble, General Motors, IBM, Eli Lilly,

Colgate‐Palmolive, and Philips.

Soon it became apparent that, in an Open Innovation model, there is a key issue: balancing

value capture against value creation (Andrews 2003), (Laursen, Salter 2006). These two

dimensions of value (capture and creation), plus the duality of internal and external resources,


20

configures a map that is very relevant when thinking about measurements. This concept is

discussed later in this paper.

It is important to note that Open Innovation exhibits a diffuse definition, as recognised by the

concept’s own originator (Chesbrough 2006b). The latest Chesbrough work claims that the

new model requires some changes even at the strategy level for full exploitation, which he

calls “Open Strategy” (Chesbrough, Appleyard 2007).

Interestingly, current key trends in the research on Open Innovation do not focus on

measurement. Rather, the main five themes are business models (balancing value creation and

value capture), inclusion of external technologies, identification/assessment of knowledge,

Start‐ups and Intellectual Property Management (Chesbrough 2006a). One exception to this

lack of focus on measurement and metrics is presented in Chesbrough (2004), where the need

for a change in metrics is clearly stated, but is focused in the issue of managing “false

negatives” (i.e., projects that would have been discarded under a closed model but may be

pursued under an open model).

2.3 Research questions The general question from the introduction section, “Can Open Innovation be measured in the

same way as traditional closed Innovation?” can now be split into more specific questions:

1. Does Open innovation require the same sort of measurement approaches as

traditional innovation?

2. What metrics are useful when the Open Innovation model is applied?

3. What changes are needed regarding metrics when the Open Innovation model is

embraced?


21

Chapter 3 Methodology

3.1 Methods and techniques selected First, a comprehensive literature review responds to the first objective of the research:

• Determine the current practices regarding the measurement of innovation.

A cross‐sectional case study has been used to explore Open Innovation, with a focus on the

measurement practices in place or processes related to the practices identified in the first

literature review. A review of the currently available examples of Open Innovation has been

used to respond to the second objective of the research:

• Determine, based on Open Innovation principles and examples available in the

literature, the basic criteria for the applicability of measurement practices.

Analysis and synthesis of the previous deliverables have generated the deliverable for

Objective 3:

• Generate a proposal for Open Innovation measurement practices based on

correlation of the deliverables from Objective 1 and Objective 2.

3.2 Justification This research is empirical and based on observation. Some sources for review on the general

problem of measuring innovation have been identified as a necessary starting point, but the

available references examining the specific problem of measuring Open Innovation are very

limited at the time of this writing . Other, alternative approaches have also been evaluated, as

discussed in the next two paragraphs.

Experiments do not seem appropriate because it would be very difficult within the research

timeframe to conduct any reasonable experiments. The typical period for the measurement of

a business process is one year because this is the typical budgeting cycle, even in R&D

concepts. It would make sense to make observations over longer periods of time as innovation

matures (3‐5 years). Similarly, it does not seem possible to find an organisation that would be

willing to test any conclusions regarding enhancements in the measurement process.

Surveys would have presented unmanageable risk because the population is unknown and

dispersed. An attempt was undertaken, using unstructured interviews at a workshop on Open

Innovation conducted in Madrid in May 2009, but it was difficult to collect more than informal


22

feedback, and that feedback was difficult to compare or analyse. The survey was only useful

for qualitative input about the interest in the research topic, not as a data collection

technique.

Case studies, however, seemed to fit well with the scope of this research. Thanks to the

Internet, it was possible to access information regarding real examples of Open Innovation

adoption at different companies. As stated, Open Innovation is new enough to discard any

longitudinal case study. Due to the general applicability of the research, the exemplar case

study also does not apply. Therefore, a cross‐sectional case‐study has been selected and takes

into account a variety of organisations as they operate today.

3.3 Research procedures The data collected for this research come from two main sources:

• Public‐domain company records on Open Innovation adoption. The use of such records

avoids potential confidentiality issues.

• Literature available regarding the measurement of innovation.

As mentioned in section 3.2, an initial set of unstructured interviews was attempted, with

limited benefit. Therefore, no additional questionnaires or interviews were undertaken.

Use of the “ISI Web of Knowledge” tool has allowed me to identify relevant existing literature

on both the measurement of innovation and the themes of Open Innovation.

Other relevant sources have been used to collect information:

• The OpenInnovation.eu platform founded by Prof. Dr. Wim Vanhaverbeke, in

partnership with Henry Chesbrough.

• The UK Innovation Index Project, led by the National Endowment for Science,

Technology and the Arts (NESTA).

• The European Industrial Research Management Association (EIRMA).

3.4 Ethical considerations The main ethical consideration had to do with the disclosure of private information used

during the research. However, the final data used are publicly available, so no disclosure of

private information has taken place, to the best of my knowledge. Details of the responses to

the informal survey have not been included, so no ethical issues are apparent.


23

Chapter 4 Analysis and interpretation

4.1 Summary of data collected

Current practices

This section provides an overview of current practices as cited in the literature on measuring

innovation. These practices are later analysed to determine their suitability for the Open

Innovation approach.

Fundamentally, every approach follows the same process for measurement, which is well

defined by KerssensvanDrongelen and Cook (1997) and is shown in Figure 4‐1.

Figure 4‐1 The measurement and control process (KerssensvanDrongelen, Cook 1997)

The main differences are more related to what is measured (i.e., the metrics used). Metrics are

sometimes referenced as Key Performance Indicators (KPIs). Multiple factors influence the

selection of metrics, and some research has already addressed the identification of these

factors. These factors are what KerssensvanDrongelen and Cook (1997) call “contingency

factors” or what Chiesa et al. (2008) refer to as “contextual factors”. Our research accepts the


24

type of research as a factor influencing metrics and will focus on the particular case of Open

Innovation.

When a complete framework is used, these metrics are aligned with dimensions (usually a

dimension can be measured with a set of metrics) and a formal process with norms and rules

about who does the measurement, how, with what frequency, and so on. A good example of a

framework is the Balanced Scorecard (Bremser, Barsky 2004).

Reviewing the vast array of metrics proposed (see section 2.2 Existing relevant knowledge), we

find metrics that can be grouped into different types, according to different characteristics. For

further analysis, it seems useful to examine groups of metrics, instead of each one individually.

Several groups were identified for this research:

Absolute metrics and ratio metrics: Absolute metrics are expressed in units of some magnitude

(e.g., Euros or months) and can be compared to target or reference values. Ratio metrics are

expressed in units per another unit or as percentages. Ratio units are normally used for better

comparison over a period of time or between different organisations.

Input, intermediate or outcome metrics: This classification is more relevant for our research.

Metrics can be found relative to the input of the innovation process (e.g., a budget for R&D or

expenditure), the output (e.g., the number of new products) or,most relevant for us, the

process itself or intermediate processes (e.g., the number of patents).

According to the type of magnitude, we found metrics for speed, time, financials or

performance‐related measurements (e.g., the execution of projects or market share).

Quantitative and qualitative (or objective and subjective): Some metrics are an exact measure

of a magnitude, but others are estimations or rankings based on expert judgement or

subjective validation.

Metrics at the organisation level, group level or individual level: Some metrics can be valid at

all three levels, whilst others only make sense at the aggregated or individual level.

Metrics related to the creativity phase or to the value capture phase: These are related to the

phases for innovation from Davila, Epstein & Shelton (2005).

Furthermore, metrics can serve one or both main objectives and can be used to monitor

and/or to motivate (reward).


25

Interestingly, other authors have found very similar grouping categories for R&D measures

(Ojanen, Vuola 2003) (Table 4‐1).

Table 4‐1 The dimensions of R&D performance (Ojanen, Vuola 2003)

Based on the literature review, a collection of metrics has been selected. The metrics are

grouped consistent with listed characteristics (Table 4‐2). This table serve as a reference for

the analysis of the different groups of metrics and is helpful to understand which groups are

affected in an Open Innovation approach and to understand the nature of the effects.


26

METRIC

Absoluteor

Ratio

Phase:Input

iNtermediateOutcome Magnitude

quanTitativequaLitative

Levels:Organization

GroupIndividua

Creativityor

Valuecreation

Function:MonitoringMotivating

Time used to adopt an innovation A N Time T O/G C MTime used to develop a product A N Time T O/G C MTime used to commercialize A N Time T O V MProject related measures (EVM) R N Index T O/G C MImpact on market (market share…) R O %/€ T O V MProfitability A O € T O V MIdea generation and selection A N ideas T O/G/I C RTime to market A N Time T O V MR&D effectiveness and efficiency R N Index T O C MHealth of the innovation portfolio R O % T O C MLife cycle performance R N Index T O C RTime to volume A N Time T O V MCustomer satisfaction A O Index L O/G V RTotal fund invested in growth projects A I € T O C Mperformance, projected versus actual R N % T O/G C R# of projects that meet planned targets A N projects T O/G C Mdevelopment time (Average) A N Time T O/G C M

revenue realized form offering in the past X years A O € T O V MCannibalization of existing products sales by new offerings A O € T O V M% of ideas funded R N %/ideas T O C M# of projects killed A N projects T O/G C M# of new products per dollar spend on R&D R O products T O C MTRL’s Technology maturation achieved A N Index T O/G C R# product at a given stage of the stage-gate approach A N products T O/G C MTime among stages A N Time T O/G C M% passing each stage R N %/projects T O/G C M#acceptable rate R N products T O/G C MSafety incidents (rare) A N incidents T O/G/I C RSkill coverage of competencies R I % L O/G/I C RTraining hours A N Time T O/G/I C R# patents A N patents T O/G/I C R# patent citations A N citations T O/G/I C R# patents per employee R N index T O/G C REffectiveness Index (ratio output/input) R O index T O V MPerformance: present values of an accomplishment A O € T O V MR&D contribution to profit/R&D cost A O %/€ T O V MMarket share gained due to R&D A O %/€ T O V MScores on surveys-customer satisfaction A O index L O/G/I V RScores on surveys-employee satisfaction A O index L O/G/I V R% of customer driven projects R N %/projects T O/G C MEngineering hours on projects/total engineering hours R N %/hours T O/G C M% projects terminated R N %/projects T O/G C MHours on projects / total R&D hours R N %/hours T O C M

Current time to market/ reference time to market R N %/Time T O V MRate of reuse of standard designs or proven technology R N index T O C MSum of revised project duration / sum of planned duration R N %/Time T O/G C M# of times rework A N reworks T C R% budget spent internally and externally on basic and applied research R I %/€ T O V M% of projects in co-operation with third party R N %/projects T O C M%of project evaluation ideas applied in new projects R N %/ideas T O C MTalent, in recruitment, training … A I talent L O/I C RKnowledge Management Tools use R N %/time T O/G/I C MCommunication Effectiveness A N index L O C RBalance of innovation portfolio R O % T O C M

Table 4‐2 Grouped Innovation metrics


27

Basic Criteria of Applicability

Based on the Open Innovation reference material, this section identifies the criteria that

should be used to judge the applicability of current measurement practices to the Open

Innovation model.

Open Innovation principles introduced in previous sections will serve as a guide for this criteria

identification:

Not all of the smart people work for us, so we must find and tap into the knowledge and

expertise of bright individuals outside our company.

This principle refers to the fact that in the Open Innovation model, the employees of the

company (“people who work for us”) are not the only appropriate participants in the

innovation process. People from outside the company can–and should be invited to–

participate in the innovation process. However, we must ensure that the people from outside

who participate in innovation are the right ones (i.e., smart, bright, knowledgeable).

According to this principle, people‐related metrics, which usually refer to internal staff, should

be used or adapted to include external actors.

At the same time, metrics related to knowledge from people must cover not only internal

knowledge but also the knowledge accessed from people outside of the company.

Because people‐related metrics can be aimed at motivating people, the metrics should also

aim to motivate external people.

Metrics regarding “bright” people are needed to ensure that the organisation is accessing the

right people. “Bright” can be associated with talent, so talent measurement is probably

important.

External R&D can create significant value; internal R&D is needed to claim some portion of

that value.

This principle recognises that both external and internal R&D must be used in the innovation

process. Internal R&D is easy to locate and to manage in the traditional way, but external R&D

is beyond the boundaries of the company. Therefore, a different approach is needed for

external R&D. The first thing to establish is access to the external R&D. Then, one can develop

ways to measure it.


28

Metrics about the value of R&D must be applicable to external R&D.

Metrics must differentiate between external and internal R&D, which is especially relevant for

ratios and comparison metrics.

Metrics related to access to external R&D are needed as well.

We don't have to originate the research to profit from it.

This principle is based on the fact one can access external R&D at any stage. R&D initiated by

others can be relevant to an organisation at any point in time. One has access to the external

R&D, and it can be used to generate profit.

Metrics are needed to measure R&D that is not generated internally.

Metrics must differentiate internally originated researched from other types of research. Profit

should also be measured and classified according to whether it originated from internal or

external research.

Building a better business model is better than getting to market first.

This principle invites a business to focus not only on being the first but also on thinking about

the right business model to profit from the innovation.

Time‐to‐market metrics must be balanced with others. Measures of the success of the

business model are needed. Speed metrics may refer to only a single phase of the innovation

process instead of the whole process.

If we make the best use of internal and external ideas, we will win.

Ideas can be used in different ways, and one can use both internal and external frameworks.

“Use” can mean incorporate into an existing product or it can simply mean the sale of the idea.

Paying for a third‐party idea can multiply the value of your own product.

Metrics of usage are needed. Again, distinction between internal and external ideas is needed.

Metrics for creation must be balanced with metrics for usage.

We should profit from others' use of our Intellectual Property, and we should buy others'

Intellectual Property whenever it advances our own business model.


29

Intellectual property must be managed in the same way as any other asset. One can sell and

buy it, and it is even possible to rent it. This principle again focuses on profiting from

intellectual property, regardless of how it is used.

Metrics regarding intellectual property are needed. Metrics must differentiate one’s own

intellectual property versus other companies’ intellectual property. Metrics regarding the

profit related to intellectual property are needed.

4.2 Data Analysis In this section, the basic criteria and current practices are correlated to find support for

current measurement practices for Open Innovation. I also propose any enhancements for

cases where there is insufficient existing support.

Current practices show the low use of people‐related metrics. At the individual level, 11 of 54

metrics can be applied (Table 4‐3).

Absolute/Ratio

Phase of processInput /

iNntermediate / Outcome Magnitude


LevelsOrganization/

Group/Individual

PhaseCreativity/

Value creation

FunctionMonitoring/Motivating

Idea generation and selection A N ideas T O/G/I C RSafety incidents (rare) A N incidents T O/G/I C RSkill coverage of competencies R I % L O/G/I C RTraining hours A N Time T O/G/I C R# patents A N patents T O/G/I C R# patent citations A N citations T O/G/I C RScores on surveys-customer satisfaction A O index L O/G/I V RScores on surveys-employee satisfaction A O index L O/G/I V R# of times rework A N reworks T O/G/I C RTalent, in recruitment, training … A I talent L O/I C RKnowledge Management Tools use R N %/time T O/G/I C M

Table 4‐3 Innovation metrics for individuals

These metrics are most usable for external staff, but some of them need to be adapted when

applied to external people. Clear example is metrics based on surveys; the surveys need to be

extended to external collaborators. Other metrics call for special attention due to the issue of

their collecting and treating personal data from people who are not employees. In general,

metrics are applicable but may be complex if a company needs to account for external people

who do not collaborate (i.e., external people not willing to report the training they have

received).


30

Especially difficult is the measurement of talent. Are external collaborators really “bright”

people? What are the parameters by which this determination is made? Knowledge‐related

measures must be applied.

The convenience of these metrics for motivating external collaborators is still to be proven

because the reward mechanism will be different (i.e., the measured individuals are not on the

payroll).

Value of the R&D is usually measured in Euros. The problem with using these metrics for

external R&D is that the economic value is usually associated with products. However,

products will include both internal and external R&D. To use these metrics properly, it is

necessary to account for both internal and external contributions at development time, which

enables managers to later split this economic value. Typical metrics found regarding R&D value

are shown in Table 4‐4.

Absolute/Ratio

Phase of processInput /

iNntermediate / Outcome Magnitude


LevelsOrganization/

Group/Individual

PhaseCreativity/

Value creation

FunctionMonitoring/Motivating

Impact on market (market share…) R O %/€ T O V MProfitability A O € T O V MTotal fund invested in growth projects A I € T O C Mrevenue realized form offering in the past X years A O € T O V MCannibalization of existing products sales by new offerings A O € T O V MPerformance: present values of an accomplishment A O € T O V MR&D contribution to profit/R&D cost A O %/€ T O V MMarket share gained due to R&D A O %/€ T O V M% budget spent internally and externally on basic and applied research R I %/€ T O V M

Table 4‐4 Innovation metrics regarding R&D value

The ability to distinguish between external and internal value is especially relevant for metrics

like the “Balance of the innovation portfolio.” Such an ability also permits a company to

enhance metrics like “Percent of projects in co‐operation with a third party”. Balance can be

pursued on the new axis of internal/external value, and percentage of co‐operation can be

measured not only in terms of the number of projects but also in terms of the value they

provide.

What is difficult to find in a traditional measurement system are metrics regarding access to

external R&D, which is a clear gap to close when adopting Open Innovation.


31

Apart from splitting profit metrics into internal and external R&D, it is important to

differentiate metrics for the input (i.e., originating) phase of the process. Outputs of Open

Innovation are not expected to be different (new products, methods, patents), but it is the

input and intermediate phases that are executed differently. Useful tools for this splitting are

available systems for classifying the maturity level of a technology. One such system is NASA’s

Technology Readiness Levels (Mankins 1995). Metrics can be used to compare the maturity

level at the starting point and at the ending point for both internal and external R&D.

Measuring profit in terms of R&D sold (e.g., licensing and patent selling) is also important. One

can measure profit in Euros, so it is possible to easily compare the sale of R&D with the sale of

traditional products.

Metrics are also needed for inputs that come from outside the organisation (i.e., ideas

proposed to the company). The number of ideas generated should be compared with the

number of ideas selected from outside the company.

Typically, there is strong emphasis on time‐to‐market metrics. To balance these metrics with

the Open Innovation focus on business models, other independent metrics are needed (i.e.,

market adoption of products) to determine whether an organisation is the first to arrive.

“Time among stages” and “Percent passing each stage” are useful to measure speed at other

phases, and these metrics are perhaps more relevant than the duration of the whole process.

Usage metrics are needed at different stages (e.g., percent of ideas used, percent of products

used, percent of patents used and percent of trade secrets used), and such metrics should

include information on whether the usage is internal or external. New ratios are potentially

valuable in this area. For example, metrics may include internally created ideas versus

externally acquired ideas, or they may include internal ideas used versus external ideas used.

These ratios can help to detect situations where more weight is on the worst, less effective

approach.

Metrics need to reflect the transition from Closed Innovation to Open Innovation, and they

help to detect deviations. Such deviations include desorbing innovation (i.e., too little external

technology acquisition) and absorbing innovation (i.e., too little external technology

exploitation) (Lichtenthaler 2008). Lichtenthaler presents how the Open Innovation strategies

can vary from a proper balance between external technology exploitation and external

technology acquisition. He identifies groups of companies, ranging from Closed Innovators to

Open Innovators, with extreme deviations like Desorbing Innovators and Absorbing


32

Innovators, see Figure 4‐2. In his research, Lichtenthaler shows how the high profitability of

the open innovators suggests that sticking to the closed innovation model may lead to a

substantial weakening of a firm’s competitive position in the future. The right balance between

exploitation of internal and external R&D is the desired condition. Ratios can be measured in

terms of economic value and / or monetary investment.

Figure 4‐2 Balance in Open Innovation (Lichtenthaler 2008)

Intellectual Property related metrics (patents, licenses) are usually in place, but when adopting

the Open Innovation model, it is necessary to address the balance of the Intellectual Property

produced, the Intellectual Property used, the Intellectual Property sold, and the Intellectual

Property bought. Again, different ratios can be defined that are useful for detecting deviations,

like selling all of the IP a company produces or only using the IP bought. In the Open

Innovation model an organisation is supposed to blend its own knowledge with the acquired

knowledge and to then sell only a part of the knowledge it generates.


33

4.3 Interpretation in relation to the research questions

Does Open innovation require the same sort of measurement approaches as


Existing measurement approaches, based on metrics and a performance management system,

are completely valid for the Open Innovation model. There are only three considerations to

keep in mind:

• Measuring the output of the innovation process is completely valid because the

objectives of the process do not change. However, new ways of generating profit can

require additional metrics regarding economic results because there is the possibility

to sell IP or benefit in different ways from other companies’ products (including using

them in one’s own innovations).

• Measuring the process requires a new axis, namely, that of externality. Most of the

constituents coming from inside the company can come now from outside the

company as well, calling for new metrics (ratios) that compare internal and external

sources.

• Externality poses challenges when the objective pursued by a metric cannot be

achieved, as is the case with external sources. Therefore, new metrics are needed to

complement the initial ones.

What metrics are useful when the Open Innovation model is applied?

Following the approach of treating the metrics according to the groups proposed in section 0,

the answer to this question includes the following six parts:

Absolute metrics and ratio metrics: Absolute metrics remain useful, but the focus is on a new

breed of ratio metrics that will compare the internal and the external contributions of each

innovation element. In addition to the results, each metric of the process can be split into the

internal (traditional) source and the new sources.

Input, intermediate or outcome metrics: Output metrics are useful because they permit the

measurement of how the whole innovation process is working. Eventually, if all is reduced to

the economic success of the company, even using the same single metric (revenue) would be

useful. However, under a richer measurement system, the input and intermediate metrics are

very useful to determine how the Open Innovation model works. Inputs to the process are

now broader because the technology may be at different maturation levels, probably because


34

the technology was invented and developed by others who entered the innovation process at

later stages. Intermediate metrics, which can measure the process until the output of the new

technology to third parties, are considered the best option. Efficiency cannot be measured

only in terms of time‐to‐market metrics. Some phases may move at greater speeds than

others. Finally, there is a need to measure the IP (acquired or generated) at all stages .

Quantitative and qualitative (or objective and subjective): The fact that some metrics must be

translated to external sources, where direct access to the data is difficult, will drive an increase

in the use of qualitative metrics. These metrics are still valuable to manage the innovation

process when external elements not under the direct control of the company must be

evaluated.

Metrics at the organisation level, group level or individual level: These categories of metrics

are slightly less useful in the Open Innovation model because the concept of organisation must

be extended to all third parties involved. Also, the individuals and groups of individuals now

include persons from multiple organisations. Individual metrics valid for inside the company

cannot be practically (or even legally) measured.

Metrics can relate to the creativity phase or to the value‐capture phase: Open Innovation

impacts both phases of the process. During the creativity phase, one can count the ideas

coming out of the organisation. At value capture, one can decide to transfer to others an

invention at a given price or royalty. Therefore, it is interesting to maintain metrics for both

phases and to discriminate between external and internal contributions.

Metrics can serve one or both main objectives: to monitor and/or to motivate (reward):

Metrics for monitoring are useful and may need to be complemented with additional metrics

that measure specific results or contributions. Metrics for motivating face the challenge of

motivating external people who are not part of the organisation. This issue find similarities in

the purchasing process or to the supplier management field, namely, how does one motivate

suppliers? (Carr, Pearson 1999)(Krause, Ellram 1997).

What changes are needed in terms of metrics when the Open Innovation model is

adopted?

Consistent with my interpretation of previous research questions, it is clear that the changes

imply adaptation or at least changes to external metrics along with the addition of new ones.

New metrics for new results, like the profit coming from IP sold or the cost of buying others’

licenses, are needed. It will be necessary to split some metrics to reflect the contribution of


35

external sources. This way, one can trace the contribution from each part when analysing the

revenue coming from products. Finally, current metrics that are useful inside the organisation

will need complementary parameters that generate the same impact outside the organisation.

Development of these metrics will requires consideration of the different relationships

involved, and any developments should always consider the legality of the measurement

process.

The rest of the performance management system can remain. A few changes regarding

responsibilities may be required, depending on who is in the best position to measure external

contributions.

4.4 Interpretation in relation to the aim The aim of this project was to produce recommendations for metrics used to evaluate the

effectiveness of an ‘Open Innovation’ approach to research. All of the findings presented can

be condensed in this “Decalogue” that a company embracing Open Innovation model should

follow when implementing a performance measurement system:

1. Add new metrics to measure new profitability models. It is necessary to measure the

revenue derived from licensing or from selling innovation to others.

2. Split all relevant measures into those of internal and external origin. For example, a

product can make use of 10 patents; measure how many are internal and how many

are external.

3. Introduce ratio metrics for comparing internal and external contributions. For

example, determine which new products are developed in‐house versus which are the

result of product acquisition.

4. If not present, introduce a maturity level model to measure the process step by step.

5. Measure the external contributions at different stages of maturity. For example,

identify the basic ideas coming from outside, the basic inventions sold to others and

the final solutions from others that may be incorporated into a company’s products.

6. Measure efficiency at different stages of the model. For example, measure the ideas

discarded versus the ideas selected, the projects approved versus the projects

cancelled and the products in portfolio versus the products discontinued.

7. Categorise the contributions to revenues as being those of internal and external origin.


36

8. To motivate external parties, use different metrics than the ones used to motivate

internal parties.

9. Check the legality of maintaining databases regarding external people.

10. Introduce a “talent” (i.e., knowledge) metric for your external parties.

There is also a corollary to this Decalogue:

Avoid using your current innovation performance measurement system without

adaptation for the new Open Innovation model.


37

Chapter 5 Conclusions

5.1 Conclusions regarding the research questions

Does Open innovation require the same sort of measurement approaches as


The main conclusion is that the same sort of approaches can be used but it is important to

adapt the metrics to capture the whole value of the new approach. If a company embraces the

new model but uses exactly the same performance measurement system and metrics, then

there is a high risk that the real value of the new model will not be properly reflected. Such a

scenario leads to the wrong conclusions about the suitability of the new model for the

company.

What metrics are useful when the Open Innovation model is in use?

Most metrics remain useful in the Open Innovation model, but some types must be carefully

considered:

• Ratios between external and internal contributions.

• Metrics regarding new ways of profiting (e.g., licensing, venturing, spin‐offs etc.).

• Metrics regarding the efficiency at intermediate steps of the technology maturation

cycle.

• Qualitative metrics for external sources not under the control of the company.

• New metrics for motivating external people.

• Intellectual Property metrics.

What changes are needed to metrics when the Open Innovation model is embraced?

It is clear that the changes needed are related to the set of metrics already in place. It will be

necessary to add new metrics for new results, to split some metrics to reflect the contribution

of external sources and to complement some metrics to achieve the same–or at least similar–

impact inside and outside the organisation.

5.2 Conclusions regarding the research aim The aim of this project is to produce recommendations for metrics used in evaluating the

effectiveness of an ‘Open Innovation’ approach to research.


38

The first clear recommendation is to use metrics regarding innovation in your organisation for

the purpose of properly managing the technology that a company develops or acquires.

The second recommendation is to adapt the metric system when one embraces the Open

Innovation model. These changes are designed to be effective in measuring innovation and in

acquiring appropriate data for making decisions.

The third recommendation is to apply the “Decalogue” presented in section 4.4.

5.3 Further work A valuable research objective would involve taking a real performance measurement system

from a company embracing Open Innovation and applying the changes proposed herein. Then,

one could compare the metrics results from the old performance measurement system with

those from the new scheme. The purpose for such work would be to determine whether the

new measurement system better captures the value of the Open Innovation model.

Other possible paths for further research include broadening the scope of the research. The

initial scope shown in Figure 1‐1can be expanded in two directions:

• Innovation can occur outside the R&D department. For example, the Marketing

department could adopt an open model.

• The innovation may occur not only in the technology (product or process) sector, but

also in the methods used for completing any task or in the skills used in the business.

5.4 Implications of this research One interesting finding from this research is a determination of how beneficial new paradigms

for innovation might be for other technology management activities. Such paradigms would

necessarily be accompanied by the right set of metrics or measurement processes to provide

upfront demonstration of the benefits they can provide.

Another interesting question involves the fact that openness, in general, implies a blurring of

the boundaries between in‐company and out‐of‐company. Ethical and legal considerations are

needed to treat, for example, the personal data of collaborators who are not proper

employees. Special attention to confidentiality issues may be necessary in this open

environment.


39

A question arises once we accept that changes are needed in the measurement system when

the Open Innovation model is embraced: How can these changes be better planned and

implemented from the technology management point of view?

This research and work by others regarding Open Innovation still needs to be reviewed in light

of new publications on real‐life experiences of adopting the new model. While this research

was conducted, the number of publications regarding Open Innovation was growing steadily.

From a personal point of view, I am very happy to have followed this process. After working in

a research and development centre for 5 years, I now recognise that daily business pressures

can cause me to forget some of the basic principles of the research method. We should never

forget these rules while doing research and development. I have found that industrial activity

can benefit a great deal from a more academic point of view.


40

References

Adams, R., Neely, A., Yaghi, B. & Bessant, J. 2008, Proposal for Measures of Firm‐Level Innovation Performance in 12 Sectors of UK Industry, NESTA, UK.

Andrew, J.P. & Haanaes, K. 2008, Measuring Innovation 2008. Squandered Opportunities, The Boston Consulting Group.

Andrews, P. 2003, Open innovation: Using research from everywhere for new product and service development, IBM BCS.

Bremser, W.G. & Barsky, N.P. 2004, "Utilizing the balanced scorecard for R&D performance measurement", R & D Management, vol. 34, no. 3, pp. 229‐238.

Brown, M.G. & Svenson, R.A. 1988, "Measuring R‐And‐D Productivity", Research‐Technology Management, vol. 31, no. 4, pp. 11‐15.

Carr, A.S. & Pearson, J.N. 1999, "Strategically managed buyer–supplier relationships and performance outcomes", Journal of Operations Management, vol. 17, no. 5, pp. 497‐519.

Chesbrough, H. 2004, "Managing open innovation", Research‐Technology Management, vol. 47, no. 1, pp. 23‐26.

Chesbrough, H.W. 2006a, "New puzzles and new findings" in Open innovation: Researching a new paradigm, eds. H.W. Chesbrough, W. Vanhaverbeke & J. West, Oxford University Press, Oxford, pp. 15‐34.

Chesbrough, H.W. 2006b, "Open Innovation: A new paradigm for understanding industrial innovation." in Open innovation: Researching a new paradigm, eds. H.W. Chesbrough, W. Vanhaverbeke & J. West, Oxford University Press, Oxford, pp. 1‐12.

Chesbrough, H.W. 2003a, "The era of open innovation", MIT SLOAN MANAGEMENT REVIEW, vol. 44, no. 3, pp. 35‐41.

Chesbrough, H.W. 2003b, Open innovation: The new imperative for creating and profiting from technology, Harvard Business School Press, Boston, MA.

Chesbrough, H.W. & Appleyard, M.M. 2007, "Open innovation and strategy", CALIFORNIA MANAGEMENT REVIEW, vol. 50, no. 1, pp. 57.

Chiesa, V., Frattini, F., Lazzarotti, V. & Manzini, R. 2008, "Designing a performance measurement system for the research activities: A reference framework and an empirical study", J.Eng.Technol.Manag., vol. 25, no. 3, pp. 213‐226.

Cumming, B.S. 1998, "Innovation overview and future challenges", European Journal of Innovation Management, vol. 1, no. 1, pp. 21‐29.


41

Damanpour, F. & Wischnevsky, J.D. 2006, "Research on innovation in organizations: Distinguishing innovation‐generating from innovation‐adopting organizations", J.Eng.Technol.Manag., vol. 23, no. 4, pp. 269‐291.

Davila, T., Epstein, M.J. & Shelton, R. 2005, Making Innovation Work: How to Manage It, Measure It, and Profit from it, Wharton School publishing.

Driva, H., Pawar, K.S. & Menon, U. 2000, "Measuring product development performance in manufacturing organisations", International Journal of Production Economics, vol. 63, no. 2, pp. 147‐159.

EFQM 2004, EFQM Framework for Innovation , European Foundation for Quality Management.

Enkel, E. & Gassmann, O. 2007, Driving Open Innovation in the Front End, EURAM (Paris, France).

Godener, A. & Soderquist, K.E. 2004, "Use and impact of performance measurement results in R&D and NPD: an exploratory study", R & D Management, vol. 34, no. 2, pp. 191‐219.

Gwynne, P. 2007, Open Innovation's Promise and Perils.

Hauser, J.R. 1998, "Research, development, and engineering metrics", Management Science, vol. 44, no. 12, pp. 1670‐1689.

Hollanders, H. & van Cruysen, A. 2008, Rethinking the European Innovation Scoreboard: A New Methodology for 2008‐2010, INNO METRICS.

Huston, L. & Sakkab, N. 2006, "Connect and develop: Inside Procter & Gamble's new model for innovation", HARVARD BUSINESS REVIEW, vol. 84, no. 3, pp. 58.

Katila, R. 2008, "Measuring Innovation Performance" in Business Performance Measurement: Unifying Theory and Integrating Practice Cambridge University Press, , pp. 304.

Kerssens‐van Drongelen, I.C. & Bilderbeek, J. 1999, "R & D performance measurement: more than choosing a set of metrics", R & D Management, vol. 29, no. 1, pp. 35‐46.

KerssensvanDrongelen, I.C. & Cook, A. 1997, "Design principles for the development of measurement systems for research and development processes", R & D Management, vol. 27, no. 4, pp. 345‐357.

Krause, D.R. & Ellram, L.M. 1997, "Critical elements of supplier development The buying‐firm perspective", European Journal of Purchasing & Supply Management, vol. 3, no. 1, pp. 21‐31.

Kumpe, T. & Bolwijn, P.T. 1994, "Toward the Innovative Firm ‐ Challenge for Research‐And‐Development Management", Research‐Technology Management, vol. 37, no. 1, pp. 38‐44.

Laursen, K. & Salter, A. 2006, "Open for innovation: The role of openness in explaining innovation performance among UK manufacturing firms", Strategic Management Journal; 64th Annual Meeting of the Academy‐of‐Managemen.JOHN WILEY & SONS LTD,


42

CHICHESTER; THE ATRIUM, SOUTHERN GATE, CHICHESTER PO19 8SQ, W SUSSEX, ENGLAND, pp. 131.

Lichtenthaler, U. 2008, "Open Innovation in Practice: An Analysis of Strategic Approaches to Technology Transactions", IEEE TRANSACTIONS ON ENGINEERING MANAGEMENT, vol. 55, no. 1, pp. 148.

Linder, J.C., Jarvenpaa, S.L. & Davenport, T.H. 2003, Toward an Innovation Sourcing Strategy.

Mankins, J.C. 1995, Technology Readiness Levels, Advanced Concepts Office, Office of Space Access and Technology, NASA.

Mark Rogers 1998, The Definition and Measurement of Innovation, Melbourne Institute.

McGrath, M.E. & Romeri, M.N. 1994, "The R&D effectiveness index", The Journal of Product Innovation Management, vol. 11, no. 3, pp. 213.

OECD 2005, OSLO MANUAL: GUIDELINES FOR COLLECTING AND INTERPRETING INNOVATION DATA, OECD and Eurostat, Europe.

Ojanen, V. & Vuola, O. 2003, Categorizing the Measures and Evaluation Methods of R&D Performance, Telecom Business Research Center Lappeenranta, Lappeenranta.

Pappas, R.A. & Remer, D.S. 1985, "Measuring R‐And‐D Productivity", Research Management, vol. 28, no. 3, pp. 15‐22.

Pawar, K.S. & Driva, H. 1999, "Performance measurement for product design and development in a manufacturing environment", International Journal of Production Economics, vol. 60‐1, pp. 61‐68.

Pearson, A.W., Nixon, W.A. & Kerssens‐van Drongelen, I.C. 2000, "R&D as a business ‐ what are the implications for performance measurement?", R & D Management, vol. 30, no. 4, pp. 355‐366.

Sarkar, S. & Costa, A.I.A. 2008, "Dynamics of open innovation in the food industry", TRENDS IN FOOD SCIENCE \& TECHNOLOGY, vol. 19, no. 11, Sp. Iss. SI, pp. 574‐580.

Schumann, P.A., Ransley, D.L. & Prestwood, D.C.L. 1995, "MEASURING ‐ R‐AND‐D PERFORMANCE", Research‐Technology Management, vol. 38, no. 3, pp. 45‐54.

Sousa, M. 2008, Open innovation models and the role of knowledge brokers.

Werner, B.M. & Souder, W.E. 1997a, "Measuring R&D performance ‐ State of the art", Research‐Technology Management, vol. 40, no. 2, pp. 34‐42.

Werner, B.M. & Souder, W.E. 1997b, "Measuring R&D performance ‐ US and German practices", Research‐Technology Management, vol. 40, no. 3, pp. 28‐32.

Documents

Performance Metrics for the Open Innovation Model - David Scarlatti