S06 - Chiesa Et Al, 2009

8/10/2019 S06 - Chiesa Et Al, 2009

1/32

Performance measurement inR&D: exploring the interplaybetween measurement objectives,dimensions of performance and

contextual factorsVittorio Chiesa1, Federico Frattini2, ValentinaLazzarotti3 and Raffaella Manzini4

1Politecnico di Milano - Department of Management, Economics and Industrial Engineering,Piazza L. da Vinci 32, 20133 Milano, Italy. [email protected] di Milano - Department of Management, Economics and Industrial Engineering,Piazza L. da Vinci 32, 20133 Milano, Italy. [email protected]` Carlo Cattaneo LIUC, Corso Matteotti, 22, 21053 Castellanza, Varese, Italy.

[email protected]` Carlo Cattaneo LIUC, Corso Matteotti, 22, 21053 Castellanza, Varese, [email protected]

Measuring research and development (R&D) performance has become a fundamental concern

for R&D managers and executives in the last decades. As a result, the issue has been

extensively debated in innovation and R&D management literature. The paper contributes to

this growing body of knowledge, adopting a systemic and contextual perspective to look into

the problem of measuring R&D performance. In particular, it explores the interplay between

measurement objectives, performance dimensions and contextual factors in the design of a

performance measurement system (PMS) for R&D activities. The paper relies on a multiple

case study analysis that involved 15 Italian technology-intensive firms. The results indicate

that firms measure R&D performance with different purposes, i.e. motivate researchers and

engineers, monitor the progress of activities, evaluate the profitability of R&D projects, favour

coordination and communication and stimulate organisational learning. These objectives are

pursued in clusters, and the importance firms attach to each cluster is influenced by the context

(type of R&D, industry belonging, size) in which measurement takes place. Furthermore, a

firms choice to measure R&D performance along a particular perspective (i.e. financial,

customer, business processes or innovation and learning) is influenced by the classes of

objectives (diagnostic, motivational or interactive) that are given higher priority. The

implications of these results for R&D managers and scholars are discussed in the paper.

1. Introduction

M easuring performance and contribution tovalue research and development (R&D)

has become a fundamental concern for R&D

managers and executives in the last de-cades (Kerssen-van Drongelen and Bilderbeek,

1999). Since the 1990s, several phenomena have

R&D Management 39, 5, 2009. r 2009 The Authors. Journal compilation r 2009 Blackwell Publishing Ltd. 2009,4889600 Garsington Road, Oxford OX4 2DQ, UK and 350 Main Street, Malden, MA 02148, USA.
mailto:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]

8/10/2019 S06 - Chiesa Et Al, 2009

2/32

encouraged the development and adoption of

specific approaches for assessing the performance

of R&D: increasing turbulent dynamics in com-

petitive arenas, shortened life cycles, globalisa-

tion, reduced time to market, increased R&D

costs and risks (Nevens et al., 1990; Bayus,

1994; Wind and Mahajan, 1997; Wolf, 2006).

Although it is still acknowledged that the

measurement of R&D performance is a challen-

ging task that might also hinder the creative and

innovative capacity of the firm (Brown and Sven-

son, 1998), today the issue is being extensively

debated in the innovation and R&D management

literature and it raises the interest of practitioners

as well (Pappas and Remer, 1985; Brown and

Svenson, 1988; Sivathanu and Srinivasa, 1996;Werner and Souder, 1997; Hauser, 1998; Driva

and Pawar, 1999; Driva et al., 2000; Poh et al.,

2001; Loch and Tapper, 2002; Godener and

Soderquist, 2004; Ojanen and Vuola, 2006).

This paper aims at contributing to this growing

body of knowledge, adopting a systemic and

contextual perspective to look into the problem

of measuring R&D performance. In particular, it

explores the interplay between measurement ob-

jectives, performance dimensions and contextual

factors in the design of a performance measure-

ment system (PMS) for R&D activities. Morespecifically, it aims to understand: (i) which objec-

tives companies pursue when they measure R&D

activities performance and whether they can be

categorized in some ways; (ii) which approaches to

R&D performance measurement are used to pur-

sue different classes of objectives; and (iii) how the

importance attached to different classes of objec-

tives, and the approaches used to pursue them, are

affected by the context in which measurement

takes place. Although the literature on manage-

ment accounting and control has acknowledged

the importance of these topics (e.g., Simons, 2000;

Azzone, 2006), they have not been properly in-vestigated in R&D settings so far. What is more, a

better understanding of these issues would be

highly beneficial for R&D managers. The rich

empirical data presented and discussed in this

paper will provide R&D managers with a number

of insights that represent a valuable starting point

to design a PMS for R&D that is adequate to the

objectives they have in mind, and is appropriate as

well to the context in which their firm operates.

Moreover, some practical suggestions about how

to improve managers satisfaction with the PMS

are discussed in the paper.In order to pursue its objectives, the paper first

develops a reference framework, which is used as a

guide for the subsequent multiple case study

analysis. The empirical investigation involved a

number of Italian firms operating in technology-

intensive industries, for which technological inno-

vation and, therefore, the results of their R&D

efforts, are a major source of competitive advan-

tage. Because of the significant contribution of

R&D to the companys overall success, these firms

will be far more likely to systematically measure

their innovative activities performance. There-

fore, they represent an ideal empirical setting for

investigating the issues we are interested in.

The remainder of the paper is organised as

follows: Section 2 reviews the relevant literature

on R&D performance measurement, whereas

Section 3 describes the reference framework un-derlying the research. Section 4 illustrates the

methodology used for the empirical analysis,

and Section 5 discusses the result of the multiple

case study. Finally, Section 6 concludes and out-

lines some avenues for future research.

2. Literature review

Existing research into the measurement of R&D

performance in industrial firms has investigated

the topic from four different perspectives, asshown in Figure 1.

At a first level, research has basically focused on

the choice of the indicators or metrics that are best

suited to the characteristics of R&D. Brown and

Svenson (1998) find that an effective PMS for

R&D is built around a limited number of indica-

tors that measure results rather than behaviour,

and privileges objective and external metrics to

subjective and internal ones. Nixon (1998) ad-

vances that performance indicators for R&D

should have a strategic orientation and reflect the

firms critical success factors, they should be simple,

able to encourage change and to balance financialand non-financial perspectives. Werner and Souder

(1997) state that the most effective measurement

approaches for R&D are those that balance both

quantitative and qualitative metrics, as also under-

lined by Pawar and Driva (1999) and Bremser and

Barsky (2004). Hauser (1998) shows that the choice

of the most appropriate metrics should be based on

the type of R&D, whether it is applied research,

core technological development or basic research.

Other scholars in this stream of research investigate

the opportunity to use financial indicators in R&D

departments (Rockness and Shields, 1988) and tobuild a synthetic indicator of R&D productivity or

efficiency (Tipping et al., 1995).

Performance measurement in R&D

r 2009 The Authors

Journal compilation r 2009 Blackwell Publishing Ltd

R&D Management 39, 5, 2009 489

8/10/2019 S06 - Chiesa Et Al, 2009

3/32

At a second level, research has looked into the

choice of the performance dimensions, or perspec-

tives, along which the measurement of R&D

should be undertaken. Pawar and Driva (1999)

advance that the measurement of R&D perfor-

mance needs to be articulated into time, costs,

quality and flexibility dimensions. Kim and Oh

(2002) identify the following types of R&D per-

formance variables: market-oriented, R&D pro-ject-specific and R&D researcher-specific. Davila

(2000) analyses the use of cost, time and customer

(or market) information in the measurement of

new product development performance. Many

scholars in this stream of research have attempted

to apply the Balanced Scorecard (BSC) approach

(Kaplan and Norton, 1992) to R&D. Kerssen-van

Drongelen and Cook (1997), e.g., show how to

develop a measurement approach for R&D per-

formance that, integrating financial, client, inter-

nal business, innovation and learning perspectives,

allows to implement the firms R&D and compe-

titive strategy. Bremser and Barsky (2004) illus-trate how the BSC approach should be integrated

with the stage-gate system (Cooper, 1993) for the

organisation of innovation development activities.

At a third level, research has adopted a sys-

temic perspective in the study of R&D perfor-

mance measurement, assuming the whole PMS

for R&D as the unit of analysis. Kerssen-van

Drongelen et al. (2000), e.g., conceive the PMS

for R&D as comprising the following elements:

metrics organised into a consistent structure,

standards to measure performance against, fre-

quency and timing of measurement and formatfor information reporting. Similarly, Ojanen and

Vuola (2006) suggest that an effective PMS for

R&D should be an internally consistent set of

measurement perspectives, objectives, control ob-

jects and measurement process. In other words,

adopting a systemic perspective means looking at

R&D performance measurement in terms of a

system, which should be made of a set on

integrated and internally consistent elements

(Chiesa and Frattini, 2007), i.e. PMS objectives,

performance dimensions, metrics or indicators,control objects and measurement process.

Finally, a more strategy-oriented stream of

research has adopted a contextual perspective,

emphasising that a PMS for R&D should be

studied within the context in which it is used,

which is both internal and external to the firm.

This body of literature is consistent with the

largest part of the extant management accounting

and control research (e.g., Gordon and Miller,

1976; Gordon and Narayanan, 1984) and basi-

cally reminds that the PMS is used in a specific

R&D setting, being basic and applied research or

NPD (Pappas and Remer, 1985; Chiesa andFrattini, 2007), with a given amount and quality

of available resources (Godener and Soderquist,

2004), within the scope of a firms specific busi-

ness strategy, mission, values and management

style (Nixon, 1998; Kim and Oh, 2002; Loch and

Tapper, 2002) and, finally, in a broader competi-

tive, economic, social, cultural and political con-

text (Loch et al., 1996; Pillai et al., 2001).

3. Reference framework

Investigating the interplay between measurement

objectives, performance dimensions and contextual

Indicators andmetrics for R&D

performancemeasurement

Dimensions for R&D performancemeasurement

Performance MeasurementSystems for R&D

Performance MeasurementSystems for R&D within the firms internal

and external context

Figure1. Taxonomy of research streams on performance measurement in R&D.

Vittorio Chiesa, Federico Frattini, Valentina Lazzarotti and Raffaella Manzini

490 R&D Management 39, 5, 2009 r 2009 The AuthorsJournal compilation r 2009 Blackwell Publishing Ltd

8/10/2019 S06 - Chiesa Et Al, 2009

4/32

factors in the design of a PMS for R&D, the paper

contributes to the fourth level of analysis described

in the last section. Although the literature adopting

a contextual perspective has studied the impact of

several endogenous and exogenous variables on

the design of specific components of the R&D

PMS, an integrated view on how the objectives for

which the PMS is introduced affect the design of its

constitutive elements, and how they are in turn

influenced by the measurement context, has not

been advanced yet.

A company that seeks to measure the perfor-

mance of its R&D can do this with many different

purposes. According to Kerssen-van Drongelen

and Cook (1997), there are two main classes of

underlying reasons for R&D performance mea-surement, i.e. to motivate scientists and research-

ers and to diagnose activities and processes. Loch

and Tapper (2002) identify the following foremost

objectives for which firms control their R&D

performance: align behaviour and set up priori-

ties, evaluate and reward researchers, establish an

operative control and stimulate learning and im-

provement. Kerssen-van Drongelen et al. (2000)

add that performance measurement, especially in

complex new product development projects, can

serve the purpose of favouring communication

and coordination among top managers, middlemanagers and researchers.

Management accounting and control research

has repeatedly shown that the set of objectives for

which a firm measures its business performance

represents a driving force that heavily affects the

design of the other PMS constitutive elements,

e.g., dimensions of performance, indicators, con-

trol objects and measurement process (Neely,

1999; Bititici et al., 2000; Tuomela, 2005). Most

of all, Robert Simons (1994, 1995) explains that

companies pursue different objectives through

different control systems (called levers of con-

trol). Specifically, belief systems are used tocommunicate and reinforce the firms value and

the paths to be followed to identify and exploit

value creation opportunities; boundaries systems

are needed to encourage individual creativity,

within well-defined frontiers; diagnostic control

systems serve the purpose of coordinating and

motivating the implementation of the firms strat-

egy; and interactive control systems are used to

stimulate organisational learning, communication

and the emergence of ideas related to new business

opportunities. Each of these management control

systems has specific characteristics in terms ofperformance dimensions and perspectives of ana-

lysis, measurement frequencies and standards.

In our research, we wanted to understand

whether these general concepts (in particular,

the influence of measurement objectives over the

characteristics of the PMS) hold true in R&D

settings as well. Moreover, we were interested in

the role of the internal and external contextual

factors. In other words, considering that the PMS

for R&D is embedded in the firms internal and

external context, we wanted to understand

whether and how these factors are able to affect

both the importance the firm attaches to different

classes of objectives and the design of its consti-

tutive elements. This assumption is consistent

with the largest part of extant management con-

trol research (Chenhall and Morris, 1986).

Considering that the number of relevant con-textual factors is potentially countless and that a

PMS for R&D is comprised of a high number of

interrelated parts, we decided to confer a more

specific and definite scope to our research. As far

as the characteristics of the PMS are concerned,

we focused on the following elements: (i) the

dimensions along which R&D performance is

assessed and (ii) the indicators (or metrics) that

are used to measure performance along the above-

mentioned dimensions (see, e.g., Chiesa and Frat-

tini, 2007). As far as the context in which mea-

surement takes place is concerned, we focused onthe following factors that have been identified by

extant research as influential over the design of the

PMS for R&D (Pappas and Remer, 1985; Ker-

ssen-van Drongelen and Bilderbeek, 1999; Davila,

2000; Bremser and Barsky, 2004): (i) the type of

R&D activity that is measured (being it basic and

applied research or new product development); (ii)

the size of the firm and its R&D unit; and (iii) the

industrial sector the firm belongs to. The reference

framework that served as a guide for our empirical

analysis is summarised in Figure 2. It hypothesises

PMS

OBJECTIVES

PMS

CHARACTERISTICS:

-Performance dimensions

-Indicators

MEASUREMENT

CONTEXT:

-Type of R&D activity

-Size of the firm and the R&D unit

-Firms sector of activity

Figure 2. The reference framework.


r 2009 The Authors



8/10/2019 S06 - Chiesa Et Al, 2009

5/32

that the objectives for which a firm adopts the

PMS influence the design of its constitutive ele-

ments. Furthermore, it advances that the context

in which measurement takes place has a potential

effect over the choice of the PMS objectives as well

as the design of its constitutive elements. The

empirical analysis will help shed light on the

strength and nature of these hypothesised rela-

tionships.

4. Research methodology

We decided to use case study research as an

overall methodological approach for our empiri-

cal investigation. As suggested by a number ofscholars, this is in fact a very powerful method for

building a rich understanding of complex phe-

nomena (Eisenhardt and Graebner, 2007) that

requires the capability to answer to how and

why questions (Yin, 2003). In particular, we

used a multiple case study approach, which was

chosen because it allows both an in-depth exam-

ination of each case and the identification of

contingency variables that distinguish each case

from the other. Furthermore, multiple case stu-

dies are appropriate when attempting to exter-

nally validate the findings from a single casestudy, through cross-case comparisons (Eisen-

hardt, 1989). Therefore, they typically yield

more robust, generalisable and testable interpre-

tations of a phenomenon than single case study

research (Eisenhardt and Graebner, 2007).

The study involved 15 Italian firms from dif-

ferent industries (e.g., aerospace, pharma-biotech,

pharmaceuticals, machining centres, chemicals)

that were studied during the last 2 years (see

Table 1, where real names have been blinded for

confidentiality reasons). As a unit of analysis for

our case study, we considered the PMS used in the

firms R&D unit. In a number of cases, thestudied company had more than a single organi-

sational unit devoted to R&D activities; unfortu-

nately, in these instances, we had the opportunity

to study only one of them. This impeded us from

undertaking an embedded analysis that could

have provided richer information. We adopt the

largely applied and broad distinction between

Research and Development, including in the for-

mer basic and applied research activities and in

the latter the development of both incremental

and radical new products. We were able to

classify the activities undertaken in the R&Dunits that we considered as either Research and

Development, this indicating a widespread orga-

nisational separation between Research and

Development activities (Chiesa, 1996).

We gathered information basically through

direct interviews; in particular, we followed these

steps:

At the outset of each case, a relationship was

established with a senior manager from the

selected firm. This person was informed about

the research project through a written sum-

mary and a telephone meeting. During this

meeting, we asked the respondent to introduce

ourselves to the head of the firms R&D

function or to another R&D manager who

was responsible for the performance of the

R&D unit and the operation of the PMS; Then we personally interviewed the selected

R&D managers; we undertook two semi-

structured interviews for each respondent

(each interview lasted on average one and a

half hour) in order to gather the information

required to pursue the papers research objec-

tives. Direct interviews followed a semi-struc-

tured replicable guide (see Appendix A),

which comprised a set of open questions for

each of the relevant constructs in our reference

framework (e.g., objectives of the PMS and

dimensions of performance);

Secondary information was collected in theform of company reports and project docu-

mentation. In particular, we gathered and

analysed all the reporting documents that

were generated in support to the functioning

of the PMSs. These informed the researchers

with background information about the se-

lected firms, the type of R&D activity they

undertake and the approaches they use for

measuring R&D performance. Above all,

these secondary information sources were in-

tegrated, in a triangulation process, with data

drawn from the direct interviews, in order toavoid post hoc rationalisation and to ensure

construct validity (Yin, 2003);

All interviews were tape-recorded and tran-

scribed; generally, at this stage a telephone

follow-up with the respondents was conducted

in order to gather some important missing data.

Data and information gathered through the

case studies were manipulated before being ana-

lysed. In particular, we applied the following

techniques (Miles and Huberman, 1984): (i) data

categorisation, which requires the decomposition

and aggregation of data in order to highlightsome characteristics (e.g., objectives pursued

with the PMS or context in which measurement



8/10/2019 S06 - Chiesa Et Al, 2009

6/32

takes place) and to facilitate comparisons; (ii)

data contextualisation, which implies the analysis

of contextual factors, not included in the concep-

tual model, that may reveal unforeseen relation-

ships between events and circumstances. Then, a

preliminary within-case analysis was performed;

the purpose was to consider each case study as aseparate one and to systematically document the

variables of interest defined in the reference fra-

mework. Then, explanation-building procedures

were applied so that the relationships between the

PMS objectives, characteristics of the PMS and

context in which measurement takes place were

identified. Finally, a cross-case analysis was un-

dertaken for comparing the patterns that emerged

in each case study in order to arrive at a general

explanation of the observed phenomenon. These

structured procedures for data collection and

analysis, as well as the use of the semi-structuredinterview guide, helped enhance the reliability of

the research (Yin, 2003).

The following section reports and discusses the

empirical evidence we gathered for the 15 cases

included in our sample. It is used to illustrate the

interplay between measurement objectives, per-

formance dimensions and contextual factors in

the design of a PMS for R&D.

5. Results and discussion

The empirical evidence that was gathered for the

cases in our sample is synthesised and mapped

along the dimensions of the reference framework

in Appendix B, where some information about

the difficulties encountered in the adoption and

use of the PMS, as well as the satisfaction of

the firms executives with this tool, is reported.

Table 2 provides a synoptic view of these data to

allow a more straightforward comparison andanalysis.

An in-depth discussion of this empirical evi-

dence is reported in the following paragraphs.

5.1. PMS objectives and measurementcontext

The empirical analysis indicates that firms decide

to measure the performance of their R&D activ-

ities with multiple purposes. In particular, based

on our study of the 15 firms, it is possible to

identify the following list of major objectives a

company might aim at when it comes to measur-

ing R&D performance:

(1) Motivate researchers and engineers and im-

prove their performance in R&D activities;

(2) Monitor the progress of R&D activities with

respect to resource consumption targets, tem-

poral milestones and technical requirements;

(3) Evaluate the profitability of R&D activities

and their contribution to the firms economic

value;(4) Support the selection of the projects to be

initiated, continued or discontinued;

Table 1. The studied firms

Firm Sector of activity No. of

employees

Role of people interviewed

Company A Semiconductors 50,000 R&D projects managerCompany B Electronics for industrial

applications500 Director of R&D and quality manager

Company C Machining centres 160 Director of the technical departmentCompany D Aerospace 1,800 Planning and cost control managerCompany E Pharmaceuticals 500 Director of the development departmentCompany F Pharma-Biotech 60 Chief operating officerCompany G Chemicals 19,300 Director of innovation & technology in plastic

additivesCompany H Aerospace 9,000 Program managerCompany I Pharma-Biotech 700 Director of oncology divisionCompany L Pharmaceuticals 70 General director of research laboratoriesCompany M Household electrical appliances

and home automation

60,000 R&D platform manager

Company N Power generation technologies 2,200 Technology and business development managerCompany O Medical imaging diagnostic 1,000 Vice President for research and developmentCompany P Pharmaceuticals 3,000 Vice President for corporate drug developmentCompany Q Energy conversion 2,600 R&D director


r 2009 The Authors



8/10/2019 S06 - Chiesa Et Al, 2009

7/32

Table 2. Synoptic representation of the case study evidence

Firm Type of

R&Dactivity

Size of the

R&D unit(number ofemployees)

Sector PMS objectives Performance perspectives

associated with eachPMS objective

Company A Basic andappliedresearch

700 High-tech (1) Motivate scientistsand engineers(2) Favour coordinationand communication(3) Stimulateorganisational learning

(1) Innovation and learning(23) Business process

Company B New productdevelopment

100 High-tech (1) Monitor the progressof R&D activities(2) Evaluate theprofitability of R&Dactivities

(1a) Business process(1b) Customer(2) Financial

Company C New productdevelopment 15 High-tech (1) Motivate scientistsand engineers (1a) Innovation and learning(1b) Business processCompany D New product

development300 High-tech (1) Monitor the progress

of R&D activities(2) Evaluate theprofitability of R&Dactivities(3) Favour coordinationand communication

(1-23) Business process

Company E New productdevelopment

50 Science-based (1) Monitor the progressof R&D activities(2) Motivate scientistsand engineers

(1) Business process(2) Innovation and learning

Company F Basic andapplied

research

60 Science-based (1) Motivate scientistsand engineers

(1a) Innovation and learning(1b) Business process

Company G Basic andappliedresearch

300 High-tech (1) Motivate scientistsand engineers(2) Favour coordinationand communication3) Stimulateorganisational learning

(1a) Innovation and learning(1b) Business process(23) Business process

Company H New productdevelopment

200 High-tech (1) Monitor the progressof R&D activities(2) Evaluate theprofitability of R&Dactivities(3) Favour coordinationand communication4) Reduce uncertainty

(1a) Business process(1b) Customer(2a) Business process(2 b) Financial(2c) Customer(34) Business process

Company I Basic andappliedresearch

280 Science-based (1) Motivate scientistsand engineers(2) Select R&D projects(3) Monitor the progressof R&D activities

(1a) Innovation and learning(1b) Business process(23) Business process

Company L Basic andappliedresearch

70 Science-based (1) Motivate scientistsand engineers

(1a) Innovation and learning(1b) Business process

Company M New productdevelopment

200 High-tech (1) Monitor the progressof R&D activities(2) Evaluate theprofitability of R&Dactivities(3) Favour coordinationand communication

(4) Stimulateorganisational learning(5) Reduce uncertainty

(12a) Financial(12b) Customer(12c) Business process(345) Business process



8/10/2019 S06 - Chiesa Et Al, 2009

8/32

(5) Favour coordination and communication

among the different people and organisa-

tional units taking part in R&D activities;

(6) Reduce the level of uncertainty that sur-

rounds R&D activities; and

(7) Stimulate and support individual and organi-

sational learning.

Moreover, our analysis reveals that these objec-

tives are pursued in clusters by firms. A first group

of companies can be identified that use perfor-

mance measurement in R&D with the main pur-pose of improving the degree of control they

exercise on R&D activities, and to have a support

for taking more effective management decisions.

Examples of these firms are Companies B, D, N,

O and Q, which share the following as main

objectives for R&D performance measurement:

monitor the progress of activities, select R&D

projects and evaluate the profitability of R&D

activities. These are labelled in the paper as

diagnostic objectives, as they correspond to the

reasons underlying the use of diagnostic control

systems in Robert Simonss theory (Simons,1995). A second group of firms can be identified

that use performance measurement mainly as a

tool for motivating scientists and engineers, direct-

ing their efforts toward the long-term innovation

targets of the firm and overcoming the lack of

commitment that R&Ds largely intangible results

often determine. This is the case of Companies A,

C, F, G, I and L. They believe that motivating

researchers and technicians is a pre-requisite for

improving their performance in R&D activities,

and often establish to reward them on the basis of

the performance that are estimated by the PMS

(as it happens, e.g., in Companies F and I, where

performance measurement provides the data ne-cessary to operate a management by objectives

MBO rewarding system). We consider these firms

as mainly interested in pursuing motivational

objectives through R&D performance measure-

ment, which remind us of the reasons for which

managers use belief and boundaries control

systems (Simons, 1995). Finally, our analysis

unravels the existence of a number of companies

for which performance measurement also serves a

second purpose, besides the diagnostic or the

motivational objectives mentioned above. In

particular, they believe it is a very useful meansto improve and streamline the execution of R&D

activities and processes, overcoming some of the

Table 2. (Contd.)

Firm Type of

R&Dactivity

Size of the

R&D unit(number ofemployees)

Sector PMS objectives Performance perspectives

associated with eachPMS objective

Company N New productdevelopment

100 High-tech (1) Monitor the progressof R&D activities(2) Evaluate theprofitability of R&Dactivities(3) Select R&D projects

(123a) Customer(123b) Financial(123c) Business process

Company O Basic andappliedresearch

100 High-tech (1) Monitor the progressof R&D activities(2) Evaluate theprofitability of R&Dactivities

(3) Select R&D projects

(123a) Financial(123b) Customer

Company P Basic andappliedresearch

300 Science-based (1) Favour coordinationand communication(2) Stimulateorganisational learning(3) Motivate scientistsand engineers

(1) Business process(23a) Innovation andlearning(23b) Financial

Company Q New productdevelopment

150 High-tech (1) Monitor the progressof R&D activities(2) Select R&D projects(3) Evaluate theprofitability of R&Dactivities

(1) Business process(23a) Business process(23b) Financial

PMS objectives and performance perspectives are ordered on the basis of the importance attached by each firm.


r 2009 The Authors



8/10/2019 S06 - Chiesa Et Al, 2009

9/32

organisational barriers that characterise them. In

particular, performance measurement is also used

to favour coordination and communication, sti-

mulate quick and effective organisational learning

and reduce the level of uncertainty surrounding

critical decisions in R&D. These objectives are

acknowledged as relevant by Companies A, D, G,

H, M and P, and they are called interactive in

this paper, as they closely mirror the use of

interactive control systems described by Simons

(1995). Figure 3 shows the position of the 15 cases

discussed in this paper with respect to the three

clusters of objectives. It is interesting to note that

these results are consistent with the sparse empiri-

cal evidence available in the literature (Kerssen-

van Drongelen and Cook, 1997; Kerssen-vanDrongelen et al., 2000; Loch and Tapper, 2002).

Interestingly, our analysis suggests that the

importance a firm attaches to each cluster of

objectives is influenced by some characteristics

of the context in which measurement takes place.

In particular, diagnostic objectives are predomi-

nant in those firms that decide to measure the

performance of new product development activ-

ities. The cases of Companies B, D, H, M, N and

Q suggest that the need for control is stronger in

NPD than in basic and applied research. The

main reason is that the output of NPD activities issold directly on the market; therefore, the respect

of deadlines, quality requirements and target

costs in these activities has a more direct impact

on the firms market competitiveness than in the

case of basic and applied research, whose clients

are basically internal. Moreover, the amount of

financial and human resources involved in NPD is

very large (especially if compared with basic and

applied research), this making a proper evalua-

tion of R&D profitability, and an accurate prior-

itisation of projects, critical challenges for R&D

managers. Our analysis reveals that the need for

diagnostic control in R&D is also particularly

strong in large R&D units. The larger a firms

R&D unit, the higher the number of different

(and often interrelated) projects that are contem-

porarily undertaken, the higher the number of

researchers and engineers (often belonging to

different departments or functional areas) taking

part in these projects and the larger the amount of

resources devoted to R&D activities. These con-

ditions make the need for a tight diagnostic

control particularly evident, as it is clear forinstance in the case of Companies D and H.

Nevertheless, exerting this type of control over

R&D activities requires that the latter are, at least

to some extent, predictable, that standards to

measure performance against can be easily iden-

tified and that the progression of project activities

along a sequence of stages can be a priori identi-

fied. It is clear that new product development is

more foreseeable than basic and applied research,

but predictability also depends on the character-

istics of the industry in which a firm operates.

Kodama (1995) suggests that it is possible toclassify industrial sectors on the basis of the

probability with which an R&D project is frozen,

which is a measure of the predictability of R&D

activities.1 High-tech industries are characterised

by a freezing rate that decreases throughout the

R&D process, whereas science-based industries

are those in which the freezing rate always

Company A

Company B

Company C

Company D

Company E

Company F

Company GCompany H

DIAGNOSTIC

1) Monitor the progress of activities

2) Evaluate the profitability

of R&D activities

3) Select R&D projects

MOTIVATIONAL

1) Motivate scientists and engineers

INTERACTIVE

1) Favour coordination and communication

2) Stimulate organisational learning

3) Reduce uncertainty

Company N

Company O

Company M

Company L

Company I

Company P

Company Q

Figure3. The studied companies and the emerging clusters of objectives. Within each cluster, objectives are ordered on the basis oftheir relative importance.



8/10/2019 S06 - Chiesa Et Al, 2009

10/32

remains high. Our analysis suggests that firms use

performance measurement in NPD with the main

purpose of controlling activities in those indus-

tries (high-tech, in the definition of Kodama)

where the failure rate of projects and activities,

and hence their unpredictability, is smaller. This is

evident if we consider, e.g., that Companies B, D

and H operate in the electronics and aerospace

industries, Company M in the household appli-

ance and home automation sectors and Compa-

nies N and Q in the power generation and energy

conversion industries. More interestingly, Com-

pany O mainly pursues diagnostic objectives

although engaged in basic and applied research,

and this appears to be linked to the lower degree

of uncertainty characterising the industry inwhich it operates.

On the other hand, firms tend to pursue mo-

tivational objectives through R&D performance

measurement in basic and applied research. This

is clear if we consider the cases of Companies A,

F, G, I and L. The managers we interviewed

acknowledged that the motivational aspect of

performance measurement is stressed here be-

cause the activity is very much uncertain, mostly

unforeseeable and with distant time outcomes,

which makes it difficult to align researchers

efforts with the firms strategic goals. In theseinstances, improving the performance of research-

ers is a matter of stimulating their creativity but,

at the same time, directing their efforts to the

aspects that are relevant for the firm as a whole.

The need to motivate researchers through perfor-

mance measurement seems to be influenced not

only by the type of R&D activity. Company E

exemplifies for instance a situation where the

motivational purpose of performance measure-

ment is felt as particularly critical in new product

development. This is due to the fact that Com-

pany E operates in a science-based industry

(Kodama, 1995), where failure rates and degreesof uncertainties are significantly higher than zero

also in the downward phases of the R&D process,

i.e. development and testing, that turn out to be

highly unpredictable. Finally, our empirical ana-

lysis suggests that motivation of researchers might

become the main objective for R&D performance

measurement in small organisations. In these

cases, in fact, control is very often exerted on a

personal (or clan) basis (Ouchi, 1979), and hence

its very essence lies in the capability to align

employees efforts to the firms priorities. This is

very clear for instance in the case of Company C.Whereas diagnostic and motivational objec-

tives are mutually exclusive, interactive ones are

pursued by the firms in our sample along with

another class of objectives. Moreover, it emerges

that companies that conceive performance mea-

surement in R&D as a means to streamline

communication and coordination, to stimulate

organisational learning and to overcome deci-

sion-making inertia (i.e. to reduce uncertainty in

R&D) share as a common feature the fact of

having a very large R&D unit. Our case studies

clearly indicate that, in order to pursue both

diagnostic and motivational objectives, a criti-

cal aspect is to prevent researchers and engineers

from perceiving their creativity and autonomy as

being too much constrained by the PMS. There-

fore, it is important to adopt an enabling ap-

proach in the management of their performance(Wouters and Wilderom, 2008), where researchers

and engineers are continuously involved in the

measurement process, a double-loop flow of in-

formation keeps them informed about the pro-

gress of R&D activities and coordination and

collective learning allow researchers and engi-

neers to take more autonomous and empowered

decisions. The firms in our sample acknowledge

that this fundamental objective can be actually

pursued through the PMS, and that it becomes

more critical in large R&D units, characterised by

a higher degree of organisational complexity,hierarchy, fragmentation and vertical specialisa-

tion, which raises the need for a better coordina-

tion and more effective organisational learning

processes. Figure 4 shows the conditions under

which each cluster of objectives for R&D perfor-

mance measurement grows in importance.

5.2. PMS characteristics

Our analysis indicates first of all that the dimen-

sions along which R&D performance is evaluated

can be brought back to the BSC perspectives, assuggested by a number of scholars (Kerssen-van

Drongelen and Cook, 1997; Bremser and Barsky,

2004). In fact, the companies that we studied

measure R&D performance taking into account:

The economic and financial aspects associated

with R&D (financial perspective);

The extent to which R&D identifies and

satisfies the needs of its internal and external

customers (customer perspective);

The efficiency with which specific tasks and

processes are carried out (business process

perspective); The extent to which R&D contributes to

generate new knowledge and innovation


r 2009 The Authors



8/10/2019 S06 - Chiesa Et Al, 2009

11/32

opportunities (innovation and learning per-

spective).

There are firms that combine several different

perspectives in the measurement of R&D perfor-mance, although they do not knowingly report

using a BSC system. This is the case for instance

of Companies A and C, which take into account

both the innovation and learning and the business

process perspectives, or Company B, which com-

bines the financial, business process and customer

perspectives. Other companies measure perfor-

mance along a single most important dimension.

For instance, Company D uses a PMS very much

focused on internal processes efficiency.

More interestingly, our analysis suggests that

companies tend to use different performance

dimensions to pursue different classes of objec-tives. It noticeably emerges that financial and

customer perspectives are privileged by firms

pursuing diagnostic objectives, as it is clear

from the cases of Companies B, H, N and O.

This is obvious if we consider that selection and

prioritisation of R&D projects is carried out on

the basis of a projects contribution to the firms

competitive advantage, which depends on its

economic/financial outcome and appealing in

the eyes of the customers. On the other hand,

the innovation and learning perspective is wide-

spread among firms pursuing motivational ob-jectives, like Companies A, C, F, I and L,

suggesting that researchers and engineers need

to be motivated mainly on the basis of their

capacity to contribute to the firms innovation

potential. The business process perspective is

instead used by both firms pursuing diagnostic

and motivational objectives. In the latter case,using this perspective besides the innovation and

learning one serves the purpose of introducing a

dimension of performance that can be more

directly controlled by the researcher, which is

critical for motivational purposes as also indi-

cated by theories of action, design and expecta-

tion (e.g., Pritchard, 1990; Moizer, 1991). For

instance, Company A evaluates with this aim the

efficiency (subjectively assessed by peers) with

which researchers perform specific tasks or ac-

quire specific competencies. Firms pursuing di-

agnostic objectives use instead the business

process perspective with the main purpose ofintroducing an operative form of control that

financial and customer-oriented measures do not

allow to perform. As far as interactive objectives

are concerned, it emerges that they are associated

mainly with the business process perspective.

What is interesting to note in this case is that

the efficiency in undertaking business processes is

evaluated in an interactive manner, through a

continuous involvement of engineers and re-

searchers in the measurement process, as it

emerges from the cases of Companies D, H and

M. This is in fact a pre-requisite for introducingthe enabling approach in performance manage-

ment that stimulates the individual autonomy and

DIAGNOSTIC

OBJECTIVES

- New Product Development- Large firms and R&D units

- High Tech industries

MOTIVATIONAL

OBJECTIVES

- Basic and Applied Research- Small firms and R&D units

- Science Based Industries

INTERACTIVE

OBJECTIVES

- Large firms and R&D units

Figure4. Clusters of objectives and measurement context.



8/10/2019 S06 - Chiesa Et Al, 2009

12/32

creativity mentioned above. This evidence is con-

sistent with the results reported by Simons (2000)

on the use and role of interactive management

control systems.

Figure 5 suggests therefore that three arche-

types for R&D performance measurement

emerge, each characterised by an internally con-

sistent set of objectives, performance dimensions

and characteristics of the measurement context.

As is clear from Figure 5, our analysis suggests

that the choice of the dimensions along which

R&D performance measurement is carried out is

influenced mainly by the objectives that are pur-

sued, rather than the context in which measure-

ment takes place. In other words, contextual

factors do not appear to affect the design of the

PMS constitutive elements, as instead hypothe-

sised in Section 3. This is clear from the analysis

of Table 2, which shows that companies operating

in different contexts use the same performance

dimensions to pursue identical objectives through

the PMS.

Another interesting aspect unearthed by our

analysis is that firms with large R&D units seem

to be more inclined to use the PMS for diagnos-

tic purposes rather than for motivational ones,

DIAGNOSTIC

OBJECTIVES

- New Product Development


- High Tech industries

- Financial perspective

- Customer perspective

- Business process perspective

MOTIVATIONAL

OBJECTIVES

- Basic and Applied Research

- Small firms and R&D units

- Science Based Industries

-Innovation and learning perspective


INTERACTIVE

OBJECTIVES



Figure5. Emerging archetypes for performance measurement in R&D.


r 2009 The Authors



8/10/2019 S06 - Chiesa Et Al, 2009

13/32

in comparison with firms with small R&D units.

This is true unless large firms are engaged in basic

and applied research or operate in very uncertain

(science-based) industries, where the motiva-

tional use of the PMS also grows in importance

in large R&D units (see, e.g., the cases of Com-

panies A, I and P).

Furthermore, our analysis shows that each

performance dimension requires specific indica-

tors to be properly estimated. Table 3 provides an

overview of the indicators, associated with the

different performance dimensions, that were iden-

tified in our empirical analysis. It should be noted

that the choice of the indicators is not affected

by the specific class of objectives pursued by the

firm. The same indicators are used by the firmsin our sample, e.g., to measure the business

processes perspective for diagnostic or motiva-

tional purposes.

As is clear, firms try to measure each perfor-

mance dimension combining input, process and

output indicators (Brown and Svenson, 1988;

Hauser, 1998), without any discernible correla-

tion between the type of indicator and perfor-

mance dimension. It is also interesting to

underline the predominance of quantitative ob-

jective indicators (Werner and Souder, 1997). The

managers we interviewed explained their choice toprivilege objective indicators (also at the costs of

leaving out some important intangible facets of

R&D performance) with the need to ensure the

measurability of these metrics, which is funda-

mental for both diagnostic and motivational

purposes, as the theory of task motivation and

incentives (Locke, 1968) indicates. The scarcity of

subjective metrics (both quantitative and qualita-

tive) used by the firms in our sample is, however,

partially in contrast with the evidence gathered in

previous research (e.g., Chiesa et al., 2008), and

this dissimilarity deserves special attention in

future research.

5.3. Critical aspects associated with theuse of the PMS

Analysing the problems associated with the in-

troduction and use of the PMS in the firms in our

sample, it clearly emerges that pursuing motiva-

tional objectives, especially in basic and applied

research units, is far more challenging than using

a PMS with diagnostic purposes in an NPD

organisation. In the former case, the largest partof the managers we interviewed have struggled to

make scientists and researchers positively accept

the PMS, with sporadic cases in which one or two

researchers left the organisation after the intro-

duction of the PMS. It is interesting to note that,

even in these cases, managers have not abandoned

the idea of bringing in the PMS (apart from

Company I). Rather, they have adopted a more

incremental approach to establish the PMS in the

organisation, have often re-designed its charac-

teristics (e.g., measurement frequency, definition

of targets or performance metrics) to take into

account the complaints or suggestions for im-

provement coming from researchers and have

more deeply involved them in the measurement

process. Although tangible results are difficult to

observe, and the costs to run the system are

particularly high, these managers are on averagesatisfied with the PMS. On the other hand, no

particular problems have come across in those

firms that have introduced and used a PMS

with diagnostic purposes in their NPD units,

where tangible results (in terms, e.g., of improved

timeliness of development projects and time-

to-market) can often be observed and the satisfac-

tion of managers is particularly high. It should be

noted that the design of the PMS is often con-

tinuous, with the characteristics of the system

(e.g., the monitored performance dimensions)

that are modified over time to mirror the changesin the competitive strategy and the environment

in which the firm operates (see, e.g., Companies

F, N and M). Although costly, this approach

seems to be particularly useful to improve the

effectiveness of the PMS and managers satisfac-

tion with it.

6. Conclusions

This paper adopts a systemic and contextual

perspective to look into the problem of measuring

R&D performance. In particular, it explores theinterplay between measurement objectives, per-

formance dimensions and contextual factors in

the design of a PMS for R&D activities. With this

aim, we first developed a reference framework

that identifies: (i) the main contextual factors that

might affect the importance a firm attaches to

different objectives for R&D performance mea-

surement and the approaches it uses to measure

R&D performance (i.e. type of R&D activity,

industry belonging and size); (ii) the main aspects

that should be looked at when designing a PMS

for R&D (i.e. dimensions of performance andindicators). This framework was used as a refer-

ence model for the subsequent empirical analysis,



8/10/2019 S06 - Chiesa Et Al, 2009

14/32

Table 3. Performance perspectives and indicators

Performance perspective Type of

indicator

Indicators

Financial perspective Input Total cost of each R&D project (4)R&D annual spending (3)R&D annual investment (2)

Process Cost for acquiring a new technology (2)Present Value of R&D accomplishments/R&D expenditures (1)

Output IRR or NPV due to R&D projects (5)Profits due to R&D (3)ROI due to R&D projects (3)Sales (or % of sales) from new products (2)Cost (or % of cost) reduction from new projects (2)Market share due to R&D and innovations (2)

Customer perspective Input No. of interactions with customers during the project (6)% of budget dedicated to customer analysis or verification (3)

% of customer driven projects (1)No. of customers included in the project team (1)Process Time to market (4)

Engineering hours on projects/engineering hours on projects andtroubleshooting (2)No. of training sessions signed off by customer and delivered (1)No. of problem analysis reports requested and delivered (1)

Output No. of customer complaints (4)Customer satisfaction (2)% of support requests fulfilled (2)No. of new customers (1)Response time to customer requests for specials (1)

Innovation and learningperspective

Input No. or % of people with management experience (2)No. of employees in R&D (1)

Process No. of hours of staff training (5)

% of suggestions implemented (2)No. of meeting or time dedicated to the analysis of reasons for failure ofprevious projects (2)Capability to acquire new bodies of competencies (2)

Output No. of new ideas per year (4)No. of innovations delivered to production and commercialization (4)No. of citations of the researchers publications (4)No. of publications (3)No. of patents registered/pending (3)% of patent applications that resulted in registered patents (3)Average product life-cycle length (3)No. of improvements suggestions per employee (3)Scientific excellence of the new ideas identified per year (2)Market attractiveness of the new ideas identified per year (2)International relevance of the competencies acquired during 1 year (2)

No. of products in development or projects in course (2)No. of new processes and significant enhancements per year (1)Business process perspective Input Experience of R&D employees (2)

No. or % of employees involved in goal setting (2)Availability (knowledge) of advanced managerial tools, e.g., projectmanagement techniques (2)Availability (knowledge) of advanced IT support tools, e.g., rapidprototyping and design support tools (1)

Process % of projects respecting costs and budget (6)Agreed milestones/objectives met (6)Quality of documentation to development (5)Average annual improvement in process parameters, e.g., quality cost,lead time, WIP, reliability, capability, down time (4)% of collaboration objectives fully satisfied (3)% of projects that lead to new or enhanced products or processes (2)

% of R&D expenditures that lead to new or enhanced products orprocesses (2)Time spent on changes to original product/project specification (2)


r 2009 The Authors



8/10/2019 S06 - Chiesa Et Al, 2009

15/32

which involved 15 technology-intensive firms ac-

tively engaged in R&D activities.

The results of the empirical investigation,

which were discussed at length in the previoussection, indicate the existence of three archetypal

models that companies adopt to measure the

performance of their R&D, which represent in-

ternally consistent sets of contextual variables,

objectives for performance measurement and

performance dimensions. They are schematically

represented in Figure 5. The existence of these

models suggests that firms use performance mea-

surement in R&D to pursue different types of

objectives. In particular, two distinct clusters of

firms emerge: one that uses performance measure-

ment with the main purpose of exerting control

over R&D activities and support critical manage-ment decisions (diagnostic objectives) and the

other that conceives performance measurement

mainly as a means to improve the motivation of

researchers (motivational objectives). Another

set of objectives is concerned with the capability

of performance measurement to improve coordi-

nation and communication, streamline the execu-

tion of complex interrelated tasks and favour

organisational learning (interactive objectives).

They seem to be pursued by firms in combination

with diagnostic or motivational objectives. The

analysis also reveals that the importance firmsattach to each class of objectives is significantly

influenced by the context in which measurement

takes place (see Figures 4 and 5). It also emerges a

specialisation in the performance dimensions used

to measure the different classes of objectives

(see Figure 5). In particular, diagnostic objec-tives are pursued mainly through the use of

financial- and customer-related measures,

whereas indicators associated with the innovation

and learning perspective are the most widespread

among companies pursuing motivational objec-

tives. Contextual factors do not seem to directly

affect the choice of the performance dimensions

used in the PMS. Their impact is in fact mediated

by the classes of objectives the firm decides to

pursue. The paper also provides a synoptic view

of the indicators (or metrics) that the firms

investigated in the scope of our research use to

measure each dimension of performance (seeTable 3). No relevant specialisation in the use of

a given type of indicators emerges along the

different perspectives for performance measure-

ment or classes of objectives.

6.1. Implications for managers

Although the results of the paper should be better

conceived in an exploratory fashion, we believe

they hold valuable implications for R&D man-

agers and, especially, for the heads of R&D unitsand departments who are interested in designing a

PMS for the organisation they are responsible for.

Table 3. (Contd.)

Performance perspective Type of

indicator

Indicators

% of projects using a common design platform (2)Rate of re-use of standard designs/proven technology (2)Average product/service cost variance (2)No. of collaborations stipulated/no. of collaboration opportunitiesidentified (2)Sum of revised project durations/sum of planned durations (1)

Output Average project delay (5)% of projects delayed or cancelled due to lack of funding (5)% of projects delayed or cancelled due to lack of human resources (4)Rate of successful projects, i.e. project achieving the assigned time, cost,quality (4)% of project milestones completed (4)Product quality, measured through indicators specific for each industry/

product (3)No. or % of products/projects completed (2)Degree of project completion (2)Total product development time (2)% of on time deliveries of specification to manufacturing (2)Average time of re-design (1)No. of customer detected design faults (1)

In parentheses the number of firms that used the specific metric.



8/10/2019 S06 - Chiesa Et Al, 2009

16/32

First, they are given a number of examples about

how a relevant sample of technology-intensive

firms have designed and used a PMS in their

R&D units, with the aim of pursuing different

classes of objectives under the influence of dis-

similar contextual variables. This rich body of

empirical evidence will provide R&D managers

with a number of insights useful to design a PMS

that is appropriate to the context in which they

operate. The synoptic view of the indicators used

by the firms in the sample (see Table 3) can be a

particularly helpful starting point for the defini-

tion of the set of metrics to be used in their PMSs.

Second, our analysis suggests that introducing

and using a PMS in a basic and applied research

unit to pursue motivational objectives can beparticularly challenging and costly. Some ap-

proaches that are likely to improve R&D man-

agers satisfaction with this type of PMS appear

to be: use of an incremental approach to establish

the PMS into the organisation (e.g., using it first

to measure the most repetitive and predictable

tasks); continuous re-design of the PMSs char-

acteristics to incorporate the suggestions for im-

provement coming from researchers; and a deeper

involvement of researchers in the measurement

process. Finally, the cases in our sample suggest

that a continuous re-design of the PMS to mirrorthe evolution of the firms competitive and R&D

strategy is an important ingredient of success,

notwithstanding the main purposes for which

the PMS is designed and used.

6.2. Implications for research

The paper adds to our understanding of perfor-

mance measurement in R&D because it is one of

the first contributions, to our best knowledge,

that systematically studies the objectives forwhich a firm decides to measure its R&D activ-

ities performance. Moreover, it explores whether

established concepts in management accounting

research (e.g., the relationship between objectives

for performance measurement and characteristics

of the PMS) can be applied in R&D settings as

well. Finally, the approach adopted in the paper

can encourage researchers in the field of R&D

performance measurement to investigate whether

and how the other dimensions of the performance

measurement context (e.g., the firms R&D strat-

egy or the R&D organisational structure) influ-ence the importance a firm attaches to different

classes of objectives.

6.3. Limitations and future research

The study obviously has some limitations. First,

because of the adopted research methodology,results cannot be statistically generalised; they

can only be analytically extended to other indus-

trial firms operating in technology-intensive in-

dustries. Even if the internal validity of the

empirical results is ensured by the cross-case,

explanation-building and pattern-matching ana-

lyses, the study does not explicitly take into

account the effects that other contextual factors

are likely to have on the choice of the objectives

and the characteristics of the PMS elements.

Therefore, further research should be aimed at

exploring the joint effects of other contextualfactors (e.g., the firms R&D strategy) on the

design of the PMS. Second, our analysis is ex-

ploratory in intent. Although we provide some

evidence on the satisfaction of R&D managers

and executives with their measurement system, we

do not systematically assess the capability of a

PMS with specific characteristics to accomplish,

in a given context, the objectives for which it has

been designed. This represents an interesting

avenue for future research, which would require

the development of an appropriate measure of

effectiveness for an R&D PMS and a statistical

analysis of a representative sample of firms.Finally, we believe that adopting a longitudinal

perspective to study the organisation-wide im-

pacts associated with the adoption of the R&D

PMS is another interesting avenue for future

investigation.

References

Azzone, G. (2006) Sistemi di controllo di gestione.

Metodi, strumenti e applicazioni. Milano: Etas.

Bayus, B.L. (1994) Are product life cycles really getting

shorter? Journal of Product Innovation Management,

11, 300308.

Bititici, U.S., Turner, T. and Begemann, C. (2000)

Dynamics of performance measurement systems.

International Journal of Operations and Production

Management, 20, 692704.

Bremser, W.G. and Barsky, N.P. (2004) Utilizing the

balanced scorecard for R&D performance measure-

ment. R&D Management, 34, 229238.

Brown, M.G. and Svenson, R.A. (1988) Measuring

R&D productivity. Research-Technology Manage-

ment, 31, 1115.

Brown, M.G. and Svenson, R.A. (1998) MeasuringR&D productivity. Research-Technology Manage-

ment, 41, 3035.


r 2009 The Authors



8/10/2019 S06 - Chiesa Et Al, 2009

17/32

Chenhall, R.H. and Morris, D. (1986) The impact of

structure, environment, and interdependence on the

perceived usefulness of management accounting sys-

tems. The Accounting Review,61, 1635.

Chiesa, V. (1996) Separating research from develop-

ment: evidence from the pharmaceutical industry.

European Management Journal, 14, 6, 638647.

Chiesa, V. and Frattini, F. (2007) Exploring the differ-

ences in performance measurement between research

and development: evidence from a multiple case

study. R&D Management, 37, 283301.

Chiesa, V., Frattini, F., Lazzarotti, V. and Manzini, R.

(2008) Designing a performance measurement system

for the research activities: a reference framework and

an empirical study. Journal of Engineering and Tech-

nology Management, 25, 213226.

Cooper, R. (1993) Winning at New Products: Acceler-ating the Process from Idea to Launch. Reading:

Addison-Wesley.

Davila, T. (2000) An empirical study on the drivers of

management control systems design in new product

development. Accounting, Organisations and Society,

25, 383409.

Driva, H. and Pawar, K.S. (1999) Performance mea-

surement for product design and development in a

manufacturing environment.International Journal of

Production Economics, 60, 6168.

Driva, H., Pawar, K.S. and Menon, U. (2000) Measur-

ing product development performance in manufac-

turing organizations. International Journal ofProduction Economics, 63, 147159.

Eisenhardt, K.M. (1989) Building theories from case

study research. Academy of Management review, 14,

4, 532550.

Eisenhardt, K.M. and Graebner, M.E. (2007) Theory

building from cases: opportunities and challenges.

Academy of Management Journal, 50, 2532.

Godener, A. and Soderquist, K.E. (2004) Use and

impact of performance measurement results in

R&D and NPD: an exploratory study. R&D Man-

agement, 32, 191220.

Gordon, L.A. and Miller, D.A. (1976) A contingency

framework for the design of accounting information

systems. Accounting, Organizations and Society, 1,

5969.

Gordon, L.A. and Narayanan, V.K. (1984) Manage-

ment accounting systems, perceived environmental

uncertainty and organizational structure: an empiri-

cal investigation. Accounting, Organizations an So-

ciety, 9, 3347.

Hauser, J.R. (1998) Research, development and engi-

neering metrics.Management Science,44, 16701689.

Kaplan, R.S. and Norton, D.P. (1992) The balance

scorecard measures that drive performance. Har-

vard Business Review, 70, 7179.

Kerssen-van Drongelen, I.C. and Bilderbeek, J. (1999)

R&D performance measurement: more than choos-ing a set of metrics. R&D Management, 29, 35

46.

Kerssen-van Drongelen, I.C. and Cook, A. (1997) De-

sign principles for the development of measurement

systems for research and development processes.

R&D Management, 27, 345357.

Kerssen-van Drongelen, I.C., Nixon, B. and Pearson,

A. (2000) Performance measurement in industrial

R&D.International Journal of Management Reviews,

2, 111143.

Kim, B. and Oh, H. (2002) An effective R&D PMS:

survey of Korean R&D researchers. International

Journal of Management Science, 30, 1931.

Kodama, F. (1995) Emerging Patterns of Innovation.

Boston: Harvard Business School Press.

Loch, C.H., Stein, L. and Terwiesch, C. (1996) Mea-

suring development performance in electronics in-

dustry. Journal of Product Innovation Management,

13, 320.Loch, C.H. and Tapper, S. (2002) Implementing a

strategy-driven performance measurement system

for an applied research group. Journal of Product

Innovation Management, 19, 185198.

Locke, E.A. (1968) Toward a theory of task motivation

and incentives.Organizational Behaviour and Human

Performance, 3, 157189.

Miles, M.B. and Huberman, A.M. (1984) Qualitative

Data Analysis. Newbury Park: Sage.

Moizer, P. (1991) Performance appraisal and rewards.

In Ashton, D., Hopper, T. and Scapens, R.W. (eds.),

Issues in Management Accounting. Englewood Cliffs:

Prentice Hall, pp. 126144.Neely, A. (1999) The performance measurement revo-

lution: why now and what next?International Journal

of Operations and Production Management, 19, 205

228.

Nevens, T.M., Summe, G.L. and Uttal, B. (1990)

Commercializing technology: what the best Compa-

nies do. Harvard Business Review, 68, 154163.

Nixon, B. (1998) Research and development perfor-

mance measurement: a case study. Management

Accounting Research, 9, 329355.

Ojanen, V. and Vuola, O. (2006) Coping with the

multiple dimensions of R&D performance analysis.

International Journal of Technology Management,33,

279290.

Ouchi, W. (1979) A conceptual framework for the

design of organizational control mechanisms. Man-

agement Science, 25, 9, 833848.

Pappas, R.A. and Remer, D.S. (1985) Measuring R&D

productivity. Research Technology Management, 28,

1522.

Pawar, K.S. and Driva, H. (1999) Performance mea-

surement for product design and development in a

manufacturing environment.International Journal of

Production Economics, 60-61, 6168.

Pillai, A.S., Joshi, A. and Rao, K.S. (2001)

Performance measurement of R&D projects in a

multi-project, concurrent engineering environment.International Journal of Project Management, 20,

165177.



8/10/2019 S06 - Chiesa Et Al, 2009

18/32

Poh, K.L., Ang, B.W. and Bai, F. (2001) A compara-

tive analysis of R&D project evaluation methods.

R&D Management, 31, 6376.

Pritchard, R.D. (1990) Measuring and Improving Orga-

nizational Productivity: a Practical Guide. New York:

Praeger.

Rockness, H.O. and Shields, M.D. (1988) An empirical

analysis of the expenditure budget in research and

development. Contemporary Accounting Research,4,

568581.

Simons, R. (1994) How new top managers use control

system as levers of strategic renewal. Strategic Man-

agement Journal, 15, 169189.

Simons, R. (1995)Levers of control How Managers Use

Innovative Control System to Drive Strategic Re-

newal. Boston: Harvard Business School Press.

Simons, R. (2000)Performance Measurement & ControlSystems for Implementing Strategy. Upper Saddle

River: Prentice Hall.

Sivathanu, P.A. and Srinivasa, R.K. (1996) Perfor-

mance monitoring in R&D projects. R&D Manage-

ment, 26 , 5765.

Tipping, J.W., Zeffren, E. and Fusfeld, A.R. (1995)

Assessing the value of your technology. Research-

Technology Management, 38, 2239.

Tuomela, T.S. (2005) The interplay of different levers

of control: a case study of introducing a new perfor-

mance measurement system. Management Account-

ing Research, 16, 293320.

Werner, B.M. and Souder, W.E. (1997) Measuring

R&D performance State of the art. Research-Technology Management, 40, 3442.

Wind, J. and Mahajan, V. (1997) Issues and opportu-

nities in new product development: an introduction

to the special issue. Journal of Marketing Research,

34, 112.

Wolf, M. (2006) The world must get to rips with seismic

economic shifts. Financial Express, February, 7.

(available at:http://www.financialexpress-bd.com).

Wouters, M. and Wilderom, C. (2008) Developing

performance-measurement systems as enabling for-

malization: a longitudinal field study of a logistics

department.Accounting, Organizations and Society,

33, 488516.Yin, R.K. (2003) Case study Research: Design and

Methods. London: Sage.

Notes

1. Kodama (1995) classifies industrial sectors into

dominant design, high-tech and science-based in-

dustries. These three clusters differ with respect to

their level of risk, defined as the probability that an

R&D program expenditure is frozen in an inter-

mediate phase of the R&D process (freezing rate).

Dominant design industries are characterised by a

freezing rate that dramatically decreases from basicthrough applied research, coming to zero in the

development phase. The dominant design industries

are: food, textile, pulp and paper, printing and

publishing, oil and paints, petroleum and coal,

rubber, ceramics, iron and steel, transportation,

energy. High-tech industries are characterised by a

freezing rate that decreases throughout the R&D

process, but remains 40 even in the late develop-

ment. High-Tech industries are: ordinary machin-

ery, electrical machinery, communications and

electronics, precision equipment and aerospace.

Science-based industries are those in which the

freezing rate always remains high throughout the

entire R&D process. Science-based industries are

basically pharmaceuticals and industrial chemicals.

Vittorio Chiesais Professor of R&D Strategy andOrganisation at Politecnico di Milano. He is

member of the Management Committee of MIP

the Business School of Politecnico di Milano,

where he is responsible for the Technology Strat-

egy area. His main research interests concern

R&D management and organisation, technology

strategy and international R&D. He has pub-

lished 6 books and more than 100 papers, includ-

ing 40 articles in leading international journals

such as the Journal of Product Innovation Man-

agement, IEEE Transactions on Engineering Man-

agement and International Journal of Operationsand Production Management.

Federico Frattini (Corresponding author) is Assis-

tant Professor at the Department of Manage-

ment, Economics and Industrial Engineering of

Politecnico di Milano. His research interests

concern R&D performance measurement, the

organisation of R&D activities and the commer-

cialisation of innovation in high-tech markets. He

has published more than 30 papers, including

articles in R&D Management, Journal of Engi-

neering and Technology Management, and Inter-national Journal of Technology Management.

Valentina Lazzarotti is Assistant Professor at

Universita` Carlo Cattaneo LIUC (Castellanza,

Varese). She teaches Economics and Business

Organization and Management Control Systems

at LIUC. She obtained her Master Degree in

Economics at Universita` Bocconi (Milano). Her

research interests concern R&D performance

measurement and organization. She has pub-

lished more that 30 papers including articles in

Journal of Engineering and Technology Manage-ment and International Journal of Technology

Management.


r 2009 The Authors


http://www.financialexpress-bd.com/http://www.financialexpress-bd.com/

8/10/2019 S06 - Chiesa Et Al, 2009

19/32

Raffaella Manzini is Associate Professor at

Universita` Carlo Cattaneo LIUC (Castellanza,

Varese). She teaches Economics and Business

Organization and Technology Strategy at LIUC

and Politecnico di Milano. Her research interests

concern technology strategy and planning, R&D

management and organization. She has published

more than 60 papers including articles in leading

journals such as Long Range Planning and R&D

Management.

Appendix A. Interview protocol

What kind of products or services does yourcompany supply? Which are the distinctive

characteristics of the industry in which it

operates? How high is the probability that

an R&D expenditure is frozen in an inter-

mediate phase of R&D process? How long

does this failure rate remain higher than zero?

Is it higher than zero also in the downward

phases of the R&D process, i.e. development

and testing? How do these characteristics

affect the challenges your firm is confronted

with in R&D?

Which are your companys core competences?

Which are the most critical for your competi-tive advantage?

What percentage of your companys annual

turnover is invested in research and/or devel-

opment activities?

What kind of R&D activity does your com-

pany carry out? Is it basic and applied re-

search or new product development? Is it

undertaken in distinct organisational units?

Which are the main characteristics of your

R&D organisation? Which is the scope of the

activities undertaken in the R&D unit we are

focusing on?

How many people does your company em-

ploy? How many scientists and researchers are

employed in R&D? And how many in the

units we are focusing on?

How long have you been measuring the per-

formance of your R&D unit? Which are the

main characteristics of the performance mea-

surement system you have been employing?

Have they evolved over time?

Why have you decided to measure R&D

performance? Which are the main objectives

that you pursue through R&D performance

measurement? How would you rate them on

the basis of their importance for your R&D or

competitive strategy? Are measurement objec-

tives mutually exclusive to some extent? What kind of performance dimensions does

your company measure in order to pursue the

established objectives? Are these performance

dimensions completely controllable by re-

searchers and engineers? Is the choice of the

performance dimensions influenced by the

objectives you wish to pursue? Are there any

other reasons behind the choice of these

dimensions?

Which indicators are employed to operatively

measure performance? Do you use different

indicators to measure different performancedimensions? Is the choice of the indicators

constrained by any reasons (e.g., incompat-

ibility with the corporate-level PMS)?

Did your company suffer from any con-

straints of human and financial resources in

designing and implementing the performance

measurement system? Is the set of objectives

that your firms pursue, or the performance

dimensions that it employs, somehow limited

by a lack of resources? And what about the

degree of measurement formalization?



8/10/2019 S06 - Chiesa Et Al, 2009

20/32

8/10/2019 S06 - Chiesa Et Al, 2009

21/32

8/10/2019 S06 - Chiesa Et Al, 2009

22/32

8/10/2019 S06 - Chiesa Et Al, 2009

23/32

8/10/2019 S06 - Chiesa Et Al, 2009

24/32

8/10/2019 S06 - Chiesa Et Al, 2009

25/32

8/10/2019 S06 - Chiesa Et Al, 2009

26/32

8/10/2019 S06 - Chiesa Et Al, 2009

27/32

8/10/2019 S06 - Chiesa Et Al, 2009

28/32

8/10/2019 S06 - Chiesa Et Al, 2009

29/32

8/10/2019 S06 - Chiesa Et Al, 2009

30/32

8/10/2019 S06 - Chiesa Et Al, 2009

31/32

8/10/2019 S06 - Chiesa Et Al, 2009

32/32

Documents

S06 - Chiesa Et Al, 2009