The determination of success factors in European demonstration projects for new propulsion systems and transport concepts

The determination of success factors in Europeandemonstration projects for new propulsion systems and

transport concepts

Arjan Heyma *, Peter Zwaneveld, Wim Korver

Department of Transport, TNO Inro, Delft, The Netherlands

Abstract

Based on collected data of 45 European demonstration projects for new propulsion systems andtransport concepts within the project urban transport options for propulsion systems and instruments foranalysis (UTOPIA), an analysis is performed with regard to factors that can be related to the success ofthese demonstration projects. Success rates are distinguished with respect to technical, economic, mobilityand environmental aspects, and with respect to speci®c project objectives. Correlation between projectcharacteristics and success items are used to formulate hypotheses, which are tested in a multivariate Logitanalysis. The results show that the success of demonstration projects for new propulsion systems andtransport concepts is positively a�ected by the use of more developed technologies, the use of transportmodes that are independent of transport chains, using them at short distances, involving manufacturers aspartners within the project and by a lack of governmental, political and environmental barriers for theintroduction of the demonstrated new transport concepts. In addition the results show a trade-o� betweenenvironmental and technical success. Ó 2000 Elsevier Science Ltd. All rights reserved.

1. Introduction

Urban areas su�er from the consequences of tra�c. Congestion, emissions and noise burdenboth residents and visitors. Transport solutions based on new propulsion systems could providesolutions to these problems. To investigate new transport modes, the project urban transportoptions for propulsion systems and instruments for analysis (UTOPIA) has been commissionedby the European Commission, Directorate General for Energy and Transport.

Transportation Research Part D 6 (2001) 1±20www.elsevier.com/locate/trd

* Corresponding author.

1361-9209/00/$ - see front matter Ó 2000 Elsevier Science Ltd. All rights reserved.

PII: S1361-9209(00)00008-0

The main objective of the UTOPIA project is to provide decision-makers with the necessarytools, methods, and guidelines for hastening the market introduction of the most appropriatetransport solution based on new propulsion systems. The work schedule of UTOPIA is dividedinto ®ve main steps: inventory, analysis, framework methodology, on-site testing, and guidelinesand decision support development. The duration of UTOPIA is 30 months, starting from January1998. A European consortium of 24 organisations from 12 di�erent countries carries out theproject. Partners among others are Energy Saving Trust (UK), CERTU (France), AEA Tech-nology (UK), AMOR (Austria), DITS (Italy), Volkswagen AG (Germany), University of Twente(NL), and TNO (NL).

One of the aims of the project is to learn from previous experiences with new transport modes.Therefore, detailed data has been collected from 45 demonstration projects all over Europe. Thisdata is used for many of the previously mentioned objectives and activities. The work presented inthis article report results of analysing this data. The objective of the analysis is to ®nd aspects thatcontribute to the success of projects with respect to economical, environmental, mobility andtechnical aspects.

Section 2 discusses the procedure used to collect the data and gives an overview of the dem-onstration projects included in the database. Section 3 elaborates on the determination of thesuccess of projects. Section 4 presents the main results from an analysis that relates success itemsto characteristics of the projects and the transport modes demonstrated. This analysis results inhypotheses on success factors for demonstration projects and transport modes in Section 5, wherea further discussion and investigation of these hypotheses is carried out. Finally, Section 6 givesconclusions and suggestions for further research.

2. Overview of data collection process, de®nitions, and projects

The selection process of the demonstration projects is carried out in two steps. First, basicinformation is collected by all partners of the UTOPIA project on demonstration projects withnew transport modes based on new propulsion systems in their country. This has resulted ininformation on about 184 European projects. An overview of these projects can be found on thewebsite of the UTOPIA project: http://www.utopia-eu.com/. Next, demonstration projects areselected for which in-depth information is collected.

The database is used for the analyses of success factors (which is reported in this paper), toprovide basic information for decision support systems that support the ex-ante and ex-postevaluation of demonstration projects, and to formulate policy guidelines for the market intro-duction of new transport modes. Data is collected by face-to-face interviews, backed up by ex-isting evaluation reports. Questions have been asked by means of a standardised survey withregard to the following topics:· general information;· market aspects, travel demand and industrial impact;· technical aspects of propulsion systems and vehicles and fuel infrastructure;· social, institutional and regulatory aspects;· experiment set-up (also known as Strategic Niche Management) and· general expectations of future developments in tra�c and transport.

2 A. Heyma et al. / Transportation Research Part D 6 (2001) 1±20

Although demonstration projects di�er largely in technology, organisation and legal environ-ment, the use of a standardised survey guarantees that the analysis can be performed in a con-sistent manner. The survey mainly focusses on objective circumstances and characteristics ofprojects, which makes the information comparable. The evaluation framework for the demon-stration projects is kept the same throughout.

The data was collected in the period from November 1998 up to March 1999. The databaseincludes 45 demonstration projects of which 32 projects involve only passenger transport(71%), nine projects involve only freight transport (20%) and four projects involve both pas-senger and freight transport (9%). For details on all demonstration projects, see Korver et al.(1998); Zwaneveld et al. (1998); UTOPIA (1999). Tables 1 and 2, respectively provide anoverview of fuel types and propulsion systems and vehicle concepts and transport concepts thatare involved. Table 1 clearly shows that one of the selection criteria was the use of alternativepropulsion systems. This explains the high number of electric propulsion systems and thecombination of spark ignition engines with LPG and CNG and compression ignition engineswith CNG and Biogas. ÔPopularÕ combinations of vehicle and transport concepts in Table 2 arethe urban bus used for collective transport, passenger vans in demand-responsive systems forpassenger transport, freight vans for freight distribution and small trucks for freight distri-bution.

Table 3 shows total capital costs per project together with an overview of the main vehicleconcept that is used in the project. In general, more expensive projects involve more expensivevehicle concepts. Note that this is not automatically so due to for example the number of vehiclesper project. Projects involving all-purpose cars and passenger vans range from quite cheap to veryexpensive. The cheapest project (capital or investment costs) only costs 4400 Euro (TWIP, Electricscooters at the Groningen railway station). The most expensive project amounts to 29 million

Table 1

Overview of fuel types and propulsion systemsa

Fuel type Propulsion system

Human

power

ICE

(petrol)

ICE

(diesel)

Electric Hybrid Other Total

Gasoline 1 1 2

Diesel 2 2

LPG 6 6

CNG 6 3 9

Biogas 1 2 3

Human power 3 3

Electricity (C) 16 1 2 19

Electricity (R) 10 1 11

Total 3 14 5 26 5 2 55a ICE� Internal combustion engine; Hybrid�Electric propulsion combined with ICE; LPG�Lique®ed petroleum gas;

CNG�Compressed natural gas. Electricity is divided into the conventional generating mix (C) and from renewable

sources (R).

A. Heyma et al. / Transportation Research Part D 6 (2001) 1±20 3

Ta

ble

3

Ov

erv

iew

of

tota

lca

pit

al

cost

sa

nd

veh

icle

con

cep

ts

To

tal

cap

ita

lco

sts

in1

00

0E

uro

Veh

icle

con

cep

t

Bic

ycl

eS

coo

ter

Urb

an

two

-sea

ter

All

-pu

rpo

se

car

Pa

ssen

ger

va

n

Urb

an

bu

s

Fre

igh

t

va

n

Sm

all

tru

ck

Oth

erT

ota

l

0±

50

11

13

50

±2

00

32

5

20

0±

50

02

12

11

11

9

50

0±

10

00

12

25

10

00

±3

00

01

31

5

>3

00

01

32

6

To

tal

21

29

28

13

533

Ta

ble

2

Ov

erv

iew

of

veh

icle

con

cep

tsa

nd

tra

nsp

ort

con

cep

tsa

Veh

icle

con

cep

tT

ran

spo

rtco

nce

pt

Ind

iv.

pa

ss.

Ta

xi

Dem

an

d

resp

.p

ass

.

Rid

e-

sha

rin

g

Ren

t-a

-

veh

icle

Co

ll.

pa

ss.

Inte

gr.

pa

ss.

Ind

iv.

frei

gh

t

Co

llec

tio

n/

dis

trib

.

To

tal

Bic

ycl

e1

11

3

Sco

ote

r1

1

Urb

an

two

-sea

ter

21

3

All

pu

rpo

seca

r2

12

23

12

13

Pa

ssen

ger

va

n4

11

6

Urb

an

bu

s1

31

14

Fre

igh

tv

an

14

5

Sm

all

tru

ck4

4

Oth

er1

41

6

To

tal

41

82

71

91

211

55

aIn

div

.�In

div

idu

al;

Pa

ss.�

Pa

ssen

ger

tra

nsp

ort

;R

esp

.�R

esp

on

siv

e;C

oll

.�C

oll

ecti

ve;

Inte

gr.�

Inte

gra

ted

.O

ther

incl

ud

essl

ow

mo

vin

gveh

icle

s,

peo

ple

mo

ver

sa

nd

lig

ht

rail

.


Euro (Biogas propelled busses in the city of Link�oping). Average projects costs were 2.16 millionEuro.

3. The identi®cation of success

The main research question investigated in this paper is:What factors can be related to the success of demonstration projects, both to the project itself and tothe transport mode tested in the project?

To answer this question, success is de®ned by indicators for the following aspects:· success on environmental aspects;· success on technical aspects;· success on social and mobility aspects;· success on economic aspects and· success with respect to stated project objectives.

Given the complexity of the demonstration projects and the many aspects that can cause therate of success, this analysis should:· be easy to understand for external readers,· take into account the fact that success of a project can be measured with respect to several

aspects,· enable comparison of diverse demonstration projects, and· allow for quantitative conclusions.

Given these considerations, binary success indicators are selected. Failure or success is easy tounderstand and the use of multiple success indicators allow the incorporation of multiple aspects.Furthermore, binary coding enables graphical presentation of the results and the use of variousstraightforward statistical methods. As far as we know, the construction and use of this kind ofsuccess indicators with the applied statistical procedures has not been used before.

Success can thus assume only two values, namely ÔyesÕ or ÔnoÕ. The indicators are based on moredetailed questions, which compare the transport mode tested in the project with the conventionaltransport mode. The selection of the conventional alternative was done by the project leader ofthe demonstration project. The more detailed questions are multiple choice questions, with thefollowing possible answers:· Ôstrongly disagreeÕ� 0;· Ôslightly disagreeÕ� 1;· Ôno opinionÕ� 2;· Ôslightly agreeÕ� 3;· Ôstrongly agreeÕ� 4.

Note that a score of two refers to Ôcomparable with the conventional modeÕ. A higher scoremeans an improvement, a lower score means deterioration. Details on the precise construction ofthe indicators are presented in Table 10 (see Appendix A) and in Zwaneveld et al. (2000). Ingeneral, each success indicator is an aggregation of several more detailed objectives. Mobilitysuccess is a very heterogeneously de®ned indicator, while economic success is straightforward.Table 4 provides an overview of scores on success indicators. Note the high score on environ-mental success and the low score on technical success.


Table 5 shows success scores for the di�erent aspects, distinguished by success with respect tothe project objectives. Although di�erences between the two columns are not statistically signif-icant, 1 one cannot disregard the correlation between success on project objectives and technical,mobility, and economic success. The high scores on environmental success are most likely due tothe mechanism of project selection. The UTOPIA project mainly aims at investigating new en-vironmentally friendly transport modes. The lack of statistical signi®cance may be surprising, asone would expect that better transport modes lead to more successful projects. It can be concludedthat projects are not directly aimed at testing whether particular transport modes are ready for thetransport market, but at other ÔlearningÕ aspects. A suggestion for future demonstration projects isthat if one aims at testing whether a particular transport mode is in itself a promising transportmode, then special attention should be paid to the set-up of the project and the project objectives.

To see which factors determine the success of a demonstration project and to learn both fromthe deployment of transport modes and the set-up of demonstration projects, one can look atanswers to questions in the database regarding possible causes for success or problems withinprojects. Problems that were mainly encountered were technological, concerned the planning andmonitoring of the project or the daily operation of the vehicles. Examples are the extremely in-novative concept of the hybrid-electric propulsion systems in the Centaur project, which requiredstudy, adaptation and development of new components, and the massive increase in use of the

Table 4

Scores and number of valid scores on the success indicators

Success indicator ÔYesÕ (%) ÔNoÕ (%) Number of valid scoresa

Technical 29 71 42 out of 45

Economic 44 56 41 out of 45

Mobility 69 31 42 out of 45

Environmental 83 17 42 out of 45

W.r.t. project objectives 59 41 39 out of 45a Non-valid scores on individual success indicators are due to non-response to detailed objectives as mentioned in Table

10.

Table 5

Success rates distinguished by success with respect to project objectives

Indicator Score ÔnoÕ on success w.r.t. project

objectives (15 observations)a (%)

Score ÔyesÕ on success with respect to project

objectives (23 observations)a (%)

Technical 26 30

Economic 47 36

Mobility 67 74

Environment 73 91a In total, 38 demonstration projects have valid scores on all success indicators.

1 All tests of signi®cance in this paper are based on a 90% con®dence level. In most cases, signi®cance is tested with a

two-sample Chi-square test. Low signi®cance levels are mainly due to the small sample size, which is inherent to the type

of investigation. We still believe that the analysis is valuable as the data is unique and may for the ®rst time give a

statistical basis to the evaluation of demonstration projects for new propulsion systems and transport concepts.


Docklands Light Railway, which was not accounted for in the planning of the project. To a lesserextent, problems with rules and regulations, commercial and ®nancial problems, and problemsregarding infrastructure and communication were mentioned. The UK legislation, for instance,prohibits standing passengers on vehicles of the size used in the Gulliver electric bus project. In theMerton CNG ¯eet project only part of the funding was available at the start of the project and theVillage Underground project experienced unexpected problems with the construction of a tunnel.Success factors that were mainly mentioned by respondents were the reliability and quality of theo�ered service, the co-operation with manufacturers and the good relationship with (local) au-thorities. Also important were personal e�ort and dedication of people involved in the project, thesolid preparation and planning, good project management, the fact that all partners had an in-terest in good results, communication within the project and the good co-operation betweenproject partners.

From these answers one can learn something about factors that in¯uence the success ofdemonstration projects, but whether these factors really cause success remains questionable.Respondents often have no complete picture of all elements that in¯uence the success of a project.In addition, respondents tend to look at factors that are variable during the project and not atfactors that were decided upon before the start of the project, like chosen technology, area ofapplication and transport use. Therefore, an analysis is performed in which the correlation be-tween characteristics of demonstration projects and their success is investigated more objectively.As can be seen in Table 6, it is possible to test a number of factors in a quantitative way.

It is clear that the success of demonstration projects, as de®ned above, partly depends on thesubjective assessment of respondents with respect to a selective amount of information, despite thefact that there is a large number of detailed objectives included, which are measurable and ratherobjective. There are three ways in which this can bias our results. Firstly, using subjective eval-uations of success instead of the true, but latent, objective success rate results in a reporting errorproblem that leads to an overestimation of the e�ect of success factors. Secondly, if true success isde®ned by other factors than included in our aggregated measures of success, then a measurementerror exists which shows the estimated e�ects of success factors as less signi®cant than they in factare. And thirdly, if success factors by themselves are a�ected by the success of the project, then the

Table 6

Success factors mentioned by respondents in order of importance

Subjective success factors Possibilities for testing objectively

1 Technology Extensive

2 Quality of service in daily operation Extensive

3 Planning and monitoring None

4 Co-operation with manufacturers Limited

5 Rules and regulations Limited

6 Personal e�ort and dedication None

7 Project management None

8 All partners had interest in outcome None

9 Commercial and ®nancial circumstances Extensive

10 Infrastructure Limited

11 Communication and co-operation between partners None


e�ect of success factors will be underestimated. For example, if success stimulates the marketposition of the applied technology, the e�ect of this market position on success is more di�cult tomeasure. Given the lack of information about each of the possible sources of bias, the resultingbias will be ambiguous. These considerations are to be kept in mind when analysing the results ofour assessment.

4. The determination of success factors

The correlation between characteristics of a demonstration project and more or less objectivelydetermined success rates enables the formulation of hypotheses concerning factors that in¯uencethe success of demonstration projects. These hypotheses may be used to test the importance of thesubjectively given success factors in Table 6.

4.1. Technological factors

The type of propulsion system and type of fuel are two technological factors that may in-¯uence the project success. A comparison of success rates shows that technical success is mainlyrelated to the age of the applied technology. Compression engines and LPG score high whileyounger technologies like hybrid-electric propulsion and biogas fuel are technically disappoint-ing. However, the only statistically signi®cant correlation with technical success is found forelectric and spark-ignition engines, and for renewable electricity, which all show a negative re-lation with success. Results for economic and mobility success are similar to those for techno-logical success, probably because younger technologies are more expensive and harder to ®t intodaily operation.

In case of propulsion systems, environmental success is the opposite of technical, economic andsocial success, but this seems to justify experiments with the aim to develop young technologies.Compared to project objectives, the projects with compression engines and relatively new fueltypes (biogas, CNG and electricity from renewable sources) seem to be most successful, but thecorrelation is not statistically signi®cant. Overall the compression engine seems most successful.Electric and hybrid-electric propulsion systems show the best environmental results, but scorepoor on technical, economic and mobility issues, which make them less suited for immediatemarket introduction. Out of all fuel types, CNG seems to be most successful, but still su�ers fromtechnical problems.

The hypothesis that the age of the technology applied in a demonstration project positivelya�ects the success of a project can be tested by determining success rates by the market position ofthe applied technology. Either the technology used requires ®nancial support and further devel-opment (8 cases), is commercially viable in limited applications (22 cases), or is already at a fullycommercial product stage (12 cases).

The results in Fig. 1 show that using commercial technologies may help to reach technical andeconomical objectives, but that technologies under development are at least as well in producinggood mobility and environmental results. Overall, projects seem to best ful®l objectives when acommercially viable technology is used in limited applications, but this result is not supported bystatistical signi®cance.


4.2. Operational factors

With respect to operational factors, passenger and freight vans show reasonable scores on allsuccess items, except on technical success. Other vehicles, which include people movers and lightrail, seem to give very good results, but hardly meet the project objectives, which must have beenset very ambitiously. The opposite is true for the urban two-seater, which score badly on technicaland economic issues, but completely meet the project objectives. Projects that include all-purposecars are far from successful. They may be so common that expectations and objectives are setmuch higher than for other vehicle concepts. Furthermore, there seems to be a trade-o� betweenenvironmental and technical success, which is illustrated by the urban two-seater with high en-vironmental success but low technical success. A similar trade-o� is present for transport con-cepts, illustrated by collection/distribution freight transport. Collective/continuous passengertransport is technically the most successful, but does not convince on other success items. Demandresponsive passenger transport seems to be very successful technically, economically and withrespect to mobility aspects, but scores relatively low on environmental issues. In contrast, indi-vidual passenger transport is very successful environmentally, but scores badly in comparisonwith the project objectives. Rent-a-vehicle projects do not seem to be very successful, but still meetthe project objectives relatively well. Overall, there is not one transport concept that performsbest. In addition, di�erences in success rates by vehicle or transport concepts are generally notstatistically signi®cant.

The comparison of success rates between vehicle and transport concepts is not conclusive aboutfactors for success. Neither is the variation in success between trip purposes. Only slightly morevariation is found for trip patterns. Three main conclusions can be drawn from that analysis.Firstly, projects tend to be more successful if focussed on passenger trips to and from the urbancentre. Secondly, there is a trade-o� between environmental and mobility success on the one handand technical and economical success on the other. And thirdly, projects for freight trips areeconomically more successful and environmentally less successful than projects that includepassenger trips. Since the economic element is relatively more important for freight transport, thismay be either a logical or a very satisfactory result.

Fig. 1. Success rates by market position of the technology used.


The results above suggest that trip distance is negatively related to the success of transportmodes in demonstration projects. To test this hypothesis, the average trip distance is determinedfor successful and unsuccessful projects, and depicted in Fig. 2. This ®gure seems to support thehypothesis for all success items, except for success compared to project objectives, although this isnot statistically supported. It can be carefully concluded that experimental transport modes thatfocus on shorter trip distances are more successful. If this situation is a proxy for market de-velopments, then the opportunities for new transport concepts and new propulsion systems aresmall for the long and medium distance transport market. Chances are found for the short dis-tance market, including feeder transport.

When transport modes in a demonstration project are part of a transport chain (16 out of 35cases), this may complicate the accomplishment of objectives and therefore reduce the projectsuccess. Fig. 3 shows evidence that this may be true for technical, economical and environmentalissues, and the extent to which project objectives are met. Only the latter di�erence is statisticallysigni®cant. As a result, new feeder systems may be confronted with implementation problems andopposition. Hastening the market introduction for these systems will not be easy.

Fig. 2. Average trip distances for successful and unsuccessful projects.

Fig. 3. Success rates by transport modes being part of a transport chain or not.


4.3. Financial factors

The information from the database with respect to ®nancial factors of demonstration projectsincludes capital and operational costs, both distinguished by vehicle, fuel infrastructure, non-fuel infrastructure, personnel, legal and other aspects. The overall quality of the data is ques-tionable, but for the most relevant items there seems to be fairly reliable data. Average costs percost item are derived for successful and unsuccessful projects. Table 7 gives a simpli®ed overviewof the sign of the relation between costs and success. This can be either positive (+), negative()), or there may hardly be any di�erence in average costs between successful and unsuccessfulprojects (0).

The compact overview shows that higher investments are related to higher environmentaland mobility success. Technical success is mainly found for small projects in terms of vehiclesand fuel infrastructure with relatively good support (i.e., higher costs) from non-fuel infra-structure and personnel. If capital costs are divided by the number of vehicles, then a positivecorrelation is found between vehicle costs and technical success. This points at positive resultsfrom the concentration of ®nancial resources to a few vehicles. Economic success can be di-rectly related to low capital costs, but is positively correlated with higher investments inpersonnel. Project objectives are more often reached in projects with relatively high investmentsin vehicles and fuel infrastructure and relatively low investments in project support likepersonnel.

Table 8 gives an indication of the relation between average operational costs per vehicle and thesuccess of demonstration projects. It seems that technical success can be obtained by higheroperational costs, which in practice means more service and better maintenance of vehicles andinfrastructure. Although economic success is expected to be correlated with lower capital costs, itappears to be related to higher operational costs per vehicle. Again, the maintenance of hardwarein the project seems to be important for success. Mobility success is also related to higher op-erational costs per vehicle, except for personnel. Since mobility success is also related to highertotal operational costs for personnel, this suggests that a good project team is important forsuccess, but that an ine�ective organisation may work counterproductively. Finally, projectobjectives are better reached when a larger part of the budget is used for vehicles and fuel in-frastructure, which is the same conclusion as for investments.

Table 7

Sign of relationship between average total capital costs and success rates

Cost element Technical

success

Economic

success

Mobility

success

Environmental

success

Success with respect to

project objectives

Vehicles ) ) + + +

Fuel infrastructure ) ) + + +

Other infrastructure + ) + ) )Personnel + + + + )Legal ) ) + + )Other + + + + )Total 0 ) + + +


4.4. Commercial factors

For the market introduction of new propulsion systems, new vehicle concepts and newtransport concepts, competitiveness seems to be of prime importance. Fig. 4 shows that transportmodes that replace conventional vehicle concepts (23 out of 42 cases) may be more successfulenvironmentally and with respect to the project objectives, but that they also experience moretechnical, economical and mobility problems. This makes it more di�cult to introduce these newtechnologies and concepts into the market. However, di�erences are small and only statisticallysigni®cant for economic success.

4.5. Infrastructure

In the analysis of costs it was found that investments in infrastructure and in the maintenanceof infrastructure seems to lead to higher success rates, with only a few exceptions. A comparisonof success rates between projects with and without infrastructure needs, reveals that economic andmobility success is negatively related to the need for fuel infrastructure but that environmentalsuccess and success with respect to project objectives is positively related to the need for fuelinfrastructure. This may result from the fact that the need for fuel infrastructure makes projectsmore expensive and more di�cult to handle in daily operation. At the same time, they may be

Fig. 4. Success rates when project does (not) replace transport concept.

Table 8

Correlation between operational costs per vehicle and project success

Cost element Technical

success

Economic

success

Mobility

success

Environmental

success

Success with respect to

project objectives

Vehicles + + + + +

Fuel supply + ) + ) +

Other infrastructure + + + + )Personnel + + ) + )


more directed towards environmental objectives 2 and therefore more successful with respect toproject objectives. Projects with non-fuel infrastructure needs turn out to be more successfultechnically, economically and with respect to mobility aspects. But they may also allocate toomany resources to infrastructure to be successful environmentally or with respect to the projectobjectives.

4.6. Organisational factors

Many factors that are subjectively identi®ed as a reason for success are organisational, in-cluding the planning and monitoring of projects, the co-operation with manufacturers, projectmanagement and the communication and co-operation between partners within a project. Mostof these factors can not be analysed objectively, except for the co-operation with manufac-turers. The correlation with success is analysed by comparing success rates between projectswith and without a manufacturer as project partner (23 and 17 cases respectively). The resultsin Fig. 5 suggest that all success items are positively correlated with the involvement ofmanufacturers. Despite the fact that the di�erences in success rates are not statistically sig-ni®cant, this seems to be an important conclusion from the analysis, which is supported by theopinion of respondents.

4.7. Legal factors

Since rules and regulations, and the relationship with (local) authorities are given as reasonsfor project success by many of the respondents, their correlation should be analysed. Unfor-tunately, the only crude indication of the in¯uence of legal factors on the success of demon-stration projects can be obtained from the distinction by country in which the demonstrationprojects takes place. Comparing success rates by country is very tentative, but may give at leastsome indication of the in¯uence of the (legal) setting for demonstration projects. The UK seemsto be a good environment for demonstration projects, which could possibly be contributed to thePowershift programme. It also has the most projects. There seems to be a distinction betweennorthern European countries (UK, Norway, the Netherlands and Sweden), which have verysuccessful projects, southern European countries (France and Italy), which have moderatelysuccessful projects, and German speaking countries (Austria and Germany), which experiencedisappointing results.

Additional evidence for the in¯uence of legal factors on the success of demonstration pro-jects is found for expected governmental, political or environmental barriers upon large-scalemarket introduction of new propulsion systems or vehicle concepts. Fig. 6 shows that projectsare in general less successful if any of these barriers are expected, except with respect tomobility aspects. The largest di�erence in success rates is found for technical issues, whichsuggests that rules and regulations a�ect the success of new technologies in the market. Again,

2 New fuel infrastructure opens the way for new, environmentally friendly fuels like CNG and electricity.


the result is put in perspective by the fact that none of the di�erences in Fig. 6 are statisticallysigni®cant.

5. Discussion of the impact of success factors

Based upon the analysis in the previous paragraph, a number of hypotheses concerning thee�ect of characteristics of demonstration projects on the success of these projects are formulated:

Hypothesis 1. The age of the applied technology positively a�ects the technical and economicalsuccess of a demonstration project.

Hypothesis 2. There is a trade-o� between technical and environmental success. Environmentallypromising transport modes su�er more often from technical problems and are therefore in theshort term less suited for large-scale market introduction.

Fig. 5. Success rates by involvement of a manufacturer in projects.

Fig. 6. Success rates by governmental, policy or environmental barriers.


Hypothesis 3. The average trip distance for experimental transport modes negatively a�ect theirsuccess, except when compared to project objectives. Therefore, projects that involve long tripdistances are in general less successful.

Hypothesis 4. Being part of a transport chain causes experimental transport modes to be lesssuccessful as a result of a lower degree of freedom in the application of the vehicle concept.

Hypothesis 5. Higher total investments in demonstration projects in general increase their tech-nical and mobility success, but decrease the economic and environmental success.

Hypothesis 6. Higher operational costs per vehicle, pointing at higher service and maintenance pervehicle, make projects more successful.

Hypothesis 7. Replacing conventional vehicle concepts by new vehicle concepts make projectsmore successful environmentally and with respect to mobility and project objectives, but leads tomore technical and economic problems.

Hypothesis 8. Involvement of a manufacturer as partner in a demonstration project is a guaranteefor higher success.

Hypothesis 9. The existence of governmental, political or environmental barriers for large-scalemarket introduction of new technologies reduces the project success.

The hypotheses above can be tested when included in a multivariate analysis of success. Thelimited number of demonstration projects requires an e�cient use of information. Since we wantto test whether we can classify a project as successful based on several factors at the same time, wecould use a discriminant or regression analysis. Since some of the factors for success are con-tinuous variables and success itself is a qualitative variable that only assumes the values ÔyesÕ orÔnoÕ, the use of a Logit model seems to be most appropriate. 3 The de®nition of the includedvariables is given in Table 11 (see Appendix A).

Table 9 shows results of the Logit analysis by estimating the probability of success for eachsuccess item. None of the equations is a statistically signi®cant explanation of success, which canbe mainly contributed to the low number of observations, but the equations are still informative.Success with respect to project objectives is relatively well explained, followed by technical andeconomical success. Mobility success is a very heterogeneously de®ned indicator, which results ina lack of correlated variation with the exogenous variables.

With respect to the hypotheses formulated above, a number of issues can be discussed. The ageof the applied technology positively a�ects the success of a demonstration project, not onlytechnically and economically, as suggested in Hypothesis 1, but also with respect to mobility andthe environment. Using new technologies thus increases the risks of a project. Still, commercially

3 Regression analysis has the advantage of obtaining ready to interpret parameters for both discrete and continuous

predictor variables. The choice of a Logit instead of a Probit model is arbitrary: both models give similar results.


viable technologies that are applied in limited applications seem to perform best in ful®lling theproject objectives. Apparently, projects aim at other ÔlearningÕ aspects than the readiness oftransport modes for the transport market.

Controlling for many project characteristics, the trade-o� between technical and environmentalsuccess is con®rmed by the estimation results (Hypothesis 2). This conclusion may motivate thedeployment of new technologies in demonstration projects. Hypothesis 3, that trip distancenegatively a�ects the technical, economical and mobility success of a demonstration project, isalso supported by the estimation results. It turns out that this is also true for success with respectto project objectives. For longer distances it seems hard to compete with the all-purpose car andnew transport concepts should therefore primarily be introduced in the short distance urbantransport market. However, environmental success is positively a�ected by trip distance, contraryto Hypothesis 3, which may be an explicit argument to have demonstration projects for longerdistance transport use. Hypothesis 4, that being part of a transport chain negatively a�ects the

Table 9

Estimation results for Logit models for success (t-values in brackets)

Dependent variable Technical

success

Economical

success

Mobility

success

Environmental

success

Success w.r.t.

objectives

Constant 9.57 20.19 1.93 )2.04 2.92

(0.92) (1.59) (0.42) (0.34) (0.31)

Limited applicable technology 2.78 3.44 0.35 0.24 10.96

(0.95) (1.24) (0.18) (0.10) (1.62)

Commercially applicable technology 8.11 7.58 1.62 1.17 3.70

(1.33) (1.71) (0.67) (0.52) (0.94)

Trip distance )1.53 )0.48 )0.59 0.72 )4.89

(1.25) (1.15) (1.08) (1.29) (2.14)

Part of transport chain )1.80 )2.73 ± )3.53 )15.57

(0.82) (1.21) ± (1.55) (1.62)

Total capital costs )0.69 )1.78 )0.04 0.45 0.03

(0.98) (1.56) (0.12) (0.80) (0.03)

Operational vehicle costs )0.33 0.01 )0.54 0.05 0.06

(0.99) (0.05) (0.20) (0.19) (0.09)

Other operational costs 0.74 0.52 0.22 0.01 0.84

(1.42) (1.69) (1.35) (0.05) (0.85)

Replacement of vehicle concept )4.40 )5.10 )1.06 3.28 7.89

(1.24) (1.69) (0.79) (1.51) (1.02)

Involvement of industry 0.91 4.22 1.42 0.98 5.98

(0.42) (1.29) (0.87) (0.41) (0.83)

Barriers expected )8.35 )0.37 )1.27 )2.76 )16.80

(1.44) (1.29) (0.96) (1.09) (1.34)

Environmental success )1.24 ± ± ± ±

(0.42) ± ± ± ±

Technical success ± ± ± )2.22 ±

± ± ± (0.87) ±

v2 value 16.179 18.144 20.121 29.274 8.037

Critical v2 value (P� 0.95) 7.261 7.261 10.851 7.261 6.571

Degrees of freedom 15 15 20 15 14

Number of observations 27 26 30 27 25


success of a new vehicle mode, is strongly supported by the estimation results. The lower degree offreedom seems to a�ect project success, which may have negative consequences for demonstrationprojects with feeder transport systems.

Total capital costs hardly a�ect the outcome of demonstration projects, except when looking ateconomic success. This seems to be a logical result. Hypothesis 5 therefore should be rejected.Operational vehicle costs do not appear to have in¯uence on project success either, which rejectsHypothesis 6. The conclusion can be that investment or operational costs cannot ``make or break''demonstration projects. Spending more money without a clear purpose does not lead to betterresults. If transport modes in a demonstration project replace conventional transport modes, theprobability of technical, economical and mobility success decreases, while that of environmentalsuccess and success with respect to project objectives increases (Hypothesis 7). Apparently it ishard to compete with conventional transport modes on technical, economic and mobility aspects,but easier with respect to project objectives and environmental aspects. Hypothesis 8, that theinvolvement of manufacturers as partners in a project has positive e�ects on project success, issupported by the estimation results for all success items. Expected governmental, political andenvironmental barriers for the large-scale introduction of new transport concepts and propulsionsystems are a good indication for the success of demonstration projects. The probability of successbecomes smaller, in particular in comparison with the project objectives, in accordance withHypothesis 9.

The results are not very strong in a statistical sense. However, if measurement errors andendogeneity of success rates and factors are more important than reporting errors, then the in-cluded factors in the analysis are stronger and statistically more signi®cant predictors for successthan they appear to be from the estimation results.

6. Conclusions

The analysis of success and characteristics of European demonstration projects for new pro-pulsion systems and transport concepts has resulted in a number of hypotheses that were tested ina multivariate analysis. The main conclusion is that there is evidence that success is in¯uenced by anumber of factors that can be selected at the start of a demonstration project. These successfactors should therefore be used in the assessment of new demonstration projects. For instance, ifa project proposal includes a very new technology, with no manufacturer involved, and theproposed transport concept is aiming for long distance trips and is in addition part of a transportchain, the likelihood of success will be low. In our view funding organisations (like for instanceDirectorate General for Energy and Transport, but also national ministries of environment) canuse these success factors as a checklist. If a project proposal rates negative on a certain number offactors, the project proposal should be adjusted. The information about success factors should becommunicated to people and organisations applying for funding. They can use the success factorsas a tool for setting up a better project proposal.

From our analysis we conclude that overall success is positively a�ected by the use of moredeveloped technologies, the use of transport modes that are no part of a transport chain, byapplying them at short distances, by involving manufacturers of the transport mode or propulsionsystem as partner in the project, and by making sure that there are few governmental, political and


environmental barriers for introduction of the new transport modes. Another main ®nding fromthe analysis is that there is a trade-o� between technical and environmental success. More at-tention for environmental issues by introducing new transport modes will generally imply more

Table 10

Details of the de®nition of success indicators

Indicator Surveyed aspects Decision rule for success Comments

Environmental success � Local air quality Average larger than 2. Over

50% of items score at least a

3. Missing items are ignored

New mode has to bene®cial

to the environment

� Regional/national/

internal air quality

� Greenhouse emissions

� Drive-by noise

� Liveability of cities

Technical success � Reliability Average larger or equal than

2. At most one item may

score 0 or 1 if average equals

2. Missing items are ignored

Equal score allowed as

conventional mode is

satisfactory

� Safety

� Acceleration

� Top speed

� Range

� Carrying capacity

Mobility success � Ease of use On average equal or larger

than 2. At most one item

may score 0 or 1. Missing

items are ignored

Equal score allowed as

conventional mode is

satisfactory

� Travel time

� Modal share of

transport mode

� Travel stress

� Local congestion

� Accessibility

� Passenger

compartment noise

Economic success � Whole life costing Value of at least 3. If value

of 2, the average value over

detailed costs questions has

to be at least 3

New mode has to score

better than conventional

Succes with respect to

the project objectives

Most frequent

mentioned objectives:

Score on all items at least

equal to 3. Only one missing

item allowed

Strict rule, due to the link

with project objectives

� Local air (78%)

� Liveability (68%)

� Noise (64%)

� Greenhouse (42%)

� Accessibility (36%)


technical problems, while technically mature transport modes will hardly produce environmentalgains. This is also illustrated by the e�ect of replacing conventional transport modes, which hasnegative e�ects on technical success, but positive e�ects on environmental success. A ®nal con-clusion is that discrepancies between individual success indicators and success with respect to theproject objectives point at the fact that demonstration projects in general aim at other learninge�ects than testing whether a particular transport mode is ready for introduction in the transportmarket.

Acknowledgements

The authors like to thank Michelle Bennemeer and an anonymous referee for valuable com-ments.

Appendix A

See Tables 10 and 11.

Table 11

De®nition and explanation of explanatory variables used in the Logit model

Explanatory variable De®nition (and value) Comments

Constant Standard intercept (1)

Limited applicable technology ÔYesÕ (1) or ÔNoÕ (0) Hypothesis 1

Commercially applicable technology ÔYesÕ (1) or ÔNoÕ (0) Hypothesis 1

Trip distance Average trip distance in kilometers Hypothesis 3

Part of transport chain ÔYesÕ (1) or ÔNoÕ (0) Hypothesis 4

Total capital costs ln (ÔTotal capital costs of the

project in EuroÕ +1)

Hypothesis 5. Decreasing returns to

scale are incorporated by using ÔlnÕfunction

Operational vehicle costs ln (ÔAnnual operating cost of

vehicles per vehicle in EuroÕ +1)



Other operational costs ln (ÔAnnual operating costs for

non-fuel infrastructure and

personnel per vehicle in EuroÕ +1)



Replacement of vehicle concept ÔYesÕ (1) or ÔNoÕ (0) Hypothesis 7

Involvement of industry ÔYesÕ (1) or ÔNoÕ (0) Hypothesis 8

Barriers expected ÔYesÕ (1) or ÔNoÕ (0), barriers

expected for fuel supply, environment

and policy

Hypothesis 9

Environmental success ÔYesÕ (1) or ÔNoÕ (0), see Table 10 Hypothesis 2

Technical success ÔYesÕ (1) or ÔNoÕ (0), see Table 10 Hypothesis 2


References

Korver, W., Riemersma, I., Heyma, A., Zwaneveld, P.J., 1998. UTOPIA: Overview of Dutch demonstration projects of

new environmentally friendly transport and propulsion systems, Delft, TNO report Inro/VVG 1998-20 (in Dutch).

UTOPIA, 1999. Deliverable 6A ± Summaries of case studies UTOPIA, Delft, TNO report Inro/VVG 1999-21.

Zwaneveld P.J., Heyma, A., Korver, W., Anreiter, W., Fischer, T., Marks, H., Manthey, A., 1998. Deliverable D2 ±

Overview of promising transport modes related to new propulsion systems, Delft, TNO report Inro/VVG 1998-21.

Zwaneveld, P.J., Korver, W., Heyma, A., Elzen, B., Ricci, S., Malavasi, G., Corsi, 2000. Deliverable D6 ) Analysis of

demonstration projects with new transport and propulsion systems, Delft, TNO repport Inro/VVG 2000

(forthcoming).


Documents

The determination of success factors in European demonstration projects for new propulsion systems and transport concepts