1

Ex post evaluation of the Cohesion Fund (including former

ISPA)

Work Package B: cost - benefit analysis of selected transport

projects

Jurate Vaznelyte, Evaluation Network MeetingBrussels, 20 October, 2011

2

Objectives of the study

Cohesion Fund and ISPA (2000 – 2006):

254 individual transport projects co-financed

Terms Of Reference for this study:

103 individual transport projects

10 schemes considered for ex post evaluation(38 individual projects out of 260 or 15%)

● Question 1: What were the impacts of these projects?

● Question 2: How can ex post CBA contribute to the practice of ex ante CBA?

● Question 3: What are the potential and limits of ex post CBA?

3

AVE Madrid-Barcelona

A23 Motorway

Lisbon – Algarve railway

M1 Motorway

IX B Corridor + Vilnius bypass

A2 Motorway

Bratislava Railway Upgrade

M0 Motorway

Agiou Konstantinou bypass

Thriasso-Pedio-Eleusina-Korinthos railway

Road

Rail

4

Ex ante stated objectives

Reduce travel time (6)

Increase capacity/reduce congestion (4)

Reduce operating costs (2)

Improve safety (8)

Improve connectivity (2)

Projects2 4 6 8 10

5

Ex ante impacts taken into account in CBA

Impacts on safety (10)

Impacts on the environment (4)

Impacts on transport operators (3)

Impacts on users (Time Savings, Vehicle Operating Costs) (10)

Other impacts (2)

Projects2 4 6 8 10

6

Ex ante CBA – its role in decision making

Compliance with EC CF application requirements (9)

Ensure value for money (6)

Choose alignment (2)

Choose design standards (2)

Prioritise elements of national transport strategy (1)

Allocate budget and optimise timing (0)

Projects2 4 6 8 10

7

Ex post economic evaluation – benefit-cost ratio (BCR)

Most projects show BCR larger than 1, i.e. the net project impact, as captured by the CBA is positive

Due to high capital costs, the AVE shows a BCR < 1. However, the wider socio-economic impact (not captured in the CBA) are significant

8

Ex post financial analysis – financial NPV

The financial NPV (before EC contributions) of seven of the ten projects is negative – not surprising as most projects do not generate direct commercial revenues.

The fact that most projects have an economic BCR > 1, but a negative financial NPV indicates that that the EC contributions help ‘unlock’ the economic benefits of these projects

9

Sources of benefits – railway projects

Railway projects

-20%

0%

20%

40%

60%

80%

100%

120%

Railway - Spain Railway - Portugal Railway - Greece Railway - Slovakia

Travel time VOCs Transport operator fares

Safety Carbon Air pollution

Noise Other  

10

Sources of benefits – road projects

Road projects

-40%

-20%

0%

20%

40%

60%

80%

100%

120%

140%

Road - Poland Road - Spain Road - Greece Road - Ireland Road - Lithuania Road - Hungary

Travel time VOCs Transport operator fares

Safety Carbon Air pollution

Noise Other  

11

Other key findings

Transparency and accountability

● The regular publication of ex post findings, whatever the outcome, generally increases transparency and accountability. It also help identify key learning points for future project planning

Evidence base

Feedback

● Average difference between ex ante and ex post was 13.4% - relatively low

● Factors explaining the difference: time lag, project delays, scope alterations and unforeseen circumstances

● Generally in line with objective to build in sufficient spare capacity to cover project lifetime. Two exceptions (one too high, one too low)

● Ex ante NPV generally higher than ex post: the main driver for discrepancy was the demand forecast

● Together with capital cost and utilisation rates findings, it is indicative of optimism bias

● Ex ante demand modelling would benefit from improvement in study area definition, modal coverage, inclusion of induced traffic and variable demand

● CBA practice would benefit from more consistency in scope, more granularity (e.g. business vs. non-work users) and harmonisations of assumptions, parameters and approaches

Capital costs

Utilisation rates

NPV

Modelling practice

CBA practice

12

Key issues considered

Strengths and weaknesses of CBA methodologies as applied by the Member States

1 2 3 4 5

Effectiveness of CBA as a tool to support project generation and decision of Member States and Commission

Utility of ex post CBA from the point of view of project promoters, Member States and the Commission

Ex post CBA as a tool for evaluating project impacts

Relevance and potential utility of CBA for macro-economic modelling

13

Strengths and weaknesses of CBA1 2 3 4 5

● There is a basic level of consistency in CBA methodology used for transport infrastructure projects across EU Member States

● CBA framework used in the ten projects is both consistent with DG REGIO’s guidelines and with good practice

Scope of the ex ante analysis sometimes narrower than what

would be ideal (externalities not always

considered)

Different appraisal parameters used across

EU Member States

(e.g. HEATCO values)

Weaknesses in demand modelling, counterfactual definition and modelling

of network effects

14

Effectiveness of CBA in decision-making

1 2 3 4 5

● The current contribution of CBA in the decision making process is limited

● Mainly used to confirm project’s value of money and to support funding application

CBA results Regional development and

environmental concerns

CBA does not fully take into account other key impacts Time that would be required to fully embed CBA into planning process

CBA offers more potential as decision-making tool

Technically Politically

CBA is strong in comparing projects with similar impacts. Could help in choosing design, alignment and in prioritization

CAB can add transparency to the decision-making process and improves political accountability

15

Utility of ex post CBA1 2 3 4 5

Ex post CBA can add significant value to planning process

Adds transparencyStrengthens evidence

base

Provides feedback on methods and techniques used for ex ante design

and appraisal

1 2 3

A key contribution of ex post evaluation has been a

better understanding of optimism bias and its causes at the planning

stage

16

Ex post CBA as a tool for impact evaluation

1 2 3 4 5

Advantages Limitations

Holistic approach

Delivers an unambiguous indicator of the project’s economic worth

Considers both direct impacts (e.g. time savings) as well as associated

externalities (e.g. safety benefits)

1

2

3

Due to long life of infrastructure, ex post CBA should be carried out as late as

possible

Counter-balancing the above, there is a risk of institutional memory loss

The analysis requires the definition of a counterfactual – for projects with a long

expected economic life, this may be problematic

1

2

3

17

CBA relevance and utility for macro-economic modelling

1 2 3 4 5

Ex post CBA covers a wider range of impacts than those focus of macro-

economic models (i.e. non-work traffic savings, safety and environment)

but

It provides a ‘net’ benefit indicator and does not indicate

who actually and ultimately receives this benefit.

It measures the added value of bringing new resources to use, while macroeconomic modeling

measures the gross value.

In practice

Macroeconomic impacts are best modeled by linking transport

accessibility models directly with macroeconomic models

18

Thank you for your attentionThank you for your attention

http://ec.europa.eu/regional_policy/sources/docgener/evaluation/rado_en.htmhttp://ec.europa.eu/regional_policy/sources/docgener/evaluation/rado_en.htm