Institute for Transport StudiesFACULTY OF ENVIRONMENT

Evaluation WorkshopDfT 6th June 2014

Introduction: Background and Issues

Tom Worsley

Visiting Fellow, ITS

Scale and Scope

Two broad levels of evaluation

1. Local, incremental, quick to implement, reversible options

2. Long term, high cost, spatially specific infrastructure projects

Methods and processes for type 1 evaluations well documented – Magenta Book – and widely practised.

Type 1 evaluations often used when we don’t know how to predict/model the target groups’ responses.

Questions answered by type 1 evaluations;•Is this delivering objectives?

•Should it be rolled out, modified or discontinued?

Major project/policy evaluation

All approved major projects meet the requirements of the 5 part Transport Business Case. Lessons can be learnt from evaluating each part of the TBC.

The appraisal part of the TBC is comprehensive – ministers’ duty to take all of the impacts into account.

Evaluation is not so constrained – is it efficient to cover all impacts in the evaluation of a major project?

What do we want to learn from the evaluation? What insights might this project provide?

What do we want to learn from an evaluation?

High level strategic feed-back• Is the scheme as success?

• Was it a good decision?

• A description, including a comparison against forecasts

Lessons about the management, commercial and financial cases

• Delivery, costs, risks, partnership

• Analysis of successes and failures, building experience and leading to improved guidance

How do we learn about the quality of the economic case?

What might we learn from the economic case?

The economic case comprises a model made up of:• Exogenous input forecasts – demographic, economic, environmental,

prices etc

• Relationships between these exogenous forecasts and travel demand

• Assumptions about the network in the with and without scheme cases

• Relationships between changes in the networks and changes in people’s travel patterns

• Assumptions about the value people put on changes to the networks – value of time savings, reliability, safety etc – and on changes to the external environment – emissions, air quality etc

• Assumptions about transport’s impact on the economy etc

• A Value for Money category (post modelling)

Evaluation by re-running the transport model

Risks:• Model used is outdated by time the evaluation is carried out (as are

many other assumptions)

• Network and other input assumptions likely to be dated and wrong

• Some observed changes can be within the model’s error margins.

• Is the counterfactual – eg what if the HS1 hadn’t been built? – credible?

Lessons to be learnt• Lessons about uncertainty and about better modelling of

mode/route/destination choice

Other approaches to evaluation of the appraisal

Appraisal values:• Update values frequently and re-estimate eg VoTTS every so often

Demand forecasts:• Monitor, review, update aggregate demand forecasts, by segment

(population, area type)

• Responses to transport cost changes – fuel cost/fares responses can be monitored from aggregate data but trip distribution, mode choice more difficult.

Transport and the Economy• Identify opportunities for re-estimation of transport/GVA impact, including

evaluation of schemes.

Summing up

To inform decisions about whether to roll out a local policy, to modify or to stop it, the control area based evaluation case study is a well established approach.

Evaluation provides increased accountability and lessons learnt in all parts of the business case. Evaluation of individual projects can improve our understanding of some key modelling and forecasting challenges.

For major projects the evaluation must target on the question to be answered lessons to be learnt. It is inefficient and unnecessary to attempt to mimic the full ex ante appraisal.

This should be complemented by a broader evaluation strategy aimed at updating forecasts, cost benefit values and elasticities.