Study on the Use of Cost-effectiveness Analysis in EC's Evaluations

CENTRE EUROPÉEN

D’EXPERTISE EN ÉVALUATION

CENTRE FOR EUROPEAN

EVALUATION EXPERTISE

Final report

Study on the Use of Cost-effectiveness Analysis

in EC’s Evaluations

20th July 2006

Study financed by the European Commission, DG BUDGET

Study on cost-effectiveness – Directorate General BUDGET

Executive summary

Purpose of the study

In its article 27(1) the Financial Regulation states that “budget appropriations shall be used in accordance with the principles of economy, efficiency and effectiveness”. In line with this obligation, the budgetary authority expresses a continuous demand for increasing efficiency in Community policies. Cost-effectiveness analysis (CEA) is one of the tools allowing evaluators to assess efficiency.

In this study cost-effectiveness analysis is understood as “an analysis which compares several interventions or several opt ons in terms of both implementation cost and the achievement of the intended result/impact.” This contrasts with approaches which compare costs to the outputs only.

i

This study was undertaken, among other things, to delineate the area of relevance of CEA, and to deliver recommendations on:

• how to improve CEA in its area of relevance • how to expand the area of relevance of CEA • how to evaluate efficiency when CEA is not relevant

Main findings

CEA is more frequent in prospective evaluations. This may be explained by the fact that it is easier to produce an estimate of future effects than to quantify actual effects. In the ex ante evaluation guidelines, there is also a strong request for a comparative analysis of the efficiency of alternative options.

The terminology used by the Commission and its external evaluators is all but stabilised. Surprising terms have been encountered like “cost-opportunity” or “cost-efficiency”.

The terms of reference tend to express efficiency related demands that are not fully specified and that cover only a part of the necessary items (efficiency of what? efficiency in achieving what? efficiency in comparison to what? efficiency question asked for which purpose?).

The costing approach tends to focus on the budgetary allocations. Administrative costs and overheads are seldom considered. Another issue is whether EC evaluations should systematically consider EU efficiency in the case of joint financing with Member States, something which has not been encountered in the study.

In almost all instances of quantitative assessment of effects, evaluators refer to an indicator which has already been defined in advance.

Effects are seldom assessed through an attribution analysis. Only one of the reviewed reports pays tribute to the issues of deadweight and substitution. Only one other includes a comparison between beneficiaries and non beneficiaries.

The selected reports show quite different approaches to comparing costs and effects: comparison between components of the intervention, comparison with a previous programme, comparison with an external benchmark, comparison with an artificial scenario.


The reasoning also follows varied paths: cost/effectiveness ratio higher/lower than a given benchmark, cost higher/lower for similar effects, effects higher/lower for similar cost, proportion of effective expenditure higher/lower for similar cost.

The prerequisites for applying CEA can be summarised as follows:

It is possible to concentrate on one single effect, or to use one of the available approaches for combining several effects

There is a recognised indicator or rating grid for each effect, or it is possible to develop such a measure

It is possible to quantify / rate the effect(s) of the intervention in each of the options to be compared, or to state credibly that the effect will remain unchanged whatever the option

It is possible to compare several interventions / options / scenarios.

This study has confirmed that one or more of these prerequisites are missing in many instances, which makes the area of relevance of CEA relatively narrow. It is therefore not surprising that CEA is far from frequent in EC evaluations.

The recommendations follow from these findings and are structured in three clusters

Improve the practice of CEA in its area of relevance

DGs should reflect on efficiency at the time of deciding upon their evaluation programme. In this framework some evaluations would be identified as important ones as regards efficiency. Efficiency questions would be asked in priority in such evaluations.

Evaluation managers should complement efficiency questions with specific methodological notes covering the following points: is a CEA recommended or not? Which effects should be considered in priority? Which costs should be included in the analysis? Which type of comparison is to be made? DG Budget should provide the appropriate technical support for applying this recommendation.

Evaluation managers should supervise efficiency questions in a careful and reinforced way, especially when a CEA is requested. At the inception stage, a precise agreement should be reached on the costing approach, and on the effects to be analysed. At the design stage, a precise agreement should be reached on the approaches to attribution analysis and comparison. DG Budget should develop a specific quality assessment grid applying to efficiency questions.

Expand the area of relevance of CEA

The Commission’s Evaluation Network should identify a few specific effects that could lend themselves to CEA (e.g. creation of sustainable EU-wide networks), ask evaluation questions about such effects in a series of evaluations, and capitalise upon these exercises in order to define appropriate indicators and benchmarks.

Develop an alternative when CEA is not relevant

In the instance of multiple and complex interventions, the evaluation managers and the evaluators should consider an alternative to CEA, building upon contribution analysis and case studies, and which could be called “cost-contribution analysis”. This approach is close to a lot of pragmatic approaches encountered in this study, but it has not been formalised as a “good professional practice”. DG Budget should take action in order to formalise and legitimise this approach, for instance through its framework contract for evaluation services.


Such an approach cannot answer the same kinds of questions as CEA does. CEA answers summative questions like: “was the intervention efficient enough?”. Cost-contribution analysis answers formative questions like: “which features of the intervention does make it more or less cost-effective?”. The Commission at the highest level should be aware that only formative evaluation questions can be answered in many instances.

By answering formative questions, the Commission cannot keep its promises in terms of providing the Parliament, Council and citizens with accountability for its efficient use of public resources, at least in the traditional understanding of the term accountability. The European institutions should recognise the limitations which are faced by evaluations in this respect. They should pay more attention to “accountability for learning”, which means that the Commission is responsible for understanding what makes its interventions more or less cost-effective (through formative efficiency questions), and to take action consequently.


Contents

1. Objectives of the study ................................................................... 1 Context .........................................................................................................1 Objectives and tasks ......................................................................................1

2. Definitions....................................................................................... 2 Cost-effectiveness and efficiency.....................................................................2 CEA in the case of complex interventions .........................................................3 Approaches to costing ....................................................................................5 Approaches to combining effects.....................................................................6 Cost benefit analysis, another standpoint .........................................................8

3. Review of evaluation reports ........................................................ 10 Selection of 117 potentially interesting reports ...............................................10 Review of 23 evaluations involving a CEA.......................................................12

4. Synthesis of efficiency-related conclusions .................................. 14

5. In-depth analysis of CEAs .............................................................17 Commissioner’s demand ...............................................................................17 Evaluators’ responses ...................................................................................18 Costing approach .........................................................................................19 Effects considered........................................................................................21 C&E reasoning .............................................................................................23 Contextual factors ........................................................................................24

6. In depth analysis of failed attempts to apply CEA ........................25 Commissioner’s demand ...............................................................................26 How do evaluators conclude on efficiency? ....................................................26 Difficulties encountered with CEA ..................................................................27

7. Research into good practices outside the Commission ................. 28

8. Summary of findings ..................................................................... 29

9. Recommendations.........................................................................31 Improve the practice of CEA in its area of relevance .......................................31 Expand the area of relevance of CEA .............................................................34 Develop an alternative when CEA is not relevant ............................................35 Summary of recommendations......................................................................38


Annex 1 – Methodological complements ...........................................40 Computer assisted search into the evaluation reports .....................................40 Database of 117 reports ...............................................................................40 In-depth analysis .........................................................................................42

Annex 2 – In-depth analysis of 15 CEAs ............................................ 42

Annex 3 – In-depth analysis of weak CEAs........................................71

Annex 4 – Examples of good practice outside the EC ........................ 90

Tables

Table 1 – Number of evaluation reports .......................................................................11

Table 2 – Time perspective and use of CEA..................................................................13

Table 3 –Intervention cost and use of CEA...................................................................14

Table 4 – Five examples of CEA outside the EC............................................................29

Boxes Box 1 – Operational efficiency versus Cost effectiveness .................................................2

Box 2 – Multiple and complex interventions....................................................................4

Box 3 – Approaches to costing ......................................................................................6

Box 4 – Example of an outranking option .......................................................................7

Box 5 – Example of data envelopment ...........................................................................8

Box 6 – CEA and CBA: two opposite standpoints.............................................................9

Box 7 – The quantitative analysis at a glance ...............................................................13

Box 8 - Time lag between cost and effects ...................................................................20

Box 9 – Assessing the quality of CEA ...........................................................................33

Box 10 – Relaxing the constraints which restrict the use of CEA.....................................34

Box 11 – Focusing on a core intermediary effect...........................................................35


Box 12 – Contribution analysis ....................................................................................36

Box 13 – Evaluating the effectiveness of a complex intervention....................................37

Box 14 – Dealing with effectiveness and efficiency in complex settings...........................37

Box 15 – Analysis of cost and contribution ...................................................................38


1. Objectives of the study

Context

In its article 27(1) the Financial Regulation states that “budget appropriations shall be used in accordance with the principles of economy, efficiency and effectiveness”. In line with this obligation, the budgetary authority expresses a continuous demand for increasing efficiency in Community policies.

When thinking of evaluating efficiency, what immediately comes to minds is the cost-effectiveness analysis (hereafter CEA), a powerful tool which allows evaluators to assess whether the unit cost of an achieved effect is satisfactory in comparison with an appropriate benchmark.

In practice, the use of CEA is rather limited in the real world of evaluation. This is in line with the fact that evaluators have repeated difficulties in quantifying actual effects on the basis of evidence.

It is also acknowledged that CEA studies tend to concentrate in areas where comparable indicators and benchmarks are available (e.g. jobs created, lives saved, energy saved, young farmers set up, etc.), and where there is a simple and linear model explaining the intended effects.

Objectives and tasks

The study aims to delineate the area of relevance of CEA. It delivers recommendations on:

• how to improve CEA in its area of relevance • how to expand the area of relevance of CEA • how to evaluate efficiency when CEA is not relevant

The study process has been divided into several tasks as follows1:

• Overall analysis of the evaluation reports delivered over the last five years • In-depth analysis of 2 sets of reports, respectively applying CEA or failing to

apply CEA • Search for good practices outside the Commission

These points are addressed in sections 3 to 7. They are preceded by a series of definitions in section 2 and followed by recommendations in section 8.

1 The methodological approach is described in Annex 1

Eureval-C3E – 20/04/2006 – page 1


2. Definitions

Cost-effectiveness and efficiency

CEA is often seen as the gold standard technique for evaluating efficiency. This vision may be acceptable or not, depending on how the terms are defined.

Efficiency is sometimes defined as a relationship between inputs and outputs (in the sense of the goods and services delivered by the evaluated intervention). If defined in this way, “efficiency” is completely different of and complementary to “cost-effectiveness” which relates to results and impacts, i.e. effects of the intervention beyond its outputs.

In its glossary, DG Budget defines “efficiency” as the extent to which the desired effects (= results and impacts) are achieved at a reasonable cost. If defined in this way, “efficiency” is close to “cost-effectiveness”.

In order to prevent confusion, this study uses the terms “operational efficiency” and “cost-effectiveness” as shown in Box 1.

Box 1 – Operational efficiency versus Cost effectiveness

Operational efficiency relates the resources to activities and/or outputs

Activity2: Work carried out during a certain period of time, consuming resources and producing outputs and effects in accordance with the objectives set.

Output: What is produced by the evaluated intervention, and what the lay level operators are fully responsible for, e.g. delivering goods or services to target groups, enforcing regulations, etc. Those responsible for outputs may be the EC, Member States, agencies, or any public or private operator who is accountable for the implementation of the intervention.

Cost effectiveness relates the resources to results and/or impacts

Result: Immediate and direct change induced in the behaviour (or capacity, attitude, opinion, satisfaction, accessibility, rights …) of those reached by the intervention. Such changes are visible to the lay level operators and can be monitored on a systematic and daily basis.

Impact: Further and/or indirect change generated by the intervention. Impacts are not visible to the lay level operators and need to be observed and analysed through specific surveys or studies.

DG Budget defines cost-effectiveness analysis as “a judgement technique in which the researcher quantifies the costs and benefits associated with an intervention on the basis of the same principles which apply to cost-benefit analysis, but there is no requirement to transfer benefits into common monetary units”. This definition is

2 The term “action” is used in the DG Budget’s glossary in order not to confuse with ABM terminology.

Eureval-C3E – 20/04/2006 – page 2


questionable for two reasons: (1) it refers to cost-benefit analysis rather than being self-sufficient, and (2) it is restricted to quantitative approaches.

In order to provide the reader with a self-sufficient definition, it is worth decomposing “cost-effectiveness” into its two terms. In line with the EC Financial Regulation (art. 27(2)) and DG Budget’s glossary, effectiveness should be understood as the extent to which objectives set are achieved, this being limited to objectives expressed in terms of intended results and impacts (and not outputs).

The “cost” element primarily refers to EC’s budgetary resources allocated to the intervention under evaluation, although a second and wider perimeter may be considered, which includes:

• Costs of other resources that are used indirectly, e.g. human resources at EC headquarters, organisational resources, …

• Financial contributions of other public sector partners.

In DG Budget's definition, cost-effectiveness is understood as an exclusively quantitative approach. If this definition is retained, then the feasibility of CEA is excessively limited. A wider definition is therefore adopted in this study, CEA being “an analysis which compares several interventions or several options in terms of both implementation cost and the achievement of the intended result/impact”.

With such a definition, CEA can be used to answer questions such as: "Could more effects have been achieved at the same cost?", or "Have other interventions obtained similar effects at a lower cost?"3, or “Was it possible to achieve more effects at a lower cost?”.

Finally, CEA is an approach in which cost and effect are two integrated variables. Several options or interventions are compared in terms of cost and effect simultaneously. It does not suffice that cost and effect be subject to separate analyses.

CEA in the case of complex interventions

A key assumption made at the beginning of this study is that CEA is adequate for evaluating the efficiency of simple projects or programmes, and not for evaluating complex interventions.

Two distinct concepts should be defined in order to understand the issue of complexity:

• An intervention may include multiple components / activities, using various instruments, and targeting several distinct categories of beneficiaries. It may aim to achieve one or several overarching objectives through multiple chains of causes and effects. Such causality chains tend to be linear and not too long. In current professional debates, this type of intervention is called “complicated”. The term “multiple” has been preferred in this study as it is considered more meaningful.

3 These questions are considered as typical efficiency questions in the MEANS collection (European Commission, 1999)

Eureval-C3E – 20/04/2006 – page 3


• When cause and effect chains are long and intricate, when external factors are predominant, when synergy effects and causal loops have to be considered, then the term “complex” is appropriate4, with a meaning which is close to its mathematical sense.

If the intervention logic is described with an explanatory model with boxes and arrows, as shown in Box 2 then the terms “multiple” and “complex” correspond to the width and length of the figure.

Box 2 – Multiple and complex interventions

Targeted group(s)

Instrument(s)

1

Effect(s)

1 1

Targeted group(s)

Instrument(s)

1

Effect(s)

1 1

x X X

Mul

tiple

x X X

Mul

tiple

ComplexComplex

Multiple and complex interventions raise specific difficulties for the use of CEA: (1) multiplicity implies that a number of effects have to be analysed together with multiple costs, and (2) complexity reduces the evaluability of the effects. The issue of multiplicity / complexity is seen as a growing concern in the evaluation community5, especially in the context of EC policies6.

This study uses the following definitions:

• Simple: The intervention applies one major instrument. It targets one homogeneous group of beneficiaries / addressees. It is acceptable to describe its logic through one single linear chain of causes and effects.

4 In the MEANS collection, the term “complex” was used for the two concepts that are distinguished here. 5 See Schwartz & al. (forthcoming) Evaluating the complex: attribution, contribution and beyond. New Brunswick: Transaction Publishers. The issue of evaluating complex interventions was already addressed in the volumes 1, 3 and 4 of the MEANS collection. 6 Summa, H. and Toulemonde J. (2002), ‘Evaluation in the European Union: addressing complexity and ambiguity’ in International Atlas of Evaluation, Furubo J.E, Rist R.C., Sandahl R. (ed), New Brunswick: Transaction Publishers, p407-424

Eureval-C3E – 20/04/2006 – page 4


• Multiple: The intervention applies several instruments, and/or it targets several distinct categories of beneficiaries. The intervention logic is made of simple, short, linear and convergent chains of cause and effect.

• Complex: Same, but there are long, circular and intricate chains of cause and effect, which ultimately end into several intended effects.

Approaches to costing

When the evaluation assesses efficiency, the term “cost” is equivalent to “resources” or “inputs”. The cost element includes all budgetary resources used for achieving the evaluated effect.

Four types of costs are to be considered:

• EC budgetary resources allocated to the intervention under evaluation • Other EC budgetary resources used for implementing the intervention, e.g.

human resources at EC headquarters, organisational costs … • Financial contributions of other public sector partners • Cost of coordination between the public sector partners involved in the

intervention

Several technical terms need to be defined:

Transaction cost. The concept has been developed in order to explain why goods and services are traded through the market (higher transaction costs) or through internal arrangements within an organisation (lower transaction costs). Importing the concept into the public sector7 is questionable since the market is not an available option in the case of public goods and services. The term is sometimes used as an equivalent to the fourth type of budgetary cost above (cost of running partnerships). It is also used with the meaning of compliance cost, something which belongs to a completely different logic (see below). In the context of evaluating public policies and programmes, the term is more confusing than helpful. It is not used in this study.

Opportunity cost. The concept reflects the idea that a given amount of budgetary resource may have more or less value than just its amount in euros. Much higher or lower benefits would accrue to the European citizen if the same amount had been allocated to another policy or programme. Using the concept for the evaluation of efficiency may be difficult since policy makers are in charge of allocating budgetary funds in order to optimise the benefits for the European citizen. The very idea of opportunity cost is rooted in the assumption that the budgetary allocations made by policy makers are sub-optimal. This idea is acceptable if, and only if, the evaluation process is legitimate enough for assessing the benefits that would arise from an alternative allocation of budgetary funds. This may be achieved through a pluralistic evaluation committee for instance.

Shadow prices. This concept is close to the former one. Once again, the idea is that a given amount of budgetary resource should not be counted as its value at market price, but the argument is different. The idea is rooted in the assumption that some of

7 Including by DG Budget, which gives the following definition in the case of non-expenditure policy: « enforcement cost associated with implementing, monitoring and enforcing the policy). (see EVALUATING EU ACTIVITIES: A Practical Guide for Commission Services – 2004 – p95)

Eureval-C3E – 20/04/2006 – page 5


the inputs are bought from the market at a cost that does not reflect their real economic value because the market is distorted. Shadow prices should be counted as zero in the case of permanent market surplus. International market prices are often used as shadow prices.

Compliance cost. Compliance costs are borne by the individuals and organisations having to comply with EC rules. If the concern is to evaluate efficiency, and if cost-effectiveness is defined as in Box 6, then the cost element should only refer to the cost for the public sector partners responsible for the intervention. Since compliance costs are borne by other agents, they should not be part of the cost element of the CEA. On the contrary, they should be seen as an effect. For instance, in the case of fisheries inspections, the budgetary cost of inspectors is taken as a cost in the CEA, but the pressure exerted on fishing enterprises is an effect. Another effect is the deterrent effect of inspections on fishermen, which leads to the ultimately expected effect: conservation of fish resources.

Box 3 – Approaches to costing

Stakeholders(e.g. compliance cost)

Publicsector

partners

ECbudgetDG in

charge

Cost for …

Cost Negative effect??actual, opportunity, shadow

Approaches to combining effects

CEA in its traditional sense includes a comparison of cost-effectiveness ratios. It implies that effectiveness be summarised in one dimension, something which is made difficult in the case of multiple interventions. Several approaches have been recommended in this context. They all share three prerequisites:

• The assessed effects are independent from one another, which means that they belong to separate cause-and-effect chains.

• The number of assessed effects is reasonable, eight is usually seen as a maximum

• All effects are either quantified or rated, a constraining condition which often reduces the number of effects to much less than eight for practical reasons

The main available approaches to combining effects are as follows:

Multi-criteria analysis with outranking. This approach compares several options or several interventions according to several judgement criteria. In the case of this study the judgement criteria are the various effects and the cost. The analysis may be

Eureval-C3E – 20/04/2006 – page 6


based on outranking or compensation8 (see below). The outranking approach is defined as follows: “option A outranks option B if it is preferable with regards to a majority of judgement criteria (more positive effects, less compliance cost, less budgetary cost) and if it is not dramatically inferior in any of the other judgement criteria”. Freely available software allows the evaluation team to carry out the analysis and to adjust the number of criteria constituting a “majority” and the threshold to be considered as “dramatically inferior” for each criterion. The approach allows the analyst to combine quantitatively assessed effects with ratings. Simple ratings are acceptable, even a four level scale for instance. The approach is relatively feasible but there is no guarantee that all options will be ranked eventually, which means that the evaluation may remain partly inconclusive.

Box 4 – Example o an ou ranking op ion f t t

Option Effect A desirable

(high)

Effect BNot desirable

(low)

Effect CDesirable

(high)

Cost(low)

performs better thanin all dimensions, except “effect A”,and it is not dramatically worseas regards “effect A”

Data envelopment: This approach has been developed for the analysis of productivity. It may be used for CEA purposes in order to compare several options (or several interventions) according to their costs and effects9. The first step consists in identifying “internal benchmarks” among the options or alternatives under study, i.e. the options which are the most efficient for achieving each individual effect. Then a “relative cost effectiveness” ratio is computed for all other options. The ratio expresses the proportion of what is achieved by a given option in comparison with what could have been achieved at the same cost by the best possible mix of internal benchmarks. Due to the calculation approach, all internal benchmarks have a 100% ratio, which means that they cannot be ranked against one another. All other options are outranked, in a larger or smaller proportion as indicated by the ratio. The approach is quite demanding in terms of causality analysis because all effects should be assessed quantitatively.

8 See MEANS, Volume 4, p104 9 Bjoerner, J. and Keiding, H. (2004) ‘Cost effectiveness with Multiple Outcomes’ in Health Economics, 13, 1181-1190

Eureval-C3E – 20/04/2006 – page 7


Box 5 – Example of data envelopment

less efficient

than any mix

1

23

effect A / €

effe

ct B

/ €

Option 3Option 3

1 2of and 1 2of and

Multi-criteria analysis with compensation: In this approach, all effects are converted into scores, and then weighted. An overall weighted score is computed for each option (or intervention). Options are then compared with regards to a ratio of weighted score / cost. All effects have to be either rated or assessed quantitatively. A specific difficulty arises with the need to decide upon a weighting system which has to reveal the policy makers’ preferences. Specific techniques are available for that purpose10.

Cost benefit analysis, another standpoint

Cost benefit analysis (CBA) is often described as just another weighting approach since the various effects are expressed in a single monetary unit. As seen above, this view is stated in DG Budget’s glossary11.

This study takes a different view: CBA is not just about the monetisation of effects. On the contrary, the move from CEA to CBA corresponds to a radical change in the assessor’s standpoint. As suggested by Box 6, CEA refers to the policy makers’ intentions. On the contrary, CBA refers to the preferences of all winners and losers in the society, which means that:

• all effects are considered, including unintended ones, provided that they are significant

• effects are assessed from the standpoint of winners and losers themselves, in line with their own perceptions and values

t

10 Toulemonde & al. (1998) ‘Evaluation in Partnership: Practical Suggestions for Improving their Quality’, in Evaluation, he International Journal of Theory, Research and Practice, 4(2) 11 The same view can be read in Levin, H.M. (1983) Cost-effectiveness, a Primer. London: Sage, p20. And also in Gray, A.W (1995) EU Structural Funds and Other Public Sector Investments: a Guide to Evaluation Methods. Dublin: Gill & Macmilan. p 4

Eureval-C3E – 20/04/2006 – page 8


• effects are weighted with reference to stakeholders perceptions, not to political priorities

• budgetary costs are understood to be a negative effect for a certain category of losers, i.e. the taxpayers.

Cost-utility analysis is a variant of CBA where effects are just rated and weighted and not monetised. Technically, the process is exactly the same as for multicriteria analysis with compensation (see above), but the rationale is the same as for CBA: the weighting system reflects the preferences of winners and losers and not those of policy makers.

Winners and losers are stakeholders, but not all stakeholders are taken into account in CBA. Winners and losers are those ultimately affected by the intervention, e.g. students, households, farmers, enterprises, taxpayers, etc. The public sector partners that co-finance the intervention or the implementing agencies are not winners / losers in the sense of CBA, although they are stakeholders.

Monetisation in CBA or weighting in CUA has to reflect the preferences of the winners / losers. This may be done through questionnaires, focus groups, or panels. The canonical approach to monetisation consists of assessing the winners’ willingness to pay for getting the benefits, and the losers’ willingness to pay for avoiding the drawbacks12.

Box 6 – CEA and CBA: two opposite standpoints

Relevance

Intervention OutputsOutputsInputsCost

InputsCost

TaxesSocietyIntended

Needsproblems

challenges

Needsproblems

challenges

Cost-effectiveness

Cost-benefit analysis

Effectiveness

Operational efficiency

Cost-utility analysis

Objectivesintentions

Objectivesintentions

Unintended

Results & impacts

CBA and CUA rely on the compensation principle, which means that a highly valued positive effect may compensate for a negative one. In other words, an option / intervention is judged positively if the winners’ benefits are large enough for compensating the losers’ losses. Needless to say that the compensation is seldom effective. Usually, the winners don’t pay anything for benefiting from public

12 The simplest case is that of taxpayers whose loss is directly expressed in monetary terms

Eureval-C3E – 20/04/2006 – page 9


intervention, and the losers don’t receive anything in compensation of their drawbacks.

A consequence of the compensation principle is that CBA judges an option / intervention in absolute terms, i.e. the assessment is positive if the gains outweigh the losses. As a consequence, CBA does not need to compare alternatives13.

3. Review of evaluation reports

Selection of 117 potentially interesting reports

EC evaluation practice

The study covers the years 2000 to 200514. Over this period of time the DG BUDGET has recorded 604 reports produced by the EC, and being called evaluation or impact assessment. These reports are included in a database, of which an extract has been made available to the study team.

Pre-selection of potentially interesting reports

In a preliminary step to this study, DG BUDGET undertook a rapid screening of the 604 reports, and ended up with a pre-selection of 161 reports containing a substantial section or chapter on efficiency.

Second selection

The study team extracted 117 reports out of the 161 pre-selected documents. This has been done by using a set of keywords, and a research profile combining a series of relevant keywords, with an aim to extract reports likely to be interesting in terms of CEA, even when the term CEA is not quoted15.

Analysis per DG

The study team has clustered the DGs in two groups (familiar, less familiar with evaluation) according to the number of evaluations made per year, and the date of the first evaluation reports recorded (before/after 2002).

As shown in the following table, all selected reports have been commissioned or produced by a group of 17 DGs16, which have been qualified as “familiar with evaluation”.

13 On the contrary, CUA requires a comparison between cost-utility ratios. 14 The coverage of 2005 is incomplete. 15 This task was carried out using a text analysis software (Wordmapper). See annex 1 for details. 16 DG ADMIN, AGRI, EAC, ECHO, EMPL, ENTRE, FISH, INFSO, MARKT, REGIO, RTD, TRADE, TREN, ENV, ESTAT, RELEX and SANCO.

Eureval-C3E – 20/04/2006 – page 10


Table 1 – Number of evaluation reports

DGs Overall EC’s practice

Pre-selection Final selection

Familiar with evaluation

514 161 117

Less familiar with evaluation

90 0 0

Total 604 161 117

Types of interventions

The interventions considered in the set of 117 selected reports have the following features:

• The budget is implemented on a centralised basis directly by the Commission’s services or indirectly17 (50 reports). Management is also decentralised to third countries (30), or shared with the Member States (27). Joint management with international organisations is less frequent (10).

• Interventions involve small to large expenditures - three balanced groups have been made: interventions with an annual cost of under €15m (37 reports); between €15m per year and €100m per year (36 reports); more than €100m per year (34 reports).

• Finally, 51 of the interventions considered are complex, 30 multiple, and 36 simple.

Types of evaluations

The set of 117 reports selected for analysis includes:

• External evaluations mainly, only 11 evaluations out of 117 being internal; • Retrospective evaluations mainly (83 reports), prospective (10) and prospective

& retrospective (21)18.

Closeness to CEA

The study team has sorted out the 117 selected reports into the following categories19:

• Reports with a cost-effectiveness analysis (23) • Reports addressing operational efficiency through a CEA-like approach (15)

reports or another approach (33) • Reports with cost benefit analysis (7) • Reports with an analysis of cost and effect (24) • Reports with an analysis of cost or effect (12)

17 The definitions of the different types of management methods refer to art. 53 of the Financial Regulation (Reg. 1605/2002) 18 Due to the fact that this study builds upon retrospective evaluations mainly, the findings and recommendations mainly apply to this category of evaluation. Exceptions are specified in footnotes. 19 The definitions of CEA, operational efficiency and cost-benefit analysis are given in Chapter 2, see Box 1 and . Box 6

Eureval-C3E – 20/04/2006 – page 11


• Other (3)

Review of 23 evaluations involving a CEA

The 23 reports including a CEA are analysed hereafter in comparison with the set of 117 reports.

Interventions evaluated

The evaluated interventions inside the 23 reports selection have the following features:

• The budget is managed by the Commission services mainly (11), or by the Commission and Member States through shared management (7), or by third countries in a decentralised way, or by the Commission and an international organisation jointly (1).

• The annual budget of the evaluated interventions ranges from 3 billion to less than one million, with 11 reports evaluating low expenditures interventions, 2 medium, and 10 high.

• 13 interventions are complex, 4 are multiple and 6 are simple.

Features of the CEA

The CEA itself has the following features:

• The costing approach concentrates on EC budgetary expenditures mainly: 19 reports out of 23

• Effects are assessed quantitatively in 16 reports, whilst 7 reports include a CEA reasoning with no quantified estimate of effect

• 15 reports use CEA in order to arrive at a conclusion involving an assessment on the merits and worth of the intervention (value judgement).

Quality of the CEA

Finally 11 reports have been considered as “good CEAs” on the basis of the following criteria:

• Strong analysis of cost • Good enough analysis of effect(s) • Comparison involving cost and effect.

The whole analysis process can be summarised in the following figure:

Eureval-C3E – 20/04/2006 – page 12


Box 7 – The quantitative analysis a a glance t

CEA

CBA

23

7

Cost and/ oreffect

no comparison

Cost and

output

Efficiency

117

All reports604

Good CEA 11

Factors affecting CEA

A number of cross section analyses have been undertaken, with an aim to understand which factors may explain the presence or absence of CEA in an evaluation report. Many tables were constructed in this respect, mainly focusing on the scope of the intervention (complex, multiple or simple) and the type of evaluation approach (prospective, retrospective or mix). All but two analyses failed to match the statistical validity test20. These analyses are presented below.

First, CEA is more frequent in prospective evaluations as shown by Table 2. This may be explained by the fact that it is easier to produce an estimate of future effects than to quantify actual effects. There is also a strong request for a comparative analysis of the efficiency of alternative options in the ex ante evaluation guidelines.

Table 2 – Time perspective and use of CEA

% CEA Other Total Prospective 50% 50% 100% Mix 14% 86% 100% Retrospective 18% 82% 100%

Second, CEA is more frequent in the evaluation of interventions which involve smaller budgetary expenditures as shown by Table 3. This finding is counter-intuitive in the sense that one could expect a higher pressure to strengthen the evaluation of efficiency in the case of large expenditures.

20 Chi square with a less than 5% probability that random would achieve the same contrast as what has been observed.

Eureval-C3E – 20/04/2006 – page 13


Table 3 –In ervention cost and use of CEA t

% CEA Other Total Large and medium 17% 83% 100% Small 29% 71% 100%

An explanation could be that an intervention with a small budget tends to be simple, and therefore lends itself to CEA. However the analysis does not show a significant difference between simple, multiple and complex interventions as regards the use of CEA. The study team therefore ends with no explanation of this finding.

4. Synthesis of efficiency-related conclusions

This section highlights a series of noteworthy conclusions encountered in the course of the study. All conclusions arising from “good CEAs” are mentioned hereafter, plus a few other efficiency-related conclusions.

Generally speaking, the conclusions are nuanced with positive and negative points. Several reports are totally positive about the efficiency of the evaluated intervention whilst none is totally negative.

Efficiency differences explained by contextual factors

One of the reviewed reports concludes on a decrease in the efficiency of the intervention. The change is explained by contextual factors which are not under control of the Commission.

• Report 62 - Set-aside measures in the common agricultural policy - The comparison between the direct cost of overproduction and the cost of non production (set-aside) shows an important economic advantage in favour of the set-aside measures at the beginning of the implementation period. But the opposite became true in 1995- 1996. This evolution can be explained by two main factors: the fall of the overproduction costs (in the region of -25%) and the rise of the non production costs (+26% between 1993 to 1995)

In another report, the evaluator has highlighted large disparities of efficiency across geographic areas. Once again, differences seem to be explained by contextual factors which are not under control of the Commission, and the conclusion suggests that such factors could be analysed more in-depth.

• Report 275 - EU centres in non-European industrialised countries aimed a promo ing coopera ion and commercial relations - When comparing the partner countries “there appears to be a near alignment of New Zealand and Canada as being most cost-effective, closely followed by the US Centres (both the new and old ones). The Australia Centre appears the least cost-effective, yet as noted, these figures must be treated with caution, both in terms of the short length of implementation and the size of the Australian grant.” This conclusion suggests that country-related factors explain the differences in efficiency, but the report does not explore this issue.

tt t

Eureval-C3E – 20/04/2006 – page 14


Efficiency differences explained by internal factors

In the following examples there are disparities of efficiency across programme components. In these instances, the causes of the identified differences have to be found in the design of the intervention rather than in uncontrolled external factors.

• Repor 256 – Population programmes in developing countries: In Kenya, the EC supported the purchase of contraceptive supplies at competitive prices, keeping programme expenses down and very cost-effectively supporting the national family planning programme.

t

-

t

• Report 417 – European Observatory for Small and Medium Enterprises: It is clearly demonstrated that the efficiency of the Observatory is better in the new version: the effects are the same as in the last period but with a smaller global cost. Some mechanisms that have been implemented during the last period seem to have changed the efficiency balance.

• Report 30 - Food security in the Russian Federation It can be concluded that, at least for wheat and beef, the use of EU intervention stocks permitted a cost-effective supply at prices respectively 25% and more than 60% below supplies from world markets. For rice, on the contrary, the use of intervention stocks implied a 50% additional cost.”

• Repor 518 - Control of fisheries - For the Baltic and the North Sea, land-based inspections are the most cost effective at picking up offences, followed by sea inspections, and lastly air inspections. For the Mediterranean and the Atlantic, land based inspections, are the most cost effective means of detecting offences, followed by air inspections and then sea inspections. From the data provided, it costs more than three times as much to detect an offence at sea as it does to detect an offence on land. ... Of course, it is important not to forget the deterrent effect of sea surveillance, but the results might suggest that across the EU as a whole, a rebalancing of resources would be appropriate. The efficiency of aerial surveillance in the Atlantic and the Mediterranean are surprising.

In the next example below, a change in efficiency is clearly attributed to the (improved) design of the policy.

• Report 43 - Intervention on oilseeds within the Common Agricultural Policy – The intervention has been deeply redesigned in order to tackle the problems encountered during the previous period. The reform involved a decrease of EC budgetary cost but did not modify the income of oilseed producers and the competitiveness of transformation industries. There is therefore evidence that the reformed support regime represented a progress in term of efficiency in comparison with the previous one.

Efficiency differences explained by a “critical mass effect”

• Repor 583 - Feedback from citizens and businesses about their daily problems with EU policies: The efficiency of the consultation mechanism has improved a lot during its implementation period. There has been some “learning-by-doing” which can be attributed to the lowering of the costs of development.

t

Eureval-C3E – 20/04/2006 – page 15


Efficiency differences explained by a “saturation effect”

In the interesting example which follows, the efficiency has decreased, simply because the best opportunities had been taken during the first programming cycle, leaving the new programme with less cost-effective ones.

• Repor 485 - Energy saving in the transportation systems – The tonne-kilometres shifted from road to railways is lower in the current cycle than it was in the previous cycle. The external benefits per euro are also lower than previously. The evaluation report explains the reduction in efficiency by a saturation effect.

t

Mixed efficiency, partly explained by deadweight

In the example below, the intervention seems to be cost effective in a restrictive way; however deadweight and other factors reduce its efficiency.

• Repor 181 - Local employment strategies – The programme was cost effective in that a wide variety of useful and mainstreamed results were achieved for a relatively modest outlay. However there was plenty of deadweight and substitution in many areas, and in others little or no need for the type of partnership promoted by the programme to be introduced. Finally, many of the results are insufficiently cleared cut or consistent for the benefits of this particular approach to be likely to be adopted very widely.

t

Efficiency differences stated but not explained

In the following examples, the evaluation team concludes on cost-effectiveness, but no explanation is given:

• Repor 194 – Education and training – The budget of the programme has increased since the previous cycle and the management and monitoring activities have increased too. The annual administra ive cost amounted to around €26m euros in 2001 and has remained globally unchanged. Related to the previous cycle, the effectiveness of programme seems to have increased. The cost-effectiveness has therefore slightly improved.

t

t

t

• Report 118 – Olive oil – If the fair income of olive oil producers is defined as the income they could gain from alternative agricultural practices (e.g. durum wheat), then the objective of fair income is clearly achieved. Was it possible at a lower cost? Previously, it has been observed that production subsidies in the olive oil sector represent from 27% to 30% of the gross product, depending on the country. The percentages vary from 45% to 57% for durum wheat, 40% to 45% for oil seeds, and up to 66% for sunflower. Therefore, by comparison, it is possible to consider that a fair income to producers is achieved at a reasonable the cost for the EC budget”.

• Repor 125 - Tobacco: The European Union could implement other types of support to maintain the actual level of employment in tobacco growing areas. However, it seems that alternative options would need financial resources comparable to the current support cost.

Eureval-C3E – 20/04/2006 – page 16


Inefficiencies assessed without references to costs or effects

Many reports derive efficiency conclusions from a detailed analysis of the functioning of the implementation process, with the implicit assumption that good functioning entails intervention efficiency and poor functioning entails inefficiency:

• Repor 206 - Transport Sector in Third Countries: Some of the interventions have suffered from the same types of inefficiencies as other projects (restrictiveness of and delays in administrative and financial procedures, timeliness of delivery, and hiatuses between first and second phases of projects, quality of implementation).

t

t• Repor 15- Transfer of know-how in transition countries: The expected results are achieved within the available budget and time. Delegations scrutinise project budgets quite rigorously, and consequently excessive cost items are usually identified and, in consultation with the proposed contractor, adjusted to more acceptable levels. As most projects are also effective (in the sense that stated project results are achieved, see chapter 3), the evaluation team concludes that project resources have efficiently been transformed into the expected results.”

Ex ante assessment of alternative options

The next case illustrates the obligation to compare the efficiency of alternative options in the ex ante evaluations.

Repor 576 – Research in the field of transport – Four options have been considered for implementing the research programme. In one of the options, DG TREN concentrates on the programming process and it does not manage the programme directly. The cost of this option is the most cost-effective. In another option, the programme is totally managed by DG TREN. This option is expected to have a slightly lower effectiveness per Euro spent.

t

tRepor 453 – Strategy for reducing mercury emissions: The report includes an interesting comparison between two systems aimed to reduce mercury emission. The two alternatives are very close to one another in terms of their functioning and results but one is much more expensive than the other.

5. In-depth analysis of CEAs

Commissioner’s demand

Several reports refer to efficiency related demands expressed in the ToR. These demands were expressed in the following terms:

• “Attaining the objectives at reasonable cost” (43)

• “Efficiency of the concerned administrative procedures, management system, organisational set up, monitoring system, and auditing arrangements” (30).

• “Cost-opportunity analysis of the Community financial intervention in order to demonstrate its added values” (485).

Eureval-C3E – 20/04/2006 – page 17


• “Comparison between inputs (human and budgetary resources) and the first results” (194)

First, it must be noted that the terminology is all but stabilised (cost-opportunity, cost-efficient!).

Four dimensions of the demands may be identified:

• Efficiency of what? e.g. “administrative procedures, management system, organisational set up, monitoring system, and auditing arrangements” (30)

• Efficiency in achieving what? e.g. “quality of outputs and user satisfaction level” (417)

• Efficiency in comparison to what? e.g. “compared to other similar EU interventions” (417); “comparative approach with past projects and between countries” (194)

• Efficiency judgement for which purpose? e.g. “in order to demonstrate EC added value” (485), in order to “justify the cost of the intervention” (417).

In some instances, the demands are passed to the evaluators through evaluation questions. The rationale for evaluation questions is to focus on key issues, which means that questions tend to focus on some effects, and may not cover the whole range of effects. For instance, the evaluation of EC Organisation for the olive oil market focuses on the cost of achieving market equilibrium and rural employment, whilst the policy also had environmental objectives.

Evaluators’ responses

Among the sharpest conclusions are the following:

• “47.5% of the programme expenditure was effective. This should be contrasted with other innovative or even mainstream programmes where the proportion of highly positive results can be just as low, often for the deployment of significantly greater resources” (181)

• “The reform represented a progress in term of efficiency since it involved a decrease of 500 millions euros/year of EC budgetary cost, whilst producers’ income was not reduced, and price paid by transformation industries remained close to international price” (43).

• “The Modal Shift action within Marco Polo II provides the best value for money of all actions both in terms of tonne-kilometres shift per euro EC subsidy and in terms of external benefits per euro EC subsidy”, as compared to four other actions within the programme (485).

• “The Initiative as a whole is extremely cost-efficient, in that it manages to leverage substantial additional funding. Its objectives, although not always met, target funding towards activities with high visibility. In terms of the selection of academic institutions, the sustainability of knowledge created via academia is likely to be greater than that of more short-term activities, and from a longer term perspective, more cost-effective (275).

Eureval-C3E – 20/04/2006 – page 18


As compared to the commissioners’, the evaluators’ terminology is better stabilised (except for the term "cost-efficient", which is still used once!). The term “value for money” is used as a fully acceptable synonym for cost-effectiveness.

As in the previous section, several dimensions are identifiable in the evaluators’ conclusions:

• Cost-effectiveness of what? e.g. “technical execution, and monitoring” (30), “network of national correspondents” (417), the budgetary expenditures (43)

• Cost-effectiveness in achieving what? e.g. “minimal losses and diversions21” (30), “likely sustainability of knowledge created” (275), “producers’ income” and “price paid by transformation industries” (43)

• Cost-effectiveness assessed in which way? e.g. proportion of effective expenditures (181), scores (30), cost per unit of effect (485), same effects at lower cost (43)

• Cost-effectiveness in comparison to what? e.g. “four management options” (576), “five distinct actions within the programme” (485), former policy, previous to the evaluated reform (43)

• Cost-effectiveness judgement from whose standpoint? E.g. for the EC “in that it manages to leverage substantial additional funding” (275), for a given DG “provided that the management approach succeeds in influencing the outcomes in a way that they can be used by the DG” (576).

Costing approach

This section describes the various approaches to dealing with costs.

Cost for whom?

Costs are borne by various actors, as in the following examples:

• Budgetary allocation to the project + parallel support given to the project by other DGs (583)

• Costs borne by EC and Member States (518).

The first two examples show that three perimeters of budgetary costs are considered: DG in charge, EC as a whole, and EU as a whole including Member States. The legitimacy of just considering the budget of the DG in charge is questionable22, especially when it leads to conclude that a management approach is efficient “in that it

21 In a CEA approach, “minimal losses and diversion” (of food aid) is to be considered as an achievement and not a cost (see Evaluation report of EC financed programme to supply agricultural products to the Russian Federation, AIDCO, 2000, report No 30 in the database). 22 Which means that the concept of « cost for the implementing body » is also questionable. In fact, DG Budget already recommends to include “administrative costs for the Commission and public authorities, in particular the Member States” (see EVALUATING EU ACTIVITIES: A Practical Guide for Commission Services – 2004 – p95). In the same spirit, to what extent is it reasonable to write conclusions like DG X paid for the totality of the project, but “promoted 21 of the 74 activities, i.e. received less than one third of total benefits” ?

Eureval-C3E – 20/04/2006 – page 19


manages to leverage substantial additional funding” (from other DGs) (275). An interesting issue is whether EC evaluations should systematically consider EU efficiency in case of joint financing with Member States.

One example illustrates the boundary between effectiveness analysis, CEA and Cost Benefit Analysis. The expected effects are (1) reduction of mercury emissions by the EU industry and (2) limited increase in the production cost in the EU industry. In the evaluation report, this two-dimensioned objective is mainly seen from the standpoint of policy-makers. The exercise has therefore been considered as neither a Cost Benefit Analysis (the standpoint is not that of winners and losers) nor a CEA (the budgetary cost of enforcing the regulation is not evoked).

Matching costs and effects

Several examples include an explicit attempt to match costs with effects.

• This has been fully done in the following example: “The costs have then been separately allocated to the two activities. Cost drivers have been identified as software development, promotion and communication, data entry, data processing and overall supervision and co-ordination. Although the cost allocation process has been made in strict co-operation with Commission services throughout this analysis, a significant number of assumptions have had to be made on how precisely to allocate costs, especially when it came to human resources” (583).

• In another example, the costs are broken down into three types (land, sea, and air inspections) which are themselves connected to different types of effects (land inspections are good for detecting badly recorded data, sea inspections are good for detecting prohibited fishing methods) (518).

Only one of the in-depth studies show an explicit adjustment of costs and effects in order to take stock of the time needed to convert inputs into effects23.

Box 8 - Time lag be ween cost and e fects t f

Programmed

Cos

t

Committed

Paid

Outputs

Results

Impact

Effe

cts

short

long

23 Although several examples compare costs and effects altogether at different points in time.

Eureval-C3E – 20/04/2006 – page 20


As explained in Box 8, this issue can be ignored if outputs are related to payments, but it must be considered if impact is related to payments, and even more to programming.

Nature of costs

Costs are of various kinds, as in the following examples:

• Grants to project promoters, central Technical Assistance Office, dissemination materials, networking events, and seminars (181)

• Grants to project promoters, administrative procedures (576)

• Grants allocated (275)

• Value of supplied food + person days committed to the control and monitoring of operations (30)

Apart from operational expenditures in the form of grants, subsidies, supply of food, etc., the costs also include administrative expenditures in the form of management, monitoring, and auditing. However, none of the in-depth studies provide a detailed estimate of the full cost of EC’s human resources (salaries and overheads24).

Reality of costs

It is not always clear in the examples whether the costs considered are foreseen, programmed (clear in 181), committed, or disbursed (clear in 30). One would have expected to read more precise statements concerning this, since it often takes months or years to move from programming to commitments, and then to disbursement, a fact which has major consequences on how costs relate to effects.

Effects considered

All effects or some effects

Most of the analysed evaluation reports relate to interventions having several intended effects, a difficulty which is tackled in various ways as shown by the following examples:

• Each of the three CAP evaluations (tobacco, olive oil, and oilseed) considers several effects separately, with questions like “are these effects achieved at a reasonable cost?”. Effects considered are: market equilibrium, economically viable farms, rural employment, and fair standard of living for the agricultural population, competitiveness of the agro-industry (125, 43, and 118).

• The Marco Polo II programme evaluation considers two categories of effects: (1) shift from road to other environmentally friendly transport systems, and (2) a range of external benefits generated by this shift as the benefit for EU for each euro spent in Marco Polo II actions (485)

• The evaluation of territorial employment pilot projects considers various project-specific effects that are “consistent with the objectives of the programme” (181).

24 To the knowledge of the study team, DG Budget’s guidance does not cover this issue.

Eureval-C3E – 20/04/2006 – page 21


Several approaches are identifiable in the above examples:

• Separate effects on the same logical chain e.g. “economically viable farms, rural employment, fair standard of living for the agricultural population” (125)

• A core effect e.g. “shift from road to other environmentally friendly transport systems”, of which a range of benefits can be logically derived (485)

• A common activity e.g. “projects” associated with an overall qualitative assessment of whether their diverse effects are “consistent with the objectives of the programme” (181)

None of the selected reports involve a weighted sum of various effects. The study team has not encountered weighted score approaches, or a sum of valued effects25.

Close or far-reaching effects

Some of the selected examples focus on far-reaching impacts, whilst others focus on direct results, or even outputs. The variety of logical levels can be seen from the following examples (from closest to furthest):

• The main products, i.e. survey and reports (417)

• Quality of the reports (492)

• Offences detected (518)

• Shift from road to other environmentally friendly transport systems (485)

• Economically viable farms (125)

• Rural employment (125)

• Fair standard of living for the agricultural population (125)

In one instance, the nature of the evaluated intervention (pilot projects) requires a particular type of effect to be considered, i.e. learning and transferring lessons (181)

Measurement approach

The set of selected examples shows several quantitative measurement approaches:

• Number of detected offences (518)

• Tonne-kilometres shift from road to other environmentally -friendly transport systems (485)

In all instances of quantitative measurement, evaluators refer to an indicator which was already defined in advance. In the one instance, the evaluation even admits that “the quantitative analysis has mainly focused on items, for which rough proxy quantitative indicators were available” (583).

The example of the territorial employment pilot projects shows an interesting alternative which mixes counting and scoring approaches. The measurement includes

25 The example of scoring approach described in the next section involves an assessment of multiple effects altogether, and not separate assessments first and then a weighted sum. Consequently, this example does not contradict the statement made in this section.

Eureval-C3E – 20/04/2006 – page 22


a counting of the projects, which are then rated on a 0-1 scale by the evaluation team as follows:

• 1 = the project achieved substantial local results consistent with the objectives of the programme; the project promoters understood how and why these achievements had been reached; they communicated effectively to other parties such that the lessons were to some useful degree taken up and acted upon;

• 0,5 = one or two of the three conditions outlined above, but not the full set of them;

• 0 = little value for the purposes of the programme.

Effect estimates

Only one of the selected reports pays tribute to the issues of deadweight and substitution: “there was plenty of deadweight and substitution in many areas” (181)

Another report includes a comparison between beneficiaries and non beneficiaries (support to tobacco producers), one of the approaches to attribution analysis which is seen as a good practice for making quantitative estimates of effects (125).

The fact that the quantitative assessment of effects is limited is not surprising. All around the world, meta-evaluations tend to show the same picture.

C&E reasoning

Alternatives

The selected reports show quite different approaches to comparing alternatives:

• Comparisons between projects and between partner countries in the case of the promotion of cooperation and commercial relations between the EU and the industrialised countries of North America and Australasia (275)

• Comparison with alternative crops like durum wheat in the case of the CAP support to olive oil (43), with alternative job retaining measures in the case of the CAP support to tobacco (125)

• Comparison with previous programmes in the case of the evaluation of Marco Polo II (485), or with the same policy before it was reformed (43)

• Comparison with other research institutes identified as potential competitors on the market, in the case of a software project (417)

• Comparison between procurement of food aid from European and international markets (30)

Several approaches are identifiable in the above examples:

• Internal comparison between components of the intervention as in the first example (275)

• Time comparison with a previous programme as in the case of the evaluation of Marco Polo II (485)

• External benchmark as in the last example (417)

Eureval-C3E – 20/04/2006 – page 23


• Artificial scenario as in the case of durum wheat replacing olive trees (43)

Comparison approaches

Examples of approaches to the comparison of alternatives are as follows:

• Cost higher/lower than that of an alternative scenario achieving similar effects (43, 125)

• Outputs compared with a benchmark in a qualitative way covering the type, quantity and quality of the outputs and the costs (583)

• Comparison of average cost per unit of achieved effect, e.g. detected offence, tonne/kilometre shifted (518, 485)

• Higher / lower proportion of effective expenditure, with higher / lower resources (181)

Average versus marginal comparison

In the case of the Marco Polo II programme, the cost effectiveness in terms of tonne-kilometres shifted per euro is lower than for the previous Marco Polo I programme, but the evaluation team does not conclude negatively on the efficiency of the new programme.

On the contrary, a marginal reasoning is followed, building on the assumption that the most attractive projects were selected first (saturation effect) (485).

Non-comparative approaches

An interesting approach can be observed in the ex ante evaluation of the RTD programme in the field of Energy and Transport. In addition to adopting a (weak) comparative approach, the evaluator proposes a detailed analysis of the factors and mechanisms that contribute to improving efficiency (576):

• Promoting economies of scale, e.g. making use of existing dissemination channels, using common contracts and helpdesks for administrative matters, adopting common evaluation procedures, etc.

• Avoiding duplication/overlap, e.g. risk that similar RTD activities are carried out by different DGs or units without them knowing this. Reducing red tape in programming, instruments, evaluation procedures, contract procedures, subcontracting arrangements, etc.

• Improving coordination with other DGs

• Improving easy understanding by industry, users, consultants, universities, research institutes, etc.

Contextual factors

Good examples of CEAs can be explained by various factors.

Eureval-C3E – 20/04/2006 – page 24


Availability of data on cost and effect

Within the different studies, there is a good availability of data on cost and effects. They come from desk research with documents provided either by the commission or by intervention partners. Several explanations can be drawn:

• Statistical data: In 3 evaluations (43, 118, 125) the subject of the intervention is related to agriculture products, a policy domain where statistical data are abundant.

• Monitoring: In the case of the food transfer to the Russian Federation, a strong (and costly) monitoring system was set up. Data on outputs and results were therefore available (30).

Availability of a comparison scenario

The availability of a reliable comparison scenario with available data can also explain that the evaluation team undertook a good CEA:

• Comparison with a former intervention: Interventions such as Marco Polo (485) oil seed (43) and olive oil (118) were in operation for years under different forms which allows a comparison with previous policies / programmes

• Multi-projects interventions : Some interventions include a significant number of projects which may be sorted out by the evaluator into contrasted clusters, making it possible to undertake internal comparisons (275, 181)

• Easily identifiable alternative scenarios: Several evaluations (30, 43, 125) include comparisons with an obvious alternative scenario like procurement at international market price, supporting alternative income generating crops or applying traditional job creation strategies.

• Existing benchmark: in one of the evaluations (417) a similar intervention had been subject to a CEA, providing a benchmark for the evaluation

• Several envisaged options: this is a frequent case in prospective exercises like in the report 576, which makes a comparison between the different options available for the implementation of a new programme.

Evaluabililty of the effect

Within several evaluations, the evaluation teams decided (or followed the demand stressed by the evaluation question) to focus on results or intermediary impacts rather than on far-reaching impacts.

6. In depth analysis of failed attempts to apply CEA

This section displays an analysis of 10 reports selected as failed attempts to apply CEA, either because they state that CEA could not be applied or because there are elements of CEA, but not a clear use of the tool.

The purpose is to contribute to:

• Understanding what the area of relevance of CEA is (is not).

Eureval-C3E – 20/04/2006 – page 25


• Understanding why a limited number of evaluations actually apply CEA.

Commissioner’s demand

Several of the analysed reports mention efficiency-related demands expressed in the ToR:

• The findings of the evaluation are intended for accountability purposes, with a focus on the achievement of the action’s objectives, its impact, and its cost-effectiveness (5)

• Is the budgetary cost of the instrument justified with regard to the noticed effects? (62)

• What is the cost-effectiveness of the system as currently designed? How can the current cost-effectiveness be improved? (252)

Three dimensions of the demands may be identified:

• Efficiency of what? e.g. “network functioning”, “products development”, “setting aside of agricultural land” (252), (62)

• Efficiency in comparison to what? e.g. “what it would be if the set-aside were not paid” (62)

• Efficiency for what? e.g. “accountability purposes”, “in order to ensure that the cost of network functioning and Eurydice products adequately match objectives” (252), (5)

One of the reports (45) refers to a statement in the ToR that mentions difficulties regarding the use of cost-effectiveness analysis: “Cost-effectiveness in relation to the rehabilitation sector can be defined as the extent to which costs of a minimal shelter per beneficiary can be built. In principle it should be possible to make a cost comparison between different projects using different housing concepts (…) technically addressing cost-effectiveness of an intervention such as that of ECHO is rather difficult”.

How do evaluators conclude on efficiency?

By passing the problem on to others

• The evaluators recommend that the evaluation activities within ALTENER be strengthened in order to give a realistic and reliable view of the impacts and cost-effectiveness of the Actions. Costs versus benefits of the Actions seem to have been poorly covered so far and need to be strengthened (68)

• Referring to the OECD’s “Principles for Effective Aid”, there is a need to pay greater attention to the costs and cost-effectiveness of technical co-operation activities (37)

• ECHO did not establish clear objectives and results which were easily measurable and could be followed by the NGOs. ECHO should work relentlessly with its partner in order to elaborate a structured design to raise awareness concerning cost-consciousness (45).

Eureval-C3E – 20/04/2006 – page 26


• Insufficient attention had been given to ensuring cost-effectiveness in project design and implementation. Specifically: Lessons learned during project implementation have not been fully reflected in subsequent rounds of programming; SPPs with potentially high impact and limited costs have not been fully exploited (37)

By issuing vague statements

• Broadly speaking, we can but emphasise a high cost relative to achieved results (38)

• This change could lead to a significant improvement in the business orientation and deployment potential of future projects, and perhaps also to improved cost-efficiency of projects through a more focused approach to other work packages, especially market studies (5)

• It is clear that if we compare the relative amounts of the projects in relation to the results expected from the programme, we can but emphasise an overall high cost related to actual obtained results (38).

Difficulties encountered with CEA

Several types of difficulties are highlighted by evaluators, or identified by the study team.

Difficulties in assessing effects

• With only nine projects completed so far, the assessment of outputs, results, and achievement of objectives is difficult and based more on expectations than on actual achievements (5)

• The effectiveness is difficult to estimate exactly because no objective for the network was quantified in advance: there are no target levels (252)

• Judging cost-effectiveness at this stage of the Agency development is difficult. Cost-effectiveness is judging input against outcome/impact, and it is precisely the latter that is difficult to discern at this early stage of operation (81).

Difficulties with cost

• This evaluation is made extremely complex because of the absence of complete and homogeneous financial data (252)

• In general, work packages are not described in detail, and no breakdown of workload on individual tasks is provided. It is recommended that future contracts contain a more detailed description and breakdown of projected costs (5)

• Some reports include no or almost no information on costs (81), (28).

Difficulties with comparisons

• A quantitative analysis model should have been used in order to assess the intervention and simulate what could have been the results in other conditions.

Eureval-C3E – 20/04/2006 – page 27


However due to time and means constraints, the realisation of this model appeared to be unfeasible (62)

• Performing a cost-effectiveness analysis over the different projects has proven to be very difficult. Local conditions are often impossible to compare with each other, etc. (45)

Difficulties with the concept itself

• In view of the specific objective of this evaluation - evaluating the financial instrument, and not the projects financed by it - aspects relating to cost-efficiency of the interventions are not addressed (28).

7. Research into good practices outside the Commission

With an aim to identify realistic benchmarks and to formulate recommendations pertaining to the EC context, the study has included a search for relevant and successful uses of CEA outside the EC. Four channels have been used:

• Call to all members of the INTEVAL research group26, an international network of experts in evaluation

• Internal expertise in Ramboll-management and Eureval

• Search of GAO’s web site27

• Google search.

Contacted experts originate from North America (USA, Canada), Europe, Australia, Korea and multilateral institutions.

The search criteria were:

• Conclusive CEA

• Methodologically sound

• Multiple or complex intervention.

As seen in Table 4, nothing has been found except the evaluation of the Sure Start Programme (UK). However this is an on-going evaluation, only the methodological design has been consulted, and it is too early to say that the final report will be conclusive.

The poor result of the search confirms that there is a challenge in applying CEA to multiple and complex interventions.

26 www.inteval-group.org 27www.gao.gov. Among the documents accessed, a particularly interesting one is the “Highlights of a Workshop on Economic Performance Measures” (www.gao.gov/new.items/d05796sp.pdf). Economic Performance Measures means CEA and CBA. The workshop discusses the feasibility of both techniques extensively, and elaborates upon three examples of simple interventions.

Eureval-C3E – 20/04/2006 – page 28

http://www.gao.gov/

http://www.gao.gov/new.items/d05796sp.pdf


Table 4 – Five examples of CEA outside the EC

Sure start Health Drug court Minority language

Youth crime

Degree of complexity

High Low Medium Medium Medium

Timing Retrospective Retrospective Retrospective Retrospective Retrospective

Authors External External Internal External External

Purpose Summative Summative Summative and formative

Summative and formative

Formative

Issues addressed/ Approach to evaluating efficiency

Effectiveness, cost-effectiveness, cost-benefit

Cost-effectiveness

Cost-effectiveness

Cost-effectiveness

Cost-effectiveness and cost-benefit

The five reports are presented in annex 5, from a methodological point of view.

8. Summary of findings

The study team has reviewed 117 reports which include a substantial conclusion on efficiency, of which 23 of them include a CEA and 7 a cost-benefit analysis. A near majority of the reviewed reports focus on operational efficiency, which means that they focus on outputs rather than results or impact.

In the CEAs, the costing approach concentrates on EC budgetary expenditures mainly (19 reports out of 23). Effects are assessed quantitatively in 16 reports, whilst 7 reports include a CEA reasoning with no quantified estimate of effect.

Finally 11 reports have been considered as “good CEAs” on the basis of the following criteria: (1) strong analysis of cost, (2) good enough analysis of effect(s), and (3) comparison involving cost and effect.

The main findings are as follows28:

(a) CEA is more frequent in prospective evaluations. This may be explained by the fact that it is easier to produce an estimate of future effects than to quantify actual effects. In the ex ante evaluation guidelines, there is also a strong request for a comparative analysis of the efficiency of alternative options.

(b) The terminology used by the Commission and its external evaluators is all but stabilised. Surprising terms have been encountered like “cost-opportunity” or “cost-efficiency”. The term “value for money” is sometimes used as a fully acceptable synonym for cost-effectiveness.

(c) The terms of reference tend to express efficiency related demands that are not fully specified and that cover only a part of the necessary items (efficiency of

28 Findings (a) to (e) are relevant to all types of evaluations, including prospective ones. Items (f) to (l) pertain to retrospective evaluations mainly.

Eureval-C3E – 20/04/2006 – page 29


what? efficiency in achieving what? efficiency in comparison to what? efficiency question asked for which purpose?).

(d) The costing approach tends to focus on the budgetary allocations. Administrative costs and overheads are seldom considered. In one instance, alternative options are assessed by just considering the cost for the DG in charge, and not the cost for the Commission as a whole. The legitimacy of this practice is highly questionable. Another issue is whether EC evaluations should systematically consider EU efficiency in the case of joint financing with Member States, something which has not been encountered in the study.

(e) Many of the reviewed evaluation reports relate to “multiple interventions” having several intended effects29, a difficulty which is tackled in various pragmatic ways by evaluators, but never by using techniques such as multi-criteria analysis or weighted score approaches.

(f) It is not always clear in the examples whether the costs considered are foreseen, programmed, committed, or disbursed. One would have expected to read more precise statements concerning this, since it often takes months or years to move from programming to commitments, and then to disbursement, a fact which has major consequences on how costs relate to effects.

(g) In almost all instances of quantitative assessment of effects, evaluators refer to an indicator which has already been defined in advance. In an interesting exception, the evaluator has developed a simple 3 level rating scale.

(h) Effects are seldom assessed through an attribution analysis. Only one of the reviewed reports pays tribute to the issues of deadweight and substitution. Only one other includes a comparison between beneficiaries and non beneficiaries.

(i) The selected reports show quite different approaches to comparing costs and effects: comparison between components of the intervention, comparison with a previous programme, comparison with an external benchmark, comparison with an artificial scenario30.

(j) The reasoning also follows varied paths: cost/effectiveness ratio higher/lower than a given benchmark, cost higher/lower for similar effects, effects higher/lower for similar cost, proportion of effective expenditure higher/lower for similar cost.

(k) The main factors facilitating CEA are a good monitoring of expenditures, the availability of statistical indicators on effects, the availability of a comparison scenario or benchmark, and the evaluability of the effects. In several instances, the evaluators improved the evaluability of the effects by focusing on direct results or intermediary impact rather than on far-reaching impact.

(l) When they fail to answer efficiency questions properly and/or to implement a CEA, evaluators tend to pass the problem on to others and to complain about the lack of measurable objectives, the weakness of management and monitoring databases, and the difficulty of identifying a comparison scenario or benchmark. Some evaluators express discontentment of not having been

29 Definition in Chapter 2, see Box 2. 30 Comparison of alternative scenarios is the main approach in case of prospective evaluation.

Eureval-C3E – 20/04/2006 – page 30


provided with impact estimates by the programme management, and they even declare to be surprised that the programme management does not monitor cost-effectiveness. How easy would evaluation be if all the problems could be transferred to managers!

In contrast to the easy solution made in point (l), the following recommendations deal mainly with the tasks of the evaluator. They are also relevant for the Commission services when they provide guidance on evaluation or outsource evaluation studies, as well as for users of the evaluation results.

9. Recommendations

The prerequisites for applying CEA can be summarised as follows:

It is possible to concentrate on one single effect, or to use one of the available approaches for combining several effects

There is a recognised indicator or rating grid for each effect, or it is possible to develop such a measure

It is possible to quantify / rate the effect(s) of the intervention in each of the options to be compared, or to state credibly that the effect will remain unchanged whatever the option

It is possible to compare several interventions / options / scenarios.

This study has confirmed that one or more of these prerequisites are missing in many instances, which makes the area of relevance of CEA relatively narrow. It is therefore not surprising that CEA is far from frequent in EC evaluations. In addition, the study has shown a number of CEAs which do not match quality standards at all.

Three recommendations follow:

Improve the practice of CEA in its area of relevance31 Expand the area of relevance of CEA Develop an alternative when CEA is not relevant

Improve the practice of CEA in its area of relevance

Improve the demand for evaluating efficiency

Efficiency questions tend to be asked in a ritual manner, which means that they are probably asked in excessive number, at the expense of the quality of the answer. In the worst instances the terms of reference just require that the so-called five DAC criteria be addressed, and efficiency is quoted en passant.

On the contrary, a recommended approach would be to reflect on efficiency at the level of each DG and at the time of deciding upon the evaluation programme. In this framework some evaluations would be identified as important ones as regards efficiency. Efficiency questions would be asked in priority in such evaluations.

31 Recommendations are mainly targeted at retrospective evaluations. Only the first section applies to prospective evaluations (partly).

Eureval-C3E – 20/04/2006 – page 31


Efficiency questions are often asked for the purpose of being accountable to the European Parliament. Once again, thought should be given to the extent to which MPs would have the capacity to absorb efficiency information if it were possible to produce it in a systematic way.

Improve demand for CEA

As shown in this study, external evaluators have obvious difficulties in answering efficiency questions. Wherever the commissioning EC service has the adequate capacity (or is provided with the adequate support), such questions would deserve to be accompanied by specific methodological notes covering the following points:

Is a CEA recommended or not? Which effects should be considered in priority? Which costs should be included in the analysis? Which type of comparison is to be made?

It is sometimes expected that the appropriateness of the evaluation design should serve as a criterion for ranking candidate evaluators. This would no longer be possible if the above recommendation were implemented. The study team considers that the poor level of efficiency evaluation justifies a more prescriptive approach in the terms of reference. Of course, a prerequisite would be that the evaluation manager be fully skilled in the subject. DG Budget should provide the appropriate technical support.

Strengthen the evaluation management in the case of CEA

In the same spirit, and provided that the commissioning EC service has the necessary capacity, efficiency questions should be subject to a careful and reinforced supervision when a CEA is requested.

At the inception stage, a precise agreement should be reached on:

Costing approach Effects to be analysed

At the design stage, a precise agreement should be reached on:

Approach to attribution analysis Approach to comparisons

At the final stage the quality assessment should be specifically adapted to CEA, for instance by using subcriteria as proposed in Box 9.

Eureval-C3E – 20/04/2006 – page 32


Box 9 – Assessing the quality of CEA

Current quality assessment framework32 Extension to CEA 1. Relevance: Does the evaluation respond to information needs, in particular as expressed in the terms of reference?

Is the efficiency question answered in a clear and unambiguous way? Has the efficiency question been clearly connected to the intervention logic?

2. Appropriate design: Is the design of the evaluation adequate for obtaining the results needed to answer the evaluation questions?

Was the approach to CEA clearly explained? Is the design strong enough?

3. Reliable data: Are data collected adequate for their intended use and have their reliability been ascertained??

Are data suitable and reliable enough as regards - costs - effects to be analysed - benchmarks?

4. Sound analysis: Are data systematically analysed to answer evaluation questions and cover other information needs in a valid manner?

Are data correctly analysed as regards? - analysis of costs - distinction between change and effect - comparability of options, scenarios, etc.

6. Credible findings: Do findings follow logically from and are they justified by, the data/information analysis and interpretations based on pre-established criteria and rationale?

Is there a clear explanation of how costs and effects are related? Is there an explanation of the extent to which CEA findings are generalisable?

7. Valid conclusions: Are the conclusions non-biased and fully based on findings?

Is the comparison fair? If various effects are aggregated, does the weighting system reflect the policy-makers’ views?

8. Helpful recommendations: Do areas need improvements identified in coherence with the conclusions? Are the suggested options realistic and impartial? 9. Clarity: Is the report well structured, balanced and written in an understandable manner?

/

32 Updated grid, borrowed from DG Budget website, June 2006

Eureval-C3E – 20/04/2006 – page 33


Expand the area of relevance of CEA

In order to expand the area of relevance of CEA, the constraints related to the four prerequisites have to be relaxed. Box 10 below presents a proposal which relaxes the three main constraints encountered in this study.

Box 10 – Relaxing the constraints which restrict the use of CEA

Prerequisite Related constraint Relaxing the constraint One main effect It is possible to concentrate on one single effect, or to use one of the available approaches for combining several effects.

Very constraining. Most of the EC interventions have multiple objectives. Focusing on just one single effect is often unacceptable for stakeholders.

Identify a core intermediary effect which is common to most cause-and-effect paths (see Box 11). Then focus the CEA on this effect. This is the core proposal which recommends all others below.

Measurement There is a recognised indicator or rating grid for each effect, or it is possible to develop such a measure.

Relatively constraining. It is difficult to develop a completely new indicator or rating grid during the time frame of an evaluation. Multiple effects imply multiple measurements.

Causality analysis is made feasible by the fact that just one effect has to be assessed, but a specific indicator has to be developed for this purpose.

Attribution It is possible to quantify / rate the effect(s) in each of the alternatives to be compared, or to credibly state that the effect will be the same.

Very constraining. Cause-and-effect analysis is the Achilles’ heel of evaluation. And the problem is made even worse if several effects are to be assessed.

Causality analysis is made feasible by the fact that just one effect has to be assessed, and that this effect is not too far down the logical chains.

Comparison It is possible to compare several interventions / options / scenarios.

Not very constraining. Comparison against an alternative scenario is at least a feasible option.

The basic idea behind the proposal is to find an intermediary effect which is common to most cause-and-effect chains, as in Box 11 below. Preferably the core intermediary effects should be comparable across several policy domains, so that a specific measurement approach (indicator or scale) can be developed, and so that benchmark programmes can be identified.

Examples of “core intermediary effects” may be:

The creation of sustainable EU-wide networks, for instance in the case of RTD, education, justice, etc.

The initiation of sustainable twinning arrangements involving two countries, for instance in the case of accession

The initiation of sustainable cross-border partnerships, for instance in the case of Interreg

The formulation of an innovative strategy by specifically created local partnerships, for instance in the case of rural development.

Eureval-C3E – 20/04/2006 – page 34


Box 11 – Focusing on a core intermediary effect

Activities Effect(s)

Coreintermediary

effect

In all the instances above, a substantial investment would be necessary in order to develop the appropriate measurement instruments (indicator or scale). This investment is too heavy for being successfully undertaken in one single evaluation. What can be recommended is that the EC evaluators’ network identifies a list of specific effects that could lend themselves to CEA, formulate priorities and initiate specific studies through a series of pilot evaluations.

This proposal may expand the area of relevance of CEA if (and only if) it is implemented at the level of the EC as a whole, first because the development of adequate indicators and scale will be time and resource consuming as seen above, and second because only an EU-wide strategy will enable evaluators to do the comparisons which are part of the CEA technique.

A wide range of EC interventions could become evaluable through CEAs if this recommendation were implemented, but definitely not all. An alternative remains therefore necessary in order to answer efficiency questions when CEA remains out of reach.

Develop an alternative when CEA is not relevant

The problem

CEA is not feasible when at least one of the prerequisites is missing, i.e.

There are several effects which cannot be easily combined and no core intermediary effect can be found

It is difficult to identify comparable interventions / options / scenarios It is difficult to make a quantitative estimate of the effect(s), or even a rating because no recognised indicator or rating scale is available

Eureval-C3E – 20/04/2006 – page 35


It is difficult to make a quantitative estimate of the effect(s), and even a rating because the chains of causes and effects are too complex.

In the worse case, the evaluation has to answer an efficiency question about an intervention which cannot be compared to anything else, and which generates a range of effects resisting any quantification or rating effort. Such a challenging situation is not uncommon and has been encountered in several instances in the in-depth study.

A solution to a similar problem does exist

It is worth recalling that a similar challenge is faced when evaluating effectiveness in difficult contexts. Traditional evaluation methods fail to reach a conclusion about effectiveness if the intervention generates a range of effects through complex cause-and-effect chains, and if no recognised indicator or rating scale is available. It is now suggested that the problem is to be solved by combining contribution analysis and case studies33.

Contribution analysis is an evaluation design which offers an alternative to attribution. It is workable in the context of complex causality mechanisms, where the evaluator cannot build a credible counterfactual, which prevents traditional attribution approaches from being applied. Contribution analysis is being established into another “standard method” besides the highly respected experimental design.

Box 12 – Con ibu ion analysis tr t

Contribution analysis proceeds through the following steps:

present the context of the intervention (situation) together with its evolution (change or absence of change)

present a plausible explanation of how the situation and the change is due to the intervention and the way it has been implemented

gather information showing that the explanation is confirmed, i.e. that there is supporting evidence and that the underlying assumptions appear to remain valid (confirmation)

gather information showing that the explanation is disproved and that there is an alternative explanation, e.g. external factors clearly played a role (disconfirmation)

write a conclusive statement about the extent to which the intervention has contributed to explain the situation and its evolution

Based on Mayne, 1999

Case studies are the only feasible design in the case of a complicated intervention which reaches varied groups through varied implementation arrangements. A series of carefully selected case studies may provide a picture of the intervention as a whole. Moreover, the method has the power to identify lessons of general value about what works and what does not. The reasoning is called “generalisation to the theory”, as

,33 Schwartz & al. (forthcoming) Evaluating the complex: attribution contribution and beyond. New Brunswick: Transaction Publishers

Eureval-C3E – 20/04/2006 – page 36


opposed to the “generalisation to the population” which applies in the case of statistical approaches.

By mixing case studies and contribution analysis, evaluators may understand complex causality mechanisms and derive lessons of general value, even if they cannot quantify the impacts.

Box 13 – Evaluating the effectiveness of a complex intervention

The following design enables one to draw generalisable lessons from an assessment of effectiveness through a mix of contribution analysis and case studies. The process is as follows:

Select a series of case studies in order to ensure a reasonably good coverage of the varied components of the intervention

Refine the overall logic of the intervention into a specific logic model for each case study

For each case study, and for each step in the logic model, carry out a contribution analysis as described in Box 12, i.e. situation and change, plausible explanation, confirmation, disconfirmation, conclusive statement

Make potential lessons emerge from the analysis of individual cases, or from the comparison of pairs of cases

Confirm the lessons by looking into all cases

Applying the solution to the efficiency challenge

The idea is to address the efficiency challenge through the same approach which is being developed for the effectiveness challenge, i.e. a move from attribution analysis to contribution analysis. The principle is as follows:

Box 14 – Dealing with effectiveness and efficiency in complex settings

Effectiveness Efficiency

Current paradigm Attribution analysis Quantitative impact estimate Ideally through an experimental design Comparison with a quantified objective Learning lessons of general interest

Attribution analysis and cost analysis Quantitative estimate of impact and cost Ideally through an experimental design and a CEA Comparison with an alternative option Learning lessons of general interest

Alternative paradigm Contribution analysis Qualitative understanding of impact mechanisms Ideally through a case study design No comparison Learning lessons of general interest

Contribution analysis and cost analysis Qualitative understanding of impact mechanisms and cost components Ideally through a case study design No comparison Learning lessons of general interest

Eureval-C3E – 20/04/2006 – page 37


First, the concept of contribution analysis must be refined in order to investigate into costs and effects simultaneously, as shown in Box 15 below. Then the overall evaluation method can be designed as in Box 13 above.

Box 15 – Analysis of cost and contribution

The analysis proceeds through the following steps:

Contribution Cost

Present the context of the intervention (situation) together with its evolution (change or absence of change).

Present the implementation of the intervention, including the activities undertaken and their cost. Focus on costly activities.

Present a plausible explanation of how the situation and the change are due to the intervention.

Present a refined explanation of how the situation and the change are specifically caused by the costly activities.

Gather information showing that the explanation is confirmed, i.e. that there is supporting evidence and that the underlying assumptions appear to remain valid.

Gather information showing that the refined explanation is confirmed.

Gather information showing that the explanation is disproved and that there is an alternative explanation, e.g. external factors clearly played a role.

Gather information showing that the refined explanation is disconfirmed.

Write a conclusive statement about the extent to which the intervention has contributed to explain the situation and its evolution.

Write a conclusive statement about the extent to which the costly activities have specifically contributed to explain the situation and its evolution.

Summary of recommendations

DGs should reflect on efficiency at the time of deciding upon their evaluation programme. In this framework some evaluations would be identified as important ones as regards efficiency. Efficiency questions would be asked in priority in such evaluations.

Evaluation managers should complement efficiency questions with specific methodological notes covering the following points: is a CEA recommended or not? Which effects should be considered in priority? Which costs should be included in the analysis? Which type of comparison is to be made? DG Budget should provide the appropriate technical support for applying this recommendation.

Evaluation managers should supervise efficiency questions in a careful and reinforced way, especially when a CEA is requested. At the inception stage, a precise agreement should be reached on the costing approach, and on the effects to be analysed. At the design stage, a precise agreement should be reached on the approaches to attribution analysis and comparison.

DG Budget, in agreement with the Commission’s Evaluation Network, should develop a specific quality assessment grid applying to efficiency questions.

The Commission’s Evaluation Network should identify a few specific effects that could lend themselves to CEA (e.g. creation of sustainable EU-wide networks), ask evaluation questions about such effects in a series of evaluations, and capitalise upon these exercises in order to define appropriate indicators and benchmarks.

Eureval-C3E – 20/04/2006 – page 38


In the instance of multiple and complex interventions, the evaluation managers and the evaluators should consider an alternative to CEA, which build upon contribution analysis and case studies, and which could be called “cost-contribution analysis”. This approach is close to a lot of pragmatic approaches encountered in this study, but it has not been formalised as a “good professional practice”. DG Budget should take action in order to formalise and legitimise this approach, for instance through its framework contract for evaluation services.

Such an approach cannot answer the same kinds of questions as CEA does. CEA answers summative questions like: “was the intervention efficient enough?”. Cost-contribution analysis answers formative questions like: “which features of the intervention make it more or less cost-effective?”. The Commission at the highest level should be aware that only formative evaluation questions can be answered in many instances.

By answering formative questions, the Commission cannot keep its promises in terms of providing the Parliament, Council and citizens with accountability for its efficient use of public resources, at least in the traditional understanding of the term accountability. The European institutions should recognise the limitations which are faced by evaluations in this respect. They should pay more attention to “accountability for learning”, which means that the Commission is responsible for understanding what makes its interventions more or less cost-effective (through formative efficiency questions), and to take action consequently.

Eureval-C3E – 20/04/2006 – page 39


Annex 1 – Methodological complements

Computer assisted search into the evaluation reports

The set of 162 reports was analysed with the help of the WORDMAPPER software. This software was used to identify reports with elements of the cost effectiveness approach. The software identifies paragraphs containing one or several keywords which had been specified in advance through a test on a few reports.

The automatic selection has been counter checked. It has been observed that no report contain two distinct CEA related sections. It may happen that the same CEA is reported in several sections of a report (e.g. main text and executive summary). In such a case, only the main section has been recorded.

Through this analysis, 117 reports and 117 efficiency related sections have been selected.

The study team insisted to test this computer assisted search approach. In practice, such an approach has not proven to be efficient in comparison with traditional search approaches.

Database of 117 reports

The database is provided on a separate excel file. The fields of are as follows (see definitions in section 2):

Context of evaluation

Internal/external: the item shows whether the evaluation is undertaken by an external consultant or a service of the Commission.

Time perspective: whether the evaluation is:

Prospective: the evaluation considers future results and impacts Retrospective: the evaluation looks at results (and possibly impacts) that are already observable in the field

Both

Evaluated intervention

Categories of intervention cost: whether the cost of intervention per year is:

Low: under 15 million € Medium: between 15 million € and 100 million € High: over 100 million €

Management methods

Degree of complexity: simple, multiple, complex.

Eureval-C3E – 20/04/2006 – page 40


Content of the CEA

The following items have been used for the 23 reports considered as CEA or very close to CEA:

Use for the tool: it shows whether:

YES: the CEA includes a “value for money” judgment. NO: the CEA is just focused on a quantitative comparison between the cost of

intervention and a chosen effect, with no clear attempt to formulate a judgment.

Type of argumentation: this item shows whether the CEA is:

A main argument: the CEA has been the main argument for arriving at the one of the evaluator’s conclusion.

An argument among other: the cost effectiveness analysis is part of a wider reasoning about efficiency.

Type of cost analysed: it highlights whether the CEA is focused on:

• EC budget.

• Wider perimeter: the cost used within the CEA, includes the EC budget and the budget of intervention partners at the local, regional or national level.

Type of CEA reasoning: this item shows whether the CEA is:

• Quantitative.

• Narrative.

Type of effect analysed: this item shows which effect is taken into account in the CEA:

• Output.

• Result.

• Impact.

Type of comparison: this item shows whether the CEA include a comparison:

• Just effectiveness and cost: the CEA includes just an analysis of the current intervention with no comparison with similar intervention or alternative approach.

• Comparison with similar measure: The CEA compares the current intervention with another similar intervention in operation in an other region of another sector.

• Comparison with alternative approach: The CEA compares the results of the current intervention with the results of an alternative measure in operation in the same region or the same sector.

Eureval-C3E – 20/04/2006 – page 41


In-depth analysis

Set of 15 reports including interesting CEA

A set of 15 reports including especially interesting CEA conclusions have been drawn from the databases, in line with the following typology, with a deliberate bias in favour of good examples.

11 reports were considered as good CEAs and included in the in-depth study. Four other reports have been added in order to increase the diversity in the final set of 15 reports in the following dimensions:

DG Year Type of approach Average cost of the intervention evaluated Type of management Scope of the intervention Type of effects analysed in the CEA

Set of 10 reports with no CEA

11 reports among the sample of 117 reports of the database contain an explanation why there was no CEA in the evaluation work. Five of those reports have a good explanation and were included in the second set of reports to be analysed in-depth. Five other reports have been chosen among the reports which were identified as having unsuccessfully attempted to do a CEA. Once again the objective was to maximise diversity in most of the above dimensions.

Annex 2 – In-depth analysis of 15 CEAs

No 30

The intervention

Description of the intervention The intervention concerns a food aid programme to a third country.

Type of management The programme is managed by the Commission (particularly the Common Service for External Relations) and authorities in the third country.

Cost The budget is fairly high, with 470 million euros for a one year period. This covers all monitoring and management costs, which are entirely borne by the European Commission.

Degree of complexity This is considered as somewhere between a simple and a complex intervention. The main reasons are:

• One main identifiable purpose: Food supply to needy population

• Several categories of beneficiaries: the population as

Eureval-C3E – 20/04/2006 – page 42


a whole • Several instruments: food delivery, counterpart funds

(proceeds of sales used for social purposes), free distribution to specific groups

The evaluation

Internal/external The evaluation was undertaken externally, by a consultancy company

Evaluation approach The evaluation is retrospective; it considers the effects of the programme.

Specific demand in the ToR

Efficiency demand in the ToR There is a demand for efficiency but no specific demand for cost-effectiveness analysis

Dimension of the demand The demand concerns the Commission management and implementation of the food aid programme. More specifically it concerns: administrative procedures, management system, organisational set up, monitoring system and auditing arrangements.

The analysis

Name of the tool in the report Cost efficiency

Name of the tool from our definition

Cost-effectiveness

Costing approach The cost included in the analysis is the value of supplied food and the cost of monitoring operations (person days committed to control and monitoring).

Effect approach The analysis considers only the outputs and does not take into account results and far-reaching impacts. The outputs are: products effectively delivered to final destination and counterparts.

Efficiency reasoning The evaluation uses an artificial scenario. A comparison is made between the supply of food from the aid programme and from the international market, in order to establish whether the market could have reached a better result at a lower cost. This comparison is made for food supply and the counterpart funds.

Efficiency response The response is included as a conclusion of the report. It considers the efficiency of technical execution of the food supply and monitoring operations. The judgment is made from the standpoint of the European Commission.

A score on a 1-5 scale is made to aggregate results for each criterion (efficiency, effectiveness, utility, etc.). The efficiency score of 3.5 is higher than almost all other criteria. It aggregates an excellent result for programme execution with poor performance regarding the time schedule and an extremely high cost of monitoring.

Contextual factors Due to the difficult situation of the third country, a strong monitoring apparatus has been set up, involving a large number of auditors. Data on the quantity of products is

Eureval-C3E – 20/04/2006 – page 43


easier to monitor. As the cost considered is the EC budget, no specific problem is reported concerning the collection of this information.

Eureval-C3E – 20/04/2006 – page 44


No 43

The intervention

Description of the intervention The intervention concerns a community policy on oilseeds as part of the oilseed common market organisation.

Type of management The programme expenditure is managed by member states.

Cost The cost of the intervention is fairly high, with 1,000 million euros per year. The evaluation is focused on the period 1992-2000.

Degree of complexity This is considered as a complex intervention. The main reasons are:

• Two main identifiable purposes: ensure oilseed supply; ensure a fair income to producers

• Several categories of beneficiaries: producers, processing industries and final users

• Several instruments: direct subsidies, free customs duty The evaluation

Internal/external The evaluation was undertaken externally, by a consultancy company.

Evaluation approach The evaluation is retrospective; it considers the effects of the Mac Sharry reform on the oilseed market.


Efficiency demand in the ToR There is a demand for efficiency but no specific demand for cost-effectiveness analysis.

Dimension of the demand The purpose of the demand is to highlight effects of the oilseed community policy from 1993-1999, with regard to the cost of this policy. The objective is not to target the policy as a whole but to focus on the system of direct subsidies.

The analysis

Name of the tool in the report efficiency


Cost-effectiveness

Costing approach The cost included in the analysis is the EC budgetary cost of the system of direct aid transfer.

Effect approach The analysis considers the results but does not take into account far-reaching impacts. It takes into account the results on producers and processing industries: income, competitiveness, access to raw materials.

Efficiency reasoning The evaluation uses a time comparison scenario. A comparison is made between the former system based on a price support regime, and the new system based on a direct subsidies regime. The implication of this change for stakeholders, in term of supplied cost and income, is assessed.

Eureval-C3E – 20/04/2006 – page 45


Efficiency response The response is included as a conclusion of the report. It considers the change in terms of EC budgetary costs and effects.

The reform involved a decrease of EC budgetary costs but did not alter the income of oilseed producers or the competitiveness of processing industries. The evaluator therefore concludes that the new type of support represents progress in terms of efficiency, compared to the previous support system.

Contextual factors The availability of data (subsidy, product quantity, income, etc.) allows for a sound analysis.

Eureval-C3E – 20/04/2006 – page 46


No 118

The intervention

Description of the intervention

The intervention concerns the main measures of the olive oil common market organisation.

Type of management The expenditure is managed by member states (12 EU members).

Cost The annual cost of the measure is fairly low, with a dedicated budget of 2 million euros per year. The study evaluates the period 1995-2000.

Degree of complexity This is a complex intervention. The main reasons are:

• Several identifiable purposes: improve the sector's productivity; ensure a fair income; maintain economic activity in a rural zone; market stabilisation; offer quality products; prevent impacts on the environment.

• Several categories of beneficiaries: Producers and consumers

Several instruments: production subsidies; tax on imports. Before 1998 three other instruments were used: reference fixed prices; consumption aid; storage system as a minimum guaranteed price.

The evaluation


Evaluation approach The evaluation is retrospective; it considers the effects of the programme.


Efficiency demand in the ToR

There is a demand for efficiency but no specific demand for cost-effectiveness analysis.

Dimension of the demand The demand can be identified by two different evaluation questions:

• Did the measures make it possible to reach market equilibrium in terms of volume? Was it at a reasonable cost?

• Was the impact of the measures on producer income obtained at a reasonable cost?

The analysis

Name of the tool in the report

Efficiency


Cost-effectiveness

Costing approach The analysis considers the entire cost of the EC budget dedicated to CMO on olive oil.

Effect approach As stated in the evaluation questions, two direct results are considered: market stabilisation; fair income to producers.

Eureval-C3E – 20/04/2006 – page 47


Those effects are treated separately.

Efficiency reasoning With regard to the market stabilisation effect, the evaluation uses a time comparison scenario. A comparison is made between the previous situation and the one subsequent to the change in the support regime (particularly the withdrawal of several support instruments).In terms of fair income to producers, the evaluation uses an artificial scenario. It compares the income situation of olive oil producers with that of producers of durum wheat, oil seed, rapeseed and sunflower.

Efficiency response The response is included as a conclusion of the report. It considers the difference in terms of amount of aid support and the variations of effects.

With regard to market stabilisation, the withdrawal of several support instruments seems to have increased the efficiency of the olive oil policy, as no guarantee existed previously on the impact of those instruments. With regard to market stabilisation, after having demonstrated that olive oil producers have similar incomes to all other farmers, a comparison of production subsidies in the olive oil sector and various other sectors (wheat, etc.) shows the efficiency of the policy.

Contextual factors Many data were available.

Eureval-C3E – 20/04/2006 – page 48


No 62

The intervention


The intervention concerns a specific measure in agriculture.

Cost The annual cost of the intervention is especially high, as it generally is for agricultural interventions (€1,600m per year)

Type of management

This intervention is managed by the Commission but the implementation is managed by Member States.


The intervention can be considered to be fairly simple in terms of the definition we used of a simple intervention:

• There is one global objective: the reform of the PAC in general• There is one general target group: EU farmers, even if this

group can be divided into different categories (according to the size of their properties or the type of crops)

• Even if several measures can be distinguished, there is a central one which is the payment of a contribution to farmers who implement the intervention.

The evaluation

internal / external The evaluation has been performed by an independent company specialised in environmental affairs

Evaluation approach

The evaluation is retrospective.

Indications in the ToR


There is a specific request stressed in an evaluation question concerning efficiency: Is the budgetary cost of the instrument justified in terms of the observed effects.

Dimension of the demand

The answer should be constructed on the basis of two counterfactual situations: (1) no payment for the intervention; (2) different subsidies. There is not a clear request for a CEA but there is a de facto one.

The analysis


Efficiency


This is a Cost Effectiveness Analysis

Effect approach The effects considered are results:

- superfluous cereal production - no cereal production

Efficiency reasoning

The evaluation uses a comparison between the direct costs of overproduction and the costs of no production. The comparison is made here between the direct budgetary costs of the 3 superfluous cereals and the costs of the entire subsidy distributed on a pro rata basis according to the production surface area for each of the 3

Eureval-C3E – 20/04/2006 – page 49


cereals.

Description of costs The costs considered are costs of overproduction and costs of no production (budgetary cost: subsidies to farmers); the costs are costs for the Commission only.

Efficiency response The response is a conclusion of the report. It considers an economic advantage in favour of the intervention at the beginning of the period (1993) – a tendency which was reversed during the period, in 1995 and 1996, for the same effects. This evolution stems from two main factors: the decline of the costs of overproduction (of the order of - 25 %) and the increase of the costs of no production resulting first of all from the increase of the payment for the intervention.

Contextual factors for CEA

The evaluator explains: “Without a simulation model, the analysis of the counterfactual situations was not possible. Payment for [the intervention] cannot be studied independently of the compensatory payments, which are within the framework of all the measures relating to the developers' income.

Eureval-C3E – 20/04/2006 – page 50


N°125

The intervention


The intervention concerns support for the production of a specific agricultural product inside the European market and support for the product price as compared to the global price.

Type of management

The programme is managed by the Commission and a part of the financial support is managed by the third country authorities.

Cost The budget of the intervention is fairly low. The cost of support for tobacco farming varies annually. The cost mainly includes the help given to producers.


This intervention can be considered to be fairly simple:

• One main identifiable purpose: supporting the producers • One main category of beneficiary: the Community producers • One main instrument: financial help to producers to prevent them

from producing too much (impact on the market price) and to promote more lucrative products

The evaluation

Full name of the report

Evaluation of the Common Market Organisation in the sector of raw tobacco, march 2003.

internal / external

The evaluation was undertaken externally by a consultancy company.

Evaluation approach

The evaluation is retrospective: it considers the past organisation of the product support (1993-2001).



There is an evaluation question about efficiency, but there is no specific demand for Cost Effectiveness analysis.


There are three evaluation questions dealing with efficiency for each aspect of the programme, to establish whether the effects were reached at a reasonable cost.

The analysis


Efficiency


Cost-effectiveness

Costing approach

The cost included in the analysis is the support cost to producers (average in 1998/1999); costs are simplified in only one group. Not all monitoring costs, management systems etc. are taken into consideration.

Effect approach

The analysis considers the results of the programme. The results are:

• The direct result on the viability of the producers' business

Eureval-C3E – 20/04/2006 – page 51


• The impact on economic activities and employment in rural development

• The impact on the producers' quality of life. Efficiency reasoning

For each evaluation question, a comparison is made between a situation with no EC intervention and a situation with EC intervention, to determine the role of EC support. A counterfactual situation is used with a comparison group.

Efficiency response

The response is included as a conclusion:

• A judgement statement at the end of each evaluation question section

• A section in the conclusion report. The efficiency judgement gives a statement on the role of the EC support with regard to the achievement of the intervention statement.

Contextual factors

There is a high level of availability of economic and financial data from various actors involved in the intervention.

According to the evaluator, there is a weakness in data available at sub-regional level, which explains the necessity to hold interviews.

In addition, the measure is fairly simple and focused on a well-defined group, which could explain the ability to achieve good efficiency.

Eureval-C3E – 20/04/2006 – page 52


No 181

The intervention

Description of the intervention The intervention concerns an employment programme.

Type of management The programme is managed by the Commission and the member states.

Cost The budget is fairly low, with 12 million euros for a one-year period. The evaluation stretches from November 2001 to July 2002.

Degree of complexity This is an in-between intervention. The main reasons are:

• Several identifiable purposes: create jobs, enable local development, create partnerships

• Several categories of beneficiaries: projects involving multi-stakeholder partnerships

• One main instrument: finance local projects The evaluation


Evaluation approach The evaluation is retrospective, considering the effects of the programme when achieved.


Efficiency demand in the ToR NO specific demand is highlighted in the study (however, the ToR was not available).


The analysis

Name of the tool in the report Cost-effectiveness


Cost-effectiveness

Costing approach The cost included in the analysis is the cost from the European Commission's side. It considers the EU inputs to finance projects, the cost of a Central Technical Assistance Office and the cost for disseminating materials, networking and organising seminars.

Effect approach The analysis considers outputs and outcomes but does not take into account far-reaching impacts. The outputs are the number of partners involved in the project. The outcomes relate to employment and territorial development outcomes with regard to specific objectives stated in each project.

Efficiency reasoning The evaluation uses an internal scenario. A comparison is made between the achievements of each project within the programme, and the share of the EU funding effectively spent is calculated.

Eureval-C3E – 20/04/2006 – page 53


Efficiency response The response is included as a conclusion of the report. The evaluation team concludes that the cost effectiveness of projects depends on the local situation (e.g. political involvement). Where projects have added value, such as partnerships, a new way of working or innovative labour market actions, and the cost-effectiveness of the programme is higher.

On the whole, 50% of the programme spending is seen as cost-effective. An overall judgment is made from the standpoint of the European Commission.

However, the evaluation team argues that a higher level of funding might have achieved proportionally more results. The analysis of various projects pointed to the low level of funds as a hindrance to their work (only just enough to support two full-time equivalent posts plus direct costs of support and administration)

Contextual factors A significant number of projects allow for an internal scenario.

Eureval-C3E – 20/04/2006 – page 54


No 194

The intervention

Description of the intervention The intervention concerns a professional training programme

Type of management The programme is managed primarily by the Commission.

Cost The budget is fairly high, with 150 million euros for a one-year period. The evaluation stretches from 2000 to 2002.

Degree of complexity This is a complex intervention. The main reasons are:

• several identifiable purposes: promoting a Europe of knowledge, professional integration, capacity to adapt to change, competitiveness

• Several categories of beneficiaries: companies, universities, national agencies, etc.

• Several measures: pilot projects, exchange projects, studies, networking.

The evaluation


Evaluation approach The evaluation is retrospective: it considers the mid term effects of the programme.


Efficiency demand in the ToR There is a demand for cost-effectiveness in the ToR.

Dimension of the demand The demand is highlighted by the following evaluation question:

What is the comparison between inputs (human and budgetary resources) and the first results?

The request is made for a comparative approach with past projects and between countries.

The analysis

Name of the tool in the report Cost-effectiveness


Cost-effectiveness

Costing approach The cost included in the analysis considers the cost of the European Commission and the cost of partners. It considers the cost of functioning (human resources and structural cost) and the cost of projects.

Effect approach The analysis considers outputs but does not take into account results and far-reaching impacts. The evaluation team explains that the short period provided for the evaluation, along with the absence of quantified objectives, preclude an examination of the programme results and impacts. The outputs are the number of

Eureval-C3E – 20/04/2006 – page 55


projects and the number of beneficiaries.

Efficiency reasoning The evaluation uses an external benchmark scenario. A comparison is made between the achievements of the current programme, the achievements of a former programme and the achievement of similar current programmes.

Efficiency response The response is included as a conclusion of the report. Considering the difficulties of comparing the different programmes, the evaluation team concludes that the cost-effectiveness of the programme should be improved. Better tools of management should be implemented.

Contextual factors Considering the lack of data due to the short period analysed, the evaluation succeeded in making a cost effectiveness judgement. Several programmes are in operation in the same field, thus allowing for possible comparison (although this comparison should be considered carefully). In addition, the evaluation considers output only, where data are more easily available.

Eureval-C3E – 20/04/2006 – page 56


No 275

The intervention


The intervention concerns the promotion of cooperation and commercial relations between the EU and third industrialised countries.

Type of management The programme is managed by the Commission, both from the central authority and by decentralised centres based in the considered industrialised countries.

Cost The budget of this intervention is particularly low, with €1m per year.

Intervention complexity The intervention is considered to be fairly simple:

• One main identifiable purpose: promoting the EU around the world.

• One main target group: foreign students in universities based in the industrialised countries considered.

The evaluation

Full name of the report Evaluation of projects and programmes promoting cooperation and commercial relations between the EU and the industrialised countries of North America and Australasia.

internal / external The evaluation has been undertaken by a consultancy company and associated experts. It is therefore external.

Evaluation approach This evaluation is retrospective; it considers the effects of the programme up to the beginning of the study. It was launched mid-term, during the implementation of the intervention.

The analysis


Cost-effectiveness


Cost-effectiveness

Costing approach The cost considered is the grant allocated by the EC for each industrialised country, per initiative centre. It represents the cost to pay a person full-time (or half-time) to run the initiative, as well as the costs induced by some promotion activities organised in these centres.

Effect approach The analysis considers the results, which are:

• The centre leverages additional funds since the industrialised country in which it is established recognizes the utility of the Centre and participates in its functioning.

• The visibility of the Initiative Centres. Efficiency reasoning The efficiency review looks at the degree to which inputs have

been transformed into outputs and includes an analysis of cost-effectiveness and value for money. Comparisons are made

Eureval-C3E – 20/04/2006 – page 57


between Centres and partner countries to establish how far the different activities, both individually and collectively, are a cost-effective means of achieving Initiative objectives.

Efficiency response The cost-effectiveness analysis ends in a conclusion on efficiency. The judgement is made from the standpoint of the European Commission.

A classification of the different structures is made to highlight the most efficient system.

Contextual factors The fact that there were a few well-identified centres may explain that it was possible to undertake a CEA. In addition, many data were available and a few more interviews have been held to complete the results.

Eureval-C3E – 20/04/2006 – page 58


No 453

The intervention

Full name of the intervention

The intervention is the global Community strategy to fight against the use of a certain toxic product in industry.

Type of management The programme in managed by the Commission and reactivated by the Member States' specific structures to fight against the use of this toxic product.

Cost The cost of this strategy is fairly low: about €5m per year. This is the cost of strengthening the relevant European legislation and control actions, and undertaking research on the impact of this product.

Degree of complexity This intervention is considered to be fairly complex:

• Several effects to be achieved: reducing the product emissions and its uses

• Several target groups: different target group are involved, depending on the effect

• Several instruments: practical studies, surveys, constraint actions, legislation, etc.

The evaluation

internal / external The evaluation was undertaken by the Directorate General in charge of this toxic product issue.

Evaluation approach This evaluation has a prospective dimension. The main objective is to use the present assessment on the subject to deal with the future strategy of the Community.

The analysis


Cost-efficiency


Cost-benefit

Costing approach The costing analysis only takes into consideration the way to destroy the toxic product. The costs considered are therefore the costs involved in destroying the toxic product.

Effect approach The analysis considers the impact of the product emission/release, and change in technical/economic processes.

Efficiency reasoning The analysis deals with the comparative cost of two different ways to abate the product emission for industries.

Efficiency response The response ended in a strong conclusion showing which technical practice was both effective and cheap.

Which costs for whom? Costs for the targeted enterprises

Contextual factor

Eureval-C3E – 20/04/2006 – page 59


No 478

The intervention


The intervention concerns a fishing agreement between the European Community and a third country.

Type of management The intervention is managed by the European Commission with the involvement of the third country itself.

Cost The intervention cost is fairly low for a European intervention (€680,000 per year) of which 41% (280,000) is set aside for targeted actions. Payment of financial compensation for the third country amounts to €400,000/yr.

Degree of complexity The intervention is considered to be between simple and complex:

• there are two very different objectives: improving the fishing area for European fishermen and contributing to conservation of the fish species

• several instruments: financial compensation and specific interventions

• but only one main target group: fishermen. The evaluation

internal / external The evaluation was performed externally by three companies specialised in fishing issues.

Evaluation approach The evaluation is both retrospective and prospective: it is divided into two parts: the first part concerns an assessment of the initial agreement (Ex post evaluation), focussing mainly on the impacts of this intervention; the second part concerns the new agreement and the improvements that could be made in the future with new tools (Ex ante evaluation).

The analysis


Cost-effectiveness analysis


Cost-benefit analysis

Costing approach The cost included in the analysis is the EC subsidy request. The costs considered are estimated financial contribution.

Effect approach The analysis considers results of the intervention, which is the creation of jobs within the EU

Efficiency approach The evaluation uses three artificial scenarios. A comparison is made between them. One of these scenarios is the current situation. This comparison is made with different criteria: employment and costs.

Efficiency response The response is included as a conclusion of the report. It is demonstrated that the current situation is less cost effective than the other two. Each scenario is described with its

Eureval-C3E – 20/04/2006 – page 60


advantages and disadvantages.

Contextual factors Difficulty in finding an appropriate comparison

Eureval-C3E – 20/04/2006 – page 61


No 485

The intervention


The intervention concerns both modal shift actions and innovative actions in industrial transport in Europe.

Type of management The programme is managed by the Commission and implementing in cooperation with the Member States.

Cost The budget of the intervention is fairly high (with €117m per year). This cost includes all the Commission subsidies to the different actions included under the programme. These subsidies are delivered for both tonne/kilometre of goods transported by rail and for training actions (meetings, training sessions, etc.)


The intervention is considered to be between simple and complex; the main reason being that there are two general objectives: to reduce road congestion (to enhance road safety) and to improve the environmental performance of the freight transport system within the Community. To achieve these objectives there are two sorts of action or instrument: modal shift and traffic avoidance actions.

The evaluation

internal / external The evaluation was undertaken externally by a company specialised in transport issues.

Evaluation approach The evaluation is prospective as it considers effects of the previous implementation period of the programme in order to define the new objectives and instruments for the next implementation period.



In the ToR there is a reference to a practical guide for preparing proposals for expenditure programmes.


The ToR provides a cost-opportunity analysis of the Community financial intervention in order to demonstrate its added value.

The analysis

Name of the tool I the report




Costing approach The cost taken into consideration in the CEA is the cost of moving one tonne of goods one kilometre via the different means of transport

Effect approach The direct impacts refer to the number of tonnes/kilometre shifted off the road.

Efficiency reasoning The efficiency reasoning is a comparison between the different ways of transporting goods, in relation to their cost.

Efficiency response The analysis constitutes the main argument for the efficiency

Eureval-C3E – 20/04/2006 – page 62


conclusion. A ratio is calculated of the different possibilities, the most cost-effective of which is the one with the lowest ratio.

The action with the highest value for money is then analysed.

Contextual factors The actions are very similar and have the same objectives and effects. It is therefore easy to compare them. Moreover, since the previous programme is very much the same as this one, it could be easy to compare the new actions with past ones to see which are the most efficient.

Eureval-C3E – 20/04/2006 – page 63


No518

The intervention


The intervention concerns the creation of a community agency for joint inspection and coordination of all the other agencies.

Type of management The intervention is conducted by the European Commission with the involvement of the Member States' agencies that already exist in the same field.

Cost The annual budget is high: €150m per year.

Degree of complexity The intervention is considered to be between simple and complex. The main reasons are:

• One main purpose: to improve the effectiveness of the control system

• It involves many stakeholders and different implementation systems.

The evaluation

internal / external The evaluation was undertaken externally by a consortium of companies specialised in the intervention field.

Evaluation approach The evaluation is prospective

The analysis





Costing approach The costs considered in the cost-effectiveness analysis are the costs for each member state

Effect approach The effect considered in the CEA is direct results: number of actions undertaken annually by each agency for each member state.

Efficiency reasoning The different agencies are compared to see if they are profitable compared to their operating costs.

Efficiency response The response is a conclusion of the report. It is proved that the present functioning is not profitable for every agency and that their reorganisation under one overarching agency would probably be more profitable, with at least the same effectiveness.

Contextual factors

Many data were available as the rule in these agencies is to collect a large volume of useful information.

Eureval-C3E – 20/04/2006 – page 64


No 576

The evaluation intervention


The intervention is a future framework research programme (2006-2010).

Type of management The programme is directly managed by the Commission.

Cost The budget of the programme evaluated is fairly high: €200m per year.

Degree of complexity The intervention is complex:

• Objectives in different fields • Target groups with different needs • Many instruments to reach the objectives.

The evaluation

Internal / external The evaluation considered was undertaken by an independent company, specialised in the intervention field.

Evaluation approach The evaluation has a prospective approach: several alternatives can be defined to reach the defined objectives. This Ex Ante evaluation was chosen from the different alternatives before the beginning of the programme.



NA


NA

The analysis


The term “analysis” is not used; the appropriate term is “assessment for cost-effectiveness”



Costing approach The costs considered are the costs of implementing the different strategies defined: materials, research activities, administrative procedures.

Effect approach The effects considered are the results of the new intervention

Efficiency reasoning The cost-effectiveness analysis applies a scoring system based on cost-effectiveness criteria; the sum of the criteria scores is the overall cost-effectiveness score of the different intervention strategies.

Efficiency response The efficiency is based only on a CEA, and leads to a positive conclusion on the interesting approach of the new framework programme.

Contextual factors The subject of the evaluation can be considered to be favourable to a CEA since the objective is to compare the different options

Eureval-C3E – 20/04/2006 – page 65


available for the implementation of the new programme; there is already the idea of a comparison.

Eureval-C3E – 20/04/2006 – page 66


No583

The intervention


The intervention concerns the implementation of an information processing system to obtain data from programme beneficiaries throughout Europe.

Cost The annual budget for this intervention is fairly low. It covers the costs of the implementation of an information processing system and administrative (or activity) costs.

Degree of complexity This intervention can be considered to be fairly simple:

• One major objective: to know the beneficiaries' opinions on policies

• The beneficiaries of the policy constitute the target group

• the information processing system is the only instrument used to meet the objective of obtaining beneficiaries' opinions.

The evaluation

Internal / external The evaluation was undertaken externally, by a consultancy company specialised in technological tools.

Evaluation approach The evaluation was undertaken mid-term during the implementation of the tool; it has both a retrospective and a prospective dimension. It considers statements from the previous period (beginning of the implementation) and makes hypotheses about the future implementation.


Efficiency demand in the ToR ?

Dimension of the demand Not available

The Cost Effectiveness Analysis


Cost-effectiveness


Cost-effectiveness

Costing approach The costs included in the analysis of the initiative are: the special budget for the tool, the budget to change the information system from one administration to another and the support of the tool to other DGs for other interventions.

Effect approach The quantitative analysis has mainly focused on intermediate outputs, for which rough proxy quantitative indicators are available.

Efficiency reasoning It was decided that a rough comparison between “orders of magnitude” in the cost of various available instruments could convey meaningful preliminary information on the intervention cost-effectiveness as compared to other alternative

Eureval-C3E – 20/04/2006 – page 67


approaches.

Efficiency response The analysis is included as a conclusion of the report. The CEA constitutes the main argument to deal with the efficiency issue.

Contextual factors The possibility of undertaking a CEA in this evaluation stems mainly from the fact that many quantitative data from desk research sources, as well as budgetary information directly provided by the Commission, were available. Moreover, the analytical classification process has been made possible by intensive exchange of information with Commission services, which has been complemented by qualitative information gathered through the stakeholders.

Eureval-C3E – 20/04/2006 – page 68


No 417

The intervention


The Community intervention considered is the implementation of a specific structure to observe a type of organisation.

Type of management The intervention is managed by the Commission.

Cost The budget is fairly low as it only corresponds to the administrative and functioning costs of this specific structure (€2m per year).

Degree of complexity The intervention is considered to be fairly simple:

• One main identifiable purpose: following the evolution of this type of organisation

• Only one beneficiary category: indirectly there are the observed organisations

• Only one instrument: the observation structure. The evaluation

internal / external The evaluation was undertaken externally by two research structures.

Evaluation approach The evaluation is retrospective; it considers the actual effects of the programme.


Efficiency demand There is a demand for dealing with efficiency issues and even a demand to review the ‘cost efficiency’ (term used in the ToR) of the structure.


The demand concerns a comparison between the very structure evaluated and other EU publications. It is also asked to make a qualitative judgement.

The analysis


Cost-effectiveness


Cost-effectiveness

Costing approach The cost considered is the cost structure linked to its activity. The structure was not evaluated in terms of profitability but according the suitability of the costs (general cost structure and costs per single activity i.e. survey costs, report costs, etc.)

Effect approach The analysis involves the two main research activities which are outputs:

• The field survey • The drafting of the reports/thematic chapters.

Eureval-C3E – 20/04/2006 – page 69


Efficiency approach It analyses the budget structure and compares it with some benchmarks given by other research institutes identified as potential competitors in the market. The financial structure is then compared with the outputs produced by the structure.

Efficiency response The response is included as a conclusion of the report. The judgement is made from the standpoint of the market.

The structure is cost-effective in terms of administrative costs, but the overall cost of the management structure, including the network of national correspondents, is high and not justified by the final outputs.

Contextual factors There are few similar structures to compare with the evaluated structure; this may be a strong argument in favour of undertaking a CEA as it is often difficult to find similar interventions to Community interventions.

Eureval-C3E – 20/04/2006 – page 70


Annex 3 – In-depth analysis of weak CEAs

No 68

The intervention


The intervention concerns an action plan in the energy field.

Cost The budget of this intervention is considered to be low, compared to other more expensive interventions (€35m per year).

Type of management

This intervention is managed by the European Commission.


The intervention can be considered to be complex:

• Several objectives: support, monitoring, promoting, training • Several target groups: industries, citizens • Several instruments: financial instruments, information.

Evaluation

internal / external The evaluation was undertaken externally by an independent company.

Evaluation approach

This is a retrospective evaluation dealing with the past period of implementation of the intervention.

Analysis

Description of effects

The effects taken in consideration are only outputs:

- studies/standards - information, education, training, pilot actions - information networks - studies, biomass.

Description of costs The costs taken into consideration are the Community funds allocated to the intervention by the successive Council Decision Boards.

Information about effectiveness

- studies/standards - information, education, training, pilot actions - information networks - studies, biomass.

Information about efficiency

A comparison is undertaken of the funds allocated to the intervention annually. The analysis is somewhat poor.

Contextual factors for no CEA

The evaluators consider that the Commission should increase the amount of effort put into evaluating the actions of the intervention under consideration as well as its projects so that a realistic and reliable view of impacts and cost-effectiveness can be obtained.

Actually there is not enough information and data collected to enable the evaluation team to conclude.

Eureval-C3E – 20/04/2006 – page 71


No 81

The intervention


The intervention concerns the creation of a European Agency for protecting workers.

Cost Information not available in the report.

Type of management

This intervention is managed by the Commission.

Complex/ simple The intervention can be considered to be a simple measure:

• One main target group: European workers • One main objective: collection and dissemination of

information that can serve the needs of people • Homogeneous instruments: data collection.

Evaluation

internal / external This evaluation has been undertaken externally by a company specialized in social development.

Evaluation approach

This is a retrospective evaluation. The objective of the study is to make an assessment of the past period of implementation.

Analysis


The effects considered are impacts. The main impact considered is the growing awareness of social rights issues in Europe.

No analysis but some remarks about cost-effectiveness. Cost- effectiveness and -efficiency seem to be confused.

“The evaluation data point to a rather low impact at this stage but, on the other hand, quite a success story in building the portals, rising awareness of OSH-matters in Europe and establishing the Network with Focal Points and others.

No information from any source (interviews, survey, case studies) points to a complaint or a major problem in this respect. The survey (Annex I) shows that the different stakeholders see the Bilbao-office cost-effectiveness as average and the cost-effectiveness of Focal Points as clearly above average. Neither do the interviews of Commission officials point to actual problems in cost-effectiveness or -efficiency.” (p 38)

Description of costs

No costs at all are described in this report.


The effectiveness of the intervention has been calculated as the value added to national efforts in the field.

Effectiveness has been treated with qualitative data (interviews, survey, case studies).


There is no work on efficiency.

Eureval-C3E – 20/04/2006 – page 72


Contextual factors The intervention is evaluated at mid-term. As cost-effectiveness analysis is judging input against outcome/impact, it is considered to be too early a stage to identify impacts.

Eureval-C3E – 20/04/2006 – page 73


No 252

The intervention


The intervention is a network for mutual exchange of information of a documentary nature.

Type of management The intervention is managed by the European Commission and indirectly by the Member States.

Cost The cost of the intervention is low (€2m per year). It covers essentially administrative costs.

Degree of complexity The intervention is considered to be rather complex:

• Several target groups: countries' administrations, education and culture actors, etc.

• Several instruments: the network consists of many different instruments such as training, hand books, databases, etc.

• Several intermediate objectives: comparison of policies, improving mutual knowledge.

The evaluation

internal / external This evaluation has been undertaken by an external company.

Evaluation approach The evaluation has a retrospective dimension.

Specific demand in the ToR for efficiency


One of the evaluation questions concerns what in the report is called “cost efficiency of the system as it is presently conceived” and how to improve this “cost efficiency”.

The analysis

Description of effects The word analysis is not used; only cost-effectiveness.

Description of costs There is a good description of costs, with useful work on cost trends.

The costs considered include:

-the cost of the subsidy of the European Commission,

- the cost of the UN covered by European Commission subsidies,

- the cost of the UN chargeable to States.

Description of effectiveness

Two evaluation questions deal with effectiveness. The way to proceed has been to classify the effects in three parts:

- the effects corresponding to the objectives of the programme fully, partly, and not

Description of efficiency

There is no clear mention of efficiency analysis.

Contextual factors The effects of the network are not quantified beforehand.

Eureval-C3E – 20/04/2006 – page 74


Moreover, there is a complaint in the report about the lack of complete and homogeneous financial and countable data.

This could be explained by the fact that the intervention is centralised by the Commission but implemented by many different operators in many countries and at different implementation levels.

Eureval-C3E – 20/04/2006 – page 75


No 451

The intervention


This intervention concerns an initiative to combat domestic violence.

Type of management

This programme is managed by the Commission.

Cost The budget of this intervention is fairly low (€3m per year). This cost includes supplies for each accepted project manager: NGOs and the appropriate authorities.


This intervention can be considered as a complex measure:

• Several target groups: children, young people, women • Several instruments: direct help, exchange of information and good

practice, research actions • Several intermediate objectives: informing the population, rescue

for victims, etc.

The evaluation

internal / external

This evaluation has been undertaken internally by the DG responsible for the intervention itself.

Type of evaluation approach

This is a retrospective evaluation. The objective of this study is to make an assessment about the past implementation of the programme.

The analysis of effectiveness and efficiency


The effects considered are both outputs and impacts.


To deal with costs, a sample of 148 projects has been defined. It represents a total budget of nearly €20 million (up to 2001), i.e. about €133,000 per project and €23,500 per participant.

The costs considered are not only the subsidies granted by the DG but also the subsidies that the projects obtained in other ways or internal funds.

How far does the report analyse effectiveness?

The effects considered are analysed according three criteria:

• Sustainability of the projects • Quality of the outputs • Achievement of high-level objectives: generation of common

definitions or a common approach to a violence-related issue. Information about efficiency

There was nothing about efficiency, as if there was a strong reluctance to deal with the costs. There is only a description of the costs and no analysis.

Eureval-C3E – 20/04/2006 – page 76


Contextual factors

A programme like the one described is considered to be not only a production unit, the costs and benefits of which can be precisely estimated in monetary terms. Although most of the inputs are known – in terms of funds – it is meaningless to put a monetary value on the outputs or impacts: e.g. the value of enabling children and women to protect themselves better and prevent violence cannot be expressed in commercial terms. However, gain indications of the usefulness of the programme’s outputs and impacts are given.

Eureval-C3E – 20/04/2006 – page 77


No 492

The intervention


The intervention is the evaluation of the same programme in each Member State.

Type of management

The intervention is entirely managed by the European Commission and especially by the DG undertaking this programme.

Cost The total cost of the intervention is €35m per year, which is considered to be low for the general cost of an intervention in the European Commission but which could be considered to be high compared to other such interventions.


The intervention is considered to be complex:

• Several objectives: the aim of the intervention is to judge the evaluations that have been undertaken on the programme, to focus on the companies involved, to synthesise the good practices all over Europe, etc.

• Several target groups: the DG, the institutions implementing the programme

• One instrument: assessing the reports. The evaluation

internal / external The intervention has been undertaken internally by the Unit in charge of assessments at the DG concerned.

Evaluation approach

This intervention is a retrospective evaluation.

The analysis


The effects considered are results: the quality of evaluations.

Description of costs The costs taken into consideration are the costs of evaluations, so that here they are the costs for the Commission, indirectly paid by the countries.


The effectiveness of the evaluations is not described in broad terms. The evaluations have not all been completed, they do not take effectiveness into consideration. However, the evaluation provides feedback on the likely achievement of the projected impacts of the intervention.


Substantial resources were allocated to the mid-term evaluation exercise, which contributed to its improved quality compared to the past.

There was no correlation between the cost of the evaluations and the Commission’s assessment of their quality. The size of the programme was a more important factor in determining the budget for the evaluation, although here the evident variation across Member States was too great.

Eureval-C3E – 20/04/2006 – page 78


Contextual factors for no CEA

Clearly, it is difficult to assess the cost-effectiveness of the mid-term evaluation process. “It is undoubtedly true that a very significant amount of resources was allocated to the mid-term evaluation. The ultimate cost-effectiveness relates to the usefulness of the evaluations in improving the quality of the programmes, and in providing added value to managing authorities and to Member States and the Commission, by pointing to possible improvements to programmes and accounting for what had been achieved to date.”

Eureval-C3E – 20/04/2006 – page 79


No 5



This intervention concerns an initiative to support the launching of trans-European applications and services in sectors of public interest (particularly Internet and telecommunications networks)

Type of management

The programme is mainly managed by the Commission.

Cost The budget of the intervention is considered as average with €25m per year. The evaluated period stretches from 1997 to 2000.


This is a simple intervention. The main reasons are:

• One main objective: bridge the gap between research and markets • Several target groups: industrial actors, users, public entities • One instrument: project funding.



In the terms of reference, there is an indication that specifies the necessity to deal with efficiency and cost effectiveness in the evaluation.


No specific dimension is stated.

The evaluation

internal / external

This evaluation has been undertaken externally by a consultancy company.


This is a retrospective evaluation. The objective of this study is to make a mid-term assessment of the achievements of the programme.

The evaluation


The effects considered are outputs (number of projects), results (aim of the projects) and impacts (on market, growth, etc.).


The evaluation considers the entire cost from the Commission's standpoint. It includes the programme's operating costs and the cost of project support.


Outputs are fairly well described according to criteria, while results and impacts are rather based on expectations, due to the small number of projects. A survey was undertaken to feed the effectiveness section.


The evaluation compares the cost of the different projects and states that for some projects there is considerable room for improvement of efficiency with regard to the number of person-days spent on individual work packages. However, the small number of projects being evaluated and the lack of budgetary resources per project weaken the analysis.

Contextual factors

The evaluation team does not explain why it does not focus on cost-effectiveness. However, it is possible to underline the following factors:

Eureval-C3E – 20/04/2006 – page 80


• Difficulties of assessing effects due to the small number of projects (nine) being evaluated. The effects are based more on expectations than on actual achievements

• Difficulties of assessing cost because projects are not described in detail and little information is available.

Eureval-C3E – 20/04/2006 – page 81


No 38



This intervention concerns an EC strategy for the health sector of a developing country.

Type of management

The programme is managed jointly with international organisations.

Cost The budget of the intervention is fairly low, with €7m per year.


This is a simple intervention. The main reasons are:

• One main objective: Creation of municipal health units • Several target groups: the population as a whole, public health

bodies • One instrument: project funding.



There is a demand to evaluate efficiency but no specific demand for cost-effectiveness.


The evaluation

internal / external



This is a retrospective evaluation. The objective of this study is to make an assessment about the achievements of the programme.

The evaluation


The considered effects are the outputs (number of health units set up) and the results (e.g. type and quality of health services)


The cost considered in the analysis is the cost of implementing the strategy. It includes human resources, building activities and equipment.


The evaluation undertakes an effectiveness analysis with regard to:

• the strategy • the different projects • support-building capacity with regard to administrative and financial

management of the national health institution. Information about efficiency

A rather weak comparison is made between the costs of the projects with regard to the outputs expected from the programme (in particular with regard to the results corresponding to the implementation of a health unit). The evaluation team concludes that projects involve high costs compared to results.

Contextual The analysis of this strategy is rather difficult because there is no clear stated logical framework. The strategy is an uncoordinated group of

Eureval-C3E – 20/04/2006 – page 82


factors concepts, recommendations, projects and institutional aspects. In addition there is no specific organisation in charge of monitoring and assessing the evolution of the programme.

Eureval-C3E – 20/04/2006 – page 83


No 28



This intervention concerns an initiative to aid uprooted people.

Type of management

The programme is a joint management with an international organisation.

Cost The budget of the intervention is considered as average, with €60m per year. The evaluated period stretches from 1997 to 1999.


This is a complex intervention. The main reasons are:

• One main objective: assist uprooted people in the intermediate stage between emergency and stabilisation, with a view to encouraging democratisation and promoting human rights

• Several target groups: uprooted people, surrounding populations, etc.

• One instrument: project funding. Specific demand in the ToR


In the terms of reference there is an indication that specifies the necessity to deal with efficiency and cost effectiveness in the evaluation.



The evaluation

internal / external



This is a retrospective evaluation. The objective of this study is to assess the achievements of the programme.

The evaluation


The effects considered are results (integration, improved skills, etc.) and impacts (e.g. environment)


Data on costs are scarce; only some project costs are known.


The effectiveness of the projects financed by the programme is analysed for the following:

• welfare and protection • improved skills and technical capacities • self-sufficiency • integration and reintegration • peace-building, democratisation and institutional building • gender • environment.

Eureval-C3E – 20/04/2006 – page 84



The efficiency analysis is hardly developed in the report.

All this requires proximity to the field and the capacity to respond appropriately. The centralised structure of the Commission has not allowed appropriate decision-making at local level. Although some decision-making authority has recently been devolved to Delegations, it has made little impact on the problems identified by this evaluation.

The conditions, under which AUP operated, particularly in refugee and open conflict contexts, increased the need for flexibility.

Improving the modalities of management of AUP will ensure increased efficiency and effectiveness of EC aid, but also enhance the EC’s capacity to play a leading and co-ordinating role, particularly at EU level.

The modalities of management mentioned below require appropriate levels, skills and personal attitudes of staff. It should nevertheless be recognised that elements related to organisational culture are key determinants of performance.

Contextual factors

In view of the specific objective of this evaluation - evaluating the financial instrument and not the projects financed by it – aspects related to cost-efficiency of the interventions financed by the line are not addressed.

Eureval-C3E – 20/04/2006 – page 85


No 45



This intervention concerns an initiative to support the victims of a natural disaster

Type of management

The programme is managed jointly with international organisations.

Cost The budget of the intervention is considered as medium, with €25m per year. The evaluated period stretches from 1997 to 2000.


It is a simple intervention. The main reasons are:

• One main objective: bridge the gap between research and markets • Several target groups: industrial actors, users, public entities • one instrument: project funding



In the terms of reference, there is an indication that specifies the necessity to deal with efficiency and cost-effectiveness in the evaluation.



The evaluation

internal / external



This is a retrospective evaluation. The objective of this study is a mid-term assessment of the achievements of the programme.

The evaluation


The effects considered are impacts and results.


The cost considered is the cost of projects.


The evaluation team held interviews to evaluate the effectiveness of the programme. The analysis considered the followings:

• Effect on dependency on external aid • Effects on local economy and population incomes • Effects on the preparation, mitigation and prevention of

catastrophes • Effects on health and nutritional practices • Environmental effects • Effects on capacity-building:

Information about

An efficiency analysis is undertaken for the following:

• Quality of planning and mobilisation of aid

Eureval-C3E – 20/04/2006 – page 86


efficiency • The timing of interventions • Financial management • Logistics management • Monitoring mechanisms.

Contextual factors

The factors which could explain that the evaluation team did not undertake a cost-effectiveness analysis are the following:

• There is a lack of monitoring (particularly performance indicators) • No clear objectives and results were stated at the beginning, which

could easily be measurable and monitored by NGOs • Local conditions are often impossible to compare (as performing a

cost-effectiveness analysis requires a comparison between projects).

Eureval-C3E – 20/04/2006 – page 87


No 37



This intervention concerns an initiative to facilitate the transition of a country to a market economy and to reinforce democracy

Type of management

The programme is managed by the third country.

Cost The budget of the intervention is considered as average, with €35m per year.


This is a complex intervention. The main reasons are:

• Several objectives: (i) human resources development (education, public administration reform, and legal assistance); business restructuring and development (privatisation and restructuring); infrastructure development; energy; food production; and environment.

• Several target groups: institutions, citizens, policy makers • Several instruments: technical assistance, project funding.



There is demand for an efficiency analysis but no specific demand for a CEA.


The evaluation

internal / external



This is a retrospective and prospective evaluation. The objective is first to make an assessment of the achievements of the programme and secondly to assess the programme in relation to the reform process.

The evaluation


Effects considered in the evaluation are mainly impacts and outcomes with regard to sectorial effects (infrastructure, energy, food production, etc.)


The cost considered is the implementation of the strategy.


Effectiveness was measured in terms of the extent to which sector programme objectives were achieved and the sustainability of the initiatives supported. It drew on information from project terms of reference and subsequent monitoring reports, particularly the end-of-project assessments and the findings of the evaluation of the regional programme.

From this was derived a ranking for each project reflecting achievement of objectives and sustainability. This ranking reflected the “scores” achieved in the monitoring reports averaged across the projects in a sector. This analysis concluded that the programme had been most successful in the sectors of transport and energy legislation.

Eureval-C3E – 20/04/2006 – page 88



The evaluation concludes that insufficient attention had been given to ensuring cost-effectiveness in project design and implementation. Specifically, lessons learned during project implementation have not been fully reflected in subsequent rounds of programming. Small Project Programmes with potentially high impacts and limited costs have not been fully exploited.

Contextual factors

Eureval-C3E – 20/04/2006 – page 89


Annex 4 – Examples of good practice outside the EC

Cost Effectiveness of Implementing SSLPs: An Interim Report

Report by the National Evaluation of Sure Start Team at the Institute for the Study of Children, Families & Social Issues, University of London

Several authors from different universities and public agencies

More information available at: http://www.ness.bbk.ac.uk

Description

The cost-effectiveness analysis is part of a national evaluation of the UK Sure Start programme that encompasses early education, childcare, health and family support, to deliver the best start in life for every child.

Sure Start can be thought of as an investment in young children and their families, which is rather like an investment in education. Costs are incurred in the short term in the expectation that there will be a return on that investment in the longer term. One of the main reasons for investing early is that in the development of children’s potential, they need to have achieved lower level milestones before they can move onto higher ones.

At this stage only an interim report has been released, the measurement of inputs. The analysis of the relationships between costs and effects will take place at a later stage only.

Also available are methodological notes and an impact report with early assessments of outcomes.

The overall aim of the evaluation is to produce a comprehensive picture of the first 260 Sure Start programmes in terms of process and components, so that programme activity can be linked to outcomes for individual users and communities.

Approach

The study, which incorporates quantitative and qualitative methods, has three components:

• A National Survey of all 260 Sure Start programmes. • An in-depth study of 26 programmes. • A series of thematic evaluations.

The cost-effectiveness evaluation aims to answer these core questions:

1. What is the total level of resources being spent on Sure Start?

Eureval-C3E – 20/04/2006 – page 90

http://www.ness.bbk.ac.uk/


2. What is the true cost of a particular initiative?

3. What are the benefits of Sure Start for children, their families, the local community and the wider public, which can be quantified in monetary terms?

4. Did the outcomes justify the investment of resources?

5. Was this the most efficient way to achieve the identified outcomes?

As part of the evaluation, the study will consider both the cost-effectiveness and the cost-benefits of the use of Sure Start resources.

Costs will be derived from information supplied by each programme to the Sure Start Unit. The information is supposed to include the costs of other services already being provided in the area, in order both to measure deadweights and to recognise that the achievement of individual targets may be influenced by the level of resources already being provided to support existing mainstream services.

The second stage will relate costs to multiple short-term outcomes. Indicators will be established for costs of achieving particular outcomes through different routes.

Finally, a cost-benefit analysis will use a human capital framework to consider the potential range of benefits from the intervention.

The evaluation will assess the value of the outcomes achieved by Sure Start for:

• Children • Families • The local community • The wider public (including taxpayers)

Overall, the evaluation will consider potential outcomes and benefits for the following:

• Educational outcomes and skill development. • Health (physical and socio-emotional) outcomes. • Behavioural outcomes (crime, domestic violence, substance use or

delinquency). • Employment outcomes (employment rates and earnings). • The economic and social well-being of the area.

Strengths and weaknesses

Sure Start is a good example of a complex intervention in the sense that it includes multiple objectives, target groups, interventions and outcomes.

Although it is an interim report that does not contain a full cost-effectiveness analysis, the report (including methodological annexes) is interesting as it provides a very systematic methodology for assessing costs and outcomes.

Eureval-C3E – 20/04/2006 – page 91


Cost-effectiveness with multiple outcomes

Article in Health Economics, 13: 1181–1190 (2004)

Authors:

Jakob BJØRNER - National Institute of Occupational Health, Copenhagen, Denmark and Hans KEIDING - University of Copenhagen, Denmark

Description

The article discusses analysis of cost-effectiveness in cases where outcome can be captured only by several measures. The method is applied to an analysis of the cost-effectiveness of alternative health plans using data from the Medical Outcome Study (JAMA 1996; 276: 1039–1047), where outcome is measured as improvement in mental and physical health.

In a large number of situations, activities in health care have to be measured in terms of outcome and cost. However, there are relatively few cases where outcome is fully captured by a single measure. An outcome score is therefore used. This score is computed by weighing together several outcome measures, using subjective and somewhat arbitrary weights. In the paper the authors propose an approach to cost-effectiveness analysis where such artificial aggregation is avoided. This is achieved by assigning to each activity the weights which are the most favourable in a comparison with the other options available, so that activities which have a poor score in this method are guaranteed to be inferior. The method corresponds to applying data envelopment analysis –from the theory of productivity – to the context of health-related economic evaluations.

Approach

The study approach addresses an important problem in the field of cost-effectiveness analysis related to the dimensionality of outcomes. If outcome is one-dimensional, then in cases such as new treatments or drugs it can be compared by means of cost-effectiveness ratios, whereas this method fails in the case of multidimensional outcomes for the simple reason that the cost-effectiveness ratio is undefined.

As pointed out by the authors, there are more subtle, underlying reasons. Attempts to replace inherently multidimensional outcomes by synthetic one-dimensional outcomes (a single index of health or quality-of-life) and to compute cost-effectiveness ratios with such outcome indexes (so-called cost-utility analysis), run into problems of consistency since they use aggregation over individuals, which is at best arbitrary and at worst misleading.

However, aggregation of outcome measures may be avoided if absolute cost-effectiveness is replaced by relative cost-effectiveness, where performance of a project is measured in relation to a given set of projects. The results of the analysis may be used to rank projects as more or less desirable from society's point of view, but the ranking will not be complete, as in the case of classical cost-effectiveness analysis, since some pairs of projects may not be comparable.

Eureval-C3E – 20/04/2006 – page 92


The DEA analysis results in a score which is 1 if there is an aggregation method such that the project is as good as any of the other projects when this aggregation method is used, and less that 1 in all other cases.

In the article, the example of a comparison of several health oriented projects on the basis of relative cost effectiveness (RCE) to be considered is taken from a previous study, the Medical Outcomes Study (see Tarlov et al. and Ware et al). The study was carried out in order to compare physical and mental outcomes of chronically ill adults treated in health maintenance organizations (HMO) and fee-for-service systems (FFS) in three USA cities (Boston, Chicago and Los Angeles). In the study, 2,235 patients with hypertension, non-insulin-dependent diabetes mellitus, recent acute myocardial infarction, congestive heart failure, and depressive disorder were sampled from HMO and FFS systems in the USA in 1986 and followed up through 1990.

The present analysis concerns all patients with data on membership of one of 11 specific Health plans. The purpose is to illustrate a comparison of the relative cost-effectiveness of different plans. The outcome measure was differences between initial and 4-year follow-up scores of summary physical and mental health from the Medical Outcomes Study 36-item Short Form Health Survey. Their analysis therefore covers only two outcomes.

For a cost-effectiveness comparison of the health plans, they use cost data operationalised as the number of days of hospitalisation and the number of visits to the physician (doctors in the study include family practitioners, cardiologists, endocrinologists and psychiatrists).

The computation was done using the DEA-software EMS (Efficiency Measurement System) developed by Scheel.


The article applies DEA (Data Envelopment Analysis) as an approach to cost-effectiveness analysis. DEA analysis can handle several types of input and outcomes (although in the article only two different outcomes are used) and does not require an assumption of a functional form relating inputs to outputs. It is therefore a very interesting method for assessing the cost-effectiveness of complex programmes and policies, where there are often multidimensional outcomes and where the ratios cannot be directly compared.

By using DEA, cost effectiveness is measured as relative cost-effectiveness, where the cost-effectiveness of one intervention is benchmarked with the cost-effectiveness of a given set of interventions.

The evaluation contains a theoretical basis for undertaking a cost-efficiency evaluation when there are multiple outcomes. However, its theoretical insights are the most valuable contribution. The specific analysis is more of a tentative quality to illustrate their method.

From the perspective of the CEA study, the primary weakness of the study is that DEA analysis is fairly complex and requires specialised software to perform. It may therefore be difficult to integrate in mainstream evaluations although the outcomes of DEA analysis would certainly be applicable to such evaluations.

Eureval-C3E – 20/04/2006 – page 93


Diverting children from a life of crime: Measuring costs and benefits

Authors:

Peter W. GREENWOOD, Karyn MODEL, C. Peter RYDELL, James CHIESA of the RAND Cooperation

The report can be downloaded chapter by chapter at:

http://www.rand.org/pubs/monograph_reports/MR699-1/

Description

The report assesses the cost-effectiveness of several crime-prevention strategies that involve early intervention in the lives of people at risk of pursuing a criminal career. The report compares the effectiveness of averting serious crimes per million dollars spent of five different types of intervention. The focus of the analysis is on prevention – not rehabilitation and punishment.

The study takes a first step toward answering the question on how much future crime will be prevented if money is diverted from incarceration to early intervention approaches. Evaluations of early interventions completed to date are described as small and scattered, and the purpose of the study is to identify approaches with enough promise to warrant further analysis of data in the literature and possibly further demonstration on a larger scale.

Approach

The study considers four different approaches to intervening early in the lives of children at some risk of future trouble with the law:

1) Home visits by child-care professionals in order to provide guidance in infant care and ward off the likelihood of abuse or neglect, both of which are associated with troubled childhoods.

2) Training for parents and therapy for families with very young school-age children who have shown aggressive behaviour or otherwise begun to “act out” in school.

3) Four years of cash and other incentives to induce disadvantaged high school students to graduate.

4) Monitoring and supervising high-school-age youths who have already exhibited delinquent behaviour.

Explicitly, the study seeks to answer the following questions:

• What percentage of the population is to be treated, and how much crime do they commit?

• What is the cost per treatment? • How effective is each programme at preventing crime? • How will effectiveness change if the programme is expanded? • How long do effects persist after treatment has ceased?

Eureval-C3E – 20/04/2006 – page 94

http://www.rand.org/pubs/monograph_reports/MR699-1/


Programme effectiveness is calculated as reductions in “serious” crime, as defined by California’s Penal Code. Reductions are in turn calculated using control group comparisons (outcomes of treatment groups compared with outcomes for total population).

Costs are calculated as the direct costs per person treated (the programme costs divided by number of participants).

Outcomes in terms of reductions in serious crime are subsequently monetarised. Accordingly, the study is both a cost-effectiveness and a cost-benefit analysis, where both inputs and outcomes are expressed in monetary terms.


The reason for selecting this study is that it is an example of a cost-effectiveness analysis that builds on previous research to establish a number of assumptions, which are then used for assessing the cost-effectiveness and subsequently cost-benefits. Moreover, their methodology is well described.

While the study has some methodological limitations in terms of scope and reliability of data, it tries to compensate for these shortcomings by thoroughly analysing the sensitivity of the results to the assumptions made about programme efficacy and other factors.

The programmes that are studied are complex and include several types of intervention. However, from the point of view of the CEA study, the main weakness is the fact that the outcome variable is standardised (reductions in serious crime), which will very often not be possible in evaluations of complex programmes.

Eureval-C3E – 20/04/2006 – page 95


The cost-effectiveness evaluation of minority language policies: Case studies on Wales, Ireland and the Basque Country

Report by François GRIN François VAILLANCOURT from European Centre for Minority Issues (ECMI) (1997)

Short description of the study

The report considers the measurement of the cost-effectiveness of various policies in favour of minority languages in the form of four case studies (two in Wales, one in Euskadi and one in the City of Galway).

The research was initiated in 1997 by the New Zealand Government to draw on a set of systematic policy instruments with a view to designing revitalisation policies in favour of the Maori language. The tender issued by Treasury in Wellington specifically required researchers to make reference to, and preferably base their work on some existing theoretical papers in the economics of language, and to apply the framework developed there to an empirical evaluation of minority language policies. The underlying question running through the study, therefore, is the identification of what works, under what conditions, and at what price.

The emphasis of the study is on the measurement of the effects of policies, the expression of these effects in terms of a common unit of measurement, and the linkage of the resulting indicator with information on the cost of each policy. This allows them to compare very different policies in terms of one common criterion, that is, cost-effectiveness.

Approach

The researchers first developed an analytical framework connecting language policies with ultimate language outcomes, in relation to both a set of sociolinguistic indicators and a formalised model of bilinguals’ language behaviour. They then moved on to the analysis of specific policies, beginning with a description of the relevant demo-linguistic, geo-linguistic and historical backgrounds, showing how each policy “fits” into the analytical framework and identifying the key actors of the policies — institutional or not. They then used figures on the expendi ure incurred to implement the policy, and derived an indicator of its cost — the crucial difference between these two concepts being that “expenditure” is essentially an accounting concept, whereas “cost” is a notion that makes sense only in relation to whatever it is that money is being spent on. This requires a precise definition of the output, and hence the selection of an output measure which allows for systematic comparison between the policies. The unit of measurement adopted in the study is that of the increase in aggregate person-hours of minority language use that can be associated with the policy. Expenditure can then be divided by the change in language use to obtain the per-hour cost of minoritylanguage use under a given policy. These calculations are then integrated in an overall assessment of the policy.

t

The in-depth analysis of the cost-effectiveness dimension of a set of language policies, along with the establishment of a methodology to this end, is the first of the goals pursued in the study. However, another goal is being pursued, namely the

Eureval-C3E – 20/04/2006 – page 96


development of a systematic approach to the selection of language policies among various options.

The comparison of policy options proceeds as follows: having calculated per hour costs, they use them as one of two terms in the construction of a best practice index. The other term is provided by a set of four impact indicators of the policy, namely: (i) the competence of speakers of a minority language; (ii) the number of speakers; (iii) their attitudes — particularly towards the minority language; and (iv) their degree of language use. These four indicators are not constructed formally, and they represent not absolute impacts but an estimated ranking of impacts on a ten-point scale. They are subsequently combined in an overall impact indicator given by their rounded arithmetic average. Finally, the “best practice index” is the simple average, for each policy, of the latter indicator and of its per-hour cost.

The researchers used data from a wide range of public and private agencies and official bodies from the UK, Europe, New Zealand and Australia.

Strengths and weaknesses of the study

The report tries to measure costs and outcomes related to minority languages. It does so in a structured way based on a clear theoretical model on how language policies lead to specific outcomes.

The report contains an interesting (although tentative) comparative assessment of the cost-effectiveness of the four studied policies.

The report’s main strength is that it demonstrates that it is possible to assess the outcomes and costs of (language) policies with no standardised outcome variables in such a way as to allow an explicit evaluation and ranking in terms of their effectiveness and cost-effectiveness.

A weakness may be that the policy costs taken into account in the report are restricted to direct and indirect financial costs (that is, direct money outlays plus “opportunity” costs). In other words, the “symbolic” or other non-material costs of policies are deliberately omitted.

The researchers have had to replace actual data on language use (or, more specifically, on increased language use as a result of a particular policy) by estimates of the latter. These estimates are rough. However, they consider them to be better than no estimates at all. Besides, their main point is not to establish the absolute, but the relative cost-effectiveness of different policies, in order to provide a ranking of cost-effectiveness.

Eureval-C3E – 20/04/2006 – page 97


Cost-effectiveness analysis of the New South Wales Adult Drug Court Program

Article published in the Evaluation Review, vol. 28 No. 1, February 2004 3-27

Authors:

Marion SHANAHAN - National Drug and Alcohol Research Centre, University of New South Wales

Emily LANCSAR and Marion HAAS - Centre for Health Economics Research and Evaluation

University of Technology, Sydney

Bronwyn LIND, Don WEATHERBURN and Shuling CHEN - Bureau of C ime Statistics rand Research, Sydney

Description

The study is a cost-effectiveness evaluation of an adult drug court programme as an alternative to jail for criminal offenders addicted to illicit drugs.

Compared with other legally imposed sanctions, drug court programmes are a form of coercive treatment. The treatment is usually implemented by a team led by a judge, and includes the defendant’s legal representative as well as representatives from the prosecution, probation, and drug-treatment services.

The goal of the NSW ADC was to decrease the level of criminal activity resulting from drug dependency, by diverting offenders into programmes designed to reduce or eliminate drug dependence via a combination of close supervision and therapeutic treatment.

Approach

Two measures of effectiveness were examined, both relating to offences committed during the follow-up period. These were the 'time to the first offence' and 'offending frequency per time unit ' .Both were calculated for each type of offence considered. Because the aim of the ADC is to reduce drug-related crime, theft and drug offences were the only types of offence examined in this study. The analysis provides a direct comparison of the ADC and control groups, assuming they have equal opportunities to offend.

Three sources of data were used to obtain information on effectiveness. The primary source, the ADC database, contained information on gender, date of birth, previous imprisonment, and prior conviction episodes. Time spent in custody was determined from data provided by the NSW Department of Corrective Services. Personal identifying details from the ADC database were matched with these records to provide dates of entry and exit from jail for every episode of imprisonment for the matched individuals. Data on offences committed during the follow-up period were drawn from the database of criminal matters dealt with by the NSW local court and maintained by the NSW Bureau of Crime Statistics and Research.

Calculation of cost-effectiveness

Eureval-C3E – 20/04/2006 – page 98


The measures of effectiveness were the time to the first offence and offending frequency per time unit, both based on the time spent out of custody in the follow-up period for each participant. Survival analysis techniques were used to analyse the time to the first offence because the data were censored (i.e. a person may not have committed an offence by the end of the follow-up period) and because the follow-up periods were of different durations for each person. Kaplan-Meier survival functions for the treated and control groups were compared for each type of offence. In each case a log-rank test was conducted to test the hypothesis of equality of the two survival functions.

The approach in the evaluation follows standard costing techniques used for economic evaluations: identify the activities for which cost will be assessed, identify the resources used in those activities, measure in physical units the volume of resources used and, finally, apply a standard unit cost to those resources to estimate a value of resource use.

A proportional hazards model was used to examine the time to the first offence, and a negative binomial regression model was used to examine offending frequency. Predictor variables were age, gender, prior convictions and prior imprisonment. In addition, two dichotomous indicator variables were included as predictors, one for the treatment effect (treated group vs. control group) and one for the termination effect (terminated vs. others).


The methodological approach of the cost-effectiveness analysis is well described and systematic. The study uses a randomly selected control group to assess outcomes.

The intervention can be considered a complex intervention in the sense that it involves several types of policy instrument.

The costing perspective that informs this study was limited to the measurement of costs that could be called treatment costs within the ADC programme and the control group. Thus, broader societal benefits that may flow from the ADC programme, such as reduced long-term demand for health and criminal justice services, reduced insurance claims, social security outlays, and pain and suffering from criminal victimisation, have not been included. Nor have potential societal costs such as public health costs that can arise because of imprisonment. Had it been possible to quantify these benefits and costs, the gap between the ADC and conventional sanctions in terms of cost effectiveness may have been larger.

Although the evaluation was conducted in line with the traditional steps of a cost-effectiveness analysis, the complexity of the programme and data limitations meant that it was not always possible to adhere strictly to the guidelines which are accepted as the gold standard (authors’ own comments).

Eureval-C3E – 20/04/2006 – page 99

Documents

Study on the Use of Cost-effectiveness Analysis in EC's Evaluations