3rd Interim Evaluation of the Education, Audiovisual, and Culture

1

May 2016

Educationand Culture

Final Report

3RD INTERIM EVALUATION OF THE EACEA

2

May 2016


Europe Direct is a service to help you find answers

to your questions about the European Union.

Freephone number (*):

00 800 6 7 8 9 10 11

(*) The information given is free, as are most calls (though some operators, phone boxes or hotels may charge you).

More information on the European Union is available on the internet (http://europa.eu). Luxembourg: Publications Office of the European Union, 2016 ISBN: 978-92-9492-212-0 doi: 10.2797/203152 European Union, 2016 Reproduction is authorised provided the source is acknowledged.

http://europa.eu.int/citizensrights/signpost/about/index_en.htm#note1#note1http://europa.eu/

3

May 2016


3rd Interim Evaluation of the

EACEA

Final Report

Prepared by: PPMI Group

LEGAL NOTICE

Neither the European Commission nor any person acting on behalf of the Commission is responsible for the use which might be made of the following information. The views expressed in this publication are the sole responsibility of the author and do not necessarily reflect the views of the European Commission.

EUROPEAN COMMISSION

Directorate-General for Education and Culture

4

May 2016


5

May 2016


Table of Contents

List of tables, figures and boxes ................................................................................ 6 List of acronyms .................................................................................................... 8

Introduction ............................................................................................................ 9 1. Background, approach and methodology .......................................................... 10

1.1. Background .................................................................................................. 10 1.1.1. Purpose and scope of the evaluation ......................................................... 10 1.1.2. Evaluation tasks and questions ................................................................. 10

1.2. An overall evaluation approach ....................................................................... 12 2. Overall performance of the Agency, quantitative and qualitative CBA ................... 14

2.1. Overall performance of the EACEA .................................................................... 14 2.1.1. The Agencys performance according to the main indicators .......................... 14 2.1.2. Quantitative benchmarking of the Commissions executive agencies .............. 17 2.1.3. Comparison between the administrative costs of the centralised an decentralised actions of Erasmus+ ......................................................................................... 19

2.2. The retrospective CBA ..................................................................................... 20 2.2.1. Background of the quantitative CBA ........................................................... 20 2.2.2. Actual staffing and costs of the EACEA ....................................................... 22 2.2.3. Cost-effectiveness of the executive agency scenario and actual savings due to

externalisation ....................................................................................................... 24 2.2.4. Workload analysis ................................................................................... 27 2.2.5. Qualitative aspects of the retrospective CBA ............................................... 29

3. Evaluation findings ........................................................................................ 32 3.1. Question 1 ..................................................................................................... 32 3.2. Question 2 ..................................................................................................... 39 3.3. Question 3 ..................................................................................................... 43 3.4. Question 4 ..................................................................................................... 67 3.5. Question 5 ..................................................................................................... 72 3.6. Question 6 ..................................................................................................... 76 3.7. Question 7 ..................................................................................................... 79

4. Conclusions and recommendations .................................................................. 89

6

May 2016


List of tables, figures and boxes

List of Tables

Table 1. The relationship between the evaluation tasks and the evaluation questions ...... 12 Table 2. Performance indicators of the EACEA, 2012-2014, EUR million or percentage .... 15 Table 3. Satisfaction with the EACEAs actual performance and the services it provides ... 17 Table 4. Main performance indicators of the Commissions executive agencies in 2014 .... 18 Table 5. EU commitments to the implementation of Erasmus+ in 2014 (EUR) and its

management cost (%) ............................................................................................ 19 Table 6. Budget managed and human resources in the EACEA compared to all executive

agencies in 2013 and 2020 ...................................................................................... 21 Table 7. The administrative budget of the EACEA in 2012-2014, EUR ............................ 24 Table 8. Estimated costs of the in-house (Commission) and the executive agency scenario,

EUR ...................................................................................................................... 26 Table 9. Qualitative items of the CBA, their correspondence to the evaluation questions and

good practices ........................................................................................................ 29 Table 10. Size, number of departments and number of units of the Commissions executive

agencies in 2014 .................................................................................................... 33 Table 11. Distribution of the EACEAs human resources between 2011 and 2014 ............ 35 Table 12. Staff Engagement Index in the EACEA and other Commission services, 2013-2014

............................................................................................................................ 39 Table 13. Evolution in the EACEAs workload indicators in 2009-2014 ........................... 48 Table 14. Satisfaction of applicants and beneficiaries with the overall quality and timeliness

of the EACEAs services ........................................................................................... 51 Table 15. Transparency of the evaluation and selection process ................................... 52 Table 16. Views on e-forms ..................................................................................... 56 Table 17. Satisfaction with the proportionality of administrative requirements and the related

electronic tools ....................................................................................................... 57 Table 18. Satisfaction of applicants and beneficiaries with the timeliness and quality of

support and guidance provided by the EACEA during various stages of the project life-cycle

............................................................................................................................ 62 Table 19. Satisfaction of beneficiaries with the monitoring activities carried out by the EACEA

............................................................................................................................ 66 Table 20. Findings of the previous evaluation in the field of unintended effects compared to

the current evaluation period ................................................................................... 76

List of Figures

Figure 1. Organisational model for the third evaluation of the EACEA ............................ 13 Figure 2. Framework of the Agency's performance ...................................................... 14 Figure 3. Estimated savings of the executive agency scenario in 2012-2014, EUR ........... 16 Figure 4. Number of staff estimated under different delegation scenarios ...................... 22 Figure 5. Evolution of the EACEAs staff during 2012-2014 .......................................... 23 Figure 6. Estimated savings of the executive agency scenario in 2012-2014, EUR ........... 27 Figure 7. Operational budget related to 2012-2014 calls, EUR million ............................ 28 Figure 8. Number of grants related to 2012-2014 calls ................................................ 28 Figure 9. The EACEA's evolution of Time-to-Award ...................................................... 44 Figure 10. The EACEA's evolution of Time-to-Contract ................................................. 45 Figure 11. The EACEA's evolution of TTP .................................................................... 46 Figure 12. Evolution of the EACEAs estimated residual error rate across the programmes

............................................................................................................................ 47 Figure 13. Execution of the operational budget of the EACEA ....................................... 50 Figure 14. Use of e-forms for submission of the proposals at the EACEA in 2009-2014 .... 55

7

May 2016


Figure 15. View on the already introduced simplifications ............................................ 58 Figure 16. Potential for future simplifications ............................................................. 59 Figure 17. Information sources on the programmes managed by the EACEA .................. 61 Figure 18. Usefulness of communication channels during the project implementation stage

............................................................................................................................ 63 Figure 19. Visibility of the EU in the programmes managed by the EACEA ..................... 65 Figure 20. Evolution in the number of the EACEA's project monitoring visits .................. 66 Figure 21. Number of officials seconded to the EACEA in the period 2012-2014 .............. 69 Figure 22. Number of coordination meetings that took place in the period 2012-2014 ..... 70 Figure 23. Extent to which the EACEAs staff are aware of the objectives of the Commission

and its parent DGs .................................................................................................. 72 Figure 24. User satisfaction with the overall quality of the Eurydice reports ................... 82 Figure 25. How much do you agree with the following statements? ............................... 83 Figure 26. To what extent do you agree with the following statements on the different

aspects of Eurypedia? ............................................................................................. 84 Figure 27. How often do you consult Eurydice reports? ................................................ 87 Figure 28. For what purposes do you use Eurydice reports? (You can choose more than one

option). ................................................................................................................. 87

List of Boxes

Box 1. Our assumptions for 2012-2013 and 2014 ....................................................... 25 Box 2. Instruments used to promote the Eurydice network products. ............................ 84

8

May 2016


List of acronyms

CAs Contracted Agents

CAF Common Assessment Framework

CBA Cost-Benefit Analysis

CHAFEA Consumers, Health and Food Executive Agency

DG Directorate-General

DG BUDG Directorate-General for the Budget

DG CNECT Directorate-General for Communications Networks, Content and Technology

DG COMM Directorate-General for Communication

DG EAC Directorate-General for Education and Culture

DG ECHO Directorate-General for Humanitarian Aid and Civil Protection

DG EMPL Directorate-General for Employment, Social Affairs and Inclusion

DG GROW Directorate-General for Internal Market, Industry, Entrepreneurship and SMEs

DG HOME Directorate-General for Migration and Home Affairs

DG RTD Directorate-General for Research and Innovation

DPO Data Protection Officer

EA Executive Agency

EACEA Education, Audiovisual and Culture Executive Agency

EASME Executive Agency for Small and Medium-sized enterprises

EC European Commission

EDF European Development Fund

ERCEA European Research Council Executive Agency

EU European Union

FTE Full-Time Equivalent

HR Human Resources

HRM Human resource management

H2020 Horizon 2020 programme

INEA Innovation and Networks Executive Agency

IT Information Technology

LLP Lifelong Learning Programme

OLAF European Anti-Fraud Office

PIC Participant Identification Code

PPMI Public Policy and Management Institute

REA Research Executive Agency

SFS Specific Financial Statement

TAs Temporary Agents

ToR Terms of Reference

TTA Time-to-Award

TTC Time-to-Contract

TTP Time-to-Pay

9

May 2016


Introduction

This report was prepared as an outcome of the 3rd interim evaluation of the EACEA (specific

contract No. EAC-2015-270, implementing Framework Contract No. EAC/22/2013-4),

carried out by the PPMI Group (Lithuania) in cooperation with the sub-contractor MATRIX

(United Kingdom). The Final Report was produced on the basis of requirements set out in

the Terms of Reference (henceforth - ToR), information gathered and analysed during the

project, as well as the minutes of project meetings with the Steering Group.

In line with requirements set out in the ToR, the Final Report provides the results of the

evaluation, its conclusions and recommendations. It is divided into the following parts:

Part 1: Background, approach and methodology;

Part 2: Overall performance of the Agency, quantitative and qualitative CBA;

Part 3: Evaluation findings;

Part 4: Conclusions and recommendations;

Annex 1: Terms of Reference;

Annex 2: Synthetic description of the changes (the results of Task 1);

Annex 3: Implementation of the actions in response to recommendations (the

results of Task 3);

Annex 4: Evaluation methodology;

Annex 5: List of interviewees;

Annex 6: Survey questionnaires and results;

Annex 7: Fiches on the selected examples of good practices;

Annex 8: Survey data/dataset and metadata;

Annex 9: Presentation summarising the contents of the report.

10

May 2016


1. Background, approach and methodology

1.1. Background

1.1.1. Purpose and scope of the evaluation

An external evaluation on the operation of each Agency has to be carried out every three

years and has to include a cost-benefit analysis (henceforth - CBA) according to Article 25

of Council Regulation (EC) No 58/2003 laying down the statute for Executive Agencies. The

CBA performed must assess the costs of coordination and checks, the impact on human

resources, possible savings within the general budgetary framework of the European Union,

efficiency and flexibility in the implementation of outsourced tasks, simplification of the

procedures used, proximity of outsourced activities to final beneficiaries, visibility of the

Community as promoter of the Community programme concerned and the need to maintain

an adequate level of know-how inside the Commission.1

The first two evaluations of the EACEAs operations covered the periods from January 2006

until April 2008 and from May 2008 until 31 December 2011 respectively. In addition, a

detailed CBA analysing four different scenarios for delegating the management of EU

programmes to the Executive Agencies (including the EACEA) was conducted in 2012-13 for

the 2014-2020 Multiannual Financial Framework.

The purpose of this assignment is to provide the third evaluation of the EACEAs

performance in the period January 2012-December 2014. This period includes the closing

phase of the 2007-2013 programmes and the starting phase of the 2014-2020 programmes

managed by the Agency. Although this evaluation assessed the EACEAs performance by the

cut-off date of 31 December 2014, the evaluator also took into account the main

developments that occurred in 2015 in formulating conclusions and recommendations.

The third evaluation of the EACEA is organised into four tasks:

1. providing a synthetic description of the changes during the period 2012-2014;

2. carrying out a retrospective quantitative and qualitative CBA;

3. assessing the implementation by the EACEA of the actions identified in its Action Plan

in response to the Agencys second evaluation;

4. drawing conclusions and providing recommendations for the future on the basis of

the evaluation questions.

Therefore, the evaluation combined a retrospective and prospective analysis:

a retrospective analysis concerned the changes during the period 2012-2014 (Task

1), CBA (Task 2), the implementation of the EACEAs Action Plan (Task 3) and the

evaluation questions of the ToR (Task 4) related to the Agencys past activities.

Where relevant, this analysis compared the situation at the end of the third

evaluation period (the end of 2014) and the situation at the end of the second

evaluation period (the end of 2011);

a prospective analysis provided recommendations for the Agencys performance in

the future based on the findings of the retrospective analysis and other evaluation

methods.

1 Article 3(1) of Council Regulation (EC) No 58/2003.

11

May 2016


1.1.2. Evaluation tasks and questions

Task 1 of the evaluation aimed to provide a synthetic description of the changes and

events that occurred during the evaluation period. We structured Task 1 according to the

organisational model presented in section 1.2. In order to synthesise these changes, we

mapped the main contextual and organisational events of the EACEAs operation, focusing

on the third evaluation period (2012-2014). Due to its descriptive nature, Task 1 did not

answer any specific questions, but it provided the necessary information and reference

points for other tasks.

Task 2 of the evaluation aimed to provide a retrospective CBA. According to Article 3(1)

of Council Regulation (EC) No 58/2003, the costs and benefits of the selected delegation

scenario as identified by the original CBA shall be tested again as part of the interim

evaluation and the results of the CBA shall be updated if need be. The CBA includes an

analysis of the quantitative aspects and the qualitative aspects. This task was related

to five of the evaluation questions in the ToR, namely Q.1, Q.2, Q.3, Q.4 and Q.5.

Task 3 consisted of an assessment of the EACEAs response (as part of its Action Plan

and other actions) to the recommendations of the second evaluation. The report made

10 recommendations. According to the ToR of this evaluation, the third evaluation of the

EACEA should give special attention to the following issues highlighted in the previous

evaluation: the implementation of the EACEAs internal mobility policy and its

effectiveness (especially that for A7 job profiles); the effectiveness and efficiency of the

EACEAs internal processes and procedures (especially monitoring, control and

reporting); the sharing of knowledge between the EACEA and policy/programme units of

DG EAC. We assessed the implementation of the Action Plan setting out the actions to

implement the recommendations of the previous evaluation and the execution of other

relevant actions.

The purpose of Task 4 was to draw conclusions and provide recommendations for the

future on the basis of the evaluation questions outlined in the ToR. Unlike other tasks, this

task provided both retrospective and prospective analysis on the basis of the gathered data.

We answered all the evaluation questions based on their operationalisation in terms of

indicators, target groups to be approached and the main data collection and analysis

methods to be applied, which was presented in the Inception Report. Conclusions of this

evaluation were organised according to the organisational model, they informed the

formulation of recommendations addressed to the EACEA and the Commission.

Table 1 outlines the links between the four tasks of the evaluation and the evaluation

questions provided in the ToR. While Task 4 (conclusions and recommendations) addressed

all the evaluation questions, other tasks of this evaluation were related to some specific

questions.

12

May 2016


Table 1. The relationship between the evaluation tasks and the evaluation

questions Evaluation

tasks Evaluation question

Task 2 Task 3 Task 4

Q.1: To what extent are the EACEA's organisation and internal structures appropriate to its new profile and to the needs it is intended to meet? What is the potential impact of the new HR policy of seconded officials, intended to maintain closer links between these officials and the parent DGs, on cooperation between the parent DGs and the Agency? To what extent has the EACEA retained and/or improved the competence and efficiency of its staff in relation to the implementation

of current and new tasks?

Task 3 Task 4

Q.2: To what extent is the EACEA operating according to the legal framework establishing it, and to what extent is there an expectations gap with regard to policy responsiveness in this context (flexibility in the implementation of outsourced tasks)?

Task 2 Task 3 Task 4

Q.3: To what extent has the EACEA led to effective management of the programmes and better services to the stakeholders and what concrete improvements can be suggested (suggestions for improvement should include an assessment of the cost-benefit ratio)?

Task 3 Task 4

Q.4: To what extent is the supervision strategy and coordination of the EACEA and the Commission services (including the parent DGs, relevant horizontal services and offices) working satisfactorily? What evidence is there for an expectations gap and how can it be eliminated in the future? Additional aspect of Q.4: To what extent are the EACEAs strategy and planning aligned with its operations and indicators?

Task 3 Task 4

Q.5: To what extent has the Commission, in the presence of the EACEA, been able to maintain an adequate level of know-how in relation to the programmes entrusted to the EACEA, while focusing on its policy role? How was this achieved while avoiding overlaps?

Task 4 Q.6: To what extent have the activities of the EACEA resulted in unintended effects (both desirable and undesirable)?

Task 3 Task 4

Q.7: Evaluate the role, performance and the output of the A7 unit in the EACEA in its provision of support to EAC's policy activities in education and youth as well as to the wider education community (national policy-makers, researchers, general public

etc.).

1.2. An overall evaluation approach The third evaluation of the EACEA was based on the holistic approach to organisational

performance analysis, identifying all important elements of organisational activities and

general relationships among them. In order to organise a diagnostic and prescriptive

analysis of the Agencys activities, we selected the CAF (Common Assessment Framework)

model for developing an organisational model tailored to the third evaluation of the EACEA.

The CAF 2013 model is divided into two main sets of organisational factors: (1) enablers

dealing with the managerial practices of an organisation and (2) various types of

organisational results. We adapted this model to fit the specificity of the executive agencies

and the ToR for the third evaluation of the EACEA. First, we introduced a new set of

organisational factors (variables) related to an external regulatory framework. Second,

we made a few modifications to the enablers of the CAF model by splitting resources from

partnerships of the Agency and by integrating leadership into the factor of people. Third,

social responsibility results were replaced by programme and policy results, which are more

relevant to the ToR of this assignment. Fourth, we divided all organisational results into

intended and unintended results in accordance with the ToR.

As a result, our model for the evaluation of the EACEA includes 12 organisational factors

(variables) grouped into the three sets: (i) regulatory and operational framework, (ii)

13

May 2016


enablers and (iii) (intended and unintended) results (see Figure 1 below). The executive

agencies have a clear legal mandate, certain autonomy and are controlled by their parent

DGs of the Commission. The set of organisational factors related to the regulatory and

operational framework of the Agency includes the following items: (1) the mandate and

responsibilities of the Agency, (2) its external governance (autonomy and

control/supervision by the parent DGs), as well as (3) its organisational structure. Figure 1. Organisational model for the third evaluation of the EACEA

Source: adapted from the CAF 2013 model by PPMI.

The so-called enablers determine what the organisation does and how it executes its tasks

to achieve the desired results within the existing regulatory framework and structure.

Organisational factors 4-8 include the following items: (4) strategy and planning, (5) people,

(6) other resources, (7) partnerships and (8) processes of the Agency. Finally, following the

organisational model the agency results are broken down into the following types of

(intended and unintended) results: (9) key performance results, (10) customer-oriented

results, (11) people results and (12) programme and policy results.

The evaluation relied on the evidence gathered and analysed using qualitative and

quantitative methods:

- desk research, including literature review, analysis of the monitoring data and

comparative analysis;

- 35 interviews with EU officials, staff of the EACEA and other executive agencies, as

well as several of the Agencys beneficiaries;

- surveys of the Agencys applicants and beneficiaries with a total response rate of

23.5 %;

- a series of 15 good practice examples allowing for benchmarking of the executive

agencies activities (their fiches are presented in Annex 7 to the report);

- retrospective cost-benefit analysis (involving both quantitative and qualitative

analysis);

- methods of retrospective analysis.

Each of the evaluation methods is described in Annex 4 to the evaluation report. The

collected qualitative and quantitative data were combined in order to answer the evaluation

questions.

14

May 2016


2. Overall performance of the Agency, quantitative and qualitative CBA

2.1. Overall performance of the EACEA

2.1.1. The Agencys performance according to the main indicators

In order to evaluate the overall performance of the Agency, we introduced a conceptual

framework which links the main elements of performance (objectives, inputs, processes,

outputs and outcomes) in a linear way. This framework allowed defining the main indicators

against which the Agencys performance can be measured.

Figure 2. Framework of the Agency's performance

Source: PPMI.

First, budget efficiency is defined as the ratio between inputs (staff) and outputs (the

budget managed by the Agency). Our performance criteria measuring budget efficiency are

budget per head (million EUR) and the ratio between the administrative and operational

budget (%). Our evaluation results indicate the ratio of budget per head decreased from

EUR 1.70 million in 2012 to EUR 1.38 million in 2014 (from EUR 1.81 million to EUR 1.47

million if the EACEAs staff in A7 unit, not managing operational appropriations, are

excluded), which means that a single employee of the Agency managed a decreasing volume

of the operational budget during this period. This is related mostly to the decreasing volume

of the operational budget (especially for 2014, which was the first year of the new

programming period). The Communication to the Commission SEC(2013)493 on the

delegation of the management of the 2014-2020 programmes to the executive agencies

provided for an estimated ratio of EUR 1.97 million for 2013, but the calculation of this ratio

did not include the additional CAs financed from the third countries and EDF contributions

(see section 2.2. of the report).

Objectives Inputs Process Outputs Outcomes

Process efficiency

Budget efficiency

Cost-effectiveness

Effectiveness

Value for money

15

May 2016


Table 2. Performance indicators of the EACEA, 2012-2014, EUR million or

percentage 2012 2013 2014

Operational budget (including third countries and EDF contributions) (according to executed commitment

appropriations)

A 728 686 602

Administrative budget B 49.33 49.82 46.13

Actual number of staff at the EACEA (at the end of the year) (including CAs financed from third countries and EDF credits)

C 428 435 437

Actual number of staff at the EACEA dealing with Eurydice D 27 27 27

Programme management cost at the EACEA: ratio between

the administrative and operational budget

E=B/A 6.78 % 7.26 % 7.66 %

Budget 'per head' excluding the A7 staff F=A/(C-

D)

1.81 1.68 1.47

Source: PPMI based on the EACEAs administrative data.

Note: during the reference period there were two staff members in A7 who dealt with youth issues.

In terms of administrative efficiency (the ratio between the administrative and

operational budget), the EACEAs cost of programme management increased from about

6.8 % in 2012 to about 7.7 % in 2014 due to the somewhat lower operational budget. This

is related largely to the lower volume of the operational budget in 2013 and 2014. Taking

into account the specificity of the Agency, its operations were moderately efficient during

our reference period. However, there is a need to monitor the indicators of administrative

efficiency closely and to discuss possible actions as to how administrative efficiency could

be enhanced in the future (including a mid-term review of the programmes managed by the

Agency).

Second, process efficiency is the extent to which programme management is simplified

and optimised. The EACEA implemented a number of efficiency-enhancing measures during

the reference period, including further implementation of simplification measures,

optimisation of the business processes, harmonisation of project management requirements

across the programmes, development of IT tools, reorganisation of the organisational

structure (see part 3 of the report and its annex 3). According to the survey results, the

EACEAs applicants and beneficiaries assessed the already introduced simplifications

positively and expressed their appreciation for further simplifications. The development of

more efficient processes at the Agency will continue in 2015 as one of the priorities in its

2015 Work Programme was related to further improving programme management.

Third, cost-effectiveness is the extent to which the Agency achieved its outcomes at a

lower cost compared with in-house management by the Commission. We measured it

according to cost savings relative to the costs of in-house management by the Commission

(million EUR). The CBA exercise found that the actual savings during the 2012-2014 period

were EUR 41.8 million (22 %) compared to the in-house scenario (see section 2.2. below).

This analysis indicates that the executive agency scenario remains more cost-effective than

the in-house scenario due lower staffing costs and a lower number of staff (without

additional staff financed from the third countries and EDF contributions).

16

May 2016


Figure 3. Estimated savings of the executive agency scenario in 2012-2014, EUR

Source: PPMI based on the EACEAs data.

Fourth, effectiveness is the extent to which the Agencys objectives were achieved. At the

level of outputs, we assessed it according to the execution of KPIs based on the

administrative data (see question 3 in part 3 of the report below). Our analysis indicated

that despite the transition to the new programming period, the EACEA satisfactorily achieved

almost all of its targets during the evaluation period, with less positive results for the targets

of error rates mostly due to the nature of the programmes delegated to the Agency (small

grants awarded to small organisations with limited administrative and accounting

capacities). The EACEA did not measure any indicators related to the quality of the

evaluation and selection procedures (e.g. redress indicators used by the executive agencies

belonging to the Research and Innovation family of the Commission). The survey data

confirmed that the Agency improved in terms of timely execution of its procedures compared

to the previous evaluation period or in terms of improvement over time during the reference

period (see question 3 in Part 3 of the report below).

At the level of outcomes, we measured effectiveness according to the satisfaction of

applicants and beneficiaries with the quality of services based on our survey data (the

specific question We are satisfied with the overall quality of services provided by the

EACEA). A total of 79.3 % of the Agencys applicants and beneficiaries agreed with this

statement (70.1 % of unsuccessful applicants and 85.6 % of beneficiaries). The respondents

who applied for/participated in Media (85.4 %), Youth in Action (84.3 %) and Lifelong

Learning (82.8 %) programmes were the most satisfied with the overall quality of the

EACEAs services. In contrast, those from the Erasmus+ Charter for Higher Education

(75.5 %), Culture programme (74.7 %) or Bilateral agreements and cooperation initiatives

with industrialised countries (57.1 %; N=14) were least positive about the overall

performance of the EACEA. The average satisfaction score was 2.5 for beneficiaries and 3.4

for unsuccessful applicants with the average of 2.85 (on a scale of 1 to 9, with one being

the most positive rating possible).

The overall satisfaction of the Agencys beneficiaries was quite stable during the evaluation

period, but there was a small increase in the overall satisfaction in 2014. While the

satisfaction score of the 2012 and 2013 beneficiaries was 2.37 and 2.38 respectively, that

of the 2014 beneficiaries was 2.33. While the overall satisfaction level among the

beneficiaries and applicants remained similar compared to the previous evaluation periods,

our survey attracted a higher share of unsuccessful applicants (40 % compared to around

30 %) who tend to be less satisfied with the services of the executive agencies.

The EACEAs applicants and beneficiaries assessed the courtesy, commitment and

competence of the Agencys personnel most positively. For instance, one respondent to the

survey of the Agencys applicants and beneficiaries noted that the quality and

professionalism of the EACEA's staff was always of a very high standard. However, a more

tangible aspect of the Agency performance pertaining to the EACEAs website and

information materials was assessed least positively. The respondents also provided a

favourable opinion on the EACEAs willingness to help them and provide individualised

189 223 896147 449 107

41 774 789

0

100 000 000

200 000 000

In-house scenario Executive agency scenario Savings of EA scenario

17

May 2016


attention. Despite a good assessment of the accessibility and clarity of information provided

by the EACEA and its ability to perform the service promptly, accurately and transparently,

some aspects of the service delivery (the information provided by the EACEA is easily

accessible and the EACEAs documents are error free) were perceived in a less positive

way (see Table 3 below). Overall, this points to the possibility of improving information

provision and communications with clients in order to improve the overall satisfaction with

the services the Agency provides.

Table 3. Satisfaction with the EACEAs actual performance and the services it

provides Specific aspects of satisfaction Index

Employees in the EACEA are consistently courteous to you 1.97

The EACEAs employees are committed to doing quality work 2.11

Employees in the EACEA are knowledgeable and competent 2.15

The EACEA strives to provide excellent programme management and high-quality service

2.29

When you have a problem, the EACEA shows a sincere interest in solving it 2.33

Employees in the EACEA are always supportive in the interest of the project 2.34

The EACEA has employees who give you personal attention 2.44

Employees in the EACEA give you prompt service 2.45

Employees in the EACEA closely cooperate with you 2.52

The events (information days, project meetings, information visits, etc.) organised by the EACEA are useful

2.55

The EACEAs communications provide relevant and useful information 2.57

The EACEAs procedures are transparent and objective 2.68

The information provided by the EACEA is easily accessible 2.82

The EACEAs documents are error free 2.89

The EACEAs website and information materials are visually appealing and user-

friendly

3.06

Average 2.48

Source: the survey of the EACEAs applicants and beneficiaries.

Fifth, value for money is the optimal use of human and financial resources to achieve the

intended outcomes. Our analysis indicated that the EACEA provided good value for money

due to a combination of the following:

the Agency effectively executed the delegated tasks and achieved almost all of its

KPIs;

the agency scenario generated actual savings of EUR 41.8 million during the 2012-

2014 period according to our retrospective CBA; and

the overall satisfaction with the quality of services among its applicants and

beneficiaries was quite high (79.3 % (70.1 % of unsuccessful applicants and 85.6 %

of beneficiaries)or the average score of 2.85), the survey respondents provided a

positive assessment of the simplifications introduced and other value-adding

novelties in project management (such as kick-off meetings or support to

dissemination); there was also a small improvement in the satisfaction of the EACEAs

applicants and beneficiaries with the specific aspects of project management

compared to the previous evaluation period.

2.1.2. Quantitative benchmarking of the Commissions executive agencies

Overall, the EACEA faces challenges in reaching the budgetary efficiency ratios achieved by

other executive agencies established by the Commission. Its programme management cost

was 7.66 % and the budget per head stood at EUR 1.38 million (EUR 1.47 million without

the staff dealing with Eurydice) in 2014. The Agencys performance was more comparable

to that of CHAFEA according to the same indicators in the same year (8.97 % and EUR 1.61

18

May 2016


million respectively). In contrast, the performance of the executive agencies belonging to

the Research and Innovation family was more efficient. For instance, the REAs cost of

programme management was 3.3 % (2.1 % if the share of costs of the central

administrative and logistical support services was excluded) and its budget per head was

EUR 2.83 million (EUR 3.46 million if these services are excluded) on the same year2.

Table 4. Main performance indicators of the Commissions executive agencies in

2014 The executive agencies of the Research

and Innovation family*

Executive agency EACEA CHAFEA ERCEA REA INEA EASME

Parent DGs (other parent DGs)

DG EAC (ECHO, HOME,

CNECT)

DG SANCO (JUST, AGRI)

DG RTD DG RTD (EAC,

GROW,

CNECT, AGRI, HOME)

DG MOVE (ENER, CNECT,

RTD)

DG GROW (RTD,

ENV, CLIMA, ENER, CNECT,

MARE)

Staff in 2014 (actually filled)

437 48 388 548 (449**)

151 283

Operational budget (million EUR), commitments

602.36 77.18 1 726 1 554 4 621.18 1 291.64

Total administrative

budget (million EUR )

46.13 6.92 36.04 51.13 13.43 24.48

Proposals received in 2014

10 058 193 8 545 11 007 1 104 8 988

Total running projects (at the end of 2014)

3 664 239 4 036 6 925 516 n.a.

Programme

management cost (ratio between the administrative and operational budget)

7.66 % 8.97 % 2.09 % 3.3 %

(2.1 %**)

0.29 % 1.90 %

Budget 'per head' (million EUR )

1.38 (1.47**

*)

1.61 4.45 2.83 (3.46**)

30.69 4.65

No. of proposals 'per head'

23.0 (24.5**

*)

4.0 22.0 20.1 (24.5**)

7.3 31.8

No. of running

projects (at the end of 2014) 'per head'

8.4

(8.9***)

5.0 10.4 12.6

(15.4**)

3.4 n.a.

Source: 2014 Activity Reports of the executive agencies, DG RTD information for the Research and Innovation family. Notes: *INEA and EASME implement other programmes besides H2020; **without central support services; ***excluding the A7 staff.

2 PPMI, Evaluation of the operation of the REA, Final Report, 2016.

19

May 2016


This difference between the EACEA and the executive agencies of the Research and

Innovation family was associated mainly with the complexity and diversity of the EACEAs

programme portfolio, the heterogeneity of its applicants and beneficiaries, the quite high

volume of applications and grants, as well as their small average size (as low as

EUR 150,000)3. In terms of number of proposals per head, the EACEA would be among the

most efficient executive agencies, while in terms of running projects per head, the Agency

would be moderately efficient compared to other executive agencies (see Table 4 above on

the main performance indicators of the executive agencies in 2014).

2.1.3. Comparison between the administrative costs of the centralised and

decentralised actions of Erasmus+

We also sought to compare the administrative costs of the centralised actions of Erasmus+

executed by the EACEA and the decentralised actions of this programme carried out by the

National Agencies. The operational grants allocated to the National Agencies in the period

2007-2008 constituted 4.3 % of the decentralised funds. Exact data on national co-financing

and other sources of funding for the operation of the National Agencies were not available,

but our survey results revealed that these sources of funding on average constituted 40-

50 % of the total budget allocated to the LLP management. Therefore, it was estimated that

the total management cost of the LLP decentralised actions in the period 2007-2008

constituted around 7-8 % of the decentralised funds4.

According to the Erasmus+ 2014 report, the management cost of the centralised actions of

Erasmus+ executed by the EACEA was calculated to be 5.94 %5 in 2014 (see Table 5 below).

The EU management cost of the decentralised actions carried out by the National Agencies

was 4.12 %, but in line with the Erasmus+ regulation, the National Authorities provide

additional co-financing for the operations of their National Agencies. Exact data on national

co-financing to the operation of these agencies are not available, but if these agencies

received 40-50 % of the total administrative budget, their total cost would rise to about 7-

8 % of the decentralised funds. Although it is not possible to accurately estimate the total

management cost of the National Agencies, the EACEAs operations in the management of

the Erasmus+ actions in the 2014-2020 period are likely to be as efficient or even more

efficient compared to those of the National Agencies.

Table 5. EU commitments to the implementation of Erasmus+ in 2014 (EUR) and

its management cost (%)

Direct management

mode (EACEA)

Indirect management mode (National

Agencies)

Heading 1 260 506 438 1 396 549 350

Management fees (National Agencies) 64 565 236

Administrative expenditure 25 332 870

International Cooperation (heading 4) 140 702 760 106 174 865

Total commitments: 426 542 068 1 567 289 451

Management cost (% of the total EU

budget commitments)

5.94 % 4.12 %

Source: Annex I (Statistical Annex) to the Erasmus+ Programme Annual Report 2014, p. 5.

3 In sports the average grant size was only somewhat more than EUR 400,000 in 2014. 4 PPMI, the Interim Evaluation of the LLP (2007-2013), the Final Report, 18 February 2011, p. 129-130. 5 This can be compared to the cost of managing all the programmes delegated to the EACEA, which stood at 7.66 % in 2014.

20

May 2016


2.2. The retrospective CBA

CBA includes both quantitative and qualitative aspects of the EACEAs performance.

2.2.1. Background of the quantitative CBA

One of the key reasons for establishing executive agencies is their potential to provide a

cost-effective solution for management of the delegated programmes. Therefore, the main

objective of the quantitative retrospective CBA was to ex-post review the validity of the

reasoning and financial calculations leading to the decision to allocate management of the

specific programmes to the EACEA and to provide insights into the EACEAs overall costs

and cost-efficiency between 2012 and 2014.

For the assessment of the quantitative (financial) aspects of the retrospective CBA, we:

i. reviewed the CBA reports and other (ex-ante) working documents (such as legislative

financial statements and specific financial statements) supporting the decision to

allocate various programmes to the Agency;

ii. screened the different scenarios (comparators) foreseen in the ex-ante assessment

(namely the in-house (Commission) and the executive agency scenarios) and the

underlying assumptions, such as the EACEAs staffing estimates and composition of

staff (TAs and CAs), average staff costs and other costs (overheads and programme

support expenditure);

iii. reviewed the estimated figures used in the ex-ante assessments taking into account

the actual data;

iv. verified whether the conclusions of the ex-ante assessments are still valid when

compared to the actual situation and what the overall possible savings are:

do the actual costs (including cost of coordination and monitoring) of the EACEA

correspond to the estimates of the respective CBAs and the legal financial

statements/specific financial statements, which were prepared in view of its

establishment (prolongation of timeframe or extension of tasks)?

is the management and execution of the programme(s) by the EACEA cost-

effective compared to the alternative options (namely the respective in-house

(Commission) scenario)?

has the establishment of the EACEA resulted in savings to the EU budget as

compared to the alternative options?

As with the previous analyses, the retrospective CBA (quantitative aspects) focused on the

comparison of financial aspects and cost-efficiency, while effectiveness indicators (such as

time to select, award and pay and client satisfaction) were analysed alongside costs but not

valued in monetary terms (such analysis is presented under question 3).

The current interim evaluation of the EACEA covers the period from 2012 to 2014, which is

related to several establishment acts and the respective financial statements against which

the actual situation was tested. The most important documents for the current retrospective

CBA are listed below:

legislative financial statement related to Commission Decision 2009/336/EC of 20

April 2009 setting up the Education, Audiovisual and Culture Executive Agency for

the management of Community action in the fields of education, audiovisual and

culture in application of Council Regulation (EC) No 58/2003 (2009/336/EC),

repealing Decision 2005/56/EC, which provides the main basis for the retrospective

CBA for 2012;

financial statement related to Commission Implementing Decision 2012/797/EU of

18 December 2012 amending Commission Decision No 2009/336/EC setting up the

21

May 2016


Education, Audiovisual and Culture Executive Agency for the management of

Community action in the fields of education, audiovisual and culture in application of

Council Regulation (EC) No 58/2003, which provides the main basis for the

retrospective CBA for 2013;

specific financial statement related to Commission Implementing Decision

2013/776/EU of 18 December 2013 establishing the Education, Audiovisual and

Culture Executive Agency and repealing Decision 2009/336/EC, which provides the

main basis for the retrospective CBA for 2014.

The current mandate of the EACEA is related to the detailed Cost Benefit Analysis for the

delegation of certain tasks regarding the implementation of the Union Programmes 2014-

2020 to the executive agencies6, which was conducted in 2013 for the 2014-2020

Multiannual Financial Framework. The CBA compared the following four scenarios based on

varying levels of delegation and distribution of programmes between different executive

agencies:

the in-house scenario assuming that the new programmes would be managed by

the Commission while the executive agencies would remain responsible for the

delivery of legacy work (20072013 MFF programmes);

the initial scenario for delegation defined by the Commission;

two alternative scenarios with different options for delegation among executive

agencies (neither of these scenarios concern the EACEA).

It was concluded that alternative scenario 2 was the most efficient in terms of cost savings

and qualitative benefits. It estimated that in order to manage EUR 13,267 million (an

increase of 127 % compared to 2013), the six agencies will need 2,887 FTEs in 2020 (an

increase of 71 %, i.e. additional 1200 FTEs compared to 2013). This compares favourably

to the in-house scenario which would require 3,088 FTEs to manage the same programmes.

The executive agencies will benefit from economies of scale as they become larger.

In order to achieve further efficiency gains, the Commission proposed a few adjustments

stemming from an improved level of productivity and aimed at containing administrative

costs through a 5 % staff reduction. This corrected the ratio of budget per head in the case

of the EACEA from EUR 1.58 million to EUR 1.97 million in 2020, making it similar to the

2013 situation.

Table 6. Budget managed and human resources in the EACEA compared to all

executive agencies in 2013 and 2020

Executive Agency

Budget managed in 2013 (million EUR )

FTEs in

2013

Budget per head

2013 (million EUR )

Budget to be

managed by EA in

2020 (million EUR )

Envisioned

FTEs in 2020

Envisioned

FTEs in 2020

(adjust

ed)

Budget per

head in 2020

(millio

n EUR )

Budget per head in 2020 (million EUR )

(adjusted)

EACEA 847 431 1.97 870 552 442 1.58 1.97

Total 5 846 1 687 3.47 13 267 2 887 2 630 4.60 5.05

Source: Communication to the Commission on the delegation of the management of the 2014-2020 programmes to executive agencies (SEC(2013)493) adapted by PPMI.

6 ICF GHK in association with Technopolis, Cost Benefit Analysis for the delegation of certain tasks

regarding the implementation of Union Programmes 2014-2020 to the Executive Agencies, Final Report, 2013.

22

May 2016


These adjusted results of the CBA were used in the Specific Financial Statement (SFS)7. With

regard to the forecasts for the administrative budget, the SFS differs from the CBA in the

following aspects:

As explained above, the number of staff at the EACEA was reduced by 14 % on

average under the executive agency scenario. The in-house scenario was not

adjusted).

The average staff-related costs were increased in the SFS by 13 % on average, while

overheads were reduced by 19 % on average. The average overheads fluctuated

within the range of 10 % over the period.

The costs in the CBA were calculated in constant 2013 prices (i.e. neutralising the

effect of inflation). However, in the SFS these estimations were used as current prices

without any further indexation. In real terms, this constituted another reduction of

the administrative budget. This could have a negative impact in the final years of the

2014-2020 period, as actual average staff costs might rise due to salary indexation,

promotions and seniority, while actual overheads might rise due to inflation.

Figure 4. Number of staff estimated under different delegation scenarios

Sources: CBA, SFS.

2.2.2. Actual staffing and costs of the EACEA

During the 2012-2014 period the overall number of the EACEAs staff increased moderately

from 428 to 437 (see Figure 5). The actual staffing level was higher than the estimations

provided in the relevant financial statements, but this was related to the hiring of additional

CAs (22 CAs in 2012-2013 and 20 CAs in 2014) financed from the third countries and EDF

contributions in order to manage the additional operational budget coming from the

respective sources. Eliminating that, the actual staffing level of the Agency was below the

initial estimations of the respective financial statements based on EU funding sources. In

line with the initial estimations, CAs constituted the majority of the EACEAs staff (about

77 % compared to the initial estimations of 75 %)8.

In 2012-2013, the administrative budget of the EACEA was structured under two titles: Title

I Staff and Title II Buildings, Equipment and Miscellaneous Operating Expenditure. Since

7 Specific financial statement related to Commission Implementing Decision 2013/776/EU of 18 December 2013 establishing the Education, Audiovisual and Culture Executive Agency and repealing Decision 2009/336/EC, which provides the main basis for the retrospective CBA for 2014. 8 A higher share of CAs primarily reflected the hiring of additional CAs financed from the third country and EDF contributions.

523 532537 537

550 557

592

499 499 501 501513 520

552

436 441 441 441 441 441 442

300

350

400

450

500

550

600

650

2014 2015 2016 2017 2018 2019 2020

In-house Executive agency - CBA Executive agency - SFS

23

May 2016


2014, the structure of the EACEAs administrative budget has been aligned with the model

agreed by the six executive agencies and now comprises three titles: Title I Staff, Title II

Buildings, Equipment and Miscellaneous Operating Expenditure and Title III Programme

Support Expenditure. In order to provide a meaningful comparison, we used the actually

implemented budget figures for 2012-2014, which are presented according to the 2014

budget structure. Such budget structure also corresponds to the budget estimations in the

respective financial statements indicated above.

Figure 5. Evolution of the EACEAs staff during 2012-2014


Our analysis shows that in 2012-2014 the actual administrative budget implemented by the

EACEA (see Table 7 for more details) amounted to EUR 145 million and was some 3.6 %

lower than the administrative budget estimations in the respective financial statements

(EUR 151 million). It is important to note that the estimations in the respective ex ante CBAs

and financial statements were based on the EU contribution only and the EACEAs

administrative budget during the 2012-2014 period also included third country and EDF

contributions (about 6 % of the EACEAs administrative budget during our reference period)

to manage the additional operational budget coming from the respective sources. Therefore,

in 2012-2014 the EACEAs administrative budget based on the EU contribution was EUR 14.6

million (9.7 %) lower than the administrative budget forecast in the relative financial

statements.

Staff related expenditure and infrastructure and operating expenditure were close to the

estimations of the relevant financial statements. Therefore, the lower actual costs of the

Agency primarily relate to programme support expenditure, which was 22 % lower than

estimated.

The actual average staff costs for CAs were in line with the initial estimations, the average

staff costs for TAs in 2014 were higher than the CBA estimations but in line with the average

costs used in the specific financial statements. The average overhead costs in 2014 were

some 8.5 % lower than the initial estimations in the specific financial statement. However,

these costs are likely to increase once the EACEA moves to new premises.

407 431 436428 435 437406 413 415

0

100

200

300

400

500

2012 2013 2014

Financial statements

Actual

Actual without CAs financed from EEA/EFTA, 3rd country and EDF contributions

24

May 2016


Table 7. The administrative budget of the EACEA in 2012-2014, EUR Administrative budget

2012 2013 2014 2012-2014

Title I. Staff

Related Expenditure

Financial

statements

28 431 000 29 963 000 30 925 000 89 319 000

Actual 29 029 693 29 755 699 31 157 442 89 942 834

Title II. Infrastructure and

Operating Expenditure


10 414 000 11 065 000 10 090 000 31 569 000

Actual 11 070 973 11 652 568 9 252 271 31 975 812

Title III. Programme Support

Expenditure


11 288.000 11 994 000 6 527 000 29 809 000

Actual 9 224 580 8 410 756 5 717 725 23 353 061

Total Financial statements

50 133 000 53 022 000 47 542 000 150 697,000

Actual 49 325 246 49 819 023 46 127 438 145 271 707

Difference (Actual - Financial statements)

-807 754 -3 202 977 -1 414 562 -5 425 293

Administrative budget related

to EEA/EFTA, 3rd country and EDF contributions*

3 132 976 3 260 698 2 823 715 9 217 389

Difference (Actual - Administrative budget related to 3rd country and

EDF contributions Financial statements)

-3 940 730 -6 463 675 -4 238 277 -14 642 682

Source: PPMI based on financial statements, final accounts, other EACEA data. Note: Budgeted estimations.

In 2014, the overall administrative budget of the EACEA decreased by 7.4 % compared to

2013, which is primarily related to the costs of independent experts (around EUR 5-6 million annually during the 2012-2014 period). Prior to 2014, these costs were charged to the

administrative budget of the EACEA, whereas they are added to the operational budget of

the respective programmes from 2014 (as in the case of other executive agencies).

2.2.3. Cost-effectiveness of the executive agency scenario and actual savings due

to externalisation

In order to assess whether the conclusions of the ex-ante assessment (the executive

agency scenario saving estimations provided in the financial statements and ex-ante CBAs)

are still valid when compared to the actual situation and what the overall possible savings

are:

1) We draw upon the actual performance of the EACEA (actual execution of the

administrative budget, actual staffing, etc.).

2) In order to ensure comparability and validity of results, we follow the assumptions

laid down in the respective CBAs and financial statements and provide estimations of

the comparable actual in-house (Commission) scenario (comparator), which would

best reflect the actual situation.

3) Based on these estimations, we assess whether the conclusions of the ex-ante

assessment are still valid when compared to the actual situation and what the overall

possible savings are.

25

May 2016


In order to deconstruct the actual in-house (Commission) scenario (comparator), we base

our estimations on the assumptions (see Box 1 below) provided for in the financial

statements and the corresponding CBAs. For the 2012-2013 period, in order to ensure

continuity and comparability, we also take account of the practices used in the previous

retrospective CBAs9.

Box 1. Our assumptions for 2012-2013 and 2014 The assumptions for 2012-2013:

Number of staff in the Commission under the in-house scenario equals the number of staff in the EACEA under the executive agency scenario.

Staff under the Commission scenario is composed of permanent officials. The Commission staff costs and overheads correspond to DG BUDG estimations used for

2012-2013: the average cost of establishment plan posts were EUR 127 000 for 2012 and EUR 131 000 for 2013 (including overheads).

The programme support expenditure (Title III) stays the same under the Commission scenario and the executive agency scenario.

The assumptions for 2014:

Number and composition of staff in the Commission and the EACEA under the in-house

scenario correspond to the CBA and the financial statement estimations10. Also, an additional 22 CAs were added to the estimated Commission staff number in 2014 in order to reflect additional staff at the EACEA financed from the contributions of third countries and EDF in order to manage the additional operational budget which was not covered in the CBA/SFS resources calculations.

The Commission staff costs and overheads correspond to DG BUDG estimations used for

2014: establishment plan posts EUR 109 000, external personnel EUR 47 000, overheads EUR 23 000.

The EACEAs average staff costs and overheads under the in-house scenario (for legacy) correspond to the actual average staff costs of the EACEA and overheads in 2014.

The programme support expenditure (Title III) stays the same under the Commission

scenario and the executive agency scenario.

Table 8 below presents the results of analysis of the estimated actual costs of the in-house

(Commission) and the executive agency scenarios.

9 COWI, Interim Evaluation of the EACEA, 2009; Ecorys, Second Interim Evaluation of the Education, Audiovisual and Culture Executive Agency (EACEA), 2013. 10 Two different approaches to deconstruct No. of staff in the actual in-house scenario for 2014 were analysed during the evaluation. The first approach was based on the number of staff indicated in the

SFS for the in-house scenario. Under the second approach, the number of staff of the in-house scenario would be calculated taking into account the actual number of staff in the EACEA and following the assumptions used in the CBA. Although the second approach might lead to more realistic staffing estimations of the in-house scenario, corresponding costs and savings estimations would be incomparable with the estimations provided for in the SFS (the SFS adjusted staffing in the externalisation scenario, however staffing of the in-house scenario was not adjusted. This provided for

a bigger staffing gap between the externalisation and the in-house scenarios than the original CBA. Therefore, it was decided to follow the first approach.

26

May 2016


Table 8. Estimated costs of the in-house (Commission) and the executive agency

scenario, EUR

2012 2013 2014 Total 2012-

2014 No. Cost No. Cost No. Cost

In-house scenario

Commission

I. Staff related expenditure

428 54 356 000 435 56 985 000 315.7 27 585 100 138 926 100

Establishment plan posts (TA)

428 54 356 000 435 56 985 000 205.6 22 410 400 133 751 400

External personnel (CA)

0 0 0 110.1 5 174 700 5 174 700

II. Infrastructure and operating expenditure

7 261 100 7 261 100

Total Commission

cost:

428 54 356 000 435 56 985 000 315.7 34 846 200 146 187 200

EACEA

I. Staff related expenditure

- - - - 229.4 14 826 722 14 826 722


57.3 6 279 666 6 279 666

External

personnel (CA)

172.1 8 547 057 8 547 057


- - - - 4 856 913 4 856 913

Total EACEA cost:

19 683 635 19 683 635

III. Programme support expenditure

9 224 580 8 410 756 5 717 725 23 353 061

TOTAL COST 428 63 580 580 435 65 395 756 545.1 60 247 560 189 223 896

Externalisation scenario

EACEA

I. Staff related

expenditure

428 29 029 693 435 29 755 699 437 31 157 442 89 942 834


99 10 411 790 99 10 258 991 101 11 068 869 31 739 651


329 15 970 798 336 16 087 928 336 16 686 875 48 745 600

Interim supportive agents and trainees

2 647 105 3 408 780 3 401 698 9 457 583


11 070 973 11 652 568 9 252 271 31 975 812

Total EACEA 428 40 100 666 435 41 408 267 40 409 713 121 918 646

Commission

I and II. Staff related expenditure, overheads

4.5 571 500 4.5 589 500 7.7 839 300 2 000 300

27

May 2016


2012 2013 2014 Total 2012-

2014 No. Cost No. Cost No. Cost


4.5 571 500 4.5 589 500 7.7 839 300 2 000 300


0 0

Title II. Infrastructure and operating expenditure

177 100 177 100

Total Commission

cost:

4.5 571 500 4 5 589 500 1 016 400 2 177 400

Title III. Programme support expenditure

9 224 580 8 410 756 5 717 725 23 353 061

TOTAL COST 432.5

49 896 746 439.5

50 408 523 444.7 47 143 838 147 449 107

ESTIMATED SAVINGS

-4.5 13 683 834 -4.5 14 987 233 100 13 103 722 41 774 789


The results of the quantitative retrospective CBA revealed that (see Table 7 and Table 8 for

more details):

The actual costs of the EACEA were lower than the initial estimates in the respective

financial statements, which were prepared in view of its establishment (prolongation

of timeframe or extension of tasks). In 2012-2014 the actual administrative budget

implemented by the EACEA(see section 2.2.2. above) amounted to EUR 136 million

based on the EU contribution and was some 9.7 % lower than the administrative

budget estimations in the respective financial statements (EUR 151 million).

The management and execution of the programme(s) by the EACEA was cost-

effective compared to the in-house (Commission) scenario. In 2012-2014 the

estimated cost savings of the executive agency scenario constituted EUR 41.8 million

(22 %) compared to the in-house scenario. The savings of the executive agency

scenario primarily resulted from a higher share of lower cost external personnel (CAs)

employed within the executive agency and a lower number of staff in 201411.

Figure 6. Estimated savings of the executive agency scenario in 2012-2014, EUR


11 The number of staff of the executive agency scenario estimated in the CBA was reduced by about 13 % in the EACEA, while the in-house scenario was not modified in the specific financial statement.

189 223 896

147 449 107

41 774 789

0

50 000 000

100 000 000

150 000 000

200 000 000

In-house scenario Executive agency scenario Savings of EA scenario

28

May 2016


2.2.4. Workload analysis

According to the financial statement estimations, the operational budget allocated by the

EACEA for 2012-2014 calls amounted to approximately EUR 1.8 billion (Figure 7). The

actual operational budget for the same period was EUR 2.3 billion12 and exceeded the

financial statement estimations by 24 %. This primarily resulted from additional third

country and EDF contributions, as well as from extensions of the EACEAs mandate that

were sometimes finalised later than the actual allocation of additional operating budget to

the EACEA13.

Figure 7. Operational budget related to 2012-2014 calls, EUR million

Source: Financial statements, the EACEAs Annual Work Programmes.

The Agencys workload is closely linked with the allocated operational budget, the number

of proposals received and the number of grants concluded. Our analysis showed that the

actual number of grants14 related to the 2012-2014 calls was very close to the initial

estimations provided in the financial statements (see Figure 8 below).

Figure 8. Number of grants related to 2012-2014 calls

Source: Financial statements, Annual Activity Reports of the EACEA.

12 Based on the EACEAs Annual Work Programmes. All funding sources. 13 This primarily refers to 2012 when the actual operational budget exceeded the financial statements estimations by 42 %, as the additional operational appropriations under the programmes that have already been delegated to the EACEA were already allocated to the Agency in 2012. However, the operational budget in the financial statement related to Commission Implementing Decision of 20 December 2012 (C(2012)9475 final) was revised only for 2013. 14 Without Erasmus/Erasmus+ university charters, Erasmus Mundus individual scholarships and Media actions Support to sales agents and Digitalisation of cinemas.

527752

543

1822

751 896 611

2258

0

500

1000

1500

2000

2500

2012 2013 2014 Total

FS estimations Actual (AWP)

4484 45193193

12196

4163 46113439

12213

0

2000

4000

6000

8000

10000

12000

14000

2012 2013 2014 Total

FS estimations Actual

29

May 2016


Overall, our analysis of the EACEAs main workload indicators showed that the Agencys

operational efficiency15 reached its peak in 2012-2013, primarily owing to the final years of

the 2007-2013 MFF programmes characterised by the highest operational budgets. In 2014,

the operational budget decreased as a result of transition to the new generation

programmes. This decrease was due primarily to the 2014-2020 Erasmus+ programme.

It is important to note that the operational budget of the 2014-2020 MFF programmes will

gradually increase and it is expected that in 2020 commitment credits will reach a level

similar to 2013 (which was the last year of the 2007-2013 MFF programmes). At the same

time, a decrease in the operational budget also indicates that in the early years of the 2014-

2020 MFF the initial EACEA workload will be lower compared to 2012-2013 (especially in

2015-2016, after a decline in the workload related to the 2007-2013 programmes legacy

management). This could provide an opportunity to improve the qualitative aspects of

programme implementation (such as improving monitoring arrangements both at

programme and project level and information flows between the Agency and the

Commission).

2.2.5. Qualitative aspects of the retrospective CBA

Qualitative aspects of the CBA indicated in the ToR (which reflect the CBA questions provided

in Article 3(1) of Regulation (EC) No 58/2003) were integrated into the overall evaluation

framework (the evaluation questions). This allowed a coherent presentation of the

evaluation results and conclusions in the final evaluation report.

As part of the qualitative CBA exercise we also developed a series of 15 good practice

examples corresponding to the specific CBA items. Each of the selected good practices was

presented in Annex 7 to the report in the form of case fiches and referred to Part 3 of the

report where relevant.

Table 9 below provides information on the links between the qualitative items of the CBA,

the evaluation questions and the titles of the selected good practices.

Table 9. Qualitative items of the CBA, their correspondence to the evaluation

questions and good practices

Qualitative item of the CBA Correspondence

to questions Concrete examples of good

practice

Level of financial and technical expertise in relation to the delegated tasks

Q.1 The Learning and Development Framework of the EACEA

Networks of specific cadre of staff in the REA

Flexibility in response to the needs of stakeholders in the implementation of the delegated tasks

Q.3, Q.2

Simplification of the procedures used (in particular the extent of widening the scope of use of flat-rate financing, electronic submission, remote selection procedures, etc.) and where possible,

impact of simplifications on the efficiency and effectiveness

Q.3 Development of simplified cost options in the programmed delegated to the EACEA The Working Smarter initiative of the EACEA

Proximity of delegated activities to final beneficiaries

Q.3 Organisation of joint events aimed at facilitating exchange of good practices and experience

15 Expressed as average programme expenditure, number of applications and open projects per staff and ratio between the administrative and operational budget.

30

May 2016


Qualitative item of the CBA Correspondence

to questions Concrete examples of good

practice

Introduction of the e-complaint form on the EACEA's website

Quality, relevance and timeliness of information provided to applicants and beneficiaries

Q.3 Organisation of kick-off meetings with newly selected beneficiaries

Visibility of the European Union as promoter of the programmes concerned

Q.3 Organisation or participation in promotional and information

events

Level of know-how retained inside the Commission (in particular the accessibility, adequacy, relevance, frequency and timeliness of information

provided to the parent DGs)

Q.5, Q.4 and Q.3

Clear delineation of tasks, clarity of roles, and lack of duplication

Q.1 and Q.4 ICE+ working arrangement

Transparency of the evaluation and selection process

Q.3 Use of IT tools for online briefing of external experts in the EACEA Monitoring quality and consistency of results produced by the external experts and harmonisation of the evaluation results

Exploitation and impact of results achieved through the activities managed by the EACEA

Q.3 and Q.5 The VALOR dissemination platform Common monitoring framework and tools introduced by DG RTD

Reporting, feed-back and cooperating to/with parent DG

Q.4, Q.5 and Q.3 The monitoring framework for EU Aid Volunteers developed by DG ECHO in cooperation with the

Agency Workload measurement process implemented by the REA

Application of the Commission's data protection and transparency rules

Q.4

During the interim period, we also reviewed the operation of the Commissions executive

agencies in the 2012-2014 period based on their Activity Reports and the available

evaluation reports. The purpose of this review was to identify recent changes to the

operation of the Commissions executive agencies and to compare them with the

performance of the EACEA.

Our review indicated that after the last extension of their mandates, the Commissions

executive agencies undertook quite similar changes aimed at increasing their efficiency and

effectiveness. First, the executive agencies introduced simplifications and optimised

their business processes (e.g. simplifications of grant management in the REA and the

ERCEA, simplification of small grants in CHAFEA or streamlining the evaluation process in

INEA), reducing administrative burden for the beneficiaries and/or increasing efficiency at

the agency level.

Second, some of the executive agencies carried out actions aimed at harmonising

management arrangements across different programmes (e.g. INEA harmonised the

administrative and technical management of all three CEF sectorial programmes delegated

it). The introduction of the common rules and IT tools provided by the Common Support

Centre contributed to an increased harmonisation of project management practices in the

Research and Innovation family (with the REA providing central administrative and logistical

31

May 2016


support services to the executive agencies, in particular for the evaluation of proposals,

management of the expert evaluators and the validation of beneficiaries' legal data).

Third, the executive agencies continued developing IT tools for application and grant

management. Corporate IT tools of the Research and Innovation family were implemented

in all the executive agencies involved in the implementation of Horizon 2020 and extended

to other executive agencies outside this family (to the EACEA and CHAFEA). Substantial

savings were expected to be generated after the introduction of these tools in CHAFEA.

Fourth, most of the executive agencies reorganised their internal tasks. For instance,

the REA and the ERCEA adapted their organisational structures to the new mandates, while

EASME regrouped financial tasks in one unit and created a centralised procurement cell

providing advice, helping the operational units with public procurement procedures and

dealing with service contracts.

Fifth, some of the executive agencies were engaged in the development of public

databases on supported projects and their results. The REA and the ERCEA provided

data to the CORDA, which is the internal data warehouse for the Commission services and

is also used to inform the EU Member States about their participation in the framework

programmes through the e-CORDA platform. Also, information on all EU-funded research

and innovation projects and their results was disseminated publicly through CORDIS, the

Commissions public repository. CHAFEA developed a public database providing information

about various projects and actions supported by the previous EU Public Health Programme

and the current EU Health Programme. The database is useful for increasing visibility and

demonstrating the results of co-financed projects to both parent DGs and the general public.

Our evaluation findings (see Part 3 of the report below) indicate that all of these changes

also occurred in the EACEA during the 2012-2014 period. Also, the Agency developed a

number of concrete cases of good practice, which are presented in the form of fiches in

Annex 7 to the report. Overall, this evidence proves that the EACEA was results orientated

and engaged in continuous learning in order to create innovation and improve its

business operations during the period 2012-2014.

32

May 2016


3. Evaluation findings

3.1. Question 1

Q.1: To what extent are the EACEAs organisation and internal structures appropriate to its new profile and to the needs it is intended to meet? What is the potential impact of the new HR policy on seconded officials, intended to maintain closer links between

these officials and the parent DGs, on cooperation between the parent DGs and the Agency? To what extent has the EACEA retained and/or improved the competence and efficiency of its staff in relation to the implementation of current and new tasks?

This evaluation question addresses the issues of organisational structure and human

resource management.

Organisational structure refers to the way in which an organisation is structured, i.e. the

division of line and staff tasks, formal chains of communication between management and

staff and the way tasks and responsibilities are divided throughout the organisation. Since

executive agencies manage EU programmes on behalf of the Commission and under its close

supervision, close relationships between the Commission and the agencies should be

reflected in the administrative and HR arrangements. Therefore, the organisational structure

of the EACEA should clearly reflect the EU programmes for which the Agency is responsible.

In this evaluation, the key issues related to organisational structure involve the extent to

which it is aligned with the tasks entrusted to the Agency, the extent to which it is clearly

documented, communicated and easy to understand. The size of the organisation and its

adequacy is also an important aspect. It is relevant to note that no shortcomings on these

issues were found in the previous evaluations.

Human resource management (HRM) involves recruitment, management and

development of people who work for an organisation. For HRM analysis, we followed three

sub-criteria of the CAF 2013 model as to how the EACEA:

plans, manages and improves its human resources transparently with regard to

strategy and planning;

identifies, develops and uses competencies of people aligning individual and

organisational goals;

Involves employees by developing open dialogue and empowerment, supporting

their well-being.

These issues are relevant to the evaluation, as a high level of staff turnover, limited staff

mobility and moderate levels of staff satisfaction may have a negative effect on the

organisational performance.

1.1. To what extent is the organisational structure aligned with the tasks entrusted to the

Agency? Is it appropriate to its new mandate and to the needs it is intended to meet?

The overall finding of our analysis is that the organisational structure of the Agency was well

aligned with the tasks delegated to the EACEA and matched well the EU programmes and

actions managed by the Agency. In addition, the appointment of the new Head of

33

May 2016


Department16 for Erasmus+ was a positive development that improved the coordination of

programme implementation.

Before our reference period, the organisational structure of the EACEA had been de facto

very flat. Up to 2013, there was a prolonged interim period when the same person acted as

both Director and Head of Department. Before that, there were no departments and all

heads of units were directly responsible to the Director of the Agency. After 2013, a separate

person as a head of department became responsible for seven units dealing with the

Erasmus+ programme and the EU Aid Volunteers programme, while the Director remained

in control of the Creative Europe and Europe for Citizens programme units, as well as units

dealing with the administrative functions. As a result of this reorganisation, the

organisational structure was better adapted to the new programmes delegated to the Agency

and corresponded well to the actual workload of their management. The roles and

responsibilities were clearly allocated within the Agencys organisational structure.

The reorganisation also improved management of this programme and made its

implementation more homogenous at the Agency level. In addition, as a result of this

reorganisation, the Agency was able to speak with one voice in various working groups,

which improved collaboration between the EACEA and the parent DGs. This improved

coordination in the implementation of Erasmus+ and facilitated standardisation of its

implementation modalities (see evaluation question 4).

Despite appointing a new Head of Department for Erasmus+ units, the o

Documents

3rd Interim Evaluation of the Education, Audiovisual, and Culture