Second interim evaluation of the Education, Audiovisual ...ec.europa.eu/dgs/education_culture/more_info/evaluations/docs/... · Second Interim Evaluation of the Education, Audiovisual

Second Interim Evaluation of the Education, Audiovisual and Culture Executive Agency (EACEA) Final Report April 2013

© European Union, 2013 Reproduction is authorised provided the source is acknowledged. Cat. No. NC-01-13-165-EN-N ISBN 978-92-79-30406-4 doi: 10.2766/4289

i

Contents

1 Introduction ............................................................................................. 1

1.1 Context, purpose and scope ..................................................................................... 1

1.2 Approach to the evaluation ....................................................................................... 2

2 Synthetic description of activity (Task 1) ............................................. 9

2.1 History ......................................................................................................................... 9

2.2 Design and set-up .................................................................................................... 11

2.3 Delegated credits and costs .................................................................................... 13

2.4 Core Responsibilities .............................................................................................. 14

2.5 Relationship with the Commission ......................................................................... 16

3 Response to previous evaluation (Task 3) ......................................... 17

4 Cost-benefit analysis (Task 2) ............................................................. 19

4.1 Introduction .............................................................................................................. 19

4.2 Relevance of role, design and set-up ..................................................................... 19

4.3 Effectiveness: service to applicants and beneficiaries ........................................ 25

4.4 Efficiency: cost of programme management ......................................................... 40

4.5 Effectiveness: better projects and outcomes ........................................................ 43

4.6 Effectiveness: enabling the Commission to fulfil its political role ...................... 51

5 Conclusions and recommendations (Task 4) ..................................... 55

List of figures

Figure 1.1 Distribution of responses by outcome of application ................................... 7

Figure 2.1 EACEA organogram (January 2013) ......................................................... 12

Figure 4.1 Effectiveness by programme ..................................................................... 28

Figure 4.2 Views on e-forms by Programme .............................................................. 31

Figure 4.3 Impact of simplification by Programme ..................................................... 32

Figure 4.4 Proximity (responsiveness) by Programme ............................................... 37

Figure 4.5 Proximity (information tools) by Programme ............................................. 39

Figure 4.6 Transparency of selection by Programme ................................................. 45

Figure 4.7 Monitoring and reporting by Programme ................................................... 48

List of tables

Table 1.1 Evaluation questions .................................................................................... 2

Table 1.2 Linking objectives, CBA issues and Evaluation Questions ........................... 3

ii

Table 1.3 Breakdown of survey results by programme and outcome of application .................................................................................................................... 6

Table 1.4 Average responses by programme, grouped by aspects of programme management (CBA sections) .................................................................... 8

Table 2.1 EACEA staff by programme and staff category (2011) ............................... 13

Table 2.2 EACEA operational budget by year (€m) ................................................... 14

Table 2.3 EACEA administrative budget by year (€m) ............................................... 14

Table 2.4 Breakdown of applications and selected projects by year (all delegated programmes) ............................................................................................. 15

Table 2.5 Breakdown of applications and selected projects by programme, 2008 to 2011 .............................................................................................................. 15

Table 4.1 Time indicators ........................................................................................... 25

Table 4.2 Payment indicators ..................................................................................... 26

Table 4.3 Effectiveness by survey .............................................................................. 27

Table 4.4 Online submissions .................................................................................... 30

Table 4.5 Views on e-forms, by survey ...................................................................... 31

Table 4.6 Views on e-forms, by year of most recent application ................................ 31

Table 4.7 Impact of simplification by survey ............................................................... 32

Table 4.8 Views on applying for funding under these programmes again (Q41) by Programme ................................................................................................. 35

Table 4.9 Proximity (responsiveness) by survey ........................................................ 36

Table 4.10 Proximity (information) by survey ............................................................. 38

Table 4.11 Comparison of staff costs of EACEA compared to in-house management by the Commission (2008-11) .............................................................. 41

Table 4.12 Key data regarding staff workload (2008-11) ........................................... 42

Table 4.13 Transparency of selection ........................................................................ 44

Table 4.14 Number of project monitoring visits by year ............................................. 47

Table 4.15 Views of monitoring and reporting by survey ............................................ 47

Table 4.16 Dissemination via EVE portal ................................................................... 50

iii

Executive Summary Subject of the evaluation The Education, Audiovisual and Culture Executive Agency (EACEA) was established in 2005 and at the beginning of 2006 assumed responsibility for the management of a number of EU programmes in the fields of education and vocational training, culture, audiovisual, citizenship and youth (plus external collaborations in some of these fields). EACEA is one of six Executive Agencies (of varying size) that have been established to date, and acts as a delegated body for three Commission Directorates General, which have ultimate responsibility for the programmes: Education and Culture (DG EAC), Communication (DG COMM) and Development and Co-operation – EuropeAid (DG DEVCO).

EACEA's core responsibilities are related to the management of programmes throughout their lifecycle, including launching and carrying out grant procedures and calls for proposals, evaluation of bids and project selection, communicating with funding recipients, obtaining financial commitments, setting up and modifying agreements and contracts, monitoring, payments, receiving final reports and ex-post activities such as monitoring missions and audits. EACEA also makes an important contribution to the successful management of programmes, through activities such as raising awareness and informing potential applicants and beneficiaries about funding opportunities, collaborating with national structures (e.g. contact points), providing project-level information and qualitative feedback to the DGs, and facilitating the dissemination of project results. EACEA has also acquired responsibilities and developed competences in policy support, mainly through the delegation of Eurydice (in 2007)1 .

In 2011, EACEA employed 433 members of staff across ten thematic/programme units and three horizontal/administrative units; and had an operational budget for commitment appropriations of €621m2.

Purpose of the evaluation

This evaluation satisfies the requirements of Council Regulation (EC) No 58/2003, for an external evaluation of each Executive Agency to be undertaken every three years. This is the second evaluation of EACEA, covering the period from 1 May 2008 to 31 December 2011 and building on the first (which covered the period from January 2006 to April 2008 and was published in early 2009). In line with the Terms of Reference (ToR), the evaluation provides an assessment of progress against the recommendations made in the first evaluation and a descriptive chapter containing a narrative of events since. The key analytical element is an assessment of the costs and benefits associated with delegating programme management to EACEA, which incorporates financial and other types of information as well as a retrospective element comparing the forecasts made in the previous evaluation with the actual situation at the end of the current period. This information is then brought together in a set of main findings and recommendations for the future.

1 This includes the management of a network of Eurydice national units, the commissioning of studies and the collection and analysis of content-related and comparative information in the areas of education and youth policy, as well as running events and disseminating relevant information. This involves a high degree of collaboration with the relevant national structures as well as other stakeholders (including national governments and research institutes) and information providers (e.g. Eurostat). The delegation of Tempus in 2009 brought additional policy support tasks, though this forms a comparatively small part of the overall programme resources. 2 For 2013, an extension in EACEA’s mandate increased the total operational (programme) budget to €896m, and total staffing to 453.

Executive Summary

iv

Main findings

EACEA has substantially achieved its objectives, with clearest progress against those related to efficient management of programmes and improving services to applicants and beneficiaries. There has been progress against all annual priorities set out in the Agency’s Annual Work Programmes: the strongest evidence of this concerns progress against the objective of streamlining application and selection processes, whereas there is less evidence of progress in terms of improving project implementation and reporting requirements. However, it should be borne in mind that this has been the main recent focus for improvements (with further improvements planned for the next generation of programmes).

Relevance of role, design and set up

EACEA’s role as a dedicated programme management body is relevant to the needs of the parent DGs and represents a clear improvement on the previous approach (including delegation to the Technical Assistance Office - TAO). EACEA is functioning in accordance with the relevant legal framework. Recent increases in both the number and size of programmes delegated to EACEA confirm the continued need for a dedicated agency deploying specialist staff to manage a range of programmes. EACEA has proved itself sufficiently flexible to respond to increased demands from the parent DGs for additional programme management services, in some cases at relatively short notice. The evaluation found a general improvement in the level of technical and financial expertise amongst EACEA staff, at the same time as overall staffing levels have increased. There is potential for securing further improvements through the strengthened training offer, provided that attendance rates at training sessions can be maximised.

The rate of occupancy of staff posts has exceeded targets since 2009 and staff turnover is comparatively low. Some staff have been recruited after completing a temporary period of employment at the Commission and others have left EACEA for posts at the Commission after successful completion of the European competition. Whilst EACEA has had to bear the cost of replacing such staff, it has served as a useful "bridge" between temporary and permanent employment for such individuals and has thus helped to retain their expertise within the EU institutions as a whole. There may be scope for EACEA to further improve both staff retention and the Agency’s overall ability to accommodate changes by promoting greater staff mobility between programme units and by exploring ways to spread its workload more evenly over the year (e.g. by co-ordinating different timescales more closely in the new generation of programmes). Staff retention could be further improved by exploring ways to create progression pathways between the different Executive Agencies.

Services to applicants and beneficiaries

EACEA has shown consistent progress against its main output indicators, notably the time taken to select projects, award grants, arrange contracts and make payments; as well as in terms of the level of operational and administrative budgets committed and paid. Stakeholders were unanimous in reporting that the service provided by EACEA represented a significant improvement on previous arrangements, while satisfaction levels amongst applicants and beneficiaries are positive and have remained relatively constant, despite increases in the Agency’s overall workload. The average satisfaction score across all respondents and programmes was 3.1 (on a scale of 1 to 9, with one being the most positive rating possible), although there were differences between programmes, with applicants and beneficiaries under the Culture Programme, Europe for Citizens and Youth in Action awarding less positive scores than those under the Lifelong Learning Programme, MEDIA and Tempus. EACEA has made significant progress towards standardising/harmonising and simplifying procedures, especially in relation to the application stage, although this has not led to a further substantial improvement in the satisfaction scores expressed by respondents to the survey of applicants and beneficiaries. It can be argued that it is not realistic to expect satisfaction to continue improving at the same rate as seen in the early years of EACEA’s operation, especially in the context of increasing competition for funding (and therefore the numbers of

v

unsuccessful applications). The least positive aspect of the current satisfaction survey relates to the quality of feedback provided on unsuccessful applications, which is regularly described as too short, too standardised or lacking detail on the relative merits or faults of an application. The potential for EACEA to minimise the administrative burden for applicants is necessarily limited by the requirements of the Financial Regulation or the design of programmes; and many of the frustrations of applicants relate to issues that are often beyond the direct control of EACEA. However, the feedback from applicants is that the administrative requirements placed on them by EACEA are, on the whole, reasonable. Applicants recognise that improvements have been made in the administrative processes, although there is scope for further, on-going improvements, for example, relating to the layout and format of e-forms, location of information on the website or in programme guides, and improvements in the information that beneficiaries have to provide in their final reports.

In terms of availability and responsiveness to beneficiaries, EACEA has performed well, with applicants reporting a high level of satisfaction in respect of knowing who to contact and EACEA's ability to respond to questions within a reasonable time. EACEA has also significantly increased the numbers of monitoring visits and improved the various communication tools it uses. Furthermore, EACEA's direct contact with beneficiaries and its day-to-day management of programmes has enabled its staff to gain a high level of policy knowledge relevant to the fields covered by their programmes. Such knowledge represents a valuable asset that is able to support the Commission in its policymaking role.

The cost of programme management and administration

Outsourcing the management of programmes continues to provide significant cost savings relative to the costs of in-house management by the Commission, primarily through the deployment of lower-paid staff than in the DGs. Staff posts created at EACEA have mostly replaced posts at the (now defunct) TAO or represented entirely new tasks (in line with the expansion of programmes). This suggests that the potential cost-savings of out-sourcing have in fact been realised, and are likely to exceed the forecasts for 2008-2011 contained in the first interim evaluation of EACEA. This included forecast savings of approximately €43m for the period 2008-2011, while calculations based on current costs and workload suggest a saving of €93m for the same period. Throughout the period covered by the evaluation, the staffing of EACEA has increased in line with its growing responsibilities, the increase in the volume of funds managed and the increase in the number of applications received. During this time, EACEA has been able to absorb a large number of new recruits whilst avoiding any reduction in staff efficiency (measured in terms of average workload per person) and any increase in staff costs relative to programme funds managed, which remains below 5% of the total operational budget. Facilitating better projects and project outcomes

Information on budget execution and the level of competition at selection stage reinforces the views of stakeholders that there has generally been no shortage of good quality project applications. Project selection processes are well defined, clear and transparent and there does not appear to be significant scope to further improve selection procedures under the current framework. The selection, training and monitoring of external experts plays a critical role in ensuring the selection of good quality projects. Given the move to carrying out briefings of external evaluators and project appraisal meetings remotely, this aspect of EACEA’s work should remain under review to ensure that quality levels are maintained.

EACEA’s monitoring and control processes are strong, though as highlighted above there has been clearer progress for application and selection procedures than for monitoring and reporting. However, a number of improvements in these areas (such as e-reporting and the use of audit certificates) have been piloted and are in the process of being implemented across programmes.

The main focus of monitoring is on the financial and ‘technical’ aspects of how programmes have been implemented. However monitoring information also plays a key role in assisting the identification of good practice

vi

and the dissemination of results. EACEA’s amount of direct contact with beneficiaries has increased, especially in relation to the number of field monitoring visits, though it is not clear that information on the results of these visits is always shared with the parent DGs. Any failure to do so represents a missed opportunity to make the most of the information collected from projects.

Despite EACEA’s efforts to ensure clarity in official documentation, the evidence suggests there remains some confusion amongst applicants and beneficiaries on the respective roles of EACEA and parent DGs, even though all are seen as part of the European Commission.

Enabling the Commission to fulfil its political role

Coordination between EACEA and the Commission is generally functioning well with representatives of both parties reporting overall satisfaction with the current situation. The formal means of co-ordination (including the Steering Committee, Co-ordination Meetings, co-operation manuals and regular reporting) have been complemented by a willingness to co-operate informally, both on a day-to-day basis and on specific occasions, including public events. Although a 2009 report by the Court of Auditors concluded that the Commission’s supervision of the Executive Agencies was quite limited, it did not suggest failure to comply with the legal framework. The Commission’s revised supervision strategy should address the report’s main findings although some issues remain to be resolved, such as the late adoption of EACEA’s Annual Work Programmes. There has been a continuous improvement in the flow of information between EACEA and the parent DGs. Staff of the parent DGs tend to receive the information they require either as a matter of course or on request. The inevitable gap created by the separation of the Commission's political role from programme management tends to be overcome by a combination of formal and informal communication mechanisms, practical tools, a degree of flexibility on both sides and the gradual development of trust and co-operation between the various individuals involved. However, some representatives of the parent DGs commented that there may be potential to improve the flow of qualitative information from EACEA, in terms of both volume and content, in order for the parent DGs to benefit more extensively from the practical knowledge held by EACEA, for example the results of field monitoring visits highlighted above.

Though arguably not a key driver of the ‘agency solution’, delegation to EACEA is likely to have a beneficial effect in terms of allowing Commission staff to better focus on their core institutional tasks. There is a clear understanding of the respective roles and responsibilities of the parent DGs and EACEA, while staff of both institutions recognise that responsibility is shared in some areas, for example in respect of the gathering and exploitation of knowledge emerging from projects or of relevance to the programmes. Recommendations

While the main focus of the evaluation is on the overall arrangements for delegating programme management tasks to EACEA, individual recommendations are directed at either the Agency or the Commission, or both institutions.

1. EACEA should consider ways to promote greater staff mobility between programme units, whilst the Commission and EACEA should together explore ways to develop career pathways with other EU bodies such as the other Executive Agencies.

2. For the forthcoming programming period, the parent DGs and EACEA should together explore ways to enable

EACEA to spread its workload more evenly over the year, by seeking closer co-ordination of the timescales of different programmes.

3. EACEA should continue to seek ways to increase the streamlining of processes still further, especially in

relation to later stages of the project life cycle such as monitoring, control and reporting, as well as any

vii

opportunities presented by the transition to the new generation of programmes from 2014. Any future evaluation should consider the extent to which both have been achieved and seek to collect evidence of any efficiency savings generated as a result.

4. EACEA should explore ways of improving feedback to unsuccessful applicants when considering future

improvements in processes. Initially this could mean applying lessons from the programmes where satisfaction with feedback is highest to the other delegated programmes wherever it is feasible to do so.

5. EACEA should continue to seek improvements in application and information tools, particularly relating to the

layout and format of e-forms and the location of information on the website and in programme guides.

6. The Commission should prioritise the revision of its supervision strategy while the Agency should include progress on its implementation as an annual item on the agenda of the EACEA Steering Committee, and in its Annual Activity Reports.

7. The Commission should take steps to ensure that the Annual Work Programme of EACEA is adopted at an

early point in each year.

8. In order to maintain the effectiveness of external experts, EACEA should keep procedures under review and consult the parent DGs on at least an annual basis regarding the suitability of experts and the quality of their advice and recommendations.

9. The impact of carrying out greater numbers of remote briefings for (and appraisal meetings with) external

experts should continue to be monitored by the Commission and EACEA in order to ensure that the quality of project evaluations is maintained.

10. The Commission and EACEA should consider further increasing the sharing of practical knowledge gained by

staff of EACEA through their close proximity to beneficiaries and involvement in the day-to-day management of programmes. In line with the findings on monitoring procedures, this could be supported by improving the quality of information provided to the Commission regarding the outcomes of field monitoring visits.

viii

Zusammenfassung Gegenstand der Evaluierung Die Exekutivagentur Bildung, Audiovisuelles und Kultur (EACEA) wurde 2005 eingerichtet und ist seit Anfang 2006 für die Verwaltung bestimmter Teile der EU-Programme in den Bereichen Bildung, Kultur, audiovisuelle Medien, Bürgerschaft und Jugend (sowie externe Zusammenarbeit in einigen dieser Bereiche) zuständig. Die EACEA ist eine der sechs eingerichteten Exekutivagenturen (mit unterschiedlicher Größe) und steht unter der Aufsicht folgender drei Generaldirektionen der Europäischen Kommission: Bildung und Kultur (DG EAC), Kommunikation (DG COMM) und EuropeAid – Entwicklu ng und Zusammenarbeit (DG DEVCO).

Die EACEA ist für die meisten Verwaltungstätigkeiten im Rahmen der Programme zuständig. Hierzu zählen: Einleitung und Abwicklung von Finanzhilfeverfahren und Ausschreibungen, Bewertung von Anträgen und Projektauswahl, Kommunikation mit Förderungsempfängern, Verfahren für Finanzierungszusagen, Gestaltung und Modifizierung von Vereinbarungen und Verträgen, Projektüberwachung, Auszahlungen, Erhalt von Abschlussberichten und Ex-post-Aktivitäten wie Überwachungsmissionen und Auditverfahren. Durch Tätigkeiten wie Bewusstseinsbildung, die Bereitstellung von Informationen über Finanzierungsmöglichkeiten für potentielle Antragsteller und Begünstigter, die Zusammenarbeit mit nationalen Strukturen (z.B. Kontaktpunkte), die Bereitstellung von Informationen auf Projektebene, qualitative Rückmeldungen an die Generaldirektionen und die Erleichterung der Verbreitung von Projektergebnissen, leistet die Exekutivagentur zudem einen wichtigen Beitrag zu der erfolgreichen Verwaltung von Programmen. Ferner hat die Exekutivagentur Aufgaben im Bereich der strategischen Unterstützung übernommen und entsprechende Kompetenzen entwickelt, insbesondere durch die Übertragung von Eurydice (in 2007)3.

Im Jahr 2011 beschäftigte die EACEA 433 Mitarbeiter in zehn thematischen Referaten und drei horizontalen Referaten/Verwaltungseinheiten, und die zugewiesene Mittelausstattung für Verpflichtungsermächtigungen belief sich auf 621 Mio. EUR4.

Zweck der Evaluierung

Diese Evaluierung erfüllt die Anforderungen der Verordnung (EG) Nr. 58/2003 des Rates in Bezug auf eine externe Bewertung jeder Exekutivagentur, die alle drei Jahre durchzuführen ist. Diese zweite Bewertung der Exekutivagentur baut auf der ersten Evaluierung auf (die den Zeitraum von Januar 2006 bis April 2008 abdeckte und Anfang 2009 veröffentlicht wurde) und deckt den Zeitraum vom 1. Mai 2008 bis zum 31. Dezember 2011 ab. Entsprechend der Aufgabenstellung umfasst die Evaluierung eine Bewertung der Fortschritte, die im Rahmen der in der ersten Evaluierung vorgelegten Empfehlungen erzielt wurden, sowie ein deskriptives Kapitel mit einer Schilderung der bisherigen Ereignisse. Das wesentliche analytische Element ist eine Kosten-Nutzen-Analyse der mit der Übertragung des Programmmanagements an die EACEA verbundenen Vorteile. Diese Analyse umfasst neben finanziellen Informationen und Informationen anderer Art auch einen retrospektiven Vergleich der in der 3 Dies umfasst die Verwaltung aller nationalen Informationsstellen des Eurydice-Netzes, die Vergabe von Studienaufträgen und die Sammlung und Analyse inhaltlicher und vergleichender Informationen in den Bereichen Bildung und Jugend, ebenso wie die Durchführung von Veranstaltungen und die Verbreitung relevanter Informationen. Dies erfordert ein hohes Maß an Zusammenarbeit mit den relevanten nationalen Strukturen sowie anderen Beteiligten (einschließlich nationale Regierungen und Forschungsinstitute) und Informationsanbietern (z. B. Eurostat). Die Übertragung von Tempus in 2009 umfasste zusätzliche Aufgaben zur Unterstützung der Politik, obgleich dies nur einen verhältnismäßig kleinen Teil der Programmressourcen insgesamt darstellt. 4 Für 2013 wurden durch eine Erweiterung des Mandates der EACEA der gesamte operative (Programm)-Haushalt auf 896 Mio. Euro und der Personalbestand auf 453 aufgestockt.

Zusammenfassung

ix

vorherigen Evaluierung getroffenen Vorhersagen mit der tatsächlichen Situation am Ende des gegenwärtigen Zeitraums. Diese Informationen werden anschließend in einer Zusammenstellung der wichtigsten Ergebnisse und Empfehlungen für die Zukunft bereitgestellt.

Wichtigste Ergebnisse

Die EACEA hat ihre Ziele im Wesentlichen erreicht. Deutliche Fortschritte gab es im Hinblick auf die Effizienz der Programmverwaltung und bei der Verbesserung von Dienstleistungen für Antragsteller und Begünstigte. Bei allen in den jährlichen Arbeitsprogrammen der Exekutivagentur niedergelegten jährlichen Prioritäten wurden Fortschritte verzeichnet. Der stärkste Nachweis dafür sind die bei der Straffung von Antragstellungs- und Auswahlverfahren erzielten Fortschritte. Bei den Projektimplementierungs- und Berichterstattungsanforderungen gab es allerdings nicht so viele Anzeichen für Verbesserungen; es ist allerdings zu beachten, dass dies kürzlich der wesentliche Fokus für Verbesserungen war (weitere Verbesserungen sind für die nächste Programmgeneration geplant).

Relevanz von Rolle, Konzeption und Gestaltung

Die Rolle der EACEA als Programmmanagementstelle ist auf die Bedürfnisse der zuständigen Generaldirektionen zugeschnitten und stellt eine deutliche Verbesserung im Vergleich zu dem vorherigen Ansatz dar (einschließlich die Übertragung zum Büro für technische Hilfe). Die Exekutivagentur arbeitet gemäß den geltenden Rechtsvorschriften. Der kürzliche Anstieg sowohl in der Anzahl als auch in der Größe der an die EACEA delegierten Programme bestätigt, dass nach wie vor Bedarf für eine Exekutivagentur besteht, die Fachpersonal beschäftigt, um eine Reihe von Programmen zu verwalten. Die Exekutivagentur hat gezeigt, dass sie ausreichend flexibel ist, um dem gesteigerten Bedarf der zuständigen Generaldirektionen an zusätzlichen Programmmanagementdiensten Rechnung zu tragen, in einigen Fällen auch relativ kurzfristig. Die Evaluierung stellte eine allgemeine Verbesserung des Niveaus der sachbezogenen und finanziellen Fachkenntnisse bei den Mitarbeitern der Exekutivagentur fest. Gleichzeitig ist der Personalbestand insgesamt gestiegen. Weitere Verbesserungen sind durch das verstärkte Angebot an Fort- und Weiterbildungsmöglichkeiten erreichbar, vorausgesetzt, dass die Anwesenheitsquoten bei den Fort- und Weiterbildungskursen maximiert werden können.

Der Bestand an Personalstellen liegt seit 2009 über dem Ziel, die Personalfluktuation ist vergleichsweise niedrig. Einige Mitarbeiter wurden nach vorübergehender Beschäftigung bei der Kommission dauerhaft eingestellt, andere Mitarbeiter sind nach erfolgreich bestandenem europäischen Auswahlverfahren von der EACEA zu Stellen bei der Kommission übergewechselt. Die Exekutivagentur musste die Kosten für den Ersatz dieser Mitarbeiter tragen, gleichwohl hat sich dies für die betreffenden Personen als nützliches Sprungbrett von einem vorübergehenden zu einem dauerhaften Arbeitsplatz erwiesen und dabei geholfen, auch weiterhin von ihrer Erfahrung innerhalb der EU-Institutionen zu profitieren. Es besteht vielleicht die Möglichkeit, sowohl die Mitarbeiterbindung als auch die Fähigkeit der Exekutivagentur, insgesamt Änderungen handzuhaben, weiter zu verbessern, indem eine größere Mitarbeitermobilität zwischen Programmeinheiten gefördert und zudem geprüft wird, wie die Arbeitsbelastung gleichmäßiger über das Jahr verteilt werden kann (z.B. durch eine engere Abstimmung von Zeitrahmen in der neuen Programmgeneration). Die Mitarbeiterbindung könnte weiter verbessert werden, indem geprüft wird, wie Karrierepfade zwischen den verschiedenen Exekutivagenturen geschaffen werden können.

Dienstleistungen für Antragsteller und Begünstigte

Die EACEA hat greifbare Fortschritte bei ihren wichtigsten Indikatoren gemacht, insbesondere im Hinblick auf die für die Projektauswahl beanspruchte Zeit, die Gewährung von Finanzhilfen, die Gestaltung von Verträgen und die Leistung von Auszahlungen sowie im Hinblick auf die Höhe der gebundenen und ausgezahlten Verwaltungs- und operationellen Mittel. Die Beteiligten vertraten einmütig die Auffassung, dass die von der Exekutivagentur bereitgestellten Dienstleistungen eine wesentliche Verbesserung im Vergleich zu den vorherigen Regelungen darstellten. Antragsteller und Begünstigte bewerteten ihren Zufriedenheitsgrad als positiv und die Zufriedenheitswerte blieben trotz einer höheren Arbeitsbelastung der Exekutivagentur relativ konstant. Der durchschnittliche Zufriedenheitswert bei allen Befragten und bei allen Programmen beträgt 3,1 (auf einer Skala von 1 bis 9, wobei 1 die höchstmögliche positive Einstufung war), obgleich es Unterschiede zwischen den Programmen

x

gab. So war die Zufriedenheit der Antragsteller und Begünstigten bei dem Kulturprogramm und den Programmen „Europa für Bürgerinnen und Bürger“ und „Jugend in Aktion“ geringer als die Zufriedenheit bei den Programmen „Lebenslanges Lernen“, MEDIA und Tempus. Die EACEA hat wesentliche Fortschritte in Richtung einer Standardisierung/Harmonisierung und einer Vereinfachung von Abläufen erzielt, insbesondere bei der Antragstellung, obwohl dies nicht zu einer weiteren erheblichen Verbesserung der von den befragten Antragstellern und Begünstigten ausgedrückten Zufriedenheit geführt hat. Es kann argumentiert werden, dass es nicht realistisch ist zu erwarten, dass sich die Zufriedenheit im gleichen Tempo erhöht wie in den ersten Jahren der Exekutivagentur, insbesondere angesichts des verschärften Wettbewerbs um Fördermittel (und eine entsprechend höhere Anzahl abgelehnter Anträge). Die Qualität der Rückmeldung bei abgelehnten Anträgen wurde in der derzeitigen Zufriedenheitsumfrage als am wenigsten positiv bewertet. Sie wird regelmäßig als zu kurz beschrieben oder es werden eine zu starke Standardisierung oder eine nicht hinreichende Detailgenauigkeit in Bezug auf die relativen Vorteile oder Mängel eines Antrags beanstandet. Die Möglichkeit für die EACEA, den Verwaltungsaufwand für Antragsteller zu minimieren, ist zwangsläufig beschränkt durch die Anforderungen der Haushaltsordnung oder die Gestaltung von Programmen. Ferner stehen viele der Frustrationen von Antragstellern oftmals mit Sachverhalten in Verbindung, die außerhalb der direkten Kontrolle der Exekutivagentur liegen. Allerdings geben Antragsteller die Rückmeldung, dass die administrativen Auflagen der Exekutivagentur insgesamt vernünftig sind. Antragsteller erkennen Verbesserungen bei den administrativen Abläufen an, obwohl Raum für weitere ständige Verbesserungen besteht, z. B. in Bezug auf das Layout und das Format der elektronischen Formulare, das Auffinden von Informationen auf der Internetseite oder in Programmführern und Verbesserungen in den Informationen, die Begünstigte in ihren Schlussberichten vorlegen müssen.

In Bezug auf Verfügbarkeit und Verständnis für Begünstigte hat die EACEA gut abgeschnitten. Antragsteller melden ein hohes Maß an Zufriedenheit, wenn es darum geht herauszufinden, wer zu kontaktieren ist, und in Bezug auf die Fähigkeit der Exekutivagentur, innerhalb einer angemessenen Zeit auf Fragen zu reagieren. Die Exekutivagentur hat zudem die Anzahl der Überwachungsbesuche erheblich erhöht und die verschiedenen von ihr verwendeten Kommunikationsmittel verbessert. Darüber hinaus hat der direkte Kontakt der Exekutivagentur mit den Begünstigten und das tägliche Management von Programmen ihren Mitarbeitern ermöglicht, einen hohen politischen Wissensstand zu erreichen, der für die durch ihre Programme abgedeckten Bereiche relevant ist. Dieses Wissen stellt eine wertvolle Ressource dar, die die Kommission bei der Ausübung ihrer Rolle in der Politikgestaltung unterstützen kann.

Programmmanagement- und Projektverwaltungskosten

Die Externalisierung des Programmmanagements führt auch weiterhin zu spürbaren Kosteneinsparungen gegenüber einem internen Management durch die Kommission. Hauptfaktor ist die Beschäftigung von Personal, das weniger gut bezahlt wird als das Personal in den Generaldirektionen. Die bei der EACEA geschaffenen Planstellen haben zum größten Teil Stellen im (inzwischen aufgelösten) Büro für technische Hilfe ersetzt oder betreuen neue Aufgaben (entsprechend dem Programmausbau). Dies legt nahe, dass die möglichen Kosteneinsparungen durch Externalisierung in der Tat realisiert wurden. Die in der ersten Zwischenevaluierung der Exekutivagentur vorhergesagten Einsparungen von etwa 43 Mio. Euro für den Zeitraum 2008-2011 werden wahrscheinlich übertroffen. Berechnungen, die auf aktuellen Daten für Kosten und Arbeitsbelastung basieren, lassen auf Einsparungen in Höhe von 93 Mio. Euro für den gleichen Zeitraum schließen. In dem durch die Evaluierung abgedeckten Zeitraum wurde der Personalbestand der EACEA entsprechend ihrer zunehmenden Aufgaben, der Erhöhung des Volumens der verwalteten Mittel und der Erhöhung der Anzahl der erhaltenen Anträge aufgestockt. Während dieser Zeit war die Exekutivagentur in der Lage, eine große Anzahl neuer Mitarbeiter ohne Minderung der Mitarbeitereffizienz (gemessen in Bezug auf durchschnittliche Arbeitsbelastung pro Person) und ohne eine Steigerung der Personalkosten im Verhältnis zu den verwalteten Programmmitteln, die unter 5 % des gesamten operativen Haushalts liegen, aufzunehmen.

xi

Bessere Projekte und Projektergebnisse

Die Informationen über die Ausführung des Haushaltsplans und die Intensität des Wettbewerbs in der Auswahlphase untermauern die Ansicht von Beteiligten, dass es generell keinen Mangel an qualitativ guten Projektanträgen gab. Die Projektauswahlverfahren waren wohldefiniert, klar und transparent und es scheint unter den derzeitigen Rahmenbedingungen keinen Raum für eine weitere Verbesserung der Auswahlverfahren zu geben. Die Auswahl, die Fort-/Weiterbildung und das Monitoring externer Mitarbeiter spielt eine wichtige Rolle dabei, die Auswahl qualitativ guter Projekte sicherzustellen. In Anbetracht der Tendenz, Briefings externer Bewerter und Projektbeurteilungssitzungen auf Entfernung durchzuführen, sollte dieser Aspekt der Tätigkeit der EACEA einer ständigen Überprüfung unterliegen, um eine gleichbleibende Qualität sicherzustellen.

Die Überwachungs- und Kontrollverfahren der Exekutivagentur sind robust, obwohl es wie obenstehend hervorgehoben bei den Antrags- und Auswahlverfahren deutlichere Fortschritte gab als bei der Überwachung und der Berichterstattung. Es wurden allerdings eine Reihe von Verbesserungen in diesen Bereichen versuchsweise eingeführt (z. B. elektronische Berichterstattung und die Verwendung von Prüfungsbescheinigungen), die zurzeit programmweit umgesetzt werden.

Schwerpunkt der Überwachung waren die finanziellen und technischen Aspekte der Umsetzung von Programmen. Die bei der Überwachung gewonnenen Informationen spielen auch bei der Identifizierung bewährter Verfahren und der Verbreitung von Ergebnissen eine wichtige Rolle. Der direkte Kontakt der Exekutivagentur mit Begünstigten und insbesondere die Anzahl der Vor-Ort-Überwachungsbesuche haben zugenommen. Gleichwohl ist nicht klar, inwieweit Informationen über die Ergebnisse dieser Besuche stets mit den zuständigen Generaldirektionen geteilt werden; ein Versäumnis würde bedeuten, dass die Chance, die aus Projekten gesammelten Informationen optimal zu nutzen, nicht wahrgenommen wird.

Trotz der Bemühungen der EACEA, Klarheit in offiziellen Dokumentationen sicherzustellen, gibt es Anzeichen dafür, dass bei den Antragstellern und Begünstigten weiterhin eine gewisse Verwirrung in Bezug auf die betreffenden Rollen der Exekutivagentur und der zuständigen Generaldirektionen besteht, obwohl alle als Teil der Europäischen Kommission betrachtet werden.

Unterstützung der Kommission bei der Ausübung ihrer politischen Funktion

Zwischen der EACEA und der Kommission besteht eine gut funktionierende Koordination. Vertreter beider Seiten zeigen sich mit der derzeitigen Situation insgesamt zufrieden. Die offiziellen Koordinationskanäle (einschließlich Lenkungsausschuss, Koordinierungssitzungen, Handbücher zur Zusammenarbeit und regelmäßige Berichterstattung) werden flankiert durch die Bereitschaft zu einer informellen Zusammenarbeit, sowohl auf täglicher Basis als auch bei besonderen Anlässen, einschließlich öffentliche Veranstaltungen. Obwohl ein Bericht des Europäischen Rechnungshofes von 2009 zu dem Schluss kam, dass die Beaufsichtigung der Exekutivagenturen durch die Kommission relativ begrenzt ist, vermittelte der Bericht nicht den Eindruck eines Versäumnisses, den rechtlichen Bestimmungen nachzukommen. Die überarbeitete Überwachungsstrategie der Kommission sollte sich auf die wichtigsten Ergebnisse des Berichtes beziehen; gleichwohl sind einige Probleme, wie zum Beispiel die späte Annahme der jährlichen Arbeitsprogramme der Exekutivagentur, noch zu lösen. Im Hinblick auf den Informationsfluss zwischen der EACEA und den zuständigen Generaldirektionen wurde eine ständige Verbesserung festgestellt. Mitarbeiter der zuständigen Generaldirektionen erhalten benötigte Informationen entweder von Amts wegen oder auf Anfrage. Die durch die Trennung der politischen Rolle der Kommission vom Programmmanagement geschaffene unvermeidbare Lücke wird in der Regel durch eine Kombination aus formellen und informellen Kommunikationsmechanismen, durch praktische Instrumente, ein gewisses Maß an Flexibilität auf beiden Seiten und die allmähliche Entwicklung von Vertrauen und Zusammenarbeit zwischen den beteiligten Personen überbrückt. Einige Vertreter der zuständigen Generaldirektionen kommentierten allerdings, dass der qualitative Informationsfluss von der Exekutivagentur in

xii

Bezug auf Volumen und Inhalt der Informationen optimiert werden könnte, damit die zuständigen Generaldirektionen umfassender von den praktischen Kenntnissen der Exekutivagentur profitieren, dies bezieht sich beispielsweise auf die Ergebnisse der oben erwähnten Überwachungsbesuche vor Ort.

Die Übertragung von Aufgaben an die EACEA wird wahrscheinlich positive Auswirkungen im Hinblick darauf haben, dass den Mitarbeitern der Kommission dadurch ermöglicht wird, sich stärker auf ihre institutionellen Aufgaben zu konzentrieren. Dies ist allerdings kein ausschlaggebendes Element der Exekutivagentur-Lösung. Es besteht ein klares Verständnis der jeweiligen Rollen und Aufgaben der zuständigen Generaldirektionen und der EACEA, und die Mitarbeiter beider Institutionen erkennen an, dass die Zuständigkeiten in einigen Bereichen, z. B. bei der Sammlung und Nutzung von Kenntnissen, die bei Projekten gewonnen werden oder die für die Programme relevant sind, geteilt sind.

Empfehlungen

Obgleich sich die Evaluierung im Wesentlichen auf die allgemeinen Rahmenbedingungen für die Übertragung von Programmemanagementaufgaben auf die EACEA konzentriert, sind einzelne Empfehlungen entweder an die EACEA oder an die Kommission oder an beide Institutionen gerichtet.

1. Die EACEA sollte für Möglichkeiten der Förderung einer größeren Personalmobilität zwischen

Programmeinheiten sorgen. Die Kommission und die EACEA sollten gemeinsam Möglichkeiten der Entwicklung von Karrierepfaden mit anderen EU-Organen, wie z. B. den anderen Exekutivagenturen prüfen.

2. Für den kommenden Programmplanungszeitraum sollten die zuständigen Generaldirektionen und die EACEA

gemeinsam Möglichkeiten prüfen, die es der EACEA erlauben würden, ihre Arbeitsbelastung gleichmäßiger über das ganze Jahr zu verteilen, indem eine engere Abstimmung der Zeitrahmen verschiedener Programme angestrebt wird.

3. Die EACEA sollte weiterhin nach Möglichkeiten suchen, die Abläufe noch weiter zu straffen, insbesondere im

Hinblick auf die späteren Phasen des Projektablaufs wie Überwachung, Kontrolle und Berichterstattung. Ferner sollte sie prüfen, welche Möglichkeiten der Übergang zur neuen Programmgeneration ab 2014 präsentiert. Eine künftige Evaluierung sollte prüfen, inwieweit beides erreicht wurde und Nachweise für dadurch erzielte Effizienzgewinne zusammentragen.

4. Die EACEA sollte bei der Planung künftiger Verbesserungen in den Abläufen Möglichkeiten prüfen, wie die

Rückmeldung an nicht erfolgreiche Antragsteller verbessert werden kann. Dies könnte zunächst bedeuten, dass die Erfahrungen aus Programmen, bei denen ein sehr hoher Grad an Zufriedenheit berichtet wurde, auf die anderen Programme übertragen werden, wenn dies machbar ist.

5. Die EACEA sollte auch weiterhin um Verbesserungen bei der Antragstellung und den Informationsmitteln

bemüht sein, insbesondere in Bezug auf das Layout und das Format von elektronischen Formularen und das Auffinden von Informationen auf der Internetseite und in Programmführern.

6. Die Kommission sollte die Überarbeitung ihrer Überwachungsstrategie vordringlich behandeln, und die EACEA sollte Fortschritte bei der Implementierung der Überwachungsstrategie als jährlichen Tagesordnungspunkt auf der Agenda des Lenkungsausschusses der Exekutivagentur und in ihre jährliche Tätigkeitsberichte aufnehmen.

7. Die Kommission sollte sicherstellen, dass das jährliche Arbeitsprogramm der EACEA zu einem frühen

Zeitpunkt in jedem Jahr angenommen wird.

xiii

8. Um die Wirksamkeit der externen Experten aufrechtzuerhalten, sollte die EACEA die Abläufe weiterhin überprüfen und die zuständigen Generaldirektionen mindestens einmal jährlich in Bezug auf die Eignung der Experten und die Qualität ihrer Beratung und Empfehlungen konsultieren.

9. Die Auswirkungen der Durchführung einer größeren Anzahl von Briefings (und Beurteilungsgesprächen) mit externen Experten auf Entfernung sollte auch weiterhin von der Kommission und der EACEA überwacht werden, um sicherzustellen, dass die Qualität der Projektevaluierungen gleich bleibt.

10. Die Kommission und die EACEA sollten Möglichkeiten einer stärkeren gemeinsamen Nutzung der praktischen Kenntnisse, die die Mitarbeiter der EACEA durch ihre Nähe zu den Begünstigten und durch ihre Beteiligung am täglichen Management von Programmen erworben haben, prüfen. Im Einklang mit den Ergebnissen zu den Überwachungsverfahren könnte dies durch eine Anhebung der Qualität der Informationen, die der Kommission in Bezug auf Ergebnisse von Vor-Ort-Überwachungsbesuchen bereitgestellt werden, gefördert werden.

xiv

Résumé Objet de l’évaluation Fondée en 2005, l’agence exécutive “Education, Audiovisuel et Culture” (EACEA) a, début 2006, assumé la responsabilité de la gestion de plusieurs programmes européens dans les domaines de l’éducation, de la formation professionnelle, de la culture, de l’audiovisuel, de la citoyenneté et de la jeunesse (auxquels se sont ajoutées des collaborations externes dans certains de ces domaines). L’EACEA est l’une des six agences exécutives (de tailles diverses) fondées à ce jour, et joue le rôle d’organe de délégation de trois Directions générales de la Commission, responsables des programmes : Education et Culture (DG EAC), Communication (DG COMM) et Développement et Coopération – EuropeAid (DG DEVCO). Les principales responsabilités de l’EACEA concernent la gestion des programmes tout au long de leur cycle de vie, y compris le lancement et la mise en œuvre des procédures associées aux bourses et des appels à propositions, l’évaluation des offres et la sélection des projets, la communication avec les bénéficiaires de fonds, l’obtention des engagements financiers, l’élaboration et la modification des contrats, le suivi, les paiements, la réception des rapports finaux, et les activités ex-post tels que le suivi des missions et des audits. L’EACEA contribue également de manière importante au succès de la gestion des programmes, par le biais d’activités telles que la sensibilisation et la communication d’informations aux candidats potentiels sur les opportunités de financement, la collaboration avec les structures nationales (ex : points de contact), la communication d’informations sur les projets et d’un feedback qualificatif auprès des DG, ainsi que la communication des résultats de projet. L’EACEA a aussi assumé des responsabilités et développé ses compétences en matière de soutien aux politiques, principalement grâce à la délégation d’Eurydice (en 2007)5 .

En 2011, l’EACEA employait 433 personnes réparties entre dix unités thématiques/de programme et trois unités horizontales/administratives. Son budget opérationnel dédié aux crédits d’engagement s’élevait à 621 millions d’euros6.

Objectif de l’évaluation

Cette évaluation satisfait les exigences du Règlement du Conseil (EC) No 58/2003, selon lequel une évaluation externe de chaque agence exécutive doit être effectuée tous les trois ans. C’est la deuxième évaluation de l’EACEA, couvrant la période du 1er mai 2008 au 31 décembre 2011. La première, publiée début 2009, a couvert la période de janvier 2006 à avril 2008. Conformément aux termes de référence, l’évaluation indique les progrès par rapport aux recommandations faites lors de la première évaluation et propose un chapitre décrivant les événements depuis la première évaluation. Le principal élément d’analyse est une évaluation des coûts et des avantages associés à la délégation de la gestion de programmes à l’EACEA. L’évaluation incorpore des données financières et d’autres types d’informations, ainsi qu’un élément rétrospectif comparant les prévisions effectuées lors de l’évaluation précédente avec la situation réelle à la fin de la période actuelle. Ces informations sont regroupées pour permettre de tirer des conclusions et faire des recommandations pour le futur.

5 Cela inclut la gestion d’un réseau d’unités nationales Eurydice, la commande d’études ainsi que la collecte et l’analyse d’informations comparatives dans le domaine de l’éducation et de la politique pour les jeunes, ainsi que l’organisation d’événements et la divulgation d’informations. Ces activités impliquent un niveau élevé de collaboration avec les structures nationales adéquates et d’autres acteurs (y compris des gouvernements nationaux et des instituts de recherche), ainsi que des fournisseurs d’informations (ex : Eurostat). La délégation de Tempus en 2009 a généré d’autres tâches de soutien aux politiques, bien que cela représente une part relativement faible des ressources de programmes. 6 En 2013, une extension du mandat de l’EACEA a fait passer le budget opérationnel à 896m €, et le nombre d’employés à 453.

Résumé

xv

Principales conclusions

L’EACEA a atteint ses objectifs. L’agence a progressé en termes d’efficacité de la gestion des programmes et de services aux candidats et bénéficiaires. Des progrès ont été réalisés dans le domaine des priorités annuelles définies dans les programmes de travaux annuels de l’agence : les plus probants étant ceux liés à la rationalisation des processus de candidatures et de sélection. Les progrès en matière de mise en œuvre des projets et de reporting sont moins notables. Toutefois, rappelons que ces derniers points ont été signalés comme éléments à améliorer récemment (d’autres améliorations sont prévues pour la prochaine génération de programmes).

Pertinence du rôle, de la structure et du fonctionnement

Le rôle de l’EACEA en tant qu’organisme de gestion de programme dédié est utile aux DG de tutelle et constitue une amélioration par rapport à l’approche précédente (délégation au Bureau d’assistance technique - TAO). L’EACEA fonctionne conformément au cadre juridique adéquat. L’augmentation récente du nombre et de la taille des programmes dédiés à l’EACEA confirme la nécessité d’une agence dédiée déployant des spécialistes pour gérer divers programmes. L’EACEA s’est révélée suffisamment souple pour satisfaire les DG de tutelle qui avaient besoin de nouveaux services de gestion de programme, parfois à relativement brève échéance. L’évaluation a constaté une amélioration dans le niveau d’expertise technique et financière du personnel de l’EACEA, alors même que le nombre d’employés a augmenté. D’autres améliorations pourraient être apportées en renforçant l’offre de formation, à condition de maximiser le taux de présence à ces stages.

Le taux d’occupation des postes a dépassé les objectifs depuis 2009 et le taux de rotation du personnel est relativement faible. Certains membres du personnel ont été recrutés après avoir eu un emploi temporaire à la Commission tandis que d’autres ont quitté l’EACEA pour un poste à la Commission après avoir réussi un concours européen. Bien que l’EACEA ait dû assumer le coût du remplacement de ces employés, il a servi de “pont” entre un emploi temporaire et permanent à ces personnes et a donc contribué à conserver leur expertise au sein des institutions de l’UE. L’EACEA pourrait encore améliorer la rétention du personnel et la capacité de l’agence à s’adapter aux changements en favorisant la mobilité du personnel entre les unités de programme et en explorant des manières de répartir sa charge de travail de manière plus équilibrée au cours de l’année (par ex. en coordonnant les délais plus étroitement avec la nouvelle génération de programmes). La rétention du personnel pourrait être meilleure en créant des ponts entre les différentes agences exécutives, permettant aux employés de progresser.

Services aux candidats et bénéficiaires

L’EACEA a bien progressé sur ses principaux indicateurs de résultat, notamment les délais de sélection de projets, l’attribution de bourses, la finalisation de contrats, les paiements, ainsi que le niveau des budgets opérationnels et administratifs établis et financés. Les parties prenantes ont dit à l’unanimité que le service fourni par l’EACEA représentait une amélioration significative par rapport aux systèmes précédents, tandis que les niveaux de satisfaction des candidats et des bénéficiaires sont positifs et sont demeurés relativement constants, malgré l’augmentation de la charge de travail de l’agence. La note de satisfaction moyenne en tenant compte de toutes les personnes interrogées et des programmes était de 3.1 (sur une échelle de 1 à 9, 1 étant la meilleure note), bien qu’il y ait des différences entre les programmes. Les candidats et bénéficiaires du Programme Culture, l’Europe aux Citoyens, et la Jeunesse en Action, ont attribué de moins bonnes notes que ceux du Programme pour l’Education et la Formation tout au long de la vie, MEDIA et Tempus. L’EACEA a bien amélioré la normalisation/l’harmonisation et la simplification des procédures, notamment à l’étape des candidatures, bien que cela n’ait pas conduit à une hausse des notes de satisfaction dans le questionnaire envoyé aux candidats et bénéficiaires. Certains avancent qu’il n’est pas réaliste de s’attendre à ce que les niveaux de satisfaction augmentent au même rythme que lors des premières années de l’EACEA, surtout dans un contexte où la concurrence pour les financements est plus vive (avec un nombre croissant de candidatures rejetées). L’aspect le moins positif de l’enquête de satisfaction porte sur la qualité du feedback donné sur les candidatures

xvi

rejetées, décrit comme trop court, trop standardisé et insuffisamment détaillé sur les points forts et les points faibles des candidatures. La faculté de l’EACEA de minimiser la charge administrative sur les candidats est nécessairement limitée par les exigences du Règlement financier ou la conception des programmes ; et nombre des frustrations des candidats sont dues à des problèmes hors du contrôle direct de l’EACEA. Néanmoins, selon les candidats, la charge administrative que demande l’EACEA est, dans l’ensemble, raisonnable. Les candidats reconnaissent que les processus administratifs sont bien meilleurs, bien qu’ils puissent être encore améliorés comme la présentation et le format des formulaires électroniques, la place des informations sur le site Web ou dans les guides de programme, et les informations que les bénéficiaires doivent fournir dans leurs rapports finaux.

En termes de disponibilité et de réactivité vis-à-vis des bénéficiaires, l’EACEA affiche de bons résultats. Les candidats sont très satisfaits de savoir qui contacter et de la capacité de l’EACEA à répondre aux questions dans des délais raisonnables. L’EACEA a également augmenté de manière significative le nombre de visites de suivi et amélioré les divers outils de communication qu’elle utilise. En outre, le contact direct de l’EACEA avec les bénéficiaires et sa gestion quotidienne des programmes ont permis à son personnel de bien connaître les politiques liées aux domaines couverts par ses programmes. Ces connaissances représentent un atout précieux qui aide la Commission à remplir son rôle de décideur de politiques.

Le coût de la gestion et de l’administration des programmes

L’externalisation de la gestion des programmes continue de diminuer les coûts de gestion interne de la Commission, en grande partie grâce au déploiement d’employés dont le salaire est plus bas que dans les DG. Les postes créés à l’EACEA ont remplacé les postes au TAO (qui n’existe plus) ou représentent des tâches entièrement nouvelles (en raison de l’expansion des programmes). Cela suggère que les économies de coût potentielles de l’externalisation ont été réalisées, et devraient dépasser les prévisions pour 2008-2011 figurant dans la première évaluation de l’EACEA. Les économies prévues pour la période 2008-2011 étaient d’environ 43m €, tandis que les calculs basés sur les coûts réels suggèrent une économie de 93m € pour la même période. Pendant la période couverte par l’évaluation, le personnel de l’EACEA a augmenté conformément à l’accroissement de ses responsabilités, du volume de fonds gérés et du nombre de candidatures reçues. L’EACEA a pu absorber un grand nombre de nouvelles recrues tout en évitant une baisse de l’efficacité du personnel (mesurée en termes de charge de travail moyenne par personne) et l’augmentation des coûts de personnel liés aux fonds de programme gérés, qui demeurent inférieurs à 5 % du budget opérationnel total. Favoriser de meilleurs projets et résultats de projet

Les informations sur l’exécution des budgets et le niveau de concurrence à l’étape de sélection confirment le point de vue des parties prenantes selon lesquelles les propositions de bonne qualité n’ont pas manqué. Les processus de sélection de projets sont bien définis, clairs et transparents et il ne semble pas nécessaire de les améliorer davantage. La sélection, la formation et le suivi des experts externes jouent un rôle critique dans le choix de projets de bonne qualité. Etant donné la décision d’effectuer les briefings des évaluateurs externes et les réunions d’évaluation de projet à distance, cet aspect du travail de l’EACEA doit continuer à être étudié pour garantir le maintien des niveaux de qualité.

Les processus de suivi et de contrôle de l’EACEA sont solides même si, comme nous l’avons indiqué plus haut, les progrès dans le domaine des procédures de candidature et de sélection ont été supérieurs à ceux réalisés dans le suivi et le reporting. Toutefois, des améliorations dans ces domaines (comme l’e-reporting et l’utilisation de certificats d’audit) ont fait l’objet de programmes-pilotes et sont en cours de mise en œuvre pour les divers programmes.

Le suivi porte principalement sur les aspects ‘techniques’ et financiers de la mise en œuvre des programmes. Toutefois, il joue également un rôle clé dans l’identification des bonnes pratiques et la divulgation des résultats.

xvii

L’EACEA a davantage de contacts avec les bénéficiaires et effectue notamment plus de visites de suivi sur le terrain. Il n’est pas sûr néanmoins que les conclusions de ces visites soient toujours partagées avec les DG de tutelle. A chaque fois que ces conclusions ne sont pas partagées, c’est une occasion manquée d’exploiter au mieux les données recueillies.

Malgré les efforts de l’EACEA pour assurer la clarté de la documentation officielle, on sait que les candidats et bénéficiaires ne comprennent pas toujours bien le rôle de l’EACEA et des DG de tutelle, même s’ils savent bien qu’elles font toutes partie de la Commission européenne.

Permettre à la Commission de remplir son rôle politique

La coordination entre l’EACEA et la Commission fonctionne généralement bien. Les représentants des deux parties sont satisfaits dans l’ensemble. Aux outils officiels de coordination (Comité de direction, Réunions de coordination, modalités de coopération et reporting régulier), s’ajoute une volonté de coopérer de manière informelle, à la fois quotidiennement et à des occasions spécifiques, comme pour des événements publics. Bien qu’un rapport de 2009 par la Cour des Auditeurs ait conclu que la supervision des agences exécutives par la Commission était relativement limitée, il n’a pas indiqué de non-conformité au cadre juridique. La stratégie de surveillance révisée de la Commission devrait résoudre les principaux problèmes signalés par le rapport, même si certaines difficultés resteront à résoudre, comme l’adoption tardive des programmes de travail annuels de l’EACEA. La circulation des informations entre l’EACEA et les DG de tutelle s’est améliorée. Le personnel des DG de tutelle reçoit généralement les informations dont il a besoin de manière automatique ou sur demande. Le fossé inévitable créé par la séparation du rôle politique de la Commission et de la gestion des programmes tend à être comblé par des mécanismes de communication formels et informels, des outils pratiques, un certain degré de flexibilité des deux côtés et l’instauration progressive de la confiance et de la coopération entre les personnes impliquées. Toutefois, des représentants des DG de tutelle ont indiqué qu’il était possible d’améliorer le flux des informations qualitatives de l’EACEA, en termes de volume et de contenu, afin que les DG de tutelle puissent profiter davantage des connaissances pratiques détenues par l’EACEA, par exemple les résultats des visites de suivi sur le terrain comme mentionné plus haut.

Bien que ce ne soit pas le principal moteur de la ‘solution agence’, la délégation à l’EACEA devrait permettre au personnel de la Commission de mieux se concentrer sur ses principales tâches institutionnelles. Les rôles et responsabilités des DG de tutelle et de l’EACEA sont bien compris. Le personnel des deux institutions reconnaît un partage des responsabilités dans certains domaines, comme par exemple la collecte et l’exploitation des connaissances émanant des projets ou utiles aux programmes.

Recommandations

Bien que l’évaluation porte essentiellement sur la délégation de la gestion des programmes à l’EACEA, les recommandations sont destinées soit à l’agence, soit à la Commission ou encore aux deux institutions.

1. L’EACEA devrait réfléchir à des manières de promouvoir une plus grande mobilité du personnel entre les

unités de programme, tandis que la Commission et l’EACEA devraient rechercher ensemble des façons de développer les progressions de carrière avec d’autres organismes de l’UE telles que les autres agences exécutives.

2. Dans la période à venir, les DG de tutelle et l’EACEA devraient explorer ensemble des moyens de permettre à

l’EACEA de répartir sa charge de travail de manière plus équilibrée sur l’année, en cherchant une meilleure coordination des calendriers des différents programmes.

xviii

3. L’EACEA devrait continuer à rechercher des manières d’améliorer encore la rationalisation des processus, notamment ceux liés aux étapes ultérieures du cycle de vie de projet tels que le suivi, le contrôle et le reporting, ainsi que les opportunités que présente la transition vers la nouvelle génération de programmes à partir de 2014. Toute évaluation future devrait étudier dans quelle mesure ces objectifs ont été atteints et indiquer les économies générées.

4. L’EACEA devrait réfléchir à des techniques pour améliorer le feedback fourni aux candidats rejetés afin

d’améliorer ses processus. Quand cela est possible, elle pourrait appliquer aux autres programmes les leçons apprises des programmes pour lesquels les candidats sont le plus satisfaits du feedback.

5. L’EACEA devrait continuer à améliorer ses outils d’application et d’information, en particulier la présentation et

le format des formulaires électroniques ainsi que la place des informations sur le site Web et dans les guides de programme.

6. La Commission devrait accorder la priorité à la révision de sa stratégie de supervision tandis que l’Agence devrait l’inclure une fois par an à l'ordre du jour du Comité de Direction de l’EACEA et ses rapports d’activité annuels.

7. La Commission devrait prendre des mesures pour s’assurer que le programme de travail annuel de l’EACEA

est adopté tôt dans l’année.

8. Afin de maintenir l’efficacité des experts externes, l’EACEA devrait faire évaluer les procédures et consulter les DG de tutelle au moins une fois par an sur la qualité des conseils des experts.

9. Les répercussions de l’augmentation des briefings à distance des experts externes ainsi que des réunions d’évaluation à distance doivent continuer à être surveillées par la Commission et l’EACEA afin de s’assurer du maintien de la qualité des évaluations de projet.

10. La Commission et l’EACEA devraient réfléchir aux moyens d’accroître le partage des connaissances pratiques acquises par le personnel de l’EACEA grâce à ses contacts avec les bénéficiaires et à son implication dans la gestion quotidienne des programmes. Conformément aux conclusions sur les procédures de suivi, cela pourrait se faire en améliorant la qualité des informations communiquées à la Commission sur les résultats des visites sur le terrain.

1

1 Introduction 1.1 Context, purpose and scope

The Education, Audiovisual and Culture Executive Agency (EACEA) was established in 2005 and at the beginning of 2006 assumed responsibility for the management of a number of EU programmes in the fields of education and vocational training, culture, audiovisual, citizenship and youth (plus external collaborations in some of these fields).

This evaluation is designed to satisfy the requirements of Council Regulation (EC) No 58/2003, which requires an external evaluation of each Executive Agency to be undertaken every three years. This will be the second such evaluation, building on the first interim evaluation published in early 2009 which covered the period from January 2006 to April 2008. This evaluation covers the period from 1 May 2008 to 31 December 2011.

1.1.1 Requirements

The Terms of Reference (ToR) define the requirements of the evaluation as follows:

1 To provide a synthetic description of what has happened during the evaluation period defined above, with a comparison to the situation at the end of the previous evaluation period, as well as the situation before the activities were externalised to EACEA to the extent still relevant (primarily the activities that have been externalised at a later stage, particularly those that did not fall within the scope of the previous evaluation). The description should provide the necessary background and reference points for Task 4 below.

2 To carry out a retrospective Cost-Benefit Analysis (CBA), focusing on a comparison between the assumptions made in the previous CBAs carried out prior to the creation of EACEA and each extension of its mandate as relevant, and the situation of today. The following issues, as specified in Council Regulation (EC) No 58/2003, should be covered by the CBA:

a. Effectiveness (in particular: the time to award and pay; client satisfaction globally) b. Efficiency (human resources needed at EACEA and the Commission, administrative cost including the

cost for coordination and checks) c. Possible savings (overall) d. Flexibility in the implementation of the outsourced tasks e. Simplification of the procedures used (in particular the extent of widening the scope of use of flat-rate

financing, electronic submission, etc.) f. Proximity of outsourced activities to final beneficiaries g. Visibility of the European Union as promoter h. Level of know-how retained inside the Commission (in particular the adequacy and type of information

provided to the parent DGs) i. Level of financial and technical expertise available at EACEA j. Exploitation of results achieved by the projects managed by EACEA7

3 To assess the implementation by EACEA and parent DGs of the actions identified in its Action Plan, which

was elaborated to respond to the recommendations put forward in the final report of February 2009 on EACEA's first interim evaluation.

7 Issue added by the Guidelines for Executive Agencies

1. Introduction

2

4 To draw conclusions and provide judgements and recommendations for the future on the basis of the evaluation questions listed under the next section. The tenderers should feel free to suggest additional questions, as appropriate, within the overall purpose of the evaluation.

1.1.2 Evaluation questions

The ToR also specify a set of evaluation questions.

Table 1.1 Evaluation questions Evaluation Questions Q.1: To what extent is EACEA relevant to the needs it is intended to meet? Q.2: To what extent is EACEA operating according to the legal framework establishing it? Q.3: To what extent has EACEA achieved its objectives? Q.4: To what extent has EACEA led to an improved management of the programmes and to better services for stakeholders and addressees as compared to the alternative options8? Q.5: To what extent is the coordination between EACEA and the Commission services (including the parent DGs, relevant horizontal services and offices) working satisfactorily? Q.6: To what extent has EACEA carried out its work efficiently9? Q7: To what extent does the procedure put in place for the selection of proposals ensure (a) that the projects of the best quality are funded, and (b) an appropriate level of transparency? Could it be further improved under the current framework? Q.8: To what extent have the mechanisms applied by EACEA for monitoring of projects ensured a proper implementation of the programmes? Q.9: To what extent have EACEA's internal organisation and procedures been conducive to its efficiency? Q.10: To what extent has EACEA enabled the Commission to better focus on its institutional tasks? Q.11: To what extent has the Commission, in the presence of EACEA, been able to maintain an adequate level of know-how in relation to the programmes entrusted to EACEA? How was this achieved?10 Q.12: To what extent have the activities of EACEA resulted in unintended effects (both desirable and undesirable)?

1.2 Approach to the evaluation

In order to approach the evaluation logically and structure both the analysis and the findings which flow from it, our approach has been to start with the definition of an overall objective for executive agencies, which in turn links to five specific objectives. We have understood the main purpose of the evaluation as being to consider the extent to which the arrangements for delegating programme management tasks to EACEA have supported the overall objective behind outsourcing to executive agencies. This has been defined during the course of the evaluation as:

"To help achieve the goals of Community programmes more efficiently and effectively"

8 Examples of alternatives to the Executive Agency: (i) management of the programme(s) by the Commission, (ii) mixed agency-Commission management, (iii) partial management by the Commission while outsourcing some activities to the extent legally possible (cf. the scenario's in section 6 of the 'Legislative Financial statement for the setting up or extension of the lifetime of an Executive Agency' http://www.cc.cec/budg/pre/legalbasis/pre-040-020_preparation_en.html#forms). 9 According to Art.3 (1) of Regulation No 58/2003, the following aspects linked to the efficiency of the agency should be analysed: possible savings within EU budget, costs of coordination and checks, efficiency in the implementation of outsourced tasks. 10 Art. 3(1) of Regulation No 58/2003 requires for the analysis of the need to maintain an adequate level of knowhow inside the Commission.

3

While the goals of EU programmes vary from one programme to the next, it is possible to define a set of specific objectives that relate to and cut across all the programmes managed by EACEA. These specific objectives are to:

Remain relevant and fit-for-purpose, in terms of role, design and set-up (Relevance) Improve the service to applicants and beneficiaries (Effectiveness and Efficiency) Reduce the cost of programme management and administration (Efficiency) Facilitate better projects and project outcomes (Effectiveness) Enable the Commission to fulfil its political role vis-à-vis Community programmes (Effectiveness)

1.2.1 Methodological framework

Building on this list of objectives, we have defined a coherent framework by which to structure the issues that will need to be covered by the evaluation, including those relating to the Cost-Benefit Analysis (CBA) and those that have been added during the course of the evaluation11. The table below shows how the issues under consideration can be grouped against the five specific objectives and also how the issues link to the twelve evaluation questions (EQs) listed in the ToR. By exploring each of the issues in turn, we have been able to generate answers to the twelve EQs.

Table 1.2 Linking objectives, CBA issues and Evaluation Questions Specific objectives Issues for CBA Evaluation Questions Relevance of role, design and set up

Relevance to need Flexibility in the implementation of outsourced tasks Level of financial and technical expertise at EACEA Appropriate internal organisations and procedures

EQ1, EQ2

Efficiency and effectiveness: Improved the service to applicants and beneficiaries

Effectiveness (time taken, client satisfaction globally, etc.) Administrative burden for applicants Simplification of procedures used Proximity of outsourced activities to final beneficiaries

EQ4, EQ12

Efficiency: Reduced the cost of programme management

Efficiency (resources needed at EACEA, DGs) Possible savings overall

EQ6, EQ9, EQ12

Effectiveness: Facilitated better projects and project outcomes

Quality and relevance of selected projects Transparency in selection Effectiveness of monitoring of projects and programmes Visibility of the EU as promoter Exploitation of results achieved by projects

EQ3, EQ4, EQ7, EQ8, EQ12

Enabled the Commission to fulfil its political role

Level of know-how retained (including flow of information and Commission role/activity) Improved focus of Commission on its institutional tasks

EQ5, EQ10, EQ11, EQ12

The common methodological framework has utilised a range of overlapping data sources for each of the four tasks highlighted above, with the data put to a different use in each case.

1.2.2 Data gathering

The research tasks have been organised into the following main phases.

11 Cost-Benefit Assessment is usually undertaken ex-ante, this evaluation instead focuses on verifying the assumptions made in previous CBAs, a process described in the Terms of Reference as “retrospective CBA”,

4

1. Inception phase. Kick-off meeting with project steering group, initial research and refinement of the evaluation framework, methodology and work-plan, initial consultations with staff in parent DGs and EACEA, submission of Inception Report and inception meeting with project steering group.

2. Main research phase. Review of literature and data provided by EACEA, available online or obtained during the interviews. This includes information relating to the programmes administered by EACEA and the functioning of the Agency. A full list is provided as an annex.

3. Main research phase. Semi-structured interviews of staff from EACEA, parent DGs and a sample of National Agencies or contact points. Topic guides and a full list of interviewees are provided in the annexes.

4. Main research phase. Online survey of applicants and beneficiaries. 4,395 organisations applying to EACEA for funding between 2008 and 2011 responded to a request to take part in the online survey (17% of nearly 26,000 email invitations delivered). A more detailed introduction on the approach to and results of the survey is provided in section 1.2.3, while information from the survey has been incorporated throughout this report. Full breakdowns by question are available as a separate annex.

Reporting phase Production of interim technical report and attendance at steering group meeting, preparation of draft final report and final reports.

1.2.3 Strengths and weaknesses of the evidence base

This approach to data collection resulted in the assembly of a substantial amount of information of various types from a range of different sources (including literature, data, interview write-ups and the results of the online survey). In general terms our approach has been to bring all this information together in a single analytical framework, categorising information not by format but by its ability to inform answers to the key issues and evaluation questions. In this way we have been able to avoid over-emphasis or bias (particularly in relation to the qualitative information) by verifying potential findings against more than one source of information. Wherever information from one source is used to provide additional detail this is made clear in the text.

Our main focus has been to evaluate the overall arrangements for delegating programme management to EACEA, and to highlight specific aspects of programme management or differences between delegated programmes wherever possible. The cooperation of EACEA and the relevant DGs (as well as national agencies and contact points) in both providing us with information and making time for interviews has been a key factor in the generation of a comprehensive evidence base for the evaluation. The willingness of applicants and beneficiaries to spend time completing the online survey has provided a rich source of both quantitative and qualitative information.

In common with most evaluations of this type, most of the qualitative evidence reflects “internal” views, in this case staff of EACEA and the parent DGs who are involved with programme management. “External” views are reflected through the online survey (and to a certain extent the interviews of national agencies and contact points). For some of the key issues (e.g. for questions relating to the administrative burden) it would also be helpful to have the views of potential applicants (e.g. those put off from applying for funding for any reason), though the lack of data on this population means there is no clear way to include them in the research.

1.2.4 Online survey of applicants and beneficiaries

The online survey of applicants and beneficiaries is one of the key sources of primary, quantitative evidence for the evaluation, especially in the areas of:

effectiveness of services

impact of simplifications

proximity and responsiveness

quality of information tools

5

transparency of processes

The survey asked respondents a number of questions that were directly comparable with previous surveys (carried out in 2008 and 2010), allowing us to look for evidence of trends in satisfaction levels over time and link the views of applicants and beneficiaries to any changes implemented by EACEA. Respondents were also able to make their own comments on the service provided by EACEA and these responses have also been analysed to add further qualitative information to the evaluation.

In order to ensure compatibility with previous (and future) surveys, many of the questions were formulated as statements, with respondents asked whether and to what extent they agreed with each using a sliding scale. Responses range from 1 (strongly agree) to 9 (strongly disagree) with a response of 5 indicating they neither agree nor disagree. This means that lower average scores are more positive, as respondents will have agreed more strongly with the statement concerned. A full set of questions and responses is provided in the annexes.

The online survey was launched on 21 March 2012 and closed on 24 April. EACEA provided details of 39,543 applicants between 2008 and 2011, including 40,877 e-mail addresses. These were cleaned to remove errors and duplication to create a panel of 28,847 unique e-mail addresses for the survey. Some 3,163 e-mails "bounced" (11% of the total) and of those that did not (25,684), the response was as follows:

Declined: 109 (0.4%) Responded: 4,395 (17.1%)12 Of the 4,395 respondents, the breakdown is as follows:

Screened out13: 53 (1.2% of responders; 0.2% of all those contacted) Partially completed: 366 (8.3% of responders; 1.4% of all those contacted) Fully completed: 3,976 (90.5% of responders; 15.5% of all those contacted)

The overall number and rate of responses exceed those of the previous satisfaction survey in 2010 and are sufficient for effective and robust analysis. The following broad patterns can be observed:

Over half of all respondents represented organisations that had submitted applications in 2011, with fewest responses from those applying in 2008 and 2009.

Responses are relatively well distributed by programme, with all except Intra-ACP Mobility (5 responses) and arguably the Bilateral Cooperation Agreements (37 responses) likely to produce sufficient numbers of responses for comparative analysis by programme. As a result, responses relating to Intra-ACP Mobility have been merged with those for Erasmus Mundus.

Responses are also well distributed geographically, with all EU Member States and participating European countries represented.

The largest group of respondents were representatives of educational institutions, which broadly reflects the profile of programmes managed by EACEA.

The following table shows the numbers of respondents and average scores for each of the main programme groups as well as the proportions of selected and non-selected14 applications in the survey for each. Although the differences between the average mean score for each programme appears relatively small, a test for statistical 12 The remainder, 21,180, did not provide any response to the email invitation 13 Qualifying questions were asked in order to ensure that all respondents were from the target group for this evaluation. Respondents who had not submitted an application to EACEA between 2008 and 2011 or for the relevant programmes were not asked to complete the survey and were directed to an alternative ‘thank you’ page 14 This is based purely on the online survey; specifically Q17 “Was your application successful?” This does not take into account the reasons an application was not selected, but seeks to illustrate the balance between selected and unselected applications (including by programme) in the final sample, and any differences in the pattern of responses.

6

significance15 shows that Culture, Europe for Citizens and Youth in Action produce less positive scores than Lifelong Learning and MEDIA (and possibly Tempus, though this is based on a smaller sample), and that these differences are not the result of random variation alone.

Table 1.3 Breakdown of survey results by programme and outcome of application Programme (All applicants and

beneficiaries) Selected Non-selected No Response

Respondents Average score (1-9)

% % (not included in %)

Bilateral Cooperation

37 3.2 82% 18% 3

Culture 478 3.3 66% 34% 31 Erasmus Mundus & Intra-ACP

209 3.1 56% 44% 9

Europe for Citizens

791 3.3 64% 36% 20

Lifelong Learning

1642 3.0 72% 28% 85

MEDIA 511 3.0 72% 28% 45 Tempus 196 3.0 51% 49% 7 Youth in Action

425 3.3 66% 34% 25

No Response 106 3.8 51% 49% 61 Total 4395 3.1 67% 33% 286

Perhaps the most significant difference in responses is between selected and non-selected applications. Although the average scores for both selected (2.9) and non-selected (3.8) applicants are positive on the scale of 1 to 9 used for the survey, there are differences in the way responses are distributed and this is demonstrated in the chart below.

15 Statistical significance testing is used to assess whether observations reflect a reliable pattern. The method applied here is the t-test for independent samples at a 5% significance level (i.e. there is at most a 5% chance of getting such a result purely by chance, given the number of responses and the difference between mean scores).

7

Figure 1.1 Distribution of responses by outcome of application

Base: Online survey (all respondents)

While there is a similar overall pattern, those whose application was selected are more likely to give answers at the more positive end of the scale (1 or 2) and less likely to give answers at the most negative end of the scale (9). It is also worth noting that selected applicants were much more likely than non-selected applicants to complete the survey. Selected applications compose 67% of the survey sample, while the success rate across actions managed by EACEA is 36% (see table 2.5)16.

Data from the survey is mainly utilised to inform answers to the key issues for the CBA (section 4) which also correspond to the main aspects of programme management. The following table shows aggregate results for these key issue areas broken down by programme. This has also been tested for statistical significance and the gaps between the programme scores coloured red and green are statistically significant, based on the number of responses and mean scores. For example the difference in the average rating of administrative burden by applicants and beneficiaries under the Culture Programme is statistically significant compared to that provided under Europe for Citizens, Lifelong Learning, MEDIA, Tempus and Youth in Action. The results for individual questions are used to inform the cost-benefit analysis (section 4) and are broken down by programme: where differences between the average scores for programmes are statistically significant this is highlighted in the accompanying text. Where the differences between scores are statistically significant this means that they cannot be explained by random variation alone, and that there are likely to be other factors at play.

It has also been possible to undertake a regression analysis17, looking for correlations between the results for Q32 “We are satisfied with the overall quality of the programme management services provided by the Executive Agency during the whole project period” and the responses for each of the main aspects of programme management listed in table 1.4. This shows that most of the variance in the score for Q32 (71%) is driven by three areas in descending order: proximity (responsiveness); effectiveness of monitoring (and reporting) and (assistance with) dissemination. This suggests that any further improvements in these areas would be most likely to generate an improved overall satisfaction score in future.

16 While it is also possible to weight the survey results to reflect this, it would mean that results would not be directly comparable with the previous satisfaction surveys. 17 Step-wise regression analysis based on Pearson correlation and two-tailed significance testing

0%

5%

10%

15%

20%

25%

30%

35%

1 2 3 4 5 6 7 8 9

Responses (1=strongly agree, 9=strongly disagree)

Non-selected

Selected

8

Table 1.4 Average responses by programme, grouped by aspects of programme management (CBA sections) Aspect of programme management

Bilateral Cooperation18

Culture Programme

Erasmus Mundus & ACP

Europe for Citizens

Lifelong Learning MEDIA Tempus

Youth in Action

Administrative burden (Q28) 4.46 4.28 3.82 3.32 3.67 3.67 3.33 3.59

Dissemination (Q31) 4.00 4.23 3.98 3.64 3.53 3.18 2.98 4.25

Effectiveness (Q9,Q11,Q32) 3.12 2.99 2.69 3.01 2.96 2.80 2.86 2.91

Effectiveness of monitoring (Q27,Q29) 3.71 3.74 3.38 3.18 3.15 3.05 3.01 3.50

Proximity – information (Q5,Q6, Q7,Q8,Q24,Q25,Q26) 2.97 2.88 2.66 3.01 2.64 2.68 2.46 2.73

Proximity – responsiveness (Q10,Q33,Q34,Q35) 2.85 3.38 3.03 3.27 2.81 2.78 2.88 3.35

Selection – transparency (Q12,Q18,Q19) 3.81 3.26 3.69 3.89 3.16 3.54 3.54 3.73

Simplification – E-forms (Q15,Q16) 3.38 3.40 3.04 3.49 3.11 3.01 3.34 3.57

Simplifications (Q37,Q38,Q39,Q40) 3.99 4.16 4.13 4.17 3.84 3.86 3.84 4.17

18 The numbers of responses for Bilateral Cooperation were too small to allow reliable significance testing

9

2 Synthetic description of activity (Task 1) 2.1 History 2.1.1 Creation of EACEA

The legal basis for the creation of Executive Agencies was established with Council Regulation (EC) No 58/2003. This process was driven by the increasing number of Community Programmes and their diversity (dealing with various fields, sets of beneficiaries and target audiences), which placed considerable pressure on staffing at the Commission and was intensified by the winding up of Technical Assistance Offices (TAOs). To date, six Executive Agencies (of varying size) have been established, dealing with a range of different programmes and initiatives and reporting to different Directorates General (DGs) of the Commission.

Executive Agencies were partly conceived as a way to make it easier for the European Commission to concentrate on its core ‘institutional tasks’ and maintain accountability to citizens, by delegating certain and defined tasks relating to the management of programmes. Such delegation was particularly important during a period in which numerous programmes were being created and renewed. Critically, it was argued that 'outsourcing certain management tasks could, moreover, be a way of achieving the goals of such Community programmes more effectively.'19 A key element in this was a desire to improve the management of programmes through greater professionalization, with agencies employing staff with specialist and technical skills in programme management. However the decisive factor in the creation of Executive Agencies was their potential to provide efficiency savings, verified through the application of Cost-Benefit Analysis (CBA) to any proposed outsourcing initiatives.

Executive Agencies are created and controlled by the Commission, which retains responsibility for ‘essential aspects’ of their structure, tasks, operation, budget system, staff, supervision (particularly governing bodies) and responsibility. They must be based in Brussels or Luxembourg and are established for a defined period of time, corresponding to the cycles of the multi-annual financial framework: EACEA is currently established until 2015, covering the 2007-13 generation of programmes with two years allowed for programme closure activities. Their actions are subject to supervision by the parent DGs and limited to executive and operational tasks under specific programmes and action lines, with costs fully covered under programme expenditure.

The European Education, Audiovisual and Culture Executive Agency (EACEA) was legally established in 200520 and became fully operational on 1 January 2006, when it assumed responsibility for the management of a number of programmes in the fields of education and vocational training, culture, audiovisual, citizenship and youth (plus external cooperation mainly in the field of education).

2.1.2 Programmes delegated in 2006

On 1st January 2006 EACEA took responsibility for managing a selection of programmes that had previously been managed mainly by various DGs of the Commission since 2000, with support from three Technical Assistance Offices (TAOs)21. As set out in Commission Decision 2005/56/EC, these included certain strands of the following programmes from the 2000-06 programming period:

19 COUNCIL REGULATION (EC) No 58/2003 of 19 December 2002 laying down the statute for executive agencies to be entrusted with certain tasks in the management of Community programmes. P1 20 COMMISSION DECISION of 14 January 2005 setting up the Education, Audiovisual and Culture Executive Agency for the management of Community action in the fields of education, audiovisual and culture in application of Council Regulation (EC) No 58/2003. (2005/56/EC) 21 Culture and Citizenship were managed in-house by the Commission

2. Synthetic description of activity (Task 1)

10

Second phase of the Socrates Community action programme; Second phase of the Leonardo da Vinci Community action programme; Youth Community action programme; Culture 2000 programme; MEDIA Plus programme - Development, Distribution and Promotion; MEDIA Training programme; Programme for the enhancement of quality in higher education and the promotion of intercultural understanding

through co-operation with third countries (Erasmus Mundus); Programme for the effective integration of information and communication technologies (ICT) in education and

training systems in Europe (e-Learning); Community action programme to promote active European citizenship; Community action programme to promote bodies active at European level in the field of youth; Community action programme to promote bodies active at European level and support specific activities in the

field of education and training; Community action programme to promote bodies active at European level in the field of culture; and Projects in the field of higher education (funded under the provisions on aid for economic cooperation with

developing countries in Asia).

2.1.3 Programmes and activities delegated from 2007 onwards

Whilst EACEA continued to manage the legacy tasks of the programmes noted above in the post-2006 period, from 2007 it also took on responsibility for certain strands within the following new programmes

Lifelong Learning; Youth in Action; Culture; Europe for Citizens; MEDIA 2007; Erasmus Mundus (to 2008).

The portfolio of programmes and activities managed by EACEA in the current period has grown incrementally since 2007, including through the addition of:

Eurydice (from 2007); Erasmus Mundus 2009-2013 (from 2009); Tempus IV and the closure of Tempus III projects (from 2009); ; Projects in the field of higher education and youth funded under the provisions of the External Co-operation

Instruments (IPA, ENPI, DCI, ICI) (from 2007); Projects promoting bilateral co-operation with industrialised countries in the field of higher education (EU-USA,

EU-Canada and ICI-ECP) (from 2007); MEDIA II; Intra-ACP mobility scheme (2010).

The Agency’s mandate was established for the period up until 31 December 2015, by Commission Decision 2009/336/EC22 of 20 April 2009.

22 http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:2009:101:0026:0030:EN:PDF

http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:2009:101:0026:0030:EN:PDF

11

2.2 Design and set-up

2.2.1 Internal structure

EACEA is overseen and managed by a Steering Committee and a Director. Steering Committee members are appointed for two year periods, whilst the Director is appointed in principle for a period of four years. Importantly however, the Director's term should take into account the period over which the programmes must be delivered, helping to ensure that terms correspond to programming periods.

The Steering Committee is made up of five members appointed by the Commission. This body adopts the annual work programme, operating budget and Annual Activity Report, as well as measures, for example, to combat fraud or irregularities. The Director is appointed by the Commission, is an official of the European Commission and is responsible for:

Representing, supervising and managing the Agency, including human resource management;

Preparing the work of the Steering Committee;

Implementing the Agency's work programme;

Financial implementation of Community programmes;

Planning and management of the Agency's operating budget.

Preparing the Agency's reports, particularly the Annual Activity Reports.

For day-to-day operations, EACEA is split into ten thematic/programme units as well as three horizontal/administrative units. Four thematic/programme units (for Lifelong Learning and Eurydice) are directly managed by the Head of Department, with the remaining six thematic/programme units and horizontal/administrative units reporting to the Director. The figure below explains the structure of EACEA.23

23 http://eacea.ec.europa.eu/about/about_eacea_en.php

http://eacea.ec.europa.eu/about/about_eacea_en.php

12

Figure 2.1 EACEA organogram (January 2013)

2.2.2 Staff

EACEA staff belong to one of two main categories:

Temporary Agents (currently 25% of staff): staff with specialist skills or holding positions of responsibility, either permanent officials seconded from the Commission or temporary staff recruited externally by EACEA; and

Contract Agents (currently 75% of staff): staff with skills in implementation, administration or project and financial management, holding long-term or permanent contracts with EACEA.

EACEA also uses temporary staff supplied by employment agencies (“intérimaires”) to help address peaks in workload or fill vacant posts. Staffing is planned and implemented according to programme area and size. Since 2009, planned levels of staffing have been broadly stable at around 430-44024. Most recently, the planned staffing for 2011 has been as follows25:

24 In 2013 staffing will reach nearly 450 due to an extension of the Agency's mandate that will allow it to manage larger budgets for some existing programmes and one additional programme, MEDIA Mundus 25 EACEA Annual Work Programme 2011

13

Table 2.1 EACEA staff by programme and staff category (2011) Activities Temporary

staff Contractual

agents Sub total AC enlargement

and bilateral agreements

(Switzerland)

Total

LLP (including Eurydice) 30 96 126 7 133

Erasmus Mundus 10 31 41 3 44

Bilateral agreements 1 4 5 0 5

Tempus 6 14 20 0 20

MEDIA/Audiovisual 12 56 68 4 72

Culture 6 20 26 2 28

Youth 6 18 24 2 26

Citizenship 5 23 28 0 28 Administrative support, coordination and communication26

26 52 78 4 82

Total 102 314 416 22 438

Source: EACEA Work Programme 2011

2.2.3 Internal operations

A key challenge has been the fact that the different programmes feature different systems, procedures and ways of working, which reflected their management by different DGs and different TAOs prior to the creation of EACEA. Over the period covered by the evaluation, EACEA has taken steps to strengthen its internal management and introduce more consistent operating procedures across the organisation. This has included:

Harmonisation of management procedures and processes across all programmes delegated to EACEA;

Continuous improvement in the quality of the IT systems and extensive efforts to harmonise them (or at least make them compatible) across the different programmes;

Prioritising the improvement of standards relating to appraisal and development of personnel, processes and procedures and financial accounting information;

Consolidation of procedures relating to minimum standards of internal control; such consolidation helped EACEA to achieve a 98% level of compliance with internal control baseline requirements in 2011;

Acceleration of the rate at which ex-ante and ex-post control checks of dossiers from previous years are carried out, leading to a reduction in the number of open dossiers; and

Development of stable ‘modalities of cooperation’ establishing working protocols with parent DGs.

2.3 Delegated credits and costs

One of the main roles of EACEA is to allocate funding to approved projects within the programmes concerned. Data from the Annual Activity Reports of EACEA highlights that such payments during the years covered by the evaluation were as follows:

26 This category includes the horizontal units of EACEA, such as human resources, finances, informatics, and logistics.

14

Table 2.2 EACEA operational budget by year (€m) Budget execution 2008 2009 2010 2011 Operational budget for commitment appropriations €488 €625 €599 €621

Operational budget for payment appropriations €406 €528 €568 €534

Source: EACEA

The administrative budget of EACEA is drawn from the financial allocation to the EU programmes that it manages. The breakdown for funding and expenditure is as follows:

Table 2.3 EACEA administrative budget by year (€m) Year 2008 2009 2010 2011 Income (credits) € 38.244 € 47.694 € 48.974 € 49.953 Expenditure – staff € 22.748 € 26.292 € 27.798 € 29.491 Expenditure - other € 14.988 € 20.082 € 21.086 € 20.364 Surplus € 0.508 € 1.321 € 0.090 € 0.098 Source: EACEA

2.4 Core Responsibilities

EACEA's core responsibilities have been related to the management of centralised elements of relevant programmes that involve '…implementation of technical projects which do not entail political decision-making and require a high level of technical and financial expertise throughout the project cycle.'27 These responsibilities are the tasks required for the management of programmes throughout their lifecycle, from launching and carrying out grant procedures and calls for proposals, evaluation of bids and project selection, communicating with funding recipients, obtaining financial commitments, setting up and modifying agreements and contracts, monitoring, payments, receiving final reports and ex-post activities such as monitoring missions and audits.

In addition to direct programme management duties, through the delegation of Eurydice and Tempus EACEA has acquired responsibilities and developed competences in policy support. This includes the commissioning of studies and the collection and analysis of content-related and comparative information in the areas of education and youth policy, as well as running events and disseminating relevant information. This involves a high degree of collaboration with the relevant national structures as well as other stakeholders and information providers (such as Eurostat).

EACEA is also mandated to carry out a set of duties linked to successful management of the delegated programmes, including raising awareness and informing potential applicants and beneficiaries about funding opportunities, collaborating with national structures (e.g. contact points), providing project-level information and qualitative feedback to the DGs and contributing to the dissemination of project results.

Over the period covered by the evaluation, there has been a steady increase in the number of programmes and scale of activities managed by EACEA. While the overall number of calls for proposals per year has remained broadly consistent, the number of individual actions under many of the calls has increased (especially in the case of LLP). When combined with the increasing numbers of applications, this highlights an increasing workload for EACEA over the course of the period covered by this evaluation. In fact, the increase in the number of applications may be linked to improvements introduced by EACEA (such as e-forms or better awareness through events such as Infodays), though it must be noted that the increased competition for funding results in lower success rates (and more unsuccessful applications) thus potentially creating greater levels of disappointment. In addition, as many

27 COMMISSION DECISION of 14 January 2005 setting up the Education, Audiovisual and Culture Executive Agency for the management of Community action in the fields of education, audiovisual and culture in application of Council Regulation (EC) No 58/2003. (2005/56/EC) p1

15

projects last for more than one year (e.g. up to five years in the case of some actions of the Culture Programme) the approval of new projects each year has served to increase the total number of projects being managed by EACEA at any one time. The tables below provide a breakdown of the relevant figures by year and by programme.

Table 2.4 Breakdown of applications and selected projects by year (all delegated programmes) Year No. of calls

for proposals No. of

applications Proportions

submitted online No. of selected

projects Success rate

2008 34 8,991 20% 3,892 38% 2009 31 11,176 25% 4,100 37% 2010 29 10,230 58% 3,473 34% 2011 29 12,687 67% 3,723 29%

Source: EACEA Annual Activity Reports

Table 2.5 Breakdown of applications and selected projects by programme, 2008 to 2011 Programme No. of

applications Proportion submitted

online

No. of selected projects

Success rate

Bilateral Cooperation Agreements 393 0% 121 31% Culture 4,060 33% 1,242 31% Erasmus Mundus & Intra-ACP 2,264 28% 863 38% Europe for Citizens 7,526 69% 2,572 34% Lifelong Learning 8,893 51% 2,385 27% LLP Erasmus Charter28 3,639 96% 2,549 70% MEDIA 12,073 11% 5,105 42% Tempus 2,113 46% 271 13% Youth in Action 6,016 17% 1,841 31% EACEA Total 46,977 40% 16,949 36% Source: Application data provided by EACEA Informatics Unit. Programmes are grouped to allow direct comparison with customer satisfaction survey

2.4.1 Services to beneficiaries

Over the period covered by the evaluation, EACEA has taken steps to improve the service to beneficiaries, in part driven by recognition of the need for simplification. Key measures have included the following:

Gradual introduction of electronic application forms across the different strands of the programmes (Culture, LLP, Europe for Citizens, Erasmus Mundus, Tempus, MEDIA, Youth in Action);

With the support of the Commission, removal of the comitology procedure29 for applications under most programme strands), thus shortening the time taken to finalise the selection of projects. Comitology only remains in place for the multi-annual co-operation projects under Culture; all MEDIA actions except FPA projects (Framework Partnership Agreement) after the first year of selection; as well as Networks of Twinned Towns and Operating Grants under Europe for Citizens. Introducing "unilateral" grant decisions in place of "bilateral" grant agreements (thus removing the need for beneficiaries to provide their written acceptance of grant funding) for certain strands of some programmes, e.g. LLP (eTwinning, Eurydice, Jean Monnet), MEDIA 2007, Culture (literary translations), Youth in Action, Europe for Citizens.

Introducing framework partnership agreements within certain strands of LLP, MEDIA 2007, Culture, Youth in Action, and Europe for Citizens, thus lightening annual renewal procedures for partnerships receiving support over a number of years;

Using fixed grants for some activities, in place of reimbursement at actual cost, e.g. LLP (Jean Monnet, Eurydice), MEDIA 2007 (digitisation of screens30), Europe for Citizens and operating grants in four programmes;

28 The Erasmus Charter is listed separately from the Lifelong Learning programme as no funding is awarded under this action and the selection process is not competitive. 29 http://europa.eu/legislation_summaries/glossary/comitology_en.htm 30 MEDIA 2007 provides support in the form of a lump sum of €20,000 per screen.

http://europa.eu/legislation_summaries/glossary/comitology_en.htm

16

Providing programme guides for most programmes, which offer a more stable framework for application through the provision of information on timetables and grant conditions many years in advance;

A reduction in the average time taken to make payments from 44 days in 2008 to 28 days in 2011; An increase in the number of monitoring visits to projects: from 241 in 2008 to 423 in 2011; Simplified reporting and budgeting processes, e.g. in the bilateral cooperation programmes and in the Tempus

programme in 2009-10; and Communication activities, including Infodays, attendance at national events, a regularly-updated website.

2.5 Relationship with the Commission

The recital to Regulation 58/2003 states that recourse to an Executive Agency (EA) does not relieve the Commission of its responsibilities under the Treaty. It goes on to state that the Commission must be able closely to circumscribe the action of each EA and maintain real control over its operation. This control is to be exercised through, inter alia, the appointment of the members of the Steering Committee and the director of an EA, approval of the EA's annual work programme, and administrative supervision (including an audit by the Court of Auditors). It is expected that each EA will collaborate intensively and continuously with the relevant Commission departments.

EACEA currently acts as a delegated body for three Commission Directorates General, which have ultimate responsibility for the programmes that it manages: Education and Culture (DG EAC), Communication (DG COMM) and Development and Co-operation – EuropeAid (DG DEVCO). These managerial or parent DGs play a supervisory role, which includes defining strategic priorities and programme plans, as well as an approval role for aspects such as control systems. This role is fulfilled by the Steering Committee, whose members are senior officials of the Commission31. Commission officials are also involved in the selection of projects through their participation in all evaluation committees for delegated programmes. The parent DGs are also responsible for commissioning external evaluations of the Agency, although such evaluations naturally require the active co-operation of EACEA. The first evaluation was completed in February 2009, covering the period from January 2006 to April 2008. EACEA must also report to the Commission at regular intervals on the progress made in implementing programmes, which it does through formal progress reports, including its Annual Activity Reports.

Ultimately, the Agency, as with all such bodies, has a degree of autonomy in the management of programmes, for example related to tasks such as authorising payments to successful bids. Such autonomy is, however, bounded by the details of the Financial Regulation applicable to the general budget of the EU,32 as well as by the rules applicable to each programme.

As well as supervising through the mandatory mechanisms noted above, the parent DGs have also worked with EACEA to put in place additional means to facilitate co-operation. These include weekly DG EAC Directors meetings attended by the Agency Director, quarterly coordination meetings at Director level and scheduled meetings between the Heads of Unit in the DGs and their counterparts at EACEA. Regular informal contact is also maintained via telephone, e-mail and visits in person. Relevant staff of the DGs also participate in communication or dissemination events implemented by EACEA, including Infodays.

31 The Director of EACEA also participates in meetings of the Steering Committee. 32 Council Regulation (EC, EURATOM) No 1605/2002 of 25 June 2002 on the Financial Regulation applicable to the general budget of the European Communities.

17

3 Response to previous evaluation (Task 3) As highlighted above, Task 3 consists of an assessment of EACEA’s response (including the detailed Action Plan) to the first interim evaluation and any actions that have been implemented to date.

The first interim evaluation of EACEA was produced in February 2009 and covered the period from January 2006 (when the Agency became operational) to April 2008. In general terms, the previous evaluation found that significant improvements had been made since 2006 in terms of the quality of service that EACEA provided to beneficiaries: a process of continuous improvement had been unfolding as processes and procedures were gradually simplified, but there remained ample scope for further improvement. The Agency met the objectives set in the 2006 and 2007 work programmes, but legacy IT systems continued to hamper efficiency and effectiveness. The report33 made eight specific recommendations:

Streamlining the interaction with applicants and beneficiaries;

Achieving further simplifications and efficiency improvements;

Solving outstanding IT issues;

Improving feedback to the Commission;

Clarifying the practical division of tasks between the Commission and the Agency;

Enhancing the correspondence between the Annual Work Programme and the Annual Activity Report with regard to objectives and achievements;

Lightening the requirements for consultative comitology; and

Keeping focus on payment times for LLP.

A table setting out in detail the actions taken by the Agency and Commission in response to these recommendations is included as an annex and this information is used throughout the report wherever relevant. In general terms the Agency and Commission have responded to the previous evaluation in a positive and systematic manner, though there is a slightly variable picture as regards the individual recommendations. These are dealt with in turn.

The Agency monitors general levels of satisfaction amongst applicants and beneficiaries, mainly through the regular surveys and assisted by direct contact (including face-to-face contact at Infodays, kick-off meetings and monitoring visits). The surveys show that satisfaction levels remain high, and while results are relatively consistent with earlier periods, there has been some improvement linked to improvements made by EACEA (i.e. programme guides, use of lump sums) in recent years.

Since the previous evaluation the Agency has made a number of further simplifications and efficiency improvements in key areas of the application and selection phase, though without the establishment of a ‘roadmap’ of priorities (as defined in previous evaluation). Subsequent sections, especially 4.2.3 on simplification, provide more detail on improvements in this area.

EACEA has placed considerable attention and invested significant resources into improving and harmonising IT tools and has rectified the problems associated with the inherited IT systems. A number of further improvements are currently in the pipeline around e-services for applicants and electronic reporting systems.

33 COWI for DG EAC: Interim evaluation of EACEA, February 2009.

3 Response to previous evaluation (Task 3)

18

Agency staff have worked hard to respond to the Commission’s needs in terms of the flow of project-level information and other programme-specific feedback. While it is clear that systems have been implemented to improve information flows, the stakeholder consultations suggest that there is scope for further improvement in terms of identifying and sharing qualitative information of use to the DGs (this issue is dealt with in more detail in relevant parts of section 4.)

The practical division of tasks between Commission and Agency has been clarified, principally through the consolidation of Modalities of Cooperation. Subsequent sections highlight a degree of doubt remaining over how well this division of tasks is understood externally, but also a lack of agreement over how critical this is to applicants’ understanding of the aims and objectives of the DGs and the programmes they oversee.

The formats of the Annual Work Programme and the Annual Activity Report have been revised in order to enable a more direct comparison between objectives and achievements.

There has been a consistent and continuing reduction in the time taken to make payments to beneficiaries; all targets are now being met, including those for LLP beneficiaries. Our consultations have suggested that the issue of delays in making pre-financing payments is strongly linked to the application of comitology in selection, and it is clear that the Agency has worked with the parent DGs to move towards more information-based procedures wherever possible. This issue remains relevant to the drawing up of detailed plans for the new generation of programmes, though the parent DGs have ultimate responsibility for this.

19

4 Cost-benefit analysis (Task 2) 4.1 Introduction Task 2 of this evaluation has required a retrospective Cost-Benefit Analysis (CBA), focusing on a comparison between the assumptions made in the previous CBAs carried out prior to the creation of EACEA and each extension of its mandate as relevant, and the situation of today. The evaluation of EACEA represents a situation in which the impacts are captured in a mix of qualitative, quantitative and monetary data and where there are varying degrees of certainty. In this situation, it has not been possible to apply a conventional cost-benefit analysis (in the narrow sense), since we cannot monetise or directly compare all (or even most) costs and benefits. As a result, this evaluation has adopted a particular approach to the CBA, defined by the Secretariat General of the Commission as “Cost-benefit thinking through multi-criteria analysis”.34 This particular form of CBA has allowed us to combine the range of positive and negative impacts into a single framework to allow easier comparison. It also provides a transparent presentation of issues at stake, as well as any distributional issues. Our role here has been thus to gather evidence that enables a clear identification of the costs and benefits.

Our approach to the CBA has covered the ten issues listed in the ToR (a to j as based on the Framework Regulation), as well as the other issues that we believe require consideration if the twelve evaluation questions are to be answered. By taking this approach, we have ensured a degree of comparability with the first interim evaluation, covered the issues required by Regulation 58/2003, and generated findings necessary to answer the evaluation questions.

4.2 Relevance of role, design and set-up

4.2.1 Relevance to need

CBA Issue Definition i: Continuous relevance of the Agency to the needs it is intended to meet

The evaluation should seek to establish whether outsourcing to EACEA is still the best way to achieve the tasks with which it has been entrusted, according to the Regulation laying down the statutes for executive agencies.

The Executive Agencies were conceived as a way to make it easier for the Commission to concentrate on its core ‘institutional tasks’ and maintain accountability to citizens by delegating certain defined tasks relating to the management of programmes. A key element in this was a desire to promote greater professionalization, with agencies employing staff with specialist and technical skills. Managerial and budgetary separation is often seen as a more responsive and effective way of delivering services, enabling specific professional and technical expertise and allowing more (or more direct) contact with beneficiaries and citizens. This is also felt by many to be a less bureaucratic solution, potentially insulating programme activity from changing political contexts and short-termism. However, the benefits of separating policy and delivery need to be weighed against a potential risk of blurred or overlapping roles and accountability.

34 European Commission (2009), Impact Assessment Guidelines SEC (2009) 92, p.47

4 Cost-benefit analysis (Task 2)

20

Whilst the Regulation establishing Executive Agencies highlights the potential gains from outsourcing, in the case of EACEA it is perhaps more correct to say that many programme management tasks have in fact been brought back within the overall institutional framework of the European Commission rather than being outsourced for the first time. Indeed, when discussing the relevance of EACEA's role, several interviewees highlighted the advantages of delegation to EACEA rather than to its predecessor, the Technical Assistance Office (TAO), which was a private body. For example, the TAO was reported to have suffered from instability, uncertain funding and (consequently) high staff turnover. Moreover, the previous situation was seen as inefficient since programme management was only partially delegated: the DGs were still required to contract with beneficiaries and make payments since the TAO was only able to prepare contracts and payments. Overall, staff members of the parent DGs were more comfortable with the delegation of programme management to a public body that is subject to the full financial and institutional requirements of the Commission.

As well as addressing weaknesses in the TAO, the evidence from the research suggests the continued relevance of delegation to an agency, such as EACEA, to the needs of the parent DGs. Firstly, the programmes of the parent DGs continue to require a large degree of management at EU level, even accounting for the significant sub-programmes that are delegated to National Agencies (notably elements of Lifelong Learning and Youth). This remains in sharp contrast to the EU Structural Funds, for example, where management is delegated almost entirely to Member States. Moreover, the number of programmes overseen by the parent DGs has increased over the period since EACEA's creation (e.g., Tempus, bilateral cooperation with third countries, education and youth activities funded by the External Co-operation Instruments), again creating the need for additional programme management capacity at EU level.

Secondly, there was unanimous agreement amongst the interviewees that programme management requires particular expertise, which is best developed through the deployment of specialist staff within a dedicated body; indeed, the experience of the last few years tended to reinforce rather than weaken this impression amongst the interviewees. Thirdly, a number of interviewees highlighted the potential of EACEA to introduce common systems and procedures across a range of programmes and also to create synergies and complementarities between programmes of different DGs, notably those related to external co-operation such as Erasmus Mundus, Tempus and actions funded by the External Co-operation Instruments.

Although not specifically mentioned in the Terms of Reference, it has also been necessary to consider how the needs of the parent DGs are evolving. Our stakeholder consultations highlighted the important work of the Eurydice unit and, to some extent, the Tempus unit in the collection and analysis of comparative information on practice and progress, suggesting there is strong and on-going demand for evidence and knowledge able to inform EU policy-making processes (especially for policy areas covered by the Open Method of Coordination).

Overall, the evidence tends to confirm the continued relevance of the Agency to the needs it was designed to meet. None of the interviewees suggested an alternative approach to programme management, such as outsourcing to a body entirely outside the Commission or bringing the role into the DGs. Indeed, the fact that the mandate of EACEA has been extended several times since its creation confirms the recognition by the parent DGs of the relevance of this type of body to the fulfilment of their programme management needs.

Main evaluation finding 1: The evidence gathered suggests that EACEA, as a dedicated programme management body operating within the overall institutional framework of the European Commission is relevant to the needs of the parent DGs and, indeed, represents an improvement on the previous approach of delegation to the TAO.

Main evaluation finding 2: The experience of the last few years (including the expansion in the number of programmes overseen by the parent DGs) confirms the continued need for a dedicated agency deploying specialist staff to manage a range of programmes. Future evaluations may need to consider how the needs of policy-makers are evolving, especially as regards the availability of evidence and knowledge, and how the Agency’s potential to provide information in support of the policy-making process can be fully exploited.

21

4.2.2 Flexibility in the implementation of outsourced tasks

CBA Issue Definition ii: Flexibility in the implementation of the outsourced tasks

Ability to react quickly and flexibly to unforeseen problems and changes

A specific requirement of the cost-benefit analyses related to outsourcing, set by Council Regulation 58/2003, is to take account of the flexibility of Executive Agencies in implementing outsourced tasks. Here, it is crucial to note that such flexibility relates to the ability to react quickly and flexibly to unforeseen problems and changes. It does not relate to flexibility in the application of rules and procedures relating to grant funding, since EACEA's freedom of action is inevitably limited, notably by the Financial Regulation.

The evidence from the current evaluation suggests that EACEA has been sufficiently flexible to respond to the increased demands from the parent DGs for additional programme management services. Indeed, none of the representatives of the parent DGs reported any particular difficulties or reluctance on the part of EACEA to respond as and when new programmes or sub-programmes emerged. Whilst some additional demands emerged gradually – as new programmes passed through the usual political decision-making process – others were made at much shorter notice, i.e. new activities within existing programmes. For example, EACEA was able to respond to a request from DG DEVCO to oversee the management of additional resources for neighbourhood countries allocated from the Erasmus Mundus and Tempus programmes in response to the "Arab Spring”35. EACEA has also been sufficiently flexible to facilitate synergies between programmes of different DGs that focus on similar areas; for example, it has facilitated synergies between EAC and DEVCO programmes related to youth policy in some third countries.

In most cases, EACEA has been able to take on responsibility for managing additional programmes through the creation of new programme units. Staff at EACEA reported that this approach had offered the benefit of retaining the flat structure – seen as crucial to facilitating effective internal communication and quick decision-making. The ability to absorb additional responsibilities has also been strengthened by the continual improvement in the "horizontal" services, notably IT.

At the same time, there may be scope for EACEA to further improve its ability to react quickly and flexibly by a re-examination of the potential for greater synergies and co-operation between the different programme units. There remain some reports of high staff workload, particularly around peak times, e.g. related to the application cycle, and there is a heavy reliance on (expensive) interim staff.

In response, EACEA has taken steps to adjust some individual portfolios of some units and interviewees reported that there is now greater mobility of staff between units. As the various internal systems (e.g. IT) and external processes (e.g. related to application and selection) become more harmonised across EACEA, it may be possible for such mobility to be increased – whether permanent or temporary – with staff more able to apply their generic programme management skills across different programmes. For this to happen, it may be necessary for the parent DGs, together with EACEA, to review the annual management cycles across the full portfolio of programmes managed by EACEA in order to minimise (or reduce) peaks and try to spread the workload more evenly over the year. Any such review must take account of the fact that much of EACEA’s portfolio relates to education and therefore merits alignment with the academic year.

35 Another example of responsiveness to political events has been the creation of Roma networks and multi-lateral projects under LLP

22

Main evaluation finding 3: EACEA has proved itself sufficiently flexible to respond to the increased demands from the parent DGs for additional programme management services, in some cases at relatively short notice.

Main evaluation finding 4: There may be scope for EACEA to further improve its ability to react by promoting greater staff mobility between programme units. There may also be ways to spread its workload more evenly over the year (e.g. by co-ordinating the different programme timescales more closely), albeit whilst maintaining alignment with the academic year for those programmes related to education.

4.2.3 Level of financial and technical expertise

CBA Issue Definition iii: Level of financial and technical expertise available at EACEA

EACEA needs a certain level of financial and technical expertise in order to manage the outsourced activity effectively and efficiently. The evaluation will analyse how this need has been catered for and assess whether the necessary level has been achieved36.

The first evaluation of EACEA concluded that the "agency staff is specialised and qualified, and many staff members have previous working experience in the Commission or the TAOs." Evidence from the current evaluation tends to support this positive finding. Indeed, there has been a general improvement in the level of technical and financial expertise at the same time as the overall staffing levels of EACEA have increased.

In the first few years after its creation, the staffing of EACEA tended to include a mix of permanent officials seconded from the DGs, former staff of the TAOs recruited by EACEA and a relatively large number of interim staff. Over the period covered by the evaluation, the situation appears to have steadily stabilised in part through the recruitment of new staff directly into EACEA, thus offering a closer fit between the needs of the various units and the skills and experience of individuals. As highlighted by the first evaluation, many new recruits bring valuable experience of working for the Commission. Such recruits have typically been on temporary contracts and are thus attracted by the potential for a long-term or permanent contract at EACEA. Many new recruits have also come from beneficiaries and other bodies at national or EU level, with such individuals offering relevant policy knowledge and practical experience of implementing projects.

Some difficulties were reported in relation to the level of staffing in the early years of the Agency (i.e. up to 2008), with the percentage of occupied posts remaining below the target of 95%.37 However, this has improved in recent years, with occupancy exceeding 97% since 2009.38 For example, in June 2011 there were just 12 vacant posts out of 438.39 One particular attraction for contract agents has been the possibility to secure a long-term or permanent contract with EACEA, compared to the maximum term of three years that is available to contract agents within the Commission. With respect to retention, one particular need highlighted by the consultation of EACEA staff has been that of creating opportunities for career progression. Indeed, where progression is limited, the risk is that staff view EACEA as a stop-gap job, enabling them to gain relevant experience before taking part in EU recruitment competition; some staff were reported to have left EACEA as a result. Another problem has also been the fact that some contract agents were recruited in the years before the European Personnel Selection Office (EPSO) test was a mandatory condition for receiving a contract of indefinite duration; as a result, EACEA risked losing a significant number of staff members at the expiry of their contracts in 2011, i.e. as many as 34 individuals, representing 10% of all contract agents. This problem was aggravated by the fact that EPSO tests were not organised annually as envisaged. However, EACEA has shown a welcome degree of pragmatism here through, for example, organising

36 This section deals with the expertise available to EACEA. Quality of the work performed is dealt with in sections 4.3 and 4.5 of the report. 37 Activity report of 2008 38 Activity report of 2009 39 Activity report of 30 June 2011

23

an ad-hoc testing in line with the future testing components of EPSO. A few units also reported that the EPSO recruitment procedures tend to result in staff that lack the specialist knowledge for areas such as the audio-visual sector – whilst others with the specialist knowledge (but not recruited via EPSO) were unable to progress beyond level 3 posts. The limited flexibility in hiring staff is in fact a problem facing all Executive Agencies, as highlighted by the Court of Auditors.40

Evidence from a recent staff survey suggests that retention of staff (and thus retention of their financial and technical expertise) has been assisted by an increased level of satisfaction, notably with respect to strong team-working and positive relations between colleagues, a good working environment, good working conditions and the opportunity to take responsibility for decision-making.41 Retention of staff has also been assisted by an increase in teleworking (covering 60 staff members) as well as a gradual increase in mobility between EACEA units (with 26 staff reported to have moved internally in the period leading up to February 201142), facilitated by increased harmonisation of programme management functions. Based on the consultations and analysis of staffing data, further promotion of internal mobility may help to increase the retention of staff. In addition, there may be potential to promote progression between the different Executive Agencies, which has tended to be very limited to date.

The evidence suggests that the level of technical and financial expertise has steadily increased amongst the staff in place at EACEA. This is reflected, for example, in a consistent reduction in the average times taken to make payments, award grants and issue contracts, an increase in the number of monitoring visits and simplifications in procedures – at the same time as the overall volume of funds managed by EACEA has risen. It was also reflected in the interviews of staff of the parent DGs, none of whom expressed any particular concern about the level of expertise available to EACEA.

There may be scope to increase the level of technical and financial expertise yet further through additional improvements in training provided to staff; the recent staff survey uncovered a decreasing level of staff satisfaction with training opportunities available and there has also been a relatively high rate of non-attendance at learning and development actions, exceeding 12% in 2011. A number of the Agency staff consulted mentioned high workloads as a contributory factor. The current EACEA Training Plan seems to suggest that the portfolio of learning and development actions available to staff is steadily widening to include all aspects of personal and professional development (in addition to the general and mandatory training courses already provided) and an emphasis on the identification of individual learning needs. The Plan also foresees improved monitoring of attendance at training as well as shorter modules of 1.5 hours that staff should find easier to attend. It will therefore be interesting for future evaluations of EACEA to explore whether such steps have ultimately been reflected in greater staff satisfaction with training.

Main evaluation finding 5: There has been a general improvement in the level of technical and financial expertise at the same time as the overall staffing levels of EACEA have increased.

Main evaluation finding 6: The low rate of occupancy of staff posts (i.e. below the target of 95%) observed at the very beginning of EACEA’s establishment has been addressed, although there is the need to further improve retention and career development perspectives by promoting greater internal mobility and exploring ways to create progression pathways between the different Executive Agencies.

Main evaluation finding 7: Improvements in technical and financial expertise are reflected in a general improvement in the performance of EACEA. Further improvements might be achieved through the strengthened training offer, provided that attendance rates at training sessions can be maximised.

40 Court of Auditors Special Report No 13/2009 ‘Delegating implementing tasks to executive agencies: a successful option?’ (2009/C 279/09) 41 EACEA Steering Committee of 25 November 2011. 42 EACEA Steering Committee of 25 February 2011.

24

4.2.4 Appropriate internal organisations and procedures

CBA Issue Definition iv: Appropriate internal organisations and procedures

This issue is in reality two-fold, as the evaluation will need to determine (i) whether the Agency operates in line with the legal framework and (ii) whether this legal framework is appropriate and allows it to carry out the tasks with which it has been entrusted.

The first evaluation concluded that "the Agency functions in accordance to the Act of Delegation and the Modalities for cooperation between the supervisory DGs and the Agency, which determine the role, responsibilities and tasks of the Agency. The Agency does not deviate from the framework set by these two documents. [...] Nevertheless, there are areas with need for further clarity in terms of roles and responsibilities, i.e. communication with the public and other entities, and the use of project results".

Since the first evaluation (and as noted above), EACEA has taken steps to further strengthen its operations, including improvements and harmonisation in IT systems, improvement of personnel appraisal and development standards, improved financial accounting, consolidation of procedures relating to internal control and acceleration of ex-post control checks of dossiers from previous years.

However, a 2009 report of the Court of Auditors (CoA) has highlighted a number of concerns, albeit addressed to Executive Agencies in general rather than specifically to EACEA. In particular, the CoA stated that "the Commission’s supervision of the agencies’ work is quite limited: the annual work programmes (subject to the Commission’s approval) are scarcely used for setting targeted objectives; the monitoring is mainly focused on indicators related to how the tasks are carried out rather than to the results produced; the reports are usually confined to budgetary execution and omit to measure progress made on a multiannual basis and identify corrective actions for the future." To address this issue, the CoA recommended that the Commission should "supervise the agencies by setting results-oriented and targeted objectives, using a limited number of relevant performance indicators which form the basis for next years’ objectives". In its response to the CoA report, the Commission stated that the tasks and targets of the EAs should be distinguished from the objectives, targets and indicators of the Community programmes and that the AWP guidelines for 2010 would be adapted accordingly in order to better reflect the different roles and to define a harmonised content for the two management instruments. It also stated the need for an intervention to revise the number of indicators and to develop new indicators to allow the DGs to improve the performance-measurement of the Executive Agencies.

The CoA report also highlighted the fact that Commission approval of agencies’ Annual Work Programmes (AWPs) tended not to occur before the beginning of each year, as required by Regulation 58/2003. One reason for this late approval, noted by the CoA, is the need for the AWPs of the relevant Community programmes to be approved before the agencies’ AWPs can be adopted. However, this reason does not entirely explain the very late adoption of EACEA’s AWP in some years, including 2012. In its response to the CoA report, the Commission highlighted the fact that guidance and monitoring continues throughout the year and, where necessary, on the basis of the draft documents. Whilst this pragmatic response is to be welcomed, the Commission should take steps to ensure an earlier approval of the AWP.

Whilst the CoA report considered all Executive Agencies rather than EACEA specifically, it is nonetheless surprising that the meetings of the EACEA Steering Committee do not appear to have given explicit consideration to the findings of the report. Similarly, the sections within EACEA Annual Activity Reports devoted to "Relations with the Court of Auditors" do not refer to the CoA report or to any response by EACEA. The reports findings were considered by the co-ordination meeting of 18 February 2011 between EACEA and DG EAC Directorate A, although this appears to be the exception; according to the minutes the co-ordination meetings with other Directorates do not appear to have given explicit consideration to the report. Directorate A used this meeting to express its satisfaction with the current mechanisms in place, such as the Steering committee, periodic

25

coordination meetings at level of directors, at level of units and at level of staff. It also committed to producing an overview table which shows which actions could be envisaged.

Given the importance of the CoA report, the Commission produced a supervision strategy in order to address the issues raised therein. Whilst some steps have already been taken, such as a reduction in the number of indicators, it would seem important for this this strategy to be updated, given the time that has elapsed since the publication of the CoA report. It would also seem appropriate for the implementation of the strategy to be reviewed by the Steering Committee on an annual basis.

Main evaluation finding 8: EACEA has further strengthened its internal operations and procedures since the first interim evaluation, although some difficulties remain, such as the late adoption of its Annual Work Programmes.

Main evaluation finding 9: The Commission has taken steps to address some of the issues raised in the 2009 Court of Auditors report regarding Executive Agencies. The report as a whole does not appear to have been considered by meetings of the EACEA Steering Committee, although the findings may have been considered in other fora.

Main evaluation finding 10: The findings of the 2009 Court of Auditors report should be addressed in the supervision strategy that the Commission is currently updating, including the report’s recommendation that the Commission should supervise the agencies by setting results-oriented and targeted objectives, using a limited number of relevant performance indicators which form the basis for next years’ objectives. It may also be appropriate for the implementation of the strategy to be reviewed on an annual basis by the EACEA Steering Committee and in EACEA's Annual Activity Reports.

4.3 Effectiveness: service to applicants and beneficiaries

4.3.1 Effectiveness

CBA Issue Definition Issue v: Effectiveness This evaluates the extent to which objectives set are achieved, specifically how well

EACEA has managed EU programmes. This includes factors such as the timeliness to select projects and pay grants, general levels of client satisfaction and how effectively the Agency has implemented its own objectives and priorities.

The previous evaluation found that the Agency had achieved its objectives, with service improvements resulting in improved levels of satisfaction. The ability to provide services to applicants and beneficiaries in a more effective and responsive manner is often cited as a benefit of executive agencies, so it is important to evaluate to what extent services have improved since the last evaluation.

Turnaround times are one element of providing an effective service, and the following table shows that EACEA is exceeding its targets and has made significant progress in most areas, especially for the time to contract (from selection).

Table 4.1 Time indicators 2008 2009 2010 2011 2011 Target Time to select projects / award grants Youth in Action and Europe for Citizens 3.2 3.1 4 months All other programmes 3.643 3.6 4 5 months Time to contract

43 2009 result is for all programmes administered by EACEA

26

2008 2009 2010 2011 2011 Target Youth in Action and Europe for Citizens 1.444 0.6 0.7 0.9 months All other programmes 1.645 1.4 1.3 2 months Source: EACEA Annual Activity Reports

The issue of payment times (especially for LLP projects) was highlighted as an area of concern in the previous evaluation, with legacy IT systems, inherited backlog of files and organisational problems highlighted as the main causes. Since then, EACEA has invested significant time and resources in addressing this, through a working group approach and the establishment of targets.

Table 4.2 displays information on the proportion of payments being made on time and in line with targets (as well as average payment times) during the study period. This shows that there has been a considerable improvement since the first interim evaluation, with all payment targets currently being exceeded (including for LLP projects).46

Table 4.2 Payment indicators 2008 2009 2010 2011 2011 Target Payments on time (%) Within time specified in contracts47 92% 87% 94% 93% 90% Within EC targets48 88% 80% Performance against EC payment targets (days) First pre-financing payment 16 14 12 20 days Most other types of payment49 33 20 19 45 days50 Grants with a single time limit 52 52 61 90 days51 Average payment times (days) Programme budgets 44 30 24 2852 Administrative budget 23 21 18 16 Source: EACEA Annual Activity Reports

Another element of effectiveness, though linked to efficiency (and of less direct relevance to applicants and beneficiaries), is EACEA’s ability to commit and pay budgets in line with plans. Information provided in the Agency’s Annual Work Programmes and Activity Reports shows that both operational and administrative budgets were committed and paid in line with annual projections between 2008 and 2011.

The online survey of applicants and beneficiaries provides arguably the most important source of evidence for considering effectiveness, as it reflects the views of EACEA’s “customers” and allows us to see trends in the way

44 All programmes with one-month target 45 All programmes with two-month target 46 Annual Activity Report 2011; EACEA 47 The maximum payment period is specified in legal commitments, and can be 30, 45 or 90 days from receipt of an admissible payment request to funds being debited from the relevant account, depending on the type of contract. 48 The new financial regulation will retain only the target for first pre-financing payment 49 Renewed pre-financing, interim and final payments 50 Will be 30 days under the new financial regulation 51 Will be 75 days under the new financial regulation 52 In its Annual Activity Report for 2011, EACEA reports that the increase in average payment times for programme budgets from 24 days in 2010 to 28 days in 2011 resulted from a lack of payment credits during the last quarter of 2011 for the Europe for Citizens programme. It goes on to note that additional payment credits were transferred in the framework of the "Virement Global" exercise and any pending delayed payments were processed by the end of that year.

27

levels of satisfaction have evolved over time. A full breakdown of the results of the online survey is provided as an annex, but the main issues in relation to effectiveness are as follows:

Table 4.3 Effectiveness by survey Question 2008 2010 2012 We are satisfied with the overall quality of the programme management services provided by the Executive Agency during the whole project period

2.8 2.2 2.8

The deadline for submission was sufficiently long for us to prepare our application

3.1 1.9 3.0

We received a reply on the selection results within the deadlines 3.9 3.0 2.9 The awarded grant was paid in good time, without significant delays 2.0 2.5 Source: Online survey (average response on scale of 1-9 – 1=strongly agree and 9=strongly disagree)

Another element of effectiveness concerns beneficiaries’ views in relation to timely receipt of the contract. This is not reproduced above as the question was asked in a different way in order to be more comparable with the previous survey. This time 80% agreed that they received the contract in good time; while in the previous survey 68% agreed that they received the contract within two months53.

The other issues relating to effectiveness display a more mixed picture when it comes to the views of applicants and beneficiaries. While there has been an improvement in the timely receipt of selection results, which is likely to be linked to the removal of comitology processes, other aspects have not changed significantly in the past few years. Overall satisfaction has returned to the level seen at the time of the first survey in 2008, which might reflect the fact that the main improvements took place between 2008 and 2010). The most negative aspect is in relation to the length of time available to prepare applications, which is a surprising result given the introduction of stable programme guides containing advance information on deadlines (and subsequent sections will demonstrate that applicants appreciate the availability of this information). Analysis of comments made by beneficiaries suggests that respondents could have linked this question to deadlines for information and document requests at other stages of the project life cycle as this is a more frequently mentioned area of concern.

The clear progress in reducing payment times is welcomed by respondents but has not translated into a clear improvement in overall satisfaction levels as regards the issue of payments. While it is not feasible to expect faster payments to completely eradicate payment-based complaints there are some recurring themes in evidence. This includes a perception that payment processes are too rigid or over-formal A number of examples of the specific comments made by respondents to the online survey are provided below, and while it should be remembered that they are not representative of the balance of responses, they do give an indication of the kinds of subjective observations that are made on a regular basis by beneficiaries. :

“At final report stage we received a large number of queries concerning items of expenditure which had been agreed at interim report stage, with documentation which had been available then having obviously been mislaid. This has delayed final payment and has created difficulties for partners.”

“The reporting system is far longer and more rigid than for other grants awarded by other funders we have worked with. They do not seem to want to hear about lessons learned in the process but want to see the project carried out exactly as written in the proposal. The payments are much speedier than before.”

“We hope to have everything to the Executive Agency by the end of May but suspect we will have to wait many months for a response or the final tranche of the grant payment”

53 This question was revised as the vast majority of respondents to the previous survey who did not agree that they had received the contract within two months either did not know or could not remember.

28

“The yardstick for financing should be only the quality of the results compared with the goals of the project. Controlling by the Executive Agency could concentrate on the quality of results. This would be much more effective than concentrating on financial rules and regulations”

This is more likely to be connected to the financial regulations of the delegated programmes than EACEAs internal procedures, though this would not necessarily be clear to beneficiaries. The following chart shows how these results can be broken down by programme, with MEDIA scoring well on overall satisfaction and deadlines for submission, and Erasmus Mundus/ACP and Tempus producing most positive scores for timely receipt of selection results.

Figure 4.1 Effectiveness by programme

Source: Online survey (all respondents)

Another facet of effectiveness is the extent to which EACEA has successfully implemented its main objectives and priorities. In response to the previous evaluation, EACEA made changes to the structure of Annual Work Programmes and Annual Activity Reports, rationalising the number of priorities in 2009 and making it much easier to monitor progress against stated objectives. These objectives are as follows:

1 2 3 4 5 6 7 8 9

The awarded grant was

paid in good time,

without s igni ficant

delays

We received a reply on

the selection results

within the deadl ines

The deadl ine for

submiss ion was

sufficiently long for us to

prepare our appl ication

We are satis fied with

the overa l l qual i ty of the

programme management

Average Response (1=strongly agree, 9=strongly disagree)

ALL RESPONDENTS

Youth in Action

Tempus

MEDIA

Li felong Learning

Europe for Ci tizens


Culture

Bi latera l Cooperation

29

1. Streamlining project selection processes

2. Providing improved guidance for project implementation and moving towards harmonised reporting requirements (for 2009 and 2010 this was defined as “improving project implementation and moving towards leaner reporting requirements”)

3. Supporting and exploiting further the achievements of EU actions

4. Supporting the Commission in the preparation of the next generation of programmes and in the extension of the Agency's mandate (between 2011 and 2014)

These issues will each be dealt with in more detail in the relevant sections (e.g. on selection process, reporting, dissemination). For the purposes of overall effectiveness, based on the information provided in the Annual Activity Reports and augmented by other research tasks, there has been some progress in all of these areas.

The 2009 Court of Auditors report54 reviewed the reporting systems for all executive agencies and found that monitoring tended to focus more on budgetary execution and how tasks have been carried out than on the results produced. While this is still true to a certain extent for EACEA’s reporting systems, there is clear evidence of progress against the first objective, with numerous examples of improvements at application and selection stage. There are fewer examples of improvements relating to implementation and reporting stages of the project lifecycle in the period covered by this evaluation (2008 to 2011), though it should be borne in mind that this has been the main focus of more recent improvements, such as the deployment of audit certificates (dealt with in more detail in section 4.5.2 on monitoring and reporting procedures).

Main evaluation finding 11: EACEA is meeting its objectives in terms of improving services for applicants and beneficiaries and is exceeding targets relating to the deadlines for selection, contracting and payments. The Agency is able to allocate operational and administrative budgets effectively.

Main evaluation finding 12: Satisfaction levels have remained positive and stable as the Agency’s workload has increased and there have been improvements in specific areas, mainly in relation to the early phases of the project life-cycle and corresponding with the improvements made by EACEA in this period (i.e. application and selection stage). There is less evidence of improving implementation and reporting, though this has been the main recent focus of activity.

4.3.2 Simplification and administrative burden

CBA Issue Definition Issue vi: Administrative burden for applicants

Administrative burden is defined as the costs in time and money incurred by entities applying for and managing EU funds from the programmes managed by EACEA. While publicly funded programmes will always demand some administration of recipients (and requirements are mainly defined by the financial regulations of the delegated programmes), our focus has been to understand whether requirements are seen as proportionate, if they exert a deterrent effect, if there is evidence of improvement and whether there are specific elements that could reasonably be addressed. .

Issue vii: Simplification of procedures

This seeks to identify and evaluate EACEAs efforts to simplify and harmonise application, monitoring and payment procedures. This includes the number of and quality of simplifications, as well as an assessment of views on their impact to date and the potential for further harmonisation.

54 European Court of Auditors, Special report No 13 (2009) Delegating Implementing Tasks to Executive Agencies: A Successful Option?

30

The previous evaluation found that the Agency was in the process of developing and adapting to simplified procedures, and the cost of this was being offset by the impact of rationalising processes. The period since 2008 has seen significant progress towards greater simplification and harmonisation (subject to the rules established for each programme), in part driven by recognition of the need to improve services to beneficiaries and keep their obligations proportionate to the size of the grant received, but also to generate efficiency savings at the Agency level. . The key measures have included the following:

Developing (or contributing to the development of) programme guides, which describe the objectives and strands of the programme and provide consistent information regarding selection procedures, general rules, terms and conditions attached to grant funding, and a calendar for the application process;

Gradual introduction of electronic application forms across most programmes and strands (Culture, LLP, Europe for Citizens, Erasmus Mundus, Tempus, MEDIA, Youth in Action)55;

Removal of the comitology procedure for applications under certain strands, thus shortening the time taken to finalise the selection of projects;

Introducing "unilateral" grant decisions in place of "bilateral" grant agreements (thus removing the need for beneficiaries to provide their written acceptance of grant funding) for certain strands of some programmes, e.g. LLP (eTwinning, Eurydice, Jean Monnet), MEDIA 2007, Culture (literary translations), Youth in Action, Europe for Citizens;

Introducing partnership framework agreements within certain strands of LLP, MEDIA 2007, Culture, Youth in Action, Erasmus Mundus and Europe for Citizens, thus providing beneficiaries with the certainty of funding over more than one year;

Using fixed grants (lump sums or flat-rates) for some activities, in place of reimbursement at actual cost, e.g. LLP (Jean Monnet, Eurydice), Culture (Literary Translations), MEDIA 2007 (Digitisation of Screens), Europe for Citizens and operating grants in four programmes;

Simplified reporting and budgeting processes, e.g. in the bilateral cooperation programmes and Tempus; and A range of additional simplifications relating to the day-to-day management of projects. This includes accepting

email correspondence rather than formal letters for simple requests and answers or minor contract amendments.

Given the legacy situation, where EACEA acquired the responsibility for managing programmes from a wide variety of organisations, many with their own procedures and processes, there was significant scope for standardising management across programmes. The fact that the Agency is a large organisation with significant capacity and expertise in programme management has also contributed to the simplification process, especially through the way that improvements can be piloted in one unit or programme before being transferred to others. In addition to the major changes mentioned above, more minor improvements can be transferred between units as a result of formal knowledge-sharing sessions or internal staff mobility. E-forms have been a key element in the process to date, and by 2011 EACEA was approaching the target of having 70% of all applications submitted online.

Table 4.4 Online submissions Indicator 2008 2009 2010 2011 Target On-line submission of applications 30% 58% 67% 70% Source: EACEA Annual Activity Reports

The survey of applicants and beneficiaries asked all respondents who had applied electronically for their views on e-forms. The following table shows that responses were generally positive on balance, though there is limited evidence of improvement over time.

55 The timetable for introduction of e-forms by programme is provided as a separate annex.

31

Table 4.5 Views on e-forms, by survey Question 2008 2010 2012 The e-form was clear and easy to complete 3.6 3.5 The e-form was easy to submit 3.0 Source: Average response for 3,362 respondents submitting electronic application, scale of 1-9

It is also possible to break down satisfaction with e-forms amongst respondents to the current survey by the year of their most recent application. Though EACEA have made improvements to e-forms in recent years, informed by the results of the previous satisfaction survey, it would appear that the changes have not translated into improved levels of satisfaction over time.

Table 4.6 Views on e-forms, by year of most recent application Question 2008 2009 2010 2011 All

Respondents The e-form was clear and easy to complete 3.3 3.8 3.7 3.5 3.5 The e-form was easy to submit 2.9 3.3 3.1 2.9 3.0 Source: Average response for 3,362 respondents submitting electronic application, scale of 1-9

The open textual comments suggest possible reasons for the limited evidence of improvements. The comments received were generally mixed, with significant numbers complaining about the complexity of e-forms, their layout (especially word limits and lack of cut-paste functionality) and technical or format problems (linked to the use of Adobe pdf or connectivity).

The fact that there has not been an improvement in overall satisfaction levels and that these comments continue to be made by more recent applicants suggests that the content and layout of e-forms should be kept under constant review. The new generation of programmes will offer an opportunity for more substantial improvements. The following chart breaks down results for the current survey by programme, showing that respondents under Lifelong Learning and MEDIA give significantly more positive results than other programme areas such as Youth in Action and Europe for Citizens.

Figure 4.2 Views on e-forms by Programme

Base: Online survey (3,362 respondents submitting electronic application)

In more general terms, staff of the Agency and parent DGs are overwhelmingly positive about the value and impact of efforts to simplify and harmonise processes and procedures, especially the move from comitology to information procedures, the increasing use of e-forms, the introduction of stable programme guides, lump sums and fixed rates

1 2 3 4 5 6 7 8 9

The e-form was easy to

submit

The e-form was clear

and easy to complete


ALL RESPONDENTS

Youth in Action

Tempus

MEDIA

Li felong Learning



Culture


32

and the move towards multi-annual agreements and framework partnerships. However, it is important to compare this picture with the results of the applicant and beneficiary survey. The following table shows the average response (on the scale of 1-9) made by all applicants making more than one application, against each of the questions linked to simplifications.

Table 4.7 Impact of simplification by survey Question 2008 2010 2012 We have noticed an improvement in the way our projects have been handled

3.4 3.0 3.7

We have noticed application procedures becoming easier or simpler

4.1

We have noticed improvements in reporting requirements 4.1 We have noticed improvements in the processing of payment claims

4.0

Source: Average response for respondents with more than one application – scale of 1-9)

While all results are broadly positive, there is little evidence of a significant improvement over time. Indeed respondents to the 2010 survey update were more likely to notice a general improvement in the way their projects had been handled than those taking part in the current survey. As highlighted in the previous section, this could be linked to the fact that the bulk of improvements took place between 2008 and 2010. The following chart breaks down responses to the current survey by programme, with respondents under Lifelong Learning and MEDIA significantly more likely to have given a more positive response than those under Europe for Citizens, Youth in Action and Culture.

Figure 4.3 Impact of simplification by Programme

Source: Online Survey: (2,793 respondents with more than one application since 2008)

1 2 3 4 5 6 7 8 9

We have noticed

improvements in the

process ing of payment

cla ims

We have noticed

improvements in

reporting requirements

We have noticed

appl ication procedures

becoming eas ier or

s impler

We have noticed an

improvement in the way

our projects have been

handled


ALL RESPONDENTS

Youth in Action

Tempus

MEDIA

Li felong Learning



Culture


33

The general approach has been to standardise approaches as far as possible, while seeking to reflect the specificities of individual programmes. The Agency has made clearest progress in terms of the improvement of application procedures, though more recent efforts have focussed on improvements at later stages of the project life cycle (see section 4.5.2). Given the fact that current programmes will only be ‘live’ for a relatively short period of time, the next wave of major changes is on hold until the precise format of future programmes has been confirmed. Nevertheless the introduction of new programmes will present additional opportunities for streamlining procedures, and Agency staff are playing an important role in these discussions.

The issue of administrative burden in applying for and managing EU funds was not one of the evaluation questions defined in the original terms of reference for , but it was highlighted in the previous evaluation and is a consistent area of concern for beneficiaries of EU programmes. This issue is arguably of particular significance to EACEA given the large numbers of small grants it administers. Although submission of an application and participation in a project is entirely voluntary, the level of the administrative burden provides one indication of the efficiency of EACEA and also the contribution to achieving the goals of Community Programmes. The main focus is on information provision and reporting requirements, specifically information that has to be transferred to EACEA or information that has to be available for inspection or supply on request. It is important to recognise that information and reporting requirements for these programmes are defined with the European Commission, though the Agency has some room for manoeuvre provided that their requirements are compliant with the Financial Regulation. The Agency has introduced a number of improvements in processes and procedures designed to reduce the time and costs associated with compliance.

The evidence collected during the various research tasks (and supported by other materials such as recent evaluations of programmes administered by EACEA) shows that in some ways applicants and beneficiaries expect to spend significant amounts of time and effort on administration. With this in mind, the key considerations are whether:

beneficiaries feel that administrative requirements are reasonable and proportionate; stakeholders and project promoters can perceive general progress or specific improvements in reducing the

administrative burden; and information and reporting requirements do not create a significant deterrent effect for future application.

As regards the first criterion, the online survey of applicants and beneficiaries included a question asking whether respondents agreed that “administrative requirements were reasonable (compared to the size of the grant and scale of the project)”. This question produced an average score of 3.7 on the scale of 1 to 9, with 63% agreeing and 22% disagreeing, suggesting a broadly positive result. This can also be broken down by programme, with Europe for Citizens56 (Tempus57 (Youth in Action, Lifelong Learning and MEDIA producing significantly more positive responses than the Culture Programme (4.3)

Judging by their specific comments, while many acknowledge recent improvements in administrative processes, others are still concerned about administrative complexity. The following broad themes are discernible:

Though documents have been standardised across programmes (and in many cases shortened or simplified), there are still numerous comments relating to the layout and format of e-forms or financial statements, including many comments recounting technical problems.

Administration is felt to be more of an issue for small projects or less ‘professionalised’ organisations and first-time applicants. The first point is not necessarily supported by responses to the specific question on

56 The higher level of satisfaction amongst beneficiaries of the Europe for Citizens programme may reflect the wider use of lump-sum grants within that programme. 57 Reporting requirements were completely revised in the move from Tempus III to Tempus IV; which may have influenced the positive responses under Tempus

34

administrative requirements, where there are only marginal differences in the responses produced by different types of beneficiary organisation.

Administrative burden is not just a matter of the amount of work required, but also the time available to complete it. A number make comments about the length of time between publication of annual priorities or calls for proposals and the deadlines for application, or perceived short turnaround times for information and documents.

Others comment on the amounts of information they need to process and difficulties around locating relevant information on the web site or in programme guides.

The issue of language is also raised in this context, with some respondents requesting that documents and forms be made available in all official languages, or in a greater selection of languages (i.e. in addition to English, French and German). Others make comments about the quality or clarity of language used, or criticise the use of jargon58.

Others comment that the time spent on administration means less time available for project work (i.e. developing content, working with partners or beneficiaries) meaning excessive administration could have a negative impact on the quality of supported projects. This point was also made by a number of national stakeholders.

It is probably unreasonable to expect respondents to make (unsolicited) positive comments about administration, and there is no evidence that the administrative burden for the programmes delegated to EACEA is worse than for other EU programmes. Furthermore it is not necessarily clear to respondents that EACEA has limited scope to make changes given the fact that administrative requirements are in many cases defined by the financial regulations. Having said that there may be scope to (continue to) address some of these issues, either in on-going improvements (e.g. format of e-forms, financial statements, layout of web site and programme guides) or in the new generation of programmes (e.g. timetables and turnaround times). The issue of languages is more problematic as extending coverage would have major cost implications.

In terms of ascertaining progress in reducing the administrative burden on applicants and beneficiaries, staff of EACEA and parent DGs (plus a number of the national stakeholders) tend to state that the process of simplification and harmonisation undertaken by the Agency has had a major positive impact. A number also commented that the Agency has completed all improvements possible under the current generation of programmes.

The current survey of applicants and beneficiaries produces a very similar result to the previous survey (from 2010) against the question on administrative requirements, though recent beneficiaries (those applying in 2011) give a slightly more positive response against this question than those from earlier years. This information suggests a broadly positive trajectory, and while a number of beneficiaries make positive comments about the improving management arrangements, it does not appear that these have translated into clear quantitative improvements in the views of applicants and beneficiaries on the overall administrative burden.

The survey also shows that the administrative requirements are not likely to deter many applicants from applying again in the future. In response to the question, “Would you consider applying for funding under these programmes again?”, only 4% of respondents to the current survey stated that they were either unlikely or very unlikely to apply for funding under these programmes again. Moreover, successful applicants (2%) – who have experienced the administrative requirements related both to application and to receipt of funds - were much less likely than unsuccessful applicants to be reluctant to apply again (9%), perhaps suggesting that the administrative requirements are not a significant deterrent.

It is also possible to break down responses to this question by programme, highlighted in the following table. The most positive responses on balance are provided by applicants under MEDIA, Youth in Action and Lifelong

58 Although we are dealing with relatively small numbers in relation to the overall survey sample, these comments are among the most frequently made

35

Learning, while (although numbers are small) applicants under Tempus and Bilateral Cooperation are slightly more likely to provide a negative response to this question than other programmes. This appears to be connected to the fact that success rates are comparatively low for these programmes (especially in the case of Tempus, see table 2.5) rather than any issue with the administration of these programmes. This point is reinforced by comparing satisfaction rates by programme contained elsewhere in this report (for example in figure 4.4. below).

Table 4.8 Views on applying for funding under these programmes again (Q41) by Programme Prog2 Certainly Probably Don’t

know / unsure

Probably Not

Certainly Not

Average for all programmes 70.8% 18.7% 6.3% 3.1% 1.1% Bilateral Cooperation 52.9% 26.5% 11.8% 8.8% 0.0% Culture Programme 65.6% 22.3% 7.7% 3.0% 1.4% Erasmus Mundus and Intra-ACP 69.3% 17.6% 8.0% 4.0% 1.0% Europe for Citizens 66.3% 21.0% 7.6% 3.9% 1.2% Lifelong Learning 72.8% 18.6% 5.7% 2.1% 0.8% MEDIA Programme 82.7% 9.5% 3.8% 2.9% 1.1% Tempus 58.8% 21.9% 9.1% 5.9% 4.3% Youth in Action 74.7% 17.2% 4.8% 3.0% 0.3%

Source: 4,027 responses to Q41, proportions for each programme vs. average

In fact, the qualitative responses offered by those applicants stating that they were either unlikely or very unlikely to apply again for funding suggest a variety of reasons for their reluctance. Of course, these comments should be seen as anecdotal rather than representative, having been offered by a self-selecting sub-set within the 4% of respondents that were unlikely to apply again. However, the 149 responses received can be grouped into a set of broad themes that offer an indication of the barriers perceived by this group of applicants. In approximate order of frequency, the main reasons offered are:

Perceived lack of transparency relating to selection process and criteria

Resources or time required for application

Skills and abilities required by applicants (especially for smaller, less professional organisations)

Excessively complex, bureaucratic or formal processes

Layout or user-friendliness of forms

Scale of information requirements

Lack of adequate feedback or advice

Perceived low chance of future success

Perception that EACEA had misunderstood or failed to appreciate the project’s concept, approach or content

Although many applicants to EU programmes expect to spend significant amounts of their time on administrative tasks, this risk is clearly weighed against the likelihood of successfully accessing funding, now or at some point in the future. The fact that competition has increased and success rates have been falling in recent years means that administrative requirements could be seen as more of an obstacle if this trend is to continue.

Main evaluation finding 13: EACEA has made significant progress towards standardising/harmonising and simplifying procedures, especially in relation to the application and selection stage. This has been driven by the Agency’s ability to recognise opportunities for simplification and greater efficiency, as well as an appreciation of the need to keep improving services.

36

Main evaluation finding 14: It is true that the current satisfaction survey does not show major improvements in levels of satisfaction with administrative requirements. However, the overall picture is broadly positive, and there is no evidence that administration is disproportionate or having a significant deterrent effect on future applications. Main evaluation finding 15: The forthcoming transition to a new generation of programmes presents EACEA and the Commission with a number of opportunities to build in major simplifications and improvements, at all stages of the project life cycle. This work is currently under way and the comments made by applicants and beneficiaries should be able to contribute to this process.

4.3.3 Proximity of outsourced activities to final beneficiaries

CBA Issue Definition Issue viii: Proximity of outsourced activities to final beneficiaries

Although EACEA manages Community programmes centrally, it aims at being closer to the programme beneficiaries, by being a "focal point" throughout the projects' life cycles. The evaluation looks for evidence around the level of direct contact with beneficiaries and stakeholders, views on approachability and responsiveness and effectiveness of communication tools

The previous evaluation highlighted the value of the Agency designating a contact person to ensure a coherent administrative follow-up on the project during its lifespan and implementation of a number of tools for improving communication with beneficiaries. The online survey of applicants and beneficiaries provides useful information on how the issue of proximity has progressed since then, as shown below

Table 4.9 Proximity (responsiveness) by survey Values 2008 2010 2012 We were satisfied with the Executive Agency's ability to respond to questions

3.1

We knew who to contact or how to contact the Executive Agency if we had a question about implementation

2.6 2.4

The answers received to questions were clear and accurate 2.5 2.7 We were satisfied with the Executive Agency's ability to respond to questions within a reasonable time.

2.4 2.7

Source: Average response for all respondents– scale of 1-9)

82% of all respondents to the current survey provided a positive response to the question about knowing who to contact, a positive result. The following chart breaks down this information by programme. It shows that respondents within MEDIA, Lifelong Learning (and to a less statistically reliable degree Tempus) tended to be significantly more satisfied than those within the Culture, Youth in Action and Europe for Citizens programmes. According to EACEA, the positive finding for MEDIA might be explained by the fact that this programme tends to have a higher proportion of repeat beneficiaries than others.

37

Figure 4.4 Proximity (responsiveness) by Programme


The survey of applicants and beneficiaries produced many complimentary comments about the professionalism or helpfulness of individual agency staff, though some commented that they were able to get more tailored advice or quicker responses from National Agencies or contact points. This should not necessarily be construed as a criticism of EACEA’s ability to be ‘close enough’ to applicants and beneficiaries as the mandate of the contact points includes the provision of support and guidance for applicants. For example, one of the tasks of the Cultural Contact Points is to “facilitate access to the Programme for, and encourage participation in its activities by, as many professionals and operators in the cultural field as possible”.59 The fact the contact points are not involved in the selection process for centralized actions (and therefore not subject to the same legal requirement to be completely impartial and respect equal treatment for all applicants under these actions) means they are perhaps better placed to give tailored guidance and support to applicants and beneficiaries in their own countries.

A key element of proximity is the level of direct contact with beneficiaries, one element being the increasing number of monitoring visits undertaken in recent years. The Agency has also developed a more formal procedure for launching projects, with kick-off meetings (for larger and multi-annual projects) explaining the steps and stages in the project life cycle, allowing beneficiaries to ask detailed questions and get to know the individuals they will be working with. A number of stakeholders comment that this is helping to reduce the numbers of subsequent problems or errors.

59 Decision No 1855/2006/EC of the European Parliament and of the Council of 12 December 2006 establishing the Culture Programme (2007 to 2013)

1 2 3 4 5 6 7 8 9

We were satis fied with

the Executive Agency's

abi l i ty to respond to

questions

We knew who to contact

or how to contact the

Executive Agency i f we

had a question about

implementation

The answers received to

questions were clear

and accurate

We were satis fied with

the Executive Agency's

abi l i ty to respond to

questions within a

reasonable time.


ALL RESPONDENTS

Youth in Action

Tempus

MEDIA

Li felong Learning



Culture


38

Analysis of documentation and the interviews provided numerous additional examples of events organised or attended by staff of the Agency, from Infodays and stakeholder events to festivals and events organised by National Agencies or contact points. While agency events are now held mainly in Brussels, significant numbers of delegates attend remotely, or download the materials provided by the Agency.

While no targets exist for numbers of events organised or attended, the quality of these events is likely to be of more direct relevance than quantity or frequency. The documentation contains many examples of positive feedback in relation to individual events, while the Agency staff interviewed for this evaluation are overwhelmingly positive about the value of participation in events and increased direct contact with beneficiaries and stakeholders in general (ensuring that this does not endanger the principle of equal and transparent treatment for all applicants). .

Effective communication is arguably another element of proximity, and the Agency has put significant effort into improving information tools, including:

Dedicated web pages and online tools about funding opportunities; Events such as information days, projects meetings and information visits; Information kits and leaflets on how to access and benefit from funding opportunities; Responding to inquiries via information mailboxes60.

The following table shows the views of applicants and beneficiaries as regards the quality of information provision by EACEA. The most positive response was made in relation to the clarity of information on deadlines, which has been supported through the introduction of stable programme guides as well as timetables and lists of frequently asked questions available on the web site. Applicants have a good level of understanding of which programmes or actions they can apply under, while the only consistent negative comments relate to the languages that information is provided in and to the ease of finding critical information in often large and complex documents.

Table 4.10 Proximity (information) by survey SURVEYS 2008 2010 2012 We understood easily under which programme (or action) our organisation could apply

2.3 2.4 2.5

The priorities and objectives of the Call for Proposals or Programme Guide were clear to us

2.9 2.9

The information provided enabled us to understand the application procedures.

2.9 3.0

We received clear information about deadlines for submitting the application

3.0 1.9

The contract was easy to understand 3.0 3.1 In the start-up phase, we were satisfied with the quality of information and advice provided

2.9 3.0

During the implementation phase, we were satisfied with the quality of information and advice provided

2.5 3.0

Source: Average response for all respondents– scale of 1-9)

The following chart shows how the results for each of these questions vary by programme. While in most cases the responses are similar across programmes, there are some differences in evidence. For example, the two questions dealing with quality of information and advice show noticeable differences between programmes, with applicants and beneficiaries under Tempus, Lifelong Learning and MEDIA providing significantly more positive responses than Culture and Europe for Citizens.

60 A small number of negative comments were made about automatic email responses, perhaps linked to the idea that personalised information is preferred.

39

Figure 4.5 Proximity (information tools) by Programme


Main evaluation finding 16: Applicants and beneficiaries provide positive responses (and numerous individual compliments) on the responsiveness and professionalism of agency staff, though they often express a dislike of standardised (i.e. non personal) responses. The more tailored advice available from national contact points plays an important role in this context.

1 2 3 4 5 6 7 8 9

During the implementation

phase, we were satisfied

with the quality of

information and advice

provided

In the start-up phase, we

were satisfied with the

quality of information and

advice provided

The contract was easy to

understand

We received clear

information about deadlines

for submitting the

application

The information provided

enabled us to understand

the application procedures.

The priorities and objectives

of the Call for Proposals or

Programme Guide were clear

to us

We understood easily under

which programme (or action)

our organisation could apply


ALL RESPONDENTS

Youth in Action

Tempus

MEDIA

Li felong Learning



Culture


40

Main evaluation finding 17: Increasing numbers of Infodays, kick-off meetings and monitoring visits have increased the level of direct contact with beneficiaries and potential beneficiaries, which is welcomed by agency staff, parent DGs and national stakeholders.

Main evaluation finding 18: EACEA has improved the quality of many communication tools, improving the availability and clarity of information through the use of tools like programme guides (especially with regard to deadlines and understanding of programmes and actions).

4.4 Efficiency: cost of programme management

CBA Issue Definition ix: Efficiency (resources needed at EACEA, DGs)

The Commission evaluation guide defines efficiency as “the extent to which the desired effects are achieved at a reasonable cost”. The efficiency question will focus on the human resources and other costs linked to the transfer of the management tasks to EACEA (costs of coordination and checks).

x: Possible savings (overall)

Prior to establishing EACEA, the CBA was concerned with identifying the potential savings induced by the transfer of programme management to the Agency. It is now possible to look at “actual” savings rather than at “possible” savings. This will be done by calculating the overall saving of the “agency solution” compared to the “All Commission solution”, as well as by identifying which are the main sources of savings.

As we have noted, an important driver in the creation of Executive Agencies has been the potential for them to reduce the financial costs of programme management, in part through the employment of less expensive staff. Indeed, this is reflected in the Court of Auditors 2009 report which stated that "the initiative of setting up the executive agencies was mainly driven by constraints on employment within the Commission rather than being based on the intrinsic features of the programmes themselves". The first interim evaluation report suggested that "on average the total costs of programme management in the Agency were approximately 20% lower than in the Commission". The evaluation went on to note that these savings mainly derived from "the difference in cost resulting from the staffing structure and from the efficiency gains achieved through economies of scale of managing several programmes”. Similarly, the 2009 CoA report concluded that "there are clear cost savings stemming from the prevalence of lower-paid contract staff, even when one considers the additional costs of the new posts created for supervision at both the Commission and the agencies".

Based on data provided by EACEA, we have updated the analysis undertaken in the first interim evaluation, which offered a comparison between the costs of EACEA compared to management by the parent DGs using average costs by categories of personnel.61 The table below offers an estimate of the average EACEA staff cost, calculated from the figures for the total staff budget and the number of staff. It also offers an average for permanent officials of the European Commission, based on figures published by DG BUDG. As with the first interim evaluation, the cost comparison assumes that the number of staff employed by EACEA to manage the programmes is equal to the number that would be employed were the programmes to be managed in-house by the parent DGs. It also assumes that in-house management would be undertaken by permanent officials of the Commission, rather than by contract agents.62 The table below also takes account of the need for the parent DGs to supervise EACEA, through the inclusion of 4.5 staff on average over the period covered by the evaluation. The evidence presented in

61 According to the Commission, applying average costs by categories of personnel is the methodology agreed with the Budget Authority for any document involving Commission human resources; European Court of Auditors, Special report No 13 (2009) Delegating Implementing Tasks to Executive Agencies: A Successful Option? 62 As a comparison, the current average annual cost of contract agents within the Commission is €64,000, according to DG BUDG.

41

the table above suggests that the cost savings identified in the first interim evaluation and in the CoA report have continued to be made, again primarily through the deployment of lower-paid staff than in the DGs.63

Table 4.11 Comparison of staff costs of EACEA compared to in-house management by the Commission (2008-11) Year 2008 2009 2010 2011 Average EACEA staff cost64 € 62 840 € 63 507 € 67 472 € 68 109 Average EC staff cost65 € 122 000 € 122 000 € 127 000 € 127 000 EACEA Staff 362 414 412 433 Total EACEA staff cost € 22 748 000 € 26 291 751 € 27 798 444 € 29 491 279 Number of EC officials involved in supervision

4.5 4.5 4.5 4.5

Total EC staff cost € 549 000 € 549 000 € 571 500 € 571 500 Total staff cost (EACEA option) € 23 297 000 € 26 840 751 € 28 369 944 € 30 062 779 EC officials 362 414 412 433 Total staff cost (EC option) € 44 164 000 € 50 508 000 € 52 324 000 € 54 991 000 Saving € 20 867 000 € 23 667 249 € 23 954 056 € 24 928 221 Source: EACEA

In considering the cost savings arising from outsourcing, the CoA report highlights the need to determine whether the corresponding Commission posts have been freed and the relevant staff redeployed. However, the same report highlights that the extent of the savings could not be verified due to a lack of reliable information on the ex ante situation; indeed, the first interim evaluation did not give consideration to this issue. Whilst the current evaluation does not cover the period during which EACEA was established, we can offer a few pointers here, based on our knowledge of that period. Firstly, since many of the EAC programmes were already outsourced to the TAO or other bodies66 in the period before 2006, it would appear that there was limited scope to free posts within the parent DGs67. Secondly, although the task of gathering data regarding staff costs at the TAO is beyond the scope of this evaluation, anecdotal evidence from the interviews suggests that salary levels of TAO staff were much lower than those of Commission staff and that similar salary differentials exist between staff of EACEA and staff of the Commission. Thirdly, the creation of new programme management posts was inevitable since the current programmes managed by EACEA are significantly larger – in terms of the numbers of additional activities delegated since 2006, the numbers of projects managed and the overall volume of funds administered – than their predecessors in the 2000-06 period. Locating these new posts within EACEA has thus created cost savings that would not have occurred if the new posts had been located within the Commission.

This helps to explain why the cost differences between the “agency solution” and “all Commission solution” for 2008-2011 are greater than that forecast in the first interim evaluation. This estimated a cost difference of €9.9m for 2008, €11.6m for 2009, €11.1m for 2010 and €10.8m for 2011, all considerably lower than the savings highlighted in table 4.12 above. As we have noted above, one further rationale for outsourcing has been that of encouraging greater professionalization, with agencies employing staff with specialist and technical skills in programme management. The 2009 CoA report confirmed that that such specialisation had occurred amongst 63 Since the costs of in-house management relate to a hypothetical scenario, there are inevitably some elements of uncertainty around the estimated savings. For example, the in-house option might involve the deployment of different numbers of staff at different levels of seniority as well as the deployment of a mix of contract agents and permanent officials. However, in its 2009 report, the Court of Auditors concluded that the savings margin attached to EAs over in-house management remained significant even including for such uncertainties. 64 Staff budget data provided by EACEA, divided by total number of staff 65 Source: DG BUDG 66 For example, Tempus was previously managed by the European Training Foundation and Eurydice by an external contractor. 67 Budgetary information provided by DG EAC estimates the extent of this at 30 posts at the Agency’s creation and an additional three for the most recent mandate extension.

42

Executive Agencies in general, resulting in better service delivery. This point was reinforced by the stakeholders interviewed for the current evaluation, amongst whom there was a consensus that the creation of specialist programme management expertise within EACEA had improved programme delivery.

The extent to which such specialisation has reduced the cost of programme management is very difficult to determine; it may be that staff time saved through efficiency gains lead to enhancement in the level or quality of service provided (e.g. more monitoring visits) rather than in costs reductions. However, by comparing key data regarding the overall workload of EACEA to the number of staff in place (as illustrated in the table below), we can draw some conclusions. First, as the number of staff has increased by 20%, i.e. from 362 to 433 (in line with the increase in the volume of funds managed and the increase in the number of applications received, albeit with some fluctuations between years), the Agency has been able to maintain broadly comparable levels of workload of staff, i.e. approaching 30 applications and more than 20 open projects (accounting for around €1.5m of expenditure) on average per individual staff member.68 This suggests the ability to recruit and deploy new staff relatively effectively without adversely affecting the quality of service provided. Second, the expansion in staff numbers over the 2008-11 period has not been at the expense of funding available for projects: staff costs as a proportion of total funds managed have remained below 5%.

Table 4.12 Key data regarding staff workload (2008-11) Year 2008 2009 2010 2011 Calls for proposals 34 31 29 29 Calls for public expressions of interest 0 6 5 1 Applications received 8 991 11 176 10 230 12 687 Applications selected 3 892 4 100 3 473 3 723 Success rate 38% 37% 34% 29% Number of projects open 8 117 8 310 8 344 8 334 Number of projects closed 5 250 5 462 4 522 4 589 Total programme expenditure69 € 488m € 625m € 600m € 621m Total staff numbers 362 414 412 433 Total staff numbers (minus Eurydice)70 334 385 381 403 Average programme expenditure per staff member

€1.5m €1.6m €1.6m €1.5m

Average number of applications per staff member

27 29 27 31

Average number of open projects per staff member

24 22 22 21

Staff costs as a proportion of total programme costs

4.7% 4.2% 4.6% 4.7%

Main evaluation finding 19: Outsourcing the management of programmes continues to provide significant cost savings relative to the costs of in-house management by the Commission. The projected cost-savings of outsourcing between 2008 and 2011 have in fact been realised and are likely to exceed the forecasts made by the first interim evaluation

68 The average workload for staff in programme units would of course be higher, since EACEA has a number of horizontal units that do not directly oversee programmes, e.g. Human Resources, Finances and Informatics, though their work clearly makes an important contribution to programme management . 69 Total commitment appropriations 70 Staff working for the Eurydice Unit have been removed from the subsequent averages, as they are not directly involved in allocating programme budgets, appraising applications or administering projects

43

Main evaluation finding 20: Staff posts created at EACEA have mostly replaced posts at the (now defunct) TAO or other bodies or represent entirely new tasks (in line with the expansion of programmes). However, a number of Commission staff posts have indeed been freed, which according to the Court of Auditors report, budgetary information provided by DG EAC and anecdotal evidence from interviews, is likely to be linked to the creation of EACEA.

Main evaluation finding 21: The staffing of EACEA has increased in line with the increase in the volume of funds managed and the increase in the number of applications received. The data suggests that EACEA has been able to absorb a large number of new recruits whilst avoiding any reduction in staff efficiency (measured in terms of average workload per person) or any increase in staff costs relative to programme funds managed.

4.5 Effectiveness: better projects and outcomes

4.5.1 Project selection

CBA Issue Definition Issue xi: Quality and relevance of selected projects

The evaluation assesses whether the management procedures put in place for the selection of projects ensured that the best projects receive funding, which is in essence the core task delegated to EACEA

Issue xii: Transparency in selection

The evaluation assess whether the management procedures put in place for the selection of projects ensured an appropriate level of transparency. A transparent selection process is defined as one that is clear, understood and seen as objective.

These issues are discussed together as they are both concerned with EACEA’s project selection processes. In terms of transparency, EACEA has a comprehensive set of arrangements in place for ensuring that selection processes are clear and objective71. This includes

guidelines on provision of information and advice, based on the principle of equal treatment (i.e. equal access to information at the various stages of the process)

description of the evaluation process, including composition of the evaluation committee and its working methods (including the use of evaluation grids and appraisal reports) avoiding conflicts of interest and preferential treatment

use of external experts, including the way they are selected and briefed, the development of consensus and quality control

the definition of eligibility, exclusion, selection and award criteria, as well as setting of quality and funding thresholds

the format of evaluation reports, including quality analysis, funding recommendations, conditional recommendations and ranking in order of quality

the process for communicating award decisions, notifying successful and unsuccessful applicants and publishing results

Results from the online survey show that applicants tend to give a positive assessment of their understanding of the selection and award criteria. While the previous evaluation noted progress in this area, it also highlighted differences between accepted and rejected applicants in relation to how they understood the principles of the selection procedure. This pattern has continued, with successful applicants (2.8) giving more positive responses than unsuccessful applicants (4.1) on their understanding of the selection procedure.

71 See EACEA Grants – Management Procedures (December 2011)

44

Table 4.13 Transparency of selection Question 2008 2010 2012 We understood the principles of the selection procedure (all applicants)

3.0 3.3 3.2

We received a clear explanation on why our application was rejected

6.0 5.0 4.9

We received useful or practical feedback on our application 5.4 Source: Average response for all respondents– scale of 1-9)

The other two questions were only asked to unsuccessful applicants, so it is perhaps unsurprising that these questions generated more negative responses. However, since the previous evaluation EACEA has sought to improve the general quality of feedback given to unsuccessful applicants and this is likely to have contributed to an increased level of satisfaction against the relevant question. The question of feedback is more problematic, as the usefulness of feedback is most likely to be judged against its potential to inform a successful application in future. Analysis of the comments made by unsuccessful applicants suggests that they are more likely to criticise experts’ understanding of their application, or apparent unfairness in the selection process, than to criticise the content of feedback.

Looking at the responses by programme, while levels of understanding remain relatively consistent, there are significant differences in the responses given in relation to the clarity of explanations and the usefulness of feedback. Responses given under Lifelong Learning, Tempus and Culture are significantly more positive than those under Europe for Citizens, Youth in Action and MEDIA.

The differences in scores by programme in the area of explanations and feedback can partly be explained by the fact that more detailed feedback is given under programmes with higher average grants (for example under LLP and Tempus). This may be more problematic for the programmes receiving large numbers of applications for smaller projects. However, respondents to the survey made frequent comments about feedback, with the most common criticisms relating to its quality or usefulness72. Feedback was often described as too short, lacking detail or overly standardised / formal, suggesting that any efforts to provide more detailed or tailored feedback on unsuccessful applications would be appreciated by applicants.

72 It is true that some respondents either disagreed with the reasons given for rejection while others (perhaps unreasonably) requested feedback that would enable them to submit successful applications in future. However most would simply have appreciated more detailed feedback on their application and its relative merits or faults, considering the time spent preparing it.

45

Figure 4.6 Transparency of selection by Programme


EACEA’s ability to select the highest quality and most relevant projects was not explored in detail in the first evaluation. Clearly a number of factors play a role in the quality of projects, such as ensuring sufficient numbers of applications; selecting the right ones; ensuring effective implementation; and also sharing good practice. Arguably the single most important determinant of quality, and therefore the main focus of this section, is the question of whether EACEA’s processes ensure the selection of high quality projects.

Information on budget executions and the on-going rise in the overall numbers of applications (i.e. the highly competitive nature of the selections) suggest that there is, in general terms, no shortage of good quality applications. Staff of parent DGs have access to information on the quality of applications (i.e. the numbers of applications meeting quality thresholds) through participation in evaluation committees and other forms of information-sharing between mirror units including the regular steering meetings. This helps to ensure that necessary remedial action can be taken.

The general issue of quality is clearly highly subjective, though the interviews of staff from EACEA, parent DGs and national stakeholders did not include negative comments on the ability of EACEA to select good projects. They did raise a number of concerns mainly related to the balance of selected projects (i.e. numbers of projects deemed innovative or from ‘young’, less established or less high-profile organisations). The most common observations can be summarised in the following way:

While the process for selecting experts is clear and transparent, a number of stakeholders have commented that the broad and multi-disciplinary nature of the fields concerned (e.g. citizenship, culture, youth policy) makes it more difficult to define experts’ levels of understanding and experience of the subject matter objectively. This could be further complicated by the turnover of experts, which is designed to balance impartiality with detailed knowledge of the policy areas.

1 2 3 4 5 6 7 8 9

We received useful or

practica l feedback on

our appl ication

We received a clear

explanation on why the

appl ication was rejected

We understood the

principles of the

selection procedure


ALL RESPONDENTS

Youth in Action

Tempus

MEDIA

Li felong Learning



Culture


46

The selection and training of experts is a critical aspect of ensuring the selection of good quality projects, so requires constant focus. Staff from the parent DGs recognise the contribution they can make in this area and are involved in work to improve selection, guidelines and briefings across a number of programmes and actions.

EACEA has general guidelines in place on the workload of external experts though in practice this depends on the size and complexity of proposals. Instances of experts being assigned a much greater workload than this are recorded in the minutes of coordination committee meetings73.

There is a move towards remote briefings and evaluations, driven mainly by a desire to make efficiency savings. Remote briefings do not necessarily impact negatively on expert’s work, and a recent evaluation of online briefings provided under five LLP actions found that satisfaction was high (95%) with experts detecting no loss of quality in comparison to face-to-face briefings74 .

Main evaluation finding 22: Information on budget execution and the level of competition at selection stage reinforces the views of stakeholders that there is generally no overall shortage of good quality applications.

Main evaluation finding 23: Project selection processes are well defined, clear and transparent. While EACEA has worked to improve the quality of feedback given to unsuccessful applicants for some programmes, the issue of feedback was the most negative single aspect of the satisfaction survey. In general terms unsuccessful applicants would appreciate more detailed, personalised feedback on their applications.

Main evaluation finding 24: The selection, training and monitoring of external evaluation experts are critical aspects of ensuring the selection of good quality projects, so should remain under constant review. The information available to us suggests that the move to remote briefings and appraisal meetings does not appear to have a negative impact on quality.

4.5.2 Effectiveness of monitoring

CBA Issue Definition Issue xii: Effectiveness of monitoring of projects and programmes

The evaluation also assesses the effectiveness of monitoring mechanisms, since the Agency is responsible for both selecting and monitoring projects.

EACEA works to a set of common guidelines on monitoring, though a number of programmes also have specific requirements, which is reflected in a degree of variation in this aspect of EACEA’s work. In general terms, this process establishes strands, countries, types of beneficiary, size and types of projects to be monitored in more detail on a programme and annual basis.

For the Agency as a whole, the main types of monitoring are desk-based (interim / progress and final reports) field-monitoring (project visits), and preventative monitoring (with guidance and advice provided on a case-by-case-basis). Reports must be accompanied by financial statements and EACEA operational officers are responsible for ensuring that reports are consistent with grant agreements (though external experts have been used in some cases, such as monitoring visits of complex cases under LLP). Agency staff select at-risk projects for further follow-up by identifying delays or other problems in implementation (especially repeat delays). A sample of completed projects is selected for ex-post audits.

73 For example - Réunion de Coordination Culture et Communication/Valorisation, 1er Juillet 2009 – MADO 17/027 74 Information supplied by EACEA Unit P1

47

There is no evidence of EACEA failing to meet the monitoring requirements of individual programmes and a number of interviewees commented that the Agency’s monitoring and control processes are strong. We also have a number of examples of programme units placing significant attention on the issue of monitoring and improving their processes. Examples include the increasing number of monitoring visits for LLP and Tempus projects and the establishment of a monitoring database of individual Erasmus Mundus beneficiaries.

EACEA has instigated improvements in a number of these areas, with a particular focus on field-monitoring. In recent years the Agency has significantly increased the numbers of monitoring visits with the table below showing that the current target of 400 project monitoring visits (around 5% of on-going projects) was exceeded in 2010 and 201175. This is universally welcomed as a way of improving project-level information held by the Agency and increasing staff members’ level of direct contact with beneficiaries and knowledge of the fields in which they operate. A number of staff from parent DGs and national contact points commented that it is important to continue to strengthen the involvement of other stakeholders in this process and the sharing of follow-up findings.

Table 4.14 Number of project monitoring visits by year 2008 2009 2010 2011 2011 Target Number of monitoring visits 241 248 412 423 400 Source: EACEA Annual Activity Reports

Another key criterion is beneficiaries’ understanding of monitoring and reporting requirements, driven by the quality of information on monitoring and reporting requirements provided by EACEA. The online survey asked two specific questions in this area and results for these are shown below.

Table 4.15 Views of monitoring and reporting by survey Values 2008 2010 2012 The instructions given on reporting procedures were clear and user-friendly

2.9 3.2

The process of monitoring our project by the Executive Agency was clear to us

3.1 3.3

Source: Average response for 2,771 successful respondents – scale of 1-9)

While broadly positive, it would appear that there has not been a significant improvement in beneficiaries’ understanding of monitoring and reporting in this period. Furthermore, a number of beneficiaries commented that the monitoring tends to be technical in nature, and is more focussed on financial compliance and quantitative outputs than content or qualitative considerations. These results can be broken down by programme, showing that there are significant differences especially for reporting instructions, with projects under Tempus, MEDIA, LLP and Europe for Citizens most likely to give responses that are more positive (to a statistically significant degree) than those under the Culture Programme.

The fact that National Tempus Offices are directly involved in monitoring may play a role in the positive result for this programme, and this model could have beneficial effects for some other delegated programmes.

75 In practice, the intensity of monitoring visits depends on the type of projects funded and the size and complexity of activity under the various programmes

48

Figure 4.7 Monitoring and reporting by Programme

Source: Online survey (2,771 successful respondents)

As highlighted earlier in previous sections, EACEA are in the process of introducing a number of improvements to monitoring and reporting procedures. This includes:

Expanding the number of kick-off meetings (for larger and multi-annual projects).

Improving the information provided to beneficiaries on financial obligations, audits and ex-post controls through the development of an information kit.

Trialling a sampling-based system for verifying participants’ expenses for small grants and eligible costs calculated on the basis of predefined ranges in the number of participants.

Introducing audit certificates for all programmes from 2013. In future beneficiaries will be able to submit a “Report of Factual Findings on the Final Financial Report” providing this is prepared by an independent, qualified auditor. This should help to smooth the process of finalising projects and claiming final payments, with beneficiaries able to reclaim auditors’ fees.

After trialling e-reporting on the ‘Networks of Twinned Towns’ measure, a system of E-final reports will be extended to cover the whole of the Europe for Citizens programme. Under this system beneficiaries are emailed a final report document pre-filled with previously supplied information. The pilot reported that this was welcomed by beneficiaries for reducing the time required to complete but was also easier for the Agency to process and ensured more accurate information that is easier to analyse and disseminate.

Main evaluation finding 25: EACEA’s monitoring and control processes are strong, though there has been less progress during the period covered by this evaluation in streamlining monitoring and reporting requirements than other elements of the project life cycle.

Main evaluation finding 26: The number of monitoring visits has increased substantially in recent years, and while it should be possible to further increase the overall numbers the cost of this should continue to be balanced against available resources. It is important that the results of field visits are always shared with the parent DGs.

Main evaluation finding 27: The main focus of monitoring is on financial and ‘technical’ aspects of the ways in which work programmes have been implemented. However monitoring information (including the results

1 2 3 4 5 6 7 8 9

The instructions given on

reporting procedures

were clear and user-

friendly

The process of

monitoring our project by

the Executive Agency was

clear to us


ALL RESPONDENTS

Youth in Action

Tempus

MEDIA

Li felong Learning



Culture


49

of project visits) is important not only for checking performance and the correct use of public funding but also plays a key role in assisting the identification of good practice and the dissemination of results.

4.5.3 Visibility of the EU as promoter

CBA Issue Definition Issue xiv: Visibility of the EU as promoter

It must be clear to stakeholders and beneficiaries that the European Union is the promoter and funder of all programmes delegated to EACEA. The Agency shall always specify that it is acting under powers delegated by the Commission, follow guidelines on guidelines on information and visibility of programmes and use of graphics and images in all documents and standard templates.

This was perhaps a more significant issue for the first evaluation, as EACEA was a new organisation, and there may have been concerns that the creation of a new agency could create uncertainty amongst stakeholders and beneficiaries over responsibility for the programme or origin of funding.

This was not raised as a general area of concern in the interviews of agency and parent DG staff and there is no evidence of any programme-related documents or materials failing to make clear reference to the EU programmes or put the role of EACEA in its proper context. The use of EU flag and other branding by the Agency is consistent, while all grants are subject to a publicity clause, where any communication by a beneficiary must acknowledge the role of EU funding.

A small number of stakeholder interviews and responses to the survey of applicants and beneficiaries suggested that there may be a degree of uncertainty (this was felt to be most true of smaller organisations or those having less direct contact with or understanding of the institutional arrangements about the respective roles of individual organisations in the management of some programmes (examples given include Europe for Citizens, Erasmus Mundus and aspects of MEDIA and LLP). This could potentially further reduce the amount of contact the DGs have with projects (or beneficiaries’ contact with policymakers) but this is not necessarily a question of the visibility of the EU. The majority of those contributing to this evaluation feel that it is clear that EACEA is part of the European Commission and that the EU is the funder of all delegated programmes.

Main evaluation finding 28: The Agency approach has not created problems linked to the visibility of the EU as promoter and funder of programmes. Despite EACEA’s efforts to ensure clarity in official documentation, there may remain some confusion amongst applicants and beneficiaries around the respective roles of EACEA and parent DGs, even though all are seen as part of the European Commission.

4.5.4 Exploitation of results

CBA Issue Definition Issue xvi: Exploitation of results achieved by the projects managed by EACEA

The promotion and dissemination and exploitation of projects’ results are part of the Executive Agency’s mission. The evaluation takes into account EACEA’s specific role in the dissemination of project results and to what extent it contributes to the process.

One of the benefits of having one single agency managing a number of different programmes is when it comes to exploiting and disseminating the results of the projects implemented under these programmes, both inside and outside the programme. Dissemination among beneficiaries can help improve the quality of projects while communication of the results to the outside community can ensure that the benefits of the programmes are recognised on a wider scale.

The previous evaluation identified "communication with the public" and "the use of project results" as areas where clarification was needed with regards to the division of tasks between EACEA and the Commission, though it would

50

appear that this situation has become clearer. In terms of communication, the Agency’s primary responsibility is towards actual and potential beneficiaries rather than the general public, and the same is true of dissemination / exploitation of results. The main responsibility for the dissemination of project results lies with the projects themselves, with DGs (and National Agencies in the case of the decentralised elements) assuming primary responsibility for disseminating programme-level results, though the Agency plays an important role in providing support.

The online survey asked beneficiaries to rate the usefulness of dissemination and promotion support provided by the Agency, with projects giving a broadly positive answer (3.7). Although this question was not asked in previous surveys, those whose most recent application was in 2011 generate a slightly more positive response (3.6) than those who applied in 2008 (3.9). It is also possible to break down responses against this question by programme, showing that applicants and beneficiaries under Tempus and MEDIA, Lifelong Learning and Europe for Citizens rate the Agency’s support for promotion and dissemination significantly more highly than those under Youth in Action or Culture. This is more likely to reflect variations in the nature or ‘specificities’ of each programme, rather than significant differences in the quality of available support.

Given the detailed knowledge of projects held in-house, the Agency has an important role in supporting dissemination by the DGs and national [agencies or] contact points. One element of this is providing information for the several ‘valorisation’ platforms such as EVE76. This information is currently transferred automatically in most cases, with the exception of MEDIA77, and as an illustration the numbers of projects uploaded to EVE is reproduced below.

Table 4.16 Dissemination via EVE portal 2008 2009 2010 2011 2011 Target Projects transferred to EVE portal 900 2,689 1,400 1,000 Project information checked by EACEA78 310 Source: EACEA Annual Activity Reports

While this shows that project-level information is being uploaded in significant quantities, it is less clear what impact this is having on the dissemination or exploitation of results. A more comprehensive assessment requires an assessment of the quality of this information and the extent to which it is accessed and utilised.

For its part, EACEA has defined an approach to sharing and disseminating information on the projects it manages, led by the unit in charge of administration and communication and named representatives in each unit. For most programmes these members of staff are the first point of contact when responding to information requests from the DGs, and are able to recommend projects and highlight success stories or high impact projects (or indeed projects not felt to be suitable for dissemination for various reasons)79.

Some of the staff of parent DGs and national contact points commented that while Agency staff are very responsive and helpful, there was scope to further improve information sharing, or make the sharing of information on funded projects and their results more proactive or systematic. This is an important issue as sharing existing good practice will help the development of high good quality projects.

76 http://ec.europa.eu/dgs/education_culture/eve/ 77 The automated upload of MEDIA projects onto EVE is currently prevented by a technical issue (MEDIA uses a different IT management tool to the other programmes), though this is being addressed by DG EAC and EACEA and will be resolved as part of an upgrade to the EVE platform. 78 Consultations suggested that project information on EVE is checked on a weekly basis. 79 In the case of LLP most requests from the DGs go directly to the responsible unit, unless the issue requires a programme-level response in which case, the response will be dealt with by the coordination team.

51

Main evaluation finding 29: The division of tasks relating to communication with the public and use of project results has been clarified since the previous evaluation, supported by the implementation of ‘Modalities of Cooperation’.

Main evaluation finding 30: The Agency supports projects in dissemination and provides project-level information to parent DGs and national contact points. Further ways of ensuring that this information sharing happens on as automatic or systematic a basis as possible should be explored, as better dissemination of results and demonstration of impact is likely to be a key priority for the future generation of programmes.

4.6 Effectiveness: enabling the Commission to fulfil its political role

4.6.1 Improved focus on the Commission on its institutional tasks

CBA Issue Definition xvii: Improved focus of Commission on its institutional tasks

One of the purposes of entrusting EACEA with the management of programmes is to allow the Commission to better focus on its institutional tasks (programme design and policy developments). The evaluation will assess whether this outcome has been achieved.

The legal basis for the creation of Executive Agencies suggested that one of the main rationales for outsourcing was to make it easier for the Commission to concentrate on its core ‘institutional tasks’80. At the time of the first evaluation of EACEA, it was found that outsourcing had provided clear benefits with respect to this issue:

"The delegation of programme management to the Agency is beneficial for the Commission as well because it can focus on policy and programme development...”

Interviewees with staff of the parent DGs and the Agency’s management suggested that this was likely to be a less important driver than the desire to achieve efficiency savings or improve the management of programmes, though a number of Commission staff confirmed that out-sourcing to EACEA had potential indirect benefits in terms of allowing them to focus on core institutional tasks.

This picture is by no means uniform, and it depends on the roles and responsibilities of individuals and units. Several units reported "efficiency gains" that allowed them more time for policy and programme work, with less time required to be devoted to routine programme management tasks; while none contradicted the overall assertion. A key benefit, reported by several units of the DGs, was the ability to focus on overall programme design and oversight without the need to address issues arising from specific projects and without any feeling of loss of control over their broad direction of their programmes. A number of specific examples of this were given in the course of the fieldwork, including:

“Commission staff do have more time for policy and programme work, as the agency solution means they have to spend much less time dealing with funding rounds, programme procedures and so on”

“When you are in charge of implementing projects with calls 3-5 times per year you are constantly focussed on this, which makes it difficult to think about changes, for example improving the monitoring regime, or you only work on it when you have finished your main priority (organising selections, signing contracts, making payments etc.). The agency solution can be seen as part of freeing your workload of your unit of these kinds of tasks and allow you to do more.”

80 Council Regulation (EC) No 58/2003

52

Overall, the interviewees tended to articulate a clear and shared understanding of the respective roles of the DGs and EACEA and none of the interviewees particularly complained of any encroachment on their role by their counterparts at ‘mirror units’. There is perhaps inevitably some degree of overlap and this tends to relate to the gathering and exploitation of policy knowledge emerging from or relevant to the programmes. This results from the fact that the programme management role at EACEA is not merely an administrative task disconnected from the content of programmes; the management of EACEA requires a degree of understanding of the relevant policy area and this understanding is in turn enhanced by day-to-day contact with projects. Moreover, interviewees at the DGs and at EACEA highlighted the fact that policy and programmes are not developed "in a vacuum" and that staff of the DGs required a practical knowledge of the implementation of the programmes. Overall, interviewees did reinforce the need for continued and regular communication between the DGs and EACEA, with EACEA staff keen for the DGs to draw on their policy knowledge (gained through exposure to the programmes) when designing future policies and programmes.

Main evaluation finding 31: Though not the primary motivating factor behind ‘the agency solution’, out-sourcing to EACEA is probably having a beneficial impact in terms of allowing Commission staff to better focus on core institutional tasks such as policy-making and programme development although it is extremely difficult to quantify the extent of this.

Main evaluation finding 32: There is a clear and shared understanding of the respective roles and responsibilities of the parent DGs and EACEA, albeit with a degree of complementarity in respect of the gathering of policy knowledge emerging from or relevant to the programmes. This aspect of the Agency’s work is comparatively new and has only recently been formalised81.

4.6.2 Level of know-how retained within the Commission

CBA Issue Definition xvi: Level of know-how retained inside the Commission

It is important for the Commission to retain a high level of knowledge of the programmes and of the sectors they aim to support, in order to keep improving the design of the Community programmes. Achieving this depends upon the flow of information between EACEA and the parent DGs.

The Council Regulation establishing the Executive Agencies also highlighted the risk of loss of know-how within the Commission and thus stipulated that cost-benefit analyses of outsourcing should take such loss of know-how into account. Again, at the time of the first evaluation, any loss of know-how appeared to have been limited:

"Most of the EU officials interviewed have answered that the close interaction between EACEA and the parent DG is a positive contribution to know how development in the Commission. [...] The interaction between the Executive Agency and the Commission allows Commission staff to get an overview of the project life cycle and benefit in terms of knowledge accumulation of programme management".

The evidence gathered for the current evaluation suggests that there have been some improvements to the positive situation identified by the last evaluation. Moreover, those staff that had been in post before 2006 tended to report a gain in know-how compared to the previous situation in which the management of some programmes was outsourced to the Technical Assistance Office (TAO). This is due in part to the formal mechanisms that have been put in place to facilitate the flow of information between EACEA and the parent DGs including:

Sector-specific modalities of co-operation that set out the respective responsibilities of and the relationship between EACEA and the parent DGs;82

81 EACEA – EAC Modalities of Cooperation in relation to Policy Support in the field of Education (including Eurydice) and Youth (November 2012) 82 Modalités de coopération, reporting et de supervision entre les DG EAC, COMM, DEVCO et l'Agence exécutive".

53

Standard reporting to the parent DGs by EACEA every quarter; Contributions by EACEA to briefings prepared by the parent DGs for their Commissioners; Regular, scheduled meetings between EACEA and the parent DGs at Director level and unit level; Attendance of the Director of EACEA as an observer at DG EAC's Board of Directors meeting; Participation of a representative of the parent DG in project selection panels; Shared libraries, e.g. "SharePoint".

The formal mechanisms have been complemented by many informal means of sharing information and communication between the relevant units at EACEA and the parent DGs. These tend to vary, reflecting the size and nature of the programmes concerned as well as the different individuals involved. Typically, they include ad hoc requests for information sent by the parent DGs (e.g. in response to requests from spokespersons of the DG and/or press criticisms) and day-to-day contact by phone, e-mail and in person as and when necessary. Some EACEA/DG units have also invited their counterparts to attend external events such as information and communication events for applicants, dissemination conferences, etc.

Staff interviewed at EACEA and the parent DGs expressed their general satisfaction with both the formal and informal means of communication. Indeed, the consensus was that they were generally fit-for-purpose without being onerous in terms of the resources required. There has also been a gradual increase in trust at all levels. Staff of the parent DGs reported their satisfaction with the formal reports provided by their counterparts at EACEA, with some highlighting instances of specific improvements, such as the more systematic provision of reports from project monitoring visits. EACEA was seen as being particularly effective and punctual in providing accurate quantitative information to the DGs, reflecting the continuous improvement in the internal systems and processes within EACEA. However, one particular difficulty highlighted by staff of EACEA and the DGs is that while the DGs have access to the databases for the programmes for which they are responsible (subject to data protection procedures), they do not currently have an easy-to-use reporting environment83.Based on interviews with staff of the parent DGs, there may be potential to improve the flow of qualitative information from EACEA, in terms of both the volume and the quality. Those interviewees offering an opinion did not suggest any lack of willingness or ability (except perhaps due to resource constraints) on the part of the relevant individuals EACEA. Indeed, there was recognition from the same interviewees that the transmission of such knowledge, e.g. related to policy learning, good practice projects, is inherently more difficult and time-consuming. At the same time, several interviewees at EACEA were keen to stress that over time they had developed policy knowledge of their field which went far beyond the technical knowledge required to fulfil a narrow programme management role. Some highlighted the knowledge of EACEA regarding the situation "on the ground" based on its day-to-day contact with projects. Those individuals, whilst recognising that the DGs retained responsibility for policymaking, were keen for this knowledge to be put to practical use in the policymaking process. Indeed, at least two units at EACEA suggested that the parent DGs could do more to exploit the knowledge held at EACEA, though there was some recognition that the DGs had in recent years become more open to doing so. For the parent DGs to make full use of the knowledge held at EACEA, it may be not necessary to introduce specific formal mechanisms across all programmes. However, the situation will vary from programme to programme and the issue should therefore be a regular item for discussion between the programme units of the DGs and their counterparts at EACEA to see if there are any specific and simple steps that could be taken. This might include routine consultation, for example, through the involvement of EACEA representatives in some internal meetings within the DGs, albeit in an advisory rather than decision-making role. Representatives of EACEA have also been actively involved in the various working groups set up to prepare the new programmes for the 2014-20 period.

Overall, we can perhaps conclude that the separation of the Commission's political role from the programme management role inevitably creates a degree of distance between the DGs and the practical implementation of the programmes for which they are responsible. However, the evidence emerging from the research suggests that this gap can be overcome by a combination of factors including formal and informal communication mechanisms, the

83 For the new generation of programmes, the Agency and DG EAC are planning to develop a joint reporting environment covering whole programmes.

54

introduction of some practical tools, a degree of flexibility on both sides and the gradual development of trust and co-operation between the various individuals involved.

Main evaluation finding 33: There has been a continuous improvement in the flow of information between EACEA and the parent DGs. Staff of the parent DGs tend to receive the information they require either as a matter of course or on request.

Main evaluation finding 34: The inevitable gap created by the separation of the Commission's political role from the programme management role tends to be overcome by a combination of factors including formal and informal communication mechanisms, the introduction of some practical tools, a degree of flexibility on both sides and the gradual development of trust and co-operation between the various individuals involved.

Main evaluation finding 35: There may be potential to improve the flow of qualitative information from EACEA, in terms of both volume and content, in order for the parent DGs to benefit more extensively from the practical knowledge held by EACEA.

55

5 Conclusions and recommendations (Task 4) The Terms of Reference (ToR) for this evaluation include a list of twelve evaluation questions to be addressed. Answers to these questions have, in effect, been provided through the analysis contained in sections 2, 3 and 4 (notably the cost-benefit analysis). However, in order to demonstrate that we have complied with the ToR, we present our conclusions here in the form of answers to the evaluation questions. These answers draw on the evidence and analysis throughout the report, particularly the "main evaluation findings". The conclusions have in turn informed a set of linked recommendations. While the main focus of the evaluation is on the overall arrangements for delegating programme management tasks to EACEA, individual recommendations are directed at either the Agency or the Commission, or both institutions, and this is made clear in each case.

Q.1: To what extent is EACEA relevant to the needs it is intended to meet?

The evidence gathered suggests that EACEA, as a dedicated programme management body operating within the overall institutional framework of the European Commission is relevant to the needs of the parent DGs and, indeed, represents an improvement on the previous approach of delegation to the TAO. Moreover, the experience of the last few years (including the expansion in the number of programmes overseen by the parent DGs) confirms the continued need for a dedicated agency deploying specialist staff to manage a range of programmes. EACEA has proved itself sufficiently flexible to respond to the increased demands from the parent DGs for additional programme management services, in some cases at relatively short notice. There may be scope for EACEA to further improve its ability to react by promoting greater staff mobility between programme units and by exploring ways to spread its workload more evenly over the year (e.g. by co-ordinating the different programme timescales more closely), albeit whilst maintaining alignment with the academic year for those programmes related to education). Recommendation 1: EACEA should consider ways to improve its ability to promote greater staff mobility between programme units, whilst the Commission and EACEA should together explore ways to develop career pathways with other EU bodies such as the other Executive Agencies. Recommendation 2: For the forthcoming programming period, the parent DGs and EACEA should together explore ways to enable EACEA to spread its workload more evenly over the year, by seeking closer co-ordination of the timescales of different programmes. Q.2: To what extent is EACEA operating according to the legal framework establishing it? The first interim evaluation concluded that EACEA was functioning in accordance with the relevant legal framework and the current evaluation has found no evidence to suggest any deviation from this framework. Indeed, steps have been to taken to further strengthen EACEA's internal operations and procedures and thus help ensure continued compliance. A 2009 report by the Court of Auditors concluded that the Commission’s supervision of Executive Agencies in general is quite limited, though it did not suggest failure to comply with the legal framework. Q.3: To what extent has EACEA achieved its objectives? EACEA has substantially achieved its objectives, with clearest progress against those related to improving management of programmes and better services (see Q4 below) and reducing the cost of programme management compared to management by the Commission (Q6).

5 Conclusions and recommendations (Task 4)

56

There has been some progress against all annual priorities set out in Annual Work Programmes, though information in Annual Activity Reports tends to focus more on budgetary execution and how tasks have been carried out than on the results produced. There is most evidence in support of progress against the objective of streamlining project selection processes and least evidence of progress in terms of improving project implementation and reporting requirements, though it should be borne in mind that this is the main current focus for improvements (with further improvements planned for the next generation of programmes).

Q.4: To what extent has EACEA led to an improved management of the programmes and better services to the stakeholders and addressees as compared to the alternative options?

Those stakeholders that were able to offer an opinion were unanimous in reporting that the service provided by EACEA represented a significant improvement on that previously offered by the TAO (although such comparisons are difficult to make given the time that has elapsed since the closure of the TAO). The potential for applicants and beneficiaries to compare the service offered by EACEA to the alternative options is necessarily limited, since few have any direct experience of such options. However, respondents to the online survey have tended to report a steady improvement in service over the period covered by the evaluation. EACEA has also shown consistent progress against its main output indicators, notably the time taken to select projects, award grants, arrange contracts and make payments, as well as the level of operational and administrative budgets committed and paid. Satisfaction levels amongst applicants and beneficiaries are positive and have remained relatively constant, despite increases in the Agency’s overall workload. EACEA has made significant progress towards standardising/harmonising and simplifying procedures, especially in relation to the application stage. These improvements have not translated into unambiguous improvements in levels of satisfaction compared to the previous satisfaction survey, although it is not clear whether it is realistic to expect satisfaction to continue improving at the same rate, especially in the context of increasing competition for funding (and therefore numbers of unsuccessful applications). However, the least positive aspect of the current satisfaction survey relates to the quality of feedback provided on unsuccessful applications, which is regularly described as too short, too standardised or lacking detail on the relative merits or faults of an application that is likely to have had significant amounts of time invested in it. The potential for EACEA to minimise the administrative burden for applicants is necessarily limited by the requirements of the Financial Regulation or the design of programmes; many of the frustrations of applicants relate to issues that are beyond the control of EACEA. However, the feedback from applicants is that the administrative requirements placed on them by EACEA are, on the whole, reasonable. Applicants recognise that improvements have been made in the administrative processes, although there is scope for further improvements, for example, relating to the layout and format of e-forms, location of information on the web site or in programme guides, and improvements in the information that beneficiaries have to provide in the final report. The new generation of programmes presents further opportunities to streamline processes and procedures. In terms of the proximity to beneficiaries, EACEA has performed well, with applicants reporting a high level of satisfaction in respect of knowing how contact at EACEA, who to contact and EACEA's ability to respond to questions within a reasonable time. EACEA has also significantly increased the numbers of monitoring visits and improved the various communication tools it uses. Recommendation 3 EACEA should continue to seek ways to increase the streamlining of processes still further, especially in relation to later stages of the project life cycle such as monitoring, control and reporting, as well as any opportunities presented by the transition to the new generation of programmes from 2014. Any future evaluation should consider the extent to which both have been achieved and seek to collect evidence of any efficiency savings generated as a result (Q6). Recommendation 4 EACEA should explore ways of improving feedback to unsuccessful applicants when considering future improvements in processes. Initially this could mean applying lessons from the

57

programmes where satisfaction with feedback is highest to the other delegated programmes wherever it is feasible to do so. Recommendation 5 EACEA should continue to seek improvements in application and information tools, particularly relating to the layout and format of e-forms and the location of information on the web site and in programme guides. Q.5: To what extent is the coordination between EACEA and the Commission services (including the parent DGs, relevant horizontal services and offices) working satisfactorily? Coordination between EACEA and the Commission services is generally functioning well with representatives of both parties reporting their overall satisfaction with the current situation. The formal means of co-ordination (including Steering Committee, Co-ordination Meetings, co-operation manuals and regular reporting) have been complemented by a willingness to co-operate informally both on a day-to-day basis and on specific occasions, including public events. Co-ordination has also been facilitated by the clear and shared understanding of the respective roles and responsibilities of the parent DGs and EACEA. A 2009 report by the Court of Auditors has concluded that the Commission’s supervision of the Executive Agencies is quite limited, though it did not suggest failure to comply with the legal framework. The Commission has taken steps to address some of the issues raised in the report, although the report as a whole does not appear to have been considered by meetings of the EACEA Steering Committee. Some issues remain to be resolved, such as the late adoption of EACEA’s Annual Work Programmes. However, the Commission’s supervision strategy should address the report’s findings, though it may be appropriate for the implementation of this strategy to be reviewed by the Steering Committee and in EACEA's Annual Activity Reports. Recommendation 6 The Commission should prioritise the revision of its supervision strategy while the Agency should include progress on its implementation as an annual item on the agenda of the EACEA Steering Committee, and in its Annual Activity Reports. Recommendation 7 The Commission should take steps to ensure that the Annual Work Programme of EACEA is adopted at an early point in each year. Q.6: To what extent has EACEA carried out its work efficiently? Outsourcing the management of programmes continues to provide significant cost savings relative to the costs of in-house management by the Commission, primarily through the deployment of lower-paid staff than in the DGs. Staff posts created at EACEA have mostly replaced posts at the (now defunct TAO) or represented entirely new tasks (in line with the expansion of programmes). This suggests that the potential cost-savings of out-sourcing have in fact been realised, and are in fact likely to exceed the forecasts for 2008-2011 contained in the first interim evaluation of EACEA. Throughout the period covered by the evaluation, the staffing of EACEA has increased in line with its growing responsibilities (e.g. Eurydice), the increase in the volume of funds managed and the increase in the number of applications received. During this time, EACEA has been able to absorb a large number of new recruits whilst avoiding any reduction in staff efficiency (measured in terms of average workload per person) and any increase in staff costs relative to programme funds managed. Q7: To what extent does the procedure put in place for the selection of proposals ensure (a) that the projects of the best quality are funded, and (b) an appropriate level of transparency? Could it be further improved under the current framework? Information on budget execution and the level of competition at selection stage reinforces the views of stakeholders that there has generally been no shortage of good quality applications. Project selection processes are well defined, clear and transparent and there does not appear to be significant scope to further improve selection

58

procedures under the current framework. The selection, training and monitoring of experts plays a critical role in ensuring the selection of good quality projects. Given the move to remote briefings and appraisal meetings this aspect of EACEA’s work should remain under review.

Recommendation 8 In order to maintain the effectiveness of external experts, EACEA should keep procedures under review and consult the parent DGs on at least an annual basis regarding the suitability of experts and the quality of their advice and recommendations.

Recommendation 9 The impact of carrying out greater numbers of remote briefings for (and appraisal meetings with) external experts should continue to be monitored by the Commission and EACEA in order to ensure that the quality of project evaluations is maintained.

Q.8: To what extent have the mechanisms applied by EACEA for monitoring of projects ensured a proper implementation of the programmes?

EACEA’s monitoring and control processes are strong and help to ensure proper implementation of the programmes, though there has been more progress for application and selection procedures than for monitoring and reporting. However, a number of improvements in these areas (such as e-reporting and the use of audit certificates) have been piloted and are in the process of being implemented across programmes.

The main focus of monitoring is on the financial and ‘technical’ aspects of the ways in which work programmes have been implemented. However monitoring information (including the results of project visits) is important not only for checking performance and the correct use of public funding but also plays a key role in assisting the identification of good practice and the dissemination of results. The amount of direct contact with beneficiaries has increased, especially in relation to the number of project monitoring visits and it is important that the results of field monitoring visits are provided to the parent DGs.

Q.9: To what extent have EACEA's internal organisation and procedures been conducive to its efficiency?

There has been a general improvement in the level of technical and financial expertise at the same time as the overall staffing levels of EACEA have increased. The rate of occupancy of staff posts, which was below the target of 95% in 2008, has exceeded the target since 2009. While turnover is comparatively low staff retention could be further improved by promoting greater internal mobility and exploring ways to create progression pathways between the different Executive Agencies (Q1). Improvements in technical and financial expertise are reflected in a general improvement in the performance of EACEA. Further improvements might be achieved through the strengthened training offer, provided that attendance rates at training sessions can be maximised, e.g. through providing modules of shorter duration, or held in different locations. Q.10: To what extent has EACEA enabled the Commission to better focus on its institutional tasks? Though arguably not a key driver of the ‘agency solution’ delegation to EACEA is probably having a beneficial effect in terms of allowing Commission staff to better focus on their core institutional tasks. There is a clear and shared understanding of the respective roles and responsibilities of the parent DGs and EACEA, albeit with an appropriate degree of complementarity in respect of the gathering and exploitation of policy knowledge emerging from or relevant to the programmes. Q.11: To what extent has the Commission, in the presence of EACEA, been able to maintain an adequate level of know-how in relation to the programmes entrusted to EACEA? How was this achieved? There has been a continuous improvement in the flow of information between EACEA and the parent DGs. Staff of the parent DGs tend to receive the information they require either as a matter of course or on request. The

59

inevitable gap created by the separation of the Commission's political role from the programme management role tends to be overcome by a combination of formal and informal communication mechanisms, practical tools, a degree of flexibility on both sides and the gradual development of trust and co-operation between the various individuals involve. However, some representatives of the parent DGS commented that there may be potential to improve the flow of qualitative information from EACEA, in terms of both volume and content, in order for the parent DGs to benefit more extensively from the practical knowledge held by EACEA. Recommendation 10 The Commission and EACEA should consider the possibility of further increasing the sharing of practical knowledge gained by staff of EACEA through their close proximity to beneficiaries and involvement in day-to-day management of programmes. In line with the findings on monitoring procedures (Q8), this could be supported by improving the quality of information provided to the Commission regarding the outcomes of monitoring visits.

Q.12: To what extent have the activities of EACEA resulted in unintended effects (both desirable and undesirable)? Some staff of EACEA have been recruited after completing a temporary period of employment at the Commission and some have left EACEA for posts at the Commission after successful completion of the recruitment competition. Whilst EACEA has had to bear the cost of replacing such staff, it has served as a useful "bridge" between temporary and permanent employment for such individuals and thus helped to retain their expertise within the EU institutions as a whole.

EACEA's close proximity to beneficiaries and its day-to-day management of programmes have enabled its staff to gain a high level of practical knowledge relevant to the fields covered by their programmes. Such knowledge represents a valuable asset that could support the Commission in its policymaking role.

Despite EACEA’s efforts to ensure clarity in official documentation, the interviews suggested there may remain some confusion amongst applicants and beneficiaries around the respective roles of EACEA and parent DGs, even though all are seen as part of the European Commission. Given the absence of clear evidence in this area, it should be possible to include a question on the clarity of institutional arrangements (as perceived by applicants and beneficiaries) in the next satisfaction survey if this is felt to be an issue of concern.

Documents

Second interim evaluation of the Education, Audiovisual ...ec.europa.eu/dgs/education_culture/more_info/evaluations/docs/... · Second Interim Evaluation of the Education, Audiovisual