European Commission Directorate-General Enlargement, E.4 Operational Audit and Evaluation Unit Evaluation Twinning versus Technical EU Pre-accession Assistance

European CommissionDirectorate-General Enlargement, E.4 Operational Audit and Evaluation Unit

Evaluation Twinning versus Technical Evaluation Twinning versus Technical EU Pre-accession AssistanceEU Pre-accession Assistance

The consortium led by COWI/AS was awarded this assignment by DG Enlargement,Operational Audit and Evaluation Unit. Within the consortium ECORYS Nederland been responsible for

implementation of the evaluation

Evaluation Twinning versus Technical Evaluation Twinning versus Technical EU Pre-accession AssistanceEU Pre-accession Assistance

The consortium led by COWI/AS was awarded this assignment by DG Enlargement,Operational Audit and Evaluation Unit. Within the consortium ECORYS Nederland been responsible for

implementation of the evaluation

Presentation of Key Findings and Recommendations

Twining National Contact Points' meeting

Brussels, 16 June 2011

Pedro Andreo. ELARG E4, Operational Audit & Evaluation

Presentation of Key Findings and Recommendations

Twining National Contact Points' meeting

Brussels, 16 June 2011

Pedro Andreo. ELARG E4, Operational Audit & Evaluation


Outline

1. Approach

2. Key findings

3. Key recommendations


Evaluation object and scope

Comparison of delivery instruments: twinning and Technical Assistance* as provided under IPA 2007-2008, PHARE 2005-2006, CARDS 2005-2006 and the Turkey pre-accession assistance instrument 2005-2006

*) TA concluded by means of restricted international tender procedures


Evaluation objectives

• To assess the contribution of twinning and Technical Assistance instruments to the overall accession process in the Western Balkans and Turkey

• To provide lessons learned and recommendations for programming and implementation of future assistance, including through the development of cost-benefit assessment templates.


Three main evaluation questions

• Is the instrument (TA or twinning) selection done appropriately?

• To what extent does the selection result in effective and efficient project implementation with a view on sustainability?

• How can the selection process be improved (recommendations)?


Evaluation approach Focus - Three levels

Broad analysis - focus: Entire twinning and TA effort in the region and reconstruction of the decision process for selection of instrument

Sector analysis - focus: Selected sectors (Agriculture & Fisheries, Energy, Internal Market and Economic Criteria, Justice and Home Affairs and Finance)

Project analysis: Pairs of comparable projects, examples of projects, beneficiaries and providers having worked with both instruments.


Evaluation approach Tools

• 154 stakeholders were interviewed in Brussels (DG-ELARG), the EU Member States (NCPs, ex-RTAs), and in all candidates and potential candidates involved (all EUDs, all NIPAC offices, three CFCUs, altogether 35 beneficiary organisations) between March and September 2010.

• Three web-surveys resulted in 165 validly filled in response sheets from beneficiary organisations, twinning providers and TA providers


Key findings EQ1: Appropriate selection of instruments? (1)

• In practice choice is based on three criteria:

- Nature of assignment (acquis related or not)

- Maturity of beneficiary institution

- Capacity of beneficiary organisation

• No formal CBAs are performed. Perceived cost differences play only a limited role in the selection process



• Selection done by beneficiary, in a dialogue with EUDs, DG-ELARG.

• Overall beneficiaries are now well placed to make an informed selection between TA and twinning, although in Albania, Bosnia and Herzegovina, Kosovo (under UNSCR 1244/99) and the Former Yugoslav Republic of Macedonia several beneficiary organisations indicated that they still have difficulties in making a proper selection.



• TA as instrument still outnumbers twinning significantly, however in some sectors the number of twinning projects is close to half of the number of projects: ‘Agriculture and Fisheries’, ‘Justice and Home Affairs’ and ‘Finance’

• The share of twinning projects increases over the years, and is larger in candidates than in potential candidates, both in number of projects (448 TA, 181 TW period 2005-08) and in terms of budget allocated (800 TA, 190 TW period 2005-08). Croatia has most experience with twinning.



Perceived intangible benefits of twinning:

• The desire for establishing a relationship with a comparable organisation in the EU is an important driver for beneficiary organisations to select twinning

• The fact that twinning introduces a different working culture is less prominent as a reason for selecting twinning.



Although twinning, when certain conditions are fulfilled, is definitely the preferred option, the following should be kept in mind:

• Acquis knowledge is not exclusively available within twinning providers

• Use of retired civil servants as RTA, however, may take away the intangible benefit of the lasting relationship

• Some mandated bodies may not have the proper profile.


Key findings EQ2 - Does the selection result in effective and efficient project implementation with a view on sustainability? (1)

No significant difference could be observed in the achievement of results in projects applying TA or twinning. Likely causes:

• Generally, the selection process has been effective

• In cases where for acquis-related assignments TA had been selected, the reasons for choosing TA were related to the limited capacity of the beneficiary. This appears to be in those cases an appropriate choice

• TA providers can also provide relevant acquis knowledge. The important effect of lasting relationship with an EU MS partner however is unique for twinning.



Strong points twinning most often mentioned:

• Appropriate knowledge

• Fostering relationship with EU MS organisation

• Contributes to change in organisational culture.

Strong points of TA:

• Appropriate knowledge

• Flexible and controllable (‘steerable’).



Weak points of TW:

• Long preparation stage

• Limited control options for beneficiary

• Administrative burden on beneficiary

Weak points of TA:

• No mandatory results

• Price

• Sometimes too supply-driven.



Cost-comparison:

• On a unit-base (i.e. fees) twinning is on average cheaper than TA (€ 46,619/Month for TW compared to € 58,493/Month for TA. i.e. 23% higher for TA – exceptionally high cost cases excluded)

• the variance in price among TA projects is much larger. some 40% of TA projects are on a monthly base cheaper than twinning.



Main factors affecting the performance of both twinning and TA projects include:

• Realistic objectives

• Ownership by beneficiary; provider should not impose solutions

• Provider has understanding of background and environment beneficiary (this makes new EU Member States very appreciated twinning providers).


Key Recommendations (1) How can the selection process be improved?

The commonly applied selection criterion, i.e.

‘twinning is only suitable for acquis related assignments in acquis related beneficiary organisations that have sufficient capacity to absorb twinning. In all other assignments technical assistance may be more effective and efficient’

is practical and realistic. There is no reason to divert from this.


Key Recommendations (2)selecting either TA or twinning

Specifying ‘maturity’ of the beneficiary organisation (criteria):

• Is the organisation already legally established?

• Has role and mandate been laid down in legislation?

• Has the organisation made a decision on the manner in which it intends to realise the acquis?

• Is there political and/or public support for the above?

• Is management of the beneficiary organisation stable?

• Does the organisation avail of sufficient capacity (staff, space, IT etc) to absorb twinning?

If a majority of these conditions are not met: opt for TA

The intangible benefit of a lasting relation with a member state is key


Key Recommendations (3)for selecting a provider

• Insisting on more detailed twinning proposals would facilitate the selection of an appropriate twinning provider

• Demanding a personal presentation from TA providers, as part of the tender procedure, would facilitate the selection of an appropriate TA provider


Key Recommendations (4)for selecting a Tw provider

It should be ascertained that:

• Mandated bodies considered as twinning providers are really part of the public administration; and

• that they are also able to secure access to relevant public bodies and relevant expertise in their own country during and after the project.

The accreditation of mandated bodies as twinning providers is the prerogative of the respective EU member states. The beneficiaries may however use above conditions to select the most appropriate

provider.


Key Recommendations (5)for implementation of twinning

• Attention should be given to more flexibility in the execution of twinning contracts

• Preparation time of twinning projects should still be shortened


The Blue Print

A blue print to support the selection of TW-TA was proposed

as part of this evaluation


Thank you for your attention!

Pedro Andreo. Head of Sector, Evaluation. ELARG E4, Operational Audit & Evaluation

Documents

European Commission Directorate-General Enlargement, E.4 Operational Audit and Evaluation Unit Evaluation Twinning versus Technical EU Pre-accession Assistance