Evaluation Units Open Days

Some experience of Implementing Agency in evaluating Pre-Accession Entrepreneurship Development ProgrammesPolish Agency for Enterprise DevelopmentEx-post evaluation of the Phare 2000 ESC Regional Component HRD – case study and future activities in that area

Jacek PokorskiEvaluation Section PAED

Rome, Italy, July 3-6 2006

Polish Agency for Enterprise Development in brief

PAED was established in the year 2001.

It is government agency responsible for implementing sectoral policyand cooperating with regional partners in managing implementation of regional programmes towards SMEs and HRD.

Scope of PAED interest: Small and medium-size enterprises development Export Promotion Regional Development Promotion of modern technologies Job creation, human resources development and counteracting

unemployment

Its activities are financed from the State Budget and European Union resources:

pre-accession programme Phare ESC Structural Funds:

- SOP Enhancement of Competitiveness of the Enterprises (European Regional Development Fund) - SOP Human Resources Development (European Social Fund)

Evaluation in PAED PAED as the Implementing Agency for Phare ESC programmes

was obliged to evaluate programmes after the end of its implementation process.

In 2004 an independent unit for evaluation matters was establish in the PAED – Evaluation Section

Activities of Evaluation Section: designing, commissioning in tender procedure on-going and ex-

post evaluation of programmes implemented by the PAED disseminating the evaluation results taking part in evaluations managed by the National Evaluation Unit evaluation capacity building in the polish public administration

area (organizing the First Evaluation Conference in Poland entitled: “Evaluation of the socio-economic programmes financed by structural funds” http://www.konferencja-ewaluacja.pl, cooperation with universities and public institutions, organizing evaluation training courses for public administration)

Phare ESC

PHARE (Poland, Hungary - Assistance for Restructuring of their Economies) – the greatest assistance EU programme for the Central and Eastern Europe countries

From 1990 to 2000, under the framework of Phare programmes, Poland recived ca. 2,5 billion EUR

New Orientation for Phare (1997) – the Phare programmes had become an instrument for european integration. The EC extended an accountability of candidating countries in a scope of programming, implementing, monitoring and evaluation of projects co-financed from the Phare Fund.

Phare ESC (Economic and Social Cohesion) – regional and sectoral assistance to HRD, SMEs and Infrastructure Development

Phare 2000 ESC Regional Component HRD Total budget: 30,7 milion EUR (75% Phare / 25% National Co-financing) Time and location: November 2002 – February 2004 / 5 Polish voivodships

(4 eastern regions with low economy development indicators and Silesia voivodship suffering from negative effects of industry restructuring process)

Types of projects: Training and business counselling for unemployed and threatened by

unemployment Training for employees of SMEs Assistance to development of local employment pacts Specialised training for future employees of Tourist Information Network and

tourist sector Promotion of entrepreneurship in secondary schools

Objectives of Component HRDIncreasing of human potential in regions including: increase of the employment possibility, development of entrepreneurship, increase of companies and thair employees adaptation capabilities, equalising of chances of men and women in the labour market

Ex-post evaluation of the Phare 2000 ESC Regional Component HRD

Objectives of evaluation obtaining of increased knowledge on the effects of the programme

PHARE 2000 Social and Economic Cohesion – Component Human Resources Development

better understanding of mechanisms contributing to obtain those effects with the aim of improvement preparation and implementation of the programs which will be financed from the means of the Phare and the ESF and will be implemented by the PAED

Evaluation criteria: effectiveness, efficiency, relevance, sustainability Three perspectives of evaluation:

total assessment of the program assessment of the HRD Component in terms of regions assessment of particular types of pojects

Judgement technique: independent specialists assessment

Methodology of evaluation

FIRST STAGE

Initial stage - the analysis of the documentation connected with the program (desk research)

SECOND STAGE

Internal evaluation – 20 individual deepened interviews (IDI) with key people managing the projects (PAED, VLOs, conractors)

THIRD STAGE

Qualitative evaluation – 17 focused group interviews (FGI) with 96 beneficiaries of the projects (one FGI for every projects in each regions)

FOURTH STAGE

Quantitative evaluation – 1365 computer assisted telephone interviews (CATI) with beneficiaries of the projects (about 60-100 CATIs for every projects in each voivodship)

A few main outcomes of evaluation

40%

39%

32%

45%35%

21%

31%

31%

23%

27%

32%

32%

66%

29%

20%

19%

26%

26%

35%

23%

0% 10% 20% 30% 40% 50% 60% 70%

Podlaskie

Podkarpackie

Warminsko-mazurskie

Lubelskie

Slaskie

TIN trainings

All - without Slaskie

All

The percentage of legally -employ ed benef iciaris, which ware unemploy ed bef ore trainings

employability assumed

employability achieved

employability VLO

Employability of beneficiaries

Training and business counselling for Training and business counselling for unemployed and threatened by unemployed and threatened by unemploymentunemployment

Source: Enclosure of the Final Report Ex-post Evaluation of the Phare 2000 ESC Component HRD

2017

2328

2766

2846

3039

2710

4516

- € 1 000 € 2 000 € 3 000 € 4 000 € 5 000 €

Podlaskie

Podkarpackie

Warminsko-mazurskie

Lubelskie

Slaskie

TIN trainings

All trainings

The average cost of creating one workplace (EUR)

Training and business counselling for Training and business counselling for unemployed and threatened by unemployed and threatened by unemploymentunemployment

Source: Enclosure of the Final Report Ex-post Evaluation of the Phare 2000 ESC Component HRD

Assistance to development Assistance to development of local employment pactsof local employment pacts

10

8

17

4

5 19

4

5

0 5 10 15 20

Warminsko-mazurskie

Lubelskie

Podkarpackie

Podlaskie

number of pacts signed number of pacts planned to be signed

Training effects:

ASSUMPTION: signing 18 local pacts

ACHIEVEMENT: 54 local pacts signed

Assumed and achieved number of pacts established

Source: Enclosure of the Final Report Ex-post Evaluation of the Phare 2000 ESC Component HRD

Difficulties in the process of working out the local pacts

50%

25%

19%

7%

14%

22%

24%

53%

69%

0% 20% 40% 60% 80%

lack of money in a local budget

beneficiaries limited ability ofdecision making

problem of little activity of oneof the parties

lack of an institutional support

lack of conviction and belief thatsigning pacts is important

lack in knowledge of the pactsidea

problems with the politicalcontext

reluctance of one of the parties

misunderstandings and personalconflicts

Assistance to development Assistance to development of local employment pactsof local employment pacts

Source: Enclosure of the Final Report Ex-post Evaluation of the Phare 2000 ESC Component HRD

Problems and limitations of the Phare 2000 ESC Regional Component HRD evaluation

insufficient experts input in assessment process

difficulties of the evaluation team to work out useful and realistic recommendations (they had some problems with understanding the specific of public policy managing process)

limited possibilities of making comparisions: the first program in Poland in such framework the first evaluation ex-post in Poland of such scope

very extensive report: evaluators had difficulties to assess and choose the priority

issues (too much focusing on details)

the lack of possibility to construct comparision groups and estimate netto effects of the programme with utilizing data from public statistics systems

Current and future activities in that area...

Evaluations of successive editions of the Phare ESC Regional Component HRD

PHARE 2001 HRD1) beneficiaries research after 6 months (from the end of

participating in the programme) - introduction to evaluation desk research (contractors reports and other programme documents) 6200 CATIs carried out by VLOs under the PAED supervision 16 IDIs with regional projects coordinators

2) evaluation ex-post after 12 months emphasising on the qualitative methods 1 FGI with VLOs employees, 5 IDIs with regional projects coordinators in

PAED 150 CATIs with start-ups-beneficiaries 115 CAPIs with entrepreneurs-beneficiaries 100 PAPIs with school manager-beneficiaries

Pilotage attempt at estimating netto effects with utilisation the propensity score matching method. Data of „untreated grup” from PULS Information System (the data base of unemployed, registered in PLO)

INNOVATION APPROCH IN POLAND!

PHARE 2002 HRD

1) beneficiaries research after 6 months – introduction to evaluation desk research 3000 CATIs with beneficiaries, carried out by contractor

2) Evaluation ex-post after 12 months emphasising on the qualitative methods 1 FGI with VLOs employees, 5 IDIs with regional projects coordinators in the PAED 150 CATIs with start-ups-beneficiaries 100 PAPIs with school manager-beneficiaris

Estimating netto effects and benchmarking the effects of the Phare 2002 HRD projects with previous editons

Implementing propensity score methode enables to assess the impact of the programme on beneficiaries after 6 and 12 months (from the end of participating in training courses)

Treated and untreated grup are going to take part in the evaluation research

Evaluations of successive editions of the Phare ESC Regional Component HRD

Thank you for your attention

For more information on the evalution in Poland (especially in the PAED), please contact Jacek

Pokorski (jacek_pokorski@parp.gov.pl)

Polish Agency for Enterprise DevelopmentUl. Panska 81/83PL 00-834 Warsaw

 http://www.parp.gov.pl