Upload
edgar-warner
View
216
Download
2
Embed Size (px)
Citation preview
Results of the midterm Results of the midterm evaluation exercise on the evaluation exercise on the Leader + programme for Leader + programme for PortugalPortugal
Special focus on evaluating innovationSpecial focus on evaluating innovation
Pedro Afonso FernandesPedro Afonso Fernandes(CIDEC – Lisbon - Portugal)(CIDEC – Lisbon - Portugal)
Expert meeting on Guidance for Common Monitoring and Expert meeting on Guidance for Common Monitoring and Evaluation FrameworkEvaluation Framework
Seminar on monitoring and evaluation of the LEADER approachSeminar on monitoring and evaluation of the LEADER approach
DG AGRI, Brussels – Room LOI 130BDG AGRI, Brussels – Room LOI 130B25 September 200625 September 2006
Pedro Afonso Fernandes ([email protected]) 2
Innovation on evaluating innovationInnovation on evaluating innovation
• The innovative starting point
• The 3 × 3 evaluation matrix
• Specific evaluation tools
• Deal with lack of self-evaluation
• Recommendations on evaluation procedures and tools
Pedro Afonso Fernandes ([email protected]) 3
The innovative starting point (I)The innovative starting point (I)
• Evaluating a innovative programme such as LEADER+ requires a innovative starting point at … two levels:
• Community level: Guidelines produced by STAR committee (Doc. VI/43503/02-Rev.1, January 2002)
Detailed Common Evaluation Questions and Criteria that cover a wide range of evaluation topics:
Implementation of the LEADER+ method
Implementation of the 3 actions (integrated pilot strategies, co-operation and networking)
Impact – Overall objectives of the Structural Funds
Impact – Specific objectives of LEADER+ (Value Added)
Financing, management and evaluation
Pedro Afonso Fernandes ([email protected]) 4
The innovative starting point (II)The innovative starting point (II)
• National level: the midterm evaluator (CIDEC) developed specific evaluation questions in order to cover the main concerns of the national authority (IDRHa – Institute of Rural Development and Hydraulics), namely:
Evaluation of the LAG’s functioning and practices
Evaluation of local strategies’ appropriateness and sustainability
Evaluation of the monitoring and information systems
• Note: the specific (national) evaluation questions complement the DG AGRI’s Common Questions
Those additional questions reinforce the evaluation of LEADER’s value added (e.g. development of competences at local level, strategies sustainability, diversification effects)
Pedro Afonso Fernandes ([email protected]) 5
The 3 The 3 × 3 evaluation matrix× 3 evaluation matrix
• Evaluating LEADER is a challenge that results from the cross of 3 levels of intervention with 3 types of action
Action 1: local strategies
Action 2: co-operation
Action 3: networking
National level (IDRHa and A.3 partners)
Local level (LAG)
Project level (promoters)
NO TARGETS
Pedro Afonso Fernandes ([email protected]) 6
Specific evaluation tools (I)Specific evaluation tools (I)
• Multiple methodological tools were mobilized in order to deal with the 3 × 3 matrix
Action 1: local strategies
Action 2: co-operation
Action 3: networking
National level (IDRHa and A.3 partners)
Interviews Interviews Interviews
Local level (LAG)
Inquiry
Case Studies
Inquiry
Case StudiesInquiry
Project level (promoters)
Inquiry
Case Studies
Inquiry
Case Studies-
Pedro Afonso Fernandes ([email protected]) 7
Specific evaluation tools (II)Specific evaluation tools (II)
• Case studies:
26 LAG covered (one half of the total)
Interview with LAG’s coordinator
Contact with 2 projects from Action 1 and interview
Analysis of the administrative processes (dossiers) of 10 projects (9 from Action 1 and one from Action 2)
Telephonic contact with some local partners
Documental analysis (e.g. local strategies, criteria to select projects)
Quantitative treatment of qualitative information
Detailed report
Pedro Afonso Fernandes ([email protected]) 8
Specific evaluation tools (III)Specific evaluation tools (III)
• Promoters Inquiry:
Main objective: find values for a set of indicators specially developed by the midterm evaluator with the support of the national authority (IDRHa)
The inquiry was applied with 9 different questionnaires according to the type of project Good response rate (51%) and reliability of the data
Quality control procedures: statistical tests that compared the data from the inquiry with the data from the case studies
Impact and results indicators were favoured in order to quantify the LEADER’s value added
Pedro Afonso Fernandes ([email protected]) 9
Specific evaluation tools (IV)Specific evaluation tools (IV)
• Examples of impacts and results measured by the Promoters Inquiry for Portugal:
63% of the projects incorporated technologies (43% ITC)
40% of the projects are innovative (product or process)
10% of the projects involved universities or polytechnics
60% of the “productive” projects mobilized local raw materials and/or services
70% of jobs created were for women
LEADER+ stimulates the development of competences on 40% of the promoters inquired and …
… the development of new different activities (no financed) on 12% of the projects – Diversification effects
Pedro Afonso Fernandes ([email protected]) 10
Deal with lack of self-evaluationDeal with lack of self-evaluation
• Only 5 on 26 LAG studied by the midterm evaluator have permanent self-evaluation practices
• Even if the LAG has this kind of practices, an independent evaluator are not mobilized typically
• The midterm evaluator deals with those problems by involving LAG’s representatives and (selected) promoters and partners in the case studies’ works
In the future, national (or even European) authorities should develop a vademecum or guide to favour the self-evaluation by the Local Action Groups
Pedro Afonso Fernandes ([email protected]) 11
Recommendations on evaluation Recommendations on evaluation procedures and tools (I)procedures and tools (I)
• At European level, it is important to develop the Common Evaluation Questions and Criteria, that is, “The Starting Point” by:
Incorporating new questions/criteria, namely, suggested by midterm evaluators
Simplifying some topics (e.g. the CEQ 4.1 “To what extent has LEADER+ contributed to promote and disseminate new integrated approaches to rural development (…)” was already covered by the action-specific CEQ)
Developing a Common Set of Indicators that would facilitate the impact and results evaluation at European and National levels (the Portuguese set already developed could be a “starting point” for that huge task)
Pedro Afonso Fernandes ([email protected]) 12
Recommendations on evaluation Recommendations on evaluation procedures and tools (II)procedures and tools (II)
• At National level, it is important:
To establish targets (quantified goals) for local strategies, co-operation and networking
To develop Complementary Evaluation Questions, Criteria and Indicators in order to reinforce the evaluation of LEADER’s value added and incorporate national concerns
To develop a mix of evaluation tools that could solve the puzzle of the 3 × 3 Matrix
Case studies should be privileged because of the innovative local-based nature of LEADER’s projects
Project inquiries are essential to collect quantitative data, namely impact and results indicators