Development of a system of common indicators for European ...ec.europa.eu/regional_policy/sources/docgener/evaluation/doc/0603… · •Gap analysis underpinning the proposed list

Framework Contract

Conceptual framework and analysis of indicators

1

Andrea Gramillano (t33) – SWECO Consortium

Evaluation network meeting

Brussels, 5th March

Development of a system of common indicators for European Regional Development Fund and Cohesion Fund interventions after 2020

Outline of the presentation

I. Methodological approach

II. What can be used from the existing list?

III. Elements for discussion: examples of proposed indicators

IV. Conclusions

2

• For output and direct result indicators

1. Identification of the most frequent intervention fields at TO level

2. Analysis of the link between the existing common output indicators and the intervention fields

3. Proposal of additional indicators (if there is a gap) tightly linked with the intervention field and IPs based on:

Programme experience (scale-up)

Literature review

3

I. Methodological approach (1/5)

• Criteria of good indicators – Better regulation toolbox

RACER criteria

1. Relevant (thematic coverage and policy responsiveness)

2. Accepted (understood by those in charge of data collection)

3. Credible (unambiguous)

4. Easy to monitor (feasible in terms of costs)

5. Robust (clearly defined and with a sound methodology)

4


• Other criteria

Time-bound (when is the right timing to measure them? – for direct result indicators)

Clear (indicate unequivocal direction of change, in particular for direct result indicators)

Debatable (can be used for benchmarking and comparison)

• FOCUS on “aggregation rules”

from project to programme

from programme to EC

5


• Gap analysis underpinning the proposed list of common indicators (example TO 1)

PROCESS - CO01 “Enterprises receiving support” is the main candidate process indicator for TO 1. An additional process indicator might be included to measure the supported research institutions

OUTPUT - CO26 “Enterprises cooperating with research institutions” can be confirmed as an output indicator (proxy of technological transfer) and completed with other indicators measuring the value of purchased equipment / services and with an indicator on built / renovated research infrastructure (surface of intervention, etc.)

6


• Gap analysis underpinning the proposed list of common indicators (example TO 1)

DIRECT RESULT

Indicators on the private investments matching public support (CO06, CO07, CO27), employment increase (CO08) and new researchers (CO24) can be confirmed and used as direct result indicators, even if CO08 and CO24 are more difficult to be measured (as highlighted by the consultation).

Indicators CO28 and CO29 measuring the number of supported enterprises to introduce innovation can be refined with other indicators measuring the (product / process) innovations resulting from the projects.

Other indicators on research activities (prototypes, articles submitted for publication) and patenting can be added.

7


8




IV. Conclusions

II. What can be used from the existing list? (1/7)

• Most frequently used indicators in TO 1 and TO 3 are either process or direct result indicators

9

Common output

indicator IP1a IP1b IP3a IP3b IP3c IP3d

CO01 10 173 139 63 77 136

CO02 2 108 90 52 69 87

CO03 49 63 12 44 89

CO04 1 48 68 19 11 63

CO05 2 40 104 8 13 40

CO06 2 46 36 33 40 43

CO07 13 40 5 23 56

CO08 1 52 102 30 50 86

CO24 50 50 100

CO25 119 11 130

CO26 32 161 1

CO27 33 102 2 2

CO28 86 5 10 22 6

CO29 2 82 10 7 35 11

Looks like a process indicator Looks like an output indicator Looks like a direct result indicator


• TO 4 is well covered (ERDF better than CF) but with differences across the IPs

10

Share common / total

IP CF ERDF CF and ERDF

4a 70%

4b 75%

4c 69%

4d 64%

4e 23%

4f 63%

4g 52%

4i 57%

4ii 50%

4iii 48%

4iv 67%

4v 22%

4vi 29%

Grand Total 45% 58% 58%

Larger use in IPs 4a(4i) and 4b (4ii) than IPs 4e (4v)

CF programmes use less common output indicators and a part of those used by ERDF programmes


• Limited used of common output indicators in TO 5

11

5a

(ERDF)

5b

(ERDF)

5i

(CF)

5ii

(CF)

(a) Common output 21 55 6 5

(b) Programme-specific 23 102 16 24

Common output share

[a /(a+b)] 48% 35% 27% 17%


• In most cases in TO 6, common output indicators measure either a direct result (e.g. CO09 on expected number of visits) or a real output (land rehabilitation - CO22)

12

IP CO 09

CO 17

CO 18

CO 19

CO 20

CO 22

CO 23

CO 29

CO 34

CO 37

CO 38

CO 39

CO 40

Total

6a 31 1 32

6b 39 46 85

6c 96 1 1 1 2 101

6d 7 75 1 83

6e 1 31 1 2 17 41 22 4 119

6f 4 1 15

6g 5 2 18

6i 10 2 12

6ii 13 13 1 27

6iii 8 8

6iv 5 1 6

ERDF

96 40 39 47 1 38 77 1 4 18 43 24 4 453

CF 10 13 13 8 8 1

Tot 96 50 52 60 1 46 85 1 5 18 43 24 4 506

13

Type COI 2014-2020 (examples)

Process indicators CO01 Number of enterprises receiving support

CO05 Number of new enterprises supported

CO02 Number of enterprises receiving grants

CO03 Number of enterprises receiving financial support other than grants

CO04 Number of enterprises receiving non-financial support

II. What can be used from the existing list? (5/7) - Examples

14


Output indicators

CO11 Total length of new railway line (km) CO12 Total length of reconstructed or upgraded railway line (km) CO14 Total length of reconstructed or upgraded roads (km) CO15 Total length of new or improved tram and metro lines (km)


15


Direct result indicators

CO06 Private investment matching public support to enterprises (grants) CO07 Private investment matching public support to enterprises (non-grants) CO24 Number of new researchers in supported entities CO27 Private investment matching public support in innovation or R&D projects CO08 Employment increase in supported enterprises CO20 Population benefiting from flood protection measures CO21 Population benefiting from forest fire protection measures


16




IV. Conclusions

• IP 1a

Name: Number of articles submitted to peer-review due to the supported operations (number)

Type: Direct result indicator

Source: programme-specific output indicators, EEA and Norway Grants 2014-2021

17

III. Elements for discussion: examples of proposed indicators – TO 1

Thematic coverage: IP 1a

Harmonisation: Reference to the “peer-reviewed publications”, without

requiring the approval and high-impact factors (similar to Horizon 2020).

Capitalisation: programme-specific output indicators

• IP 1a / 1b

Name: Built / renewed research infrastructure (square meters / number)

Type: Output indicator

Intervention field: 58, 59, 60, 61

Source: programme-specific output indicators

Harmonisation: Definition of research infrastructure – Horizon 2020

18

Thematic coverage: IP 1a / 1b

Harmonisation: other EC programmes



• IP 3b / 3d

Name: Share of supported SMEs with increased exports (%)



Rule of aggregation at EU level = (1) Number of SMEs receiving support (process indicator),

(2) Supported SMEs with increased exports (supporting indicator of the direct result indicator calculated at programme level)

(3) = (2)/(1): Share of supported SMEs with increased export (%) (direct result indicator).

19


Harmonisation: Aggregation rule at EU level (Rural programmes)


• IP 4e / 4v

Name: Public recharging points for electric vehicles installed (number)


Intervention field: 043


20


Thematic coverage: IP 4e / 4v

Harmonisation: EC Directive 2014/94 (definition of recharging points)


21


• IP 4a, 4c, 4d / 4i, 4iii, 4iv

Name: Modernised / renovated / extended heating / thermal network (km)


Intervention field: 015


Thematic coverage: TO 4


22


• IP 5a, 5b/ 5i, 5ii

Name: Average response time to emergency situations


Intervention field: 087, 088


Thematic coverage: The indicator measure the result not in terms of

additional population but in terms of “performance” of the interventions in

emergency situations


23


• IP 5a, 5b/ 5i, 5ii

Name: Local plans for risks protection adopted / revised

Type: Output indicators


Source: programme-specific output indicators and Covenant of Mayors

Thematic coverage: TO 5

Harmonisation: Covenant of Mayors


24




IV. Conclusions

IV. Conclusions

A. Continuity - Maintain / refine as much as possible the existing indicators if they have been used successfully and discard the others

B. Feasibility and harmonisation - Introduce indicators which either come from the literature review (other DGs, EU directives, practices, etc.) or from programme-specific indicators

C. Timeliness – Direct result indicators can be measured either at the project completion (e.g. private investment matching public support) or some time later

D. Simplification – Clear guidance and thematic coverage

25

Thank you for your attention

26

Andrea Gramillano

ANNEX – other indicators

27

• IP 4e / 4v

Name: Low-emission public transport vehicles purchased or refitted (number)




Link with programme-specific output indicators: other programme-specific output indicators can be added to specify the fuel and the type of vehicle.

28


29


• IP 6a, 6b, 6f, 6g, 6i, 6ii

Name: Additional population served by supported recycling facilities

(Population)


Intervention field: 017, 018, 019

Source: programme-specific output indicators, based on the existing CO17, CO18

Documents

Development of a system of common indicators for European ...ec.europa.eu/regional_policy/sources/docgener/evaluation/doc/0603… · •Gap analysis underpinning the proposed list