1

INDICATORS OF INDICATORS OF REGULATORY QUALITYREGULATORY QUALITY

Project website

Claudio M. RadaelliBrussels, 25 January 2005

2

Description of the projectDescription of the project

Regulatory quality is a complex concept. In our project we acknowledge the role of different stakeholders and institutional contexts

Quality measured in terms of design, activity-output, and real-world outcome

Assessment of indicators and indexes proposed in the literature

Analysis of better regulation programmes: focus on four tools (IA, Simplif., Access, Consultation)

Questionnaire sent to support network in May 2004 The project is about the DESIGN of indicators. We have

not calculated indicators, neither have we scored Member States

3

Why quality?Why quality?

Quality versus quantity Quality and efficiency of regulations Better regulation as a public policy (actors,

resources, instruments, and decision-making structure)

Quality and the Lisbon strategy Quality and indicators

4

Measuring regulatory quality in EU Measuring regulatory quality in EU Member States (questionnaire)Member States (questionnaire)

• Diffusion of the Mandelkern report principles• Most countries have opted for an experimental approach

to indicators – one based on the gradual introduction of simple measures

• Setting quantitative targets is an effective way to encourage public administration to implement measurement systems (as shown by the case of B, DK, NL and S)

• No systematic use of cost-benefit analysis in the EU Member States. Rather, the emphasis is on how stakeholders are affected by proposed regs

5

Diffusion of principles but different practice: Diffusion of principles but different practice: three clustersthree clusters

1. Tools of better regulation are still at the stage of pilot project. Measurement of quality has been considered, but limited activity.

2. Basic systems of quality assurance and focus on administrative burdens.

3. Sophisticated quality assurance, well-embedded impact assessment, initiatives for the measurement of quality which target the whole regulatory process

6

Combining diversity of approaches and Combining diversity of approaches and common progress in the EUcommon progress in the EU

• Different practice is compatible with common progress

• A common approach, shared beliefs about regulatory quality and its components, and sense of direction towards the Lisbon goals

• Three specific sets of indicatorsThree specific sets of indicators

7

The hard questionsThe hard questions

FROM INSTRUMENTS TO POLICY: How do we know that a good IA produces better regulation?

FROM POLICY TO REGULATORY OUTPUT: How do we know that ‘better regulation policy’ produces better regulations?

FROM REGULATORY OUTPUT TO FINAL ECONOMIC OUTCOME: What is the causal chain going from good regulation to real-world economic outcomes? How robust is it?

8

How we answer these questionsHow we answer these questions

Focus on how regulation changes the behaviour or stakeholders

Selection of five indicators that measure the quality of better regulation policy without going too far in the causal chain

We use Lisbon indicators, but we do not test the quality of better regulation policy directly on economic indicators like the growth of the knowledge-based economy

9

The three systemsThe three systems

1. Design of better regulation policy and tools. Macro and ex-ante

2. Ex-post and micro

3. Bridge between measurement of regulatory quality and the systematic evaluation of better regulation as public policy

10

Where do the indicators come Where do the indicators come from?from?

Operationalisation of the principles of better regulation

Questionnaire and consultation with support network

Experience of MS and other countries Criteria of “usable knowledge” = indicators

that can be embedded in policy processes

11

Indicators – characteristics of system 1: quality of the process

• an ex-ante macro-system - does not go down to the level of individual impact assessments or consultations

• based on objective indicators, it covers the design of better regulation policies, instruments (contents of formal guidance), and supporting and quality assurance activities

• can be managed by coordinating units in MSs and the SG in the Commission

12

Quality assuranceQuality assuranceIndicatorIndicator What is measuredWhat is measured InterpretationInterpretation Data collectionData collection

Quantitative Quantitative targetstargets

Existence or absence

Targets facilitate implem. of BR programmes

IRQ dataset

Reporting on Reporting on progressprogress

Existence or absence

Accountability and transparency

IRQ dataset

Performance of Performance of quality assurance quality assurance unitunit

Existence or absence

Establishing quantitative criteria to assess success or failure

IRQ dataset

Civil society – Civil society – public official public official documentdocument

Existence or absence

Legitimacy of better regulation programmes

Official guidance

Monitoring and Monitoring and evaluationevaluation

Existence or absence

Culture of monitoring and evaluation

IRQ database plus official guidance

13

Indicators – characteristics of system Indicators – characteristics of system 2: internal evaluation2: internal evaluation

• An ex-post micro-level system focused on what the tools of better regulation deliver in a given time-period

• We look at guidance as designed in system 1 and measure what has been achieved

14

System 2 contains the following:System 2 contains the following:

1. A simple checklist on the quality of individual impact assessments and major consultations

2. Real-world measures of the impact of better regulation policies (subjective: surveys over time)

3. Five real-world measures of the impact of better regulation policies (objective)

4. Limited number of important indicators of economic outcome – role of causal chains

15

Survey of regulators Survey of regulators Percentage of regulators who think that IA has the Percentage of regulators who think that IA has the

following effect:following effect:

[1] Makes policy formulation more transparent[2] Speeds up / delays policy implementation[3] Reduces conflict at the stage of policy formulation[4] Reduces conflict at the stage of policy implementation[5] Be more likely to represent the preferences of citizens[6] Final policy decision will be more likely to achieve goals[7] Final policy decision be more likely to represent a

consensus among affected entities[8] Final policy decision be more likely to create open

markets

16

Survey of citizens and firmsSurvey of citizens and firms[1] % of citizens who think that their views are taken on board in

the development of single market policy in the EU [2] % of firms who think that the quality of regulation in the EU

(including both EU and domestic rules) has improved over the last three years

[3] % of firms who think that participation in EU-level consultation makes an impact on the final decision

[4] % of firms who think that EU policy has an impact in opening markets, making it easier to compete, and generating competition in the professions

[5] % of firms who think that in last three years restrictions and obstacles to their business have disappeared altogether or significantly reduced

[6] % of firms who think that regulatory changes at the EU level are predictable

17

The Five Real-World measures of The Five Real-World measures of better regulation policybetter regulation policy

[1] Estimate of total regulatory costs delivered by regulations for which impact assessments (IAs) were prepared in year xxxx / Estimate of total regulatory benefits delivered by regulations for which IAs were prepared in year xxxx

[2] Net benefits delivered by regulations for which IAs were prepared in year t / Net benefits delivered by regulations for which IAs were prepared in year t-1

[3] Total cost reduction resulting from simplification [4] Annual rate of reduction in the total administrative

burdens stemming from EU legislation [5] Cost of administrative procedures eliminated in year t /

Cost of administrative procedures eliminated in year t-1

18

System 3: external evaluation “All policies can be evaluated, and better regulation is no exception”

• no matter how sophisticated, indicators are just a component of quality assurance

• the latter requires strategic and operational management, specific structures, and dedicated tools

• one important element is external evaluation• indicators must be complemented by interviews, case

studies, etc. • external evaluation should be done by think tanks,

academics, and stakeholders - notion of pluralistic evaluation

19

Indicators – which system is suitable for whom?

system 1 could be used by the cluster of Member States with basic approaches to regulatory quality and impact assessment in experimental-pilot stage. They should move as soon as possible to the second system

system 2 could be used by the cluster in which consultation, simplification, and the assessment of administrative burdens are already well embedded - however, some simple indicators of this system can be adopted by all EU governments now, no need to wait

20

Indicators – which system is suitable for whom?

finally, system 3 is applicable to countries with sophisticated quality assurance mechanisms (as well as to the Commission)

21

How to use indicatorsHow to use indicators

• Open method of coordination• High-level group on Competitiveness –

Competitiveness Council, this can be a

body that can adopt indicators• DBER can be the forum where data are

collected and technical discussions on

indicators take place

22

Why use open method of Why use open method of coordination?coordination?

• Better regulation policy is an essential component of the Lisbon agenda.

• It involves different levels of governance. • A responsibility for the EU and the

Member States

23

There is an OMC in the making:There is an OMC in the making:

1. Principles (Mandelkern)

2. National programmes of regulatory reform

3. Peer review of reforms (within DBER)

4. Timetable (Lisbon)

5. Need to complete OMC with Indicators and iterative processes