Embed Size (px)
Citation preview
Questions Which kind of systems have countries, blocs
of countries (OECD, EU, etc), or international organizations (World Bank, PNUD, IDB, etc) developed that would allow, facilitate and promote policy makers to share information on evaluation?
Which federative countries have developed efficient systems for sharing successful local experiences? How do these systems function?
Background There are several systems implemented both
by international organizations and by countries to share and promote evaluation
Surprisingly there is very little systematically documented on these systems therefore hard to say if they are successful or not
On a recent Bank publication “ Building Better Policies” we analyzed leading evaluation systems that had been in place for more than a decade visit http://go.worldbank.org/TGWVN78VR0
Classification on cross-comparison
4
Country Purpose Leadership Operation Users
Chile Budget/Control Direct Centralized Single
Mexico Budget/Accountability Delegated Decentralized Multiple
Australia Budget Direct Decentralized Single
Canada Budget Direct Decentralized Single
UK/NAO* Accountability Direct Centralized Multiple
UK/Treasury Budget/Control Direct Decentralized Single
* Supreme Auditor
Chile: Main Features
5
System is highly centralized – closely linked to and based on a highly centralized, top-down budget process
Budget office manages most details, and is main user of information – utilization fostered by close link to budget process
Management control by hierarchical oversightLittle buy-in from other (potential) stakeholdersSome impact on allocations and program management
Chile: Utilization of Government Evaluations—2000–05Effect onprogram
Minor adjustmentof program, forexample, improvedprocesses orinformation systems
Major adjustmentof managementprocesses, for example,changed targetingcriteria, or newinformation systems
Substantial redesignof program ororganizational structure
Institutional relocationof program
Programtermination
Programsaffected
24% 38% 25% 5% 8%Percentage of all evaluated programs. Source: Guzman 2007
Chile: How does it work?
6
6
Budget Office
Budget Office
Budget Office
Programs
Budget Office
•Define the evaluation plan•Decide what and how to evaluate
•Coordinate and steer evaluation system•Maintain quality control•Define monitoring parameters
Programs are evaluated directly by Budget Office
•Select and hire evaluators•Supervise evaluation implementation•Monitor indicators and targets
•Collection of evaluation results•Dissemination of evaluation results
•Utilization of evaluation results
1.
2.
3.
4.
5.
Budget Office
Congress
Australia( similar to Canada and USA): Main Features
7
Mandate to evaluate each program every 3-5 yearsPortfolio Evaluation Plans to be prepared annually
for the following three yearsDepartment of Finance: Steering & quality controlSector Departments: planning, implementationEvaluation results primarily used for budgetary
decisions: allocations of funds for new policies and reallocation of savings (i.e. the discretionary part of annual budget process)
System lasted from 1987-1996 – now system is even more decentralized
Australia: How does it work?
8
Treasury
Department of Prime
Minister and Cabinet
Department of
Finance
Parliament
Citizens
Inputs to PEPs
Inputs to PEPs
Inputs to Portfolio Evaluation Plans (PEPs)
Formal notification of
PEPs
Involvement in evaluations
Publication of department’s
evaluation reports
Reporting of keyEvaluation findings in department’s budget paper (prospective) and in their annual reports (retrospective)
Source: Mackay 2011
Sector Departmen
ts(and outrider
agencies)
How do institutional differences affect utilization and sustainability?
9
Centralization requires the right institutional structure – a centralized evaluation system design in a fragmented public sector will fail
In a system with multiple stakeholders delegation to an impartial agency might be a viable option (AEVAL or CONEVAL type)– but beware of over-engineering and objective overload
Who is to gain and who has to worry about buying into the system: The senior civil service, the legislature, ministries, service delivery units, the finance ministry, the head of government (PM or President)?
Staying in control of overall steering and quality of outputs does not mean being in charge of all aspects of evaluation implementation – strategic delegation might be smart for buy-in and workload
Is it possible to imagine a long-term sustainable, well utilized evaluation that does not have a stable link to budgetary decisions?
QuestionsIs there any consistent effort for building an
international or national system for the classification of public policies?
If so, what is this system? How does it classify public policies? How refined (how many categories does it consider) and well structured (on how many levels can the classification be conducted) is it?
Background Yes, most country systems have gone through a
process of public policy classification according to the objective of the system ( budget, accountability, etc)
For example, the USA through the PART system used a public policy classification compatible with their budget classification, similarly other countries are trying to do something similar ( Mexico, South Africa)
There are also international agencies such as 3IE (Campbell Evaluation Guidelines) which have gone through a process of program/public intervention classification
To certain extend the Bank is also following this classification in our new jobs knowledge platform www.jobsknowledge.org
Example from evaluation guidelines SATable 1: Relationship between budget programme, delivery agreements and implementation programmes using the example of Early Childhood Development (ECD) Strategic plan, APP and budget Delivery agreements Detailed implementation
programmes eg National Integrated Plan for ECD
Vote level Outcome level Department of Basic Education (could cross more than one department)
Outcome 1 Improved quality of basic education
(Budget) Programme level DBE Programme 2 Curriculum Policy, Support and Monitoring
5 year strategic objectives (several per programme)
Impact level of Implementation Programme
Strategic Objective 2.6 Improve the quality of early childhood development:
Output 3 Improve ECD Eg improved learning outcomes of children resulting from exposure to ECD
Sub programme level Suboutput Outcome level of Implementation Programme
Curriculum Implementation and Monitoring
Suboutput 3.1 Universalise access to grade R Suboutput 3.2 Improved quality of ECD
Eg All children who wish to access Grade R have access, at least at minimum standards of quality.
Sub-sub programme ECD Directorate
High level activities
Output level of Implementation Programme eg
3.1.1: Use of priority lists to ensure that public Grade R is rolled out to all eligible schools. 3.2.1: Distribution of high quality teaching and learning materials designed specifically for grade R
1. Grade R is provided in all eligible schools
2. High quality teaching and learning materials designed specifically for grade R available in ECD centres
3. Etc Low level activities Activity level
