National Evaluation and Results Management System–

Sinergia –

Two decades of lessons and experiences

Directorate of Monitoring and Evaluation of Public Policy November 2013

El Modelo Sinergia

Información efectiva para el mejoramiento de las políticas públicas

SEGUIMIENTOSINERGIA

EVALUACIÓNSINERGIA

TERRITORIALSINERGIA

RENDICIÓN DE CUENTAS

INNOVACIÓN + INVESTIGACIÓN

Our model

Evidence for the decision making process

MONITORING EVALUATION TERRITORIAL

ACCOUNTABILITY

INNOVATION AND RESEARCH

THE VALUE CHAIN: OUR CONCEPTUAL BASIS

Goals Inputs Processes Outputs Outcomes Impacts

Sinergia’s model is based on the value chain and is oriented to identify bottle necks in each link of the public policy process.

Processes Institutional Outcomes Impacts

Executive Evauation

Our portfolio includes different types of evaluations in order to respond to bottlenecks identified in each link of the value chain.

Our process brings about transparency and consistency. In order to be effective, evaluations need to

The evaluation process

Be a result of a standardized processInclude the participation of all stakeholdersAnswer decision maker questionsBe in line with the government agenda

Evaluation Schedule

• Design 3 Months• Procurement 3 Months• Development 8 Months• Use of Results 6 Months

TOTAL: 20 Months

Selección de la política a evaluar

Diseño de la Evaluación

Ejecución

Uso de resultados

Area del Gobierno

Contratación

Firmas externas

Selection of policies to

be evaluated

Evaluation Design

Procurement

Government Area

EvaluationDevelopment

Implementing Results

Through these years there has been changes and lessons learned

The system´s evolution

We are working in different sectors going beyond social inclusion areaWe have a wide evaluations portfolioWe have published methodological guidelines of evaluation Our evaluations are public on internetOur process is part of the NDP quality management system

Types of evaluation by year of implementation Evaluations by sector

We still face new challenges

It is necessary a high level champion who is aware of the importance of doing evaluations and has the capacity to disseminates its attributes within the executive level.

It is required an adequate legal framework but first it is important to know: What should be its scope?, What should regulate?

It is important to develop the evaluation culture through different levels of government, as well as improving knowledge of the M&E concepts

It is vital to involve citizens in the evaluation process, so they can use it for social control

1 Spread of the evaluation culture:

Evaluated entities should me more committed with using the evaluations results and with the agenda setting.

Each evaluation must have a Plan for transfer and implement recommendations, which should be design between Sinergia, the evaluator and the evaluated entity.

The data bases should be public and simple to be searched.

It is need to have a monitoring scheme for the imlementation of evaluation results

2 Use of evaluations:

Externally, for decision-making processes: Internally, for more influence:

Replicate evaluations in order to contrast results and evaluate evaluators.

Improve the quality of evaluations through meta-evaluation.

To do systematic reviews in order to define new lines of action based on evaluations already done

We still face new challenges

3 Quality of the evaluations:

Working with universities

Improve the evaluations process with the support of a technical expert, during the design, implementation and use of results. In house or peer reviewer?

High level advisory

Universities should play a critic role replicating evaluations in order to contrast and compare results. As well it would be important to exchange knowledge and experiences.

Regular training

Of the evaluation team, in order to implement new methodologies and improve the quality of the existing ones.

It´s not just about quantity

It doesn´t matter if we have a limited number of evaluations been done at the same time. It´s very important to have an adequate number in order to guarantee quality and rigorousness.

We still face new challenges

4 Improving evaluators market

Dialogue with consulting firms in order to improve the procurement process.

Training for better proposal’s presentation.

Prioritisation of the technical quality when it comes to qualify proposals.

Improve strategies of evaluation costing.

Promote the development of small consulting firms.

We still face new challenges

Proposed questions

1. ¿Is it possible to observe any change in the quality of evaluations? How to ensure program evaluators are impartial and consistent? How to evaluate evaluators?

Each evaluation has a Monitoring Commitee in order to guarantee impartiality and consistency.

Evaluations managers must be technically strong to guarantee the accuracy of the evaluator.

NDP technical offices and representatives of the evaluated entitty should act as a quality filter for the evaluation.

The evaluations should also be evaluated through:

Evaluations replicas done by universities. Meta-evaluation (contrasting evaluations results). Peer reviewers.

2. Is there enough transparency in evaluations? (How evaluations are assigned?, Are evaluations public?)

Selección de la política a evaluar

Diseño de la Evaluación

Ejecución

Uso de resultados

Area del Gobierno

Contratación

Firmas externas

- The extern procurement guarantees impartiality. - Evaluators are chosen by scoring upon specific criteria

The consistency of the evaluation is guarantee by its technical design

Evaluations are public in the official website:sinergia.dnp.gov.co

Selection of policies

to be evaluated

Procurement

EvaluationDevelopment

Implementing Results

Government Area

3. Still the results-based budget on the table, or the M & E systems are limited to recommending actions for improvement?

Background• The National

Development Plan 2010-2014 was not designed upon a strict relation between goals and budget

2013• Work in the design

of methodological guidelines to achieve a relation between planning and budgeting processes.

2014

i) A strategic formulation methodology for monitoring the national development plan

ii) Guidelines for the design and process of scalability of the National Development Plan.

Based on the monitoring process, determine the required inputs to acomplish the planned outcomes of a public intervention. (costing of inputs)

Identify and link the planned outcomes of the public interventions with the outputs. In this way, attain the coordination between the design of public programs and budgeting.

The evaluations allow the validation of causal relations between the links of the value chain.

In Sinergia we are working on it:

Socio-economic situation

Needs

Objectives Inputs Activities Outputs

External factors

Cost - efectivenessExpenditures economy

Efectiveness

Outcomes

Efficacy

ProductivityEfficiency

Impacts

3. Still the results-based budget on the table, or the M & E systems are limited to recommending actions for improvement?

Use of the information referred to the actions needed to implement the public interventions.

This allows to develop good practices during the productive process.

Use of the information of the delivery of good and services and the generation of strategic results. This allows to make budgeting

decisions , approve or disapprove the continuity of public interventions and influence the adoption of the recomendations

resulted from evaluations.

Use of the information about the operation and the partial results of the public interventions. This allows to desig or re-design the implementation of public policies,

make budgeting decisions and prioritize population groups.

Objectives Inputs(costs) Activities Outputs

(costing)Intermediate

results Final

results Inmediate

results

Executive entities from the national and subnational level Coordinating entities from the national,

regional and subnational levels.

Ope

rativ

eM

anag

emen

tPo

litica

l

Entities objectives

Sectorial objectives

National Objectives

Results chain Productive process

3. Still the results-based budget on the table, or the M & E systems are limited to recommending actions for improvement?

4. How to use evaluation results for the decision making process? / How to promote the use of evaluations?

Evaluated entities should me more committed with using the evaluations results and with the agenda setting.

Each evaluation must have a Plan for transfer and implement recommendations, which should be design between Sinergia, the evaluator and the evaluated entity.

The data bases should be public and simple to be searched.

It is need to have a monitoring scheme for the imlementation of evaluation results

Externally, for decision-making processes: Internally, for more influence:

Replicate evaluations in order to contrast results and evaluate evaluators.

Improve the quality of evaluations through meta-evaluation.

To do systematic reviews in order to define new lines of action based on evaluations already done

5. Is there a positive cost-benefit ratio doing evaluations?

Numbers in USD

Financial resources invested in evaluations 2010-2012

To design public policy (CONPES)

To improve existing interventions.

To improve procurement processes.

It would be worth to quantify the benefits of evaluations for the public sector

2010 2011 20120

1,000,000

2,000,000

3,000,000

4,000,000

5,000,000

6,000,000

7,000,000

8,000,000

9,000,000

5,255,586

8,182,799

6,480,970

Use of evaluations

Thank youwww.dnp.gov.co

www.sinergia.dnp.gov.co/portaldnp/@Sinergia_DNP

PBX: 3815000