Embed Size (px)
Citation preview
UNIVERSITY OF ANTWERP
INSTITUTE OF DEVELOPMENT POLICY AND MANAGEMENT
Dissertation
ASSESSING COLOMBIA’S
MONITORING AND EVALUATION
SYSTEM: The Demand Side and the Use of Information for Public
Accountability
Sara Lucía PÉREZ PERDOMO
Master of Development Evaluation and Management
Supervisor: Prof. Dr. Nathalie Holvoet
Academic Year 2012-2013
2
UNIVERSITY OF ANTWERP
INSTITUTE OF DEVELOPMENT POLICY AND MANAGEMENT
Dissertation
ASSESSING COLOMBIA’S
MONITORING AND EVALUATION
SYSTEM: The Demand Side and the Use of Information for Public
Accountability
Sara Lucía PÉREZ PERDOMO
Master of Development Evaluation and Management
Supervisor: Prof. Dr. Nathalie Holvoet
Academic Year 2012-2013
Preface
The choice of my dissertation topic was influenced by my interest regarding M&E systems
and their huge ability to influence and improve government performance, promoting at the
same time public sector effectiveness and transparency, good governance and democratic
quality; and the paradox that this affirmation represents in relation with the Colombian
situation. In this country, the existence of an internationally recognized M&E system
contradicts the also recognized’ high rates of poverty and inequality; bad performance in
governance, transparency and corruption indicators; as well as “institutional, political, and
fiscal difficulties” (Castro, 2009: vi).
In line with this and as a result of my first end of module paper, I found out that there are
some missing aspects when evaluating M&E systems’ demand side and the use of the
information for public accountability: the different frameworks, institutional and
organizational arrangements that shaped the way in which institutions demand and use the
information are not being recognized. Also, actors that guarantee the rule of law and defend
the legal framework are not considered as stakeholders of the M&E information within the
accountability function of the system.
As a result, I decided to assess Colombian M&E system, focusing on the demand side of it
and the use of the information for public accountability.
I want to express my gratefulness to my mom and my sister for being there for me always,
at all times and under all circumstances. Special gratitude to my friends in Belgium and now
in life: Anita, Allan, Patty, Carlos, Pierre, Lucia and of course Stefan. I will always be
thankful to Greet for encouraging me, trusting me and supporting me; and especially to
professor Holvoet for all the knowledge and the passion for M&E. Finally, I want to thank
the backing of my working colleges and of that special person in my heart, Mario. Love you
all.
5
Table of Contents
List of Tables ....................................................................................................................................... 6
List of Figures ..................................................................................................................................... 6
List of Graphs ...................................................................................................................................... 6
List of Acronyms ................................................................................................................................. 7
EXECUTIVE SUMMARY ................................................................................................................. 9
1. INTRODUCTION ........................................................................................................... 10
2. M&E SYSTEMS: STRUCTURE, LEARNING AND ACCOUNTABILITY ........... 15
2.1 The Supply Side ....................................................................................................................... 17
2.2 The Demand Side ..................................................................................................................... 19
2.3 Politics of M&E ....................................................................................................................... 22
3. EMPHASIZING THE ACCOUNTABILITY FUNCTION ........................................ 25
3.1 Public Accountability .................................................................................................................. 26
3.1.1 Dimensions and Institutional Arrangements ............................................................... 27
3.1.2 Types of Accountability ............................................................................................... 30
4. DIAGNOSTIC OF COLOMBIA´S M&E SYSTEM ................................................... 33
4.1 A Technocratic Government with a Well-organized System ...................................................... 35
4.1.1 Colombian M&E Context ..................................................................................................... 35
4.1.2 Assessing M&E Policy ......................................................................................................... 38
4.1.3 Indicators, Data Collection and Methodology ..................................................................... 49
4.1.4 The Importance of the System’s Organization and its Capacity Building ............................ 57
4.2 Low Participation of Actors outside the Government and the External Accountability Function
........................................................................................................................................................... 64
4.2.1 An Underutilized System: Assessing Intrastate (horizontal) Accountability ........................ 69
5. CONCLUSIONS AND POLICY RECOMMENDATIONS ....................................... 79
BIBLIOGRAPHY .................................................................................................................. 84
Appendix A: Assessing Public Accountability ................................................................................. 87
Appendix B: Assessment Benchmark ............................................................................................... 91
Appendix C: Quantitative Analysis ................................................................................................. 109
6
List of Tables
Table 1. Classification of Public Accountability by Type of forum ....................................... 32
Table 2. Number of surveys by group ...................................................................................... 42
List of Figures
Figure 1. Colombia’s General National Budget ....................................................................... 47
Figure 2. Monitoring Pyramidal Scheme ................................................................................. 50
Figure 3. Value Chain Structure – First Version ...................................................................... 51
Figure 4. Value Chain Structure – Second Version ................................................................. 52
Figure 5. Monitoring Previous Steps ........................................................................................ 54
Figure 6. Phases for the Evaluation of Strategic Policies ........................................................ 56
Figure 7. Steps to Monitor de PND .......................................................................................... 58
List of Graphs Graph 1. Percentage of people that scored SINERGIA’s General Quality as excellent or very
good (Total: 30%) .................................................................................................................... 43
Graphs 2 and 3. Percentage of people that scored SINERGIA’s quality in terms of credibility,
utility, clarity, current importance and relevance as excellent or very good ........................... 44
7
List of Acronyms
AAA Accra Agenda for Action
CODHES Consultancy for Human Rights and Displacement
CONPES National Council for Social and Economic Policy (by its Spanish acronym)
CSOs Civil Society Organizations
DANE National Statistics Administration Department (by its Spanish acronym)
DDTS Subnational Sustainable Development Directorate (by its Spanish acronym)
DEPP Public Policy Monitoring and Evaluation Directorate (by its Spanish acronym)
DIFP Public Investment and Finance Directorate (by its Spanish acronym)
DNP Department of National Planning (by its Spanish acronym)
ECI Unconstitutional State of Things (by its Spanish acronym)
GED Effective Enjoyment of Rights (by its Spanish acronym)
GNB General National Budget
IDP Internal Displaced People
NAA New Aid Approach
NCC National Consultancy Center
NGOs No Governmental Organizations
NPM New Public Management
NTI Government Transparency Index
M&E Monitoring and Evaluation
MoF Ministry of Finance
PAG Government Action Programs
PD Paris declaration
PND National Development Plan (by its Spanish acronym)
POS Political Opportunity Structure
8
PND National Development Plan (by its Spanish acronym)
PRAP Public Administration Renewal Program (by its Spanish acronym)
PRSP Poverty Reduction Strategy Paper
SIGOB Government Results Information System
SIIF Integrated Financial Management Information System
SINERGIA National Results-Based Management and Evaluation System (by its Spanish
acronym)
WB World Bank
9
EXECUTIVE SUMMARY
Colombian M&E system has being considered for many years and among different regional
levels as an excellent tool to improve government performance, due to its decision-making
guidance function and its inside and outside accountability functions. But in reality, this
system is highly decision-making oriented. It emphasizes more its learning and feedback
function as well as its internal accountability one, leaving aside important elements of its
external accountability function.
As a result, SINERGIA seems to be adopting a fragmented approach to M&E, with an
important focus upon the more technical dimension of the system, passing by the more
political and institutional ones. In general terms SINERGIA presents flaws in the policy of
M&E, specifically related to the system organization and its linkages with other government
agencies implementing monitoring processes. At the same time, little participation of actors
outside government and the reduced utilization of M&E information by the regular citizen it’s
hampering the system’s legitimacy. In line with this, SINERGA’s external accountability
function is very limited, due to its feedback, learning and internal accountability focus. While
electoral accountability is somehow recognized, and some efforts are being develop towards
the acquirements of a stronger social accountability practice; upward and horizontal
accountability are still being denial.
Nowadays, the country is starting to see the emergence of diagonal accountability practices
that are pushing social and horizontal accountability. But despite of this, there is still an
absence of networks within the horizontal accountability structures; mainly when it comes to
generate timely information to input the accountability process, which should also be one of
the main tasks of M&E systems.
M&E outcomes should inform not only political and social accountability, they should also
inform other control entities and actors interested to know about government performance like
donors, other levels of government, other sectors, congress, and civil society. All of them
should be considered among the M&E system’ stakeholders, as well as high courts and
control agencies, supporting the accountability networking structure. There is a window of
opportunity for change, and some recommendations to do so are presented at the end of the
document.
10
1. INTRODUCTION
During the last decade of the 20th century a new set of institutional arrangements arose in the
developing world “to address both developmental failures and democratic deficits” (McGee et
al., 2010:1). Specifically for Latin America, two particularly moments where important to
shape those institutional arrangements: 1)The economic structural reforms from the Bretton
Woods agencies leading to the adoption of market-oriented and competitive economies; and
2)The transition to/or consolidation of democratic regimes at the beginning of the 1990’s
which, along with the failure of the welfare state, the emerging globalization process, and the
discredit of the traditional inefficient and corrupt state agencies led to the adoption of a new
perception of governance.
Within this new governance awareness, a wide distribution of economic and political power
inside and outside governmental boundaries was put in place. The process implied the
inclusion and participation of different private actors in public matters (mainly for social
accountability and policy’s formulation, implementation and evaluation), as a strategy to
achieve “efficient, independent, accountable and open public service[s]” (Leftwich, 1994:371)
and to manage and promote development. As a result, the new institutional arrangements that
Latin America countries started to adopt included functions of monitoring and evaluation
(M&E) in their public management procedures.
Since the end of the 1990’s and the beginning of the 21st century, the establishment of M&E
systems has being supported by the adoption of a new aid approach within the development
discourse. New aid modalities like sector and general budget support and the conversion of
aid instruments “from project to more programme-oriented aid and (…) the inclusion of
‘broad-based civil society participation’ as an aid conditionality” (Holvoet and Renard,
2007:66) helped to shape the institutional transition.
Likewise, the adoption of the Paris Declaration (PD) and the Accra Agenda for Action (AAA)
redefined the “importance of M&E because of some of the newly incorporated principles”,
which specifically highlighted the importance of both the feedback and the accountability
(internal and external) functions of M&E. Within those principles, and evidence-based
approach to policy-making and its iterative learning process, a results-based orientation for
management and budgeting, and a participatory perspective with its broad-based approach
11
was introduced (Holvoet, 2012). To accomplish them, it is necessary to rely on well-
established M&E systems.
Within this scenario, the role and responsibility of national governments increased, because
according to the new aid paradigm, they became the ones in charge to manage the entire
system (Holvoet and Rombouts, 2008:9). This entailed not only ensuring all the technical
aspects to produce adequate, reliable and high quality information for both Monitoring and
Evaluation1; it also implied a shift in the focus of the public management dynamics, moving
from inputs to outcomes, where intermediate outputs are also important as part of the whole
policy chain. Therefore, the consolidation of M&E systems that emphasize not only the
implementation (Inputs-activities-outputs) but also the results (outcomes-impact) in the policy
causal chain became fundamental.
In line with this process, Latin America countries started to adopt and install functions of
monitoring and evaluation within their public management procedures; a tendency that
emerged and evolved differently from one country to another. Particularly in the Colombian
case, M&E institutional arrangements were set up under a “planning model”, by which the
M&E system was expected to “improve political decisions at high levels of government and
improve accountability”. To accomplish that purpose, institutional designs were developed
based on their “articulation with national development plans and […] inspired by ministries of
planning” (Cunill, 2010:87).
Colombia’s National Results-Based Management and Evaluation System (SINERGIA, by its
Spanish acronym),was developed base on a Constitutional mandate established 1991. The
design and organization of the system to monitor and evaluate the management and results of
the national public policies, programs and projects was designated to the National Planning
Department (DNP) by an organic law, the 152 of 1994. The system was created to support
decision-making processes and the allocation of budget by improving effectiveness in the
formulation and implementation of public policies related to the National Development Plan
(PND). The aim was to measure the results of public management not only for decision
making, fiscal sustainability and rationalization and prioritization of expenditure, but also for
accountability (DNP, 2012:3).
1 The technical dimensions of M&E systems are the “quality of statistical systems, quality of indicators and targets
(‘indicatorism’)) at the detriment of the broader institutional and M&E policy issues” (Holvoet and Rombouts, 2008:9).
12
Colombia´s M&E system was validated by the World Bank between 1994 and 1999, “and
was promoted as a model to be followed” (Cunill, 2010:82). Until today, SINERGIA has
being considered “one of Latin America’s outstanding results-based monitoring and
evaluation initiatives” (Castro, 2009: 1). But despite this recognition, the Colombian
government remains ineffective in fulfilling social needs and demands, while at the same time
corrupt practices continue to be common within the daily activities of public servants at all
levels, reinforcing the lastingness of developmental failures and democratic deficits.
M&E systems are conceived as an excellent tool to improve government performance and to
construct good governance because of their decision-making guidance function -based on
feedback and iterative learning processes-, and their inside and outside accountability
function. Therefore, one could wonder what are the main issues, problems and pitfalls that are
limiting the desirable performance of the learning and accountability functions of Colombia´s
M&E system? Also, and since the present study will focus on the demand side of the system
and the use of the information it produces to accomplish its (inside and outside) accountability
function, we will try to determine what kind of accountability is effectively being promoted
and/or neglected by SINERGIA? Furthermore, is this reflecting the adoption of a fragmented
approach to M&E, where a technocratic dimension is being imposed over a more political
and/or institutional one?
The assumption here is that most of the challenges regarding Colombia’s M&E system “are
not of a technical nature. They refer more to matters of systems governance” (Cunill,
2010:86). This means that apparently, we will find the majority of the limitations of the
system in those aspects that are related to the politics of M&E and the demand side of the
system, even more if we take into account that the latter suggests a complex process that
involves and requires incentives, an appropriate environment and governmental willingness –
institutional arrangements and power relations between providers and users- to ensure that the
information obtained is effectively use for policy action (Holvoet and Renard, 2007:73-76).
M&E systems should be something more than just a public managing tool for the national
government. They should be a source of information for civil society, donors, oversight
agencies, congress, courts, interest groups, among others stakeholders; with the aim to
enlighten accountability processes and to inform the country’s web of accountability
mechanisms (Mainwaring, 2005: 30). Therefore, M&E outcomes should inform not only
13
political and social accountability, they should also inform administrative and legal one2;
which includes not only donors, civil society and congress among their stakeholders, but also
high courts and control agencies as a fundamental part of accountability networking.
Our findings demonstrate that Colombia´s M&E system focuses on the promotion of the
learning and the internal political accountability functions, as well as the external social
accountability one, leaving aside the intrastate dimension of the concept.
The remainder of this paper is organized as follows: First, a brief description of M&E systems
is presented, discussing both their supply and demand sides. As we have already mentioned
before, we will focus on the latter, and we will pay special attention to the politics of M&E,
presenting an analytical framework than will contribute to the understanding of this particular
issue in the Colombia case. Then, chapter 3 describes the concept of public accountability;
while chapter 4 comprehends a diagnostic analysis of SINERGIA covering the period 2001 to
2011. The selection of the period was based on the statement that “[i]n Colombia, the turning
point for SINERGIA began in 2002, when the Uribe administration reoriented the system
towards a focus on monitoring at the programs and policy levels. Also noteworthy is the
emphasis currently placed on the function of accountability to society” (Cunill, 2010:86).
Uribe’s administration ended in 2010.
For the assessment we will use the checklist developed by Holvoet and Renald (2007) and
adapted later by Holvoet and Inberg (2012); combined with some elements of Holvoet and
Rombouts’ (2008) assessment framework for the politics of M&E (see chapter 2 pages 16-
19); and Bovens’ (2007) analytical and evaluative framework for public accountability (see
Appendix A)3. To ensure objectiveness in the application of the assessment instruments, a
benchmark will be presented (Appendix B) as a guideline for the different aspects of the
system that are going to be assessed. This benchmark should not be taken as a blueprint or an
effort towards standardization; it is rather a methodological support to avoid subjectivity in
the designation of the scores. Each M&E experience needs to be studied under its own
context, since a system of this nature needs to constantly adapt “to the specific political,
2 This accountability concepts were presented by Mark Bovens (2007:454-457) to distinguish between different types of
public accountability. The classification was “based on the type of the forum to which the actor is required to render
account”. The detailed explanation of the concepts will be presented in Table 1. 3 The selection of Holvoet and Renald (2007) and Holvoet and Inberg (2012) assessment instrument is derivate from the
literature review elaborated in the first end of module paper. This instrument results to be the most complete, including as a
value added, a quantitative component which facilitates comparative assessments. This instrument will also be complemented
with Bovens’ (2007) analytical and evaluative framework for public accountability, in order to include in Holvoet and
Renald’s (2007) diagnostic instrument a broader set of stakeholders, and to evaluate their use of information for public
accountability.
14
institutional, and cultural context of each country”; and this “should not be overlooked”
(Castro 2009: 35).
Along the same lines, the quantitative results of the evaluation (scores) cannot be taken as
absolute values; they are only provided as complements to the qualitative analysis of the
system and to allow comparisons between its components (Appendix C). The evaluation of
the system is based on the available data about Colombia’s M&E system, including
institutional published and unpublished documentation, internal and external reports and
evaluations, along with some other assessments of the system performed by the international
community, and other legal, institutional and governmental documents that are related to the
dynamics of SINERGIA, and our own experience.
The limitations of the study are related to the words restrictions imposed to elaborate the
document, which implied a restriction in the amount of information that can be analyzed and
include in the text. Also, and since this policy paper was at first a desk study based mainly on
secondary data, and them it was brought together with working experience, that make it
possible to capture the daily dynamics and first hand information for a more precise analysis
and conclusions, it is important to mentioned that there are some contractual limitations in the
information that is included in this paper. Finally, since the document contains a case study, it
is difficult to provide broad and general statements about the topics that are discussed here.
Nonetheless, this method provides a solid and legitimate technique to approach a topic from a
scientific perspective and to present rigorous conclusions and policy recommendations.
15
2. M&E SYSTEMS: STRUCTURE, LEARNING AND
ACCOUNTABILITY
In this part of the text we will present very briefly a description of M&E systems and their
fundamental concepts; their supply and demand sides as well as the politics of M&E. First, it
is important to understand the difference between implementation-based and results-based
M&E systems. The former concentrates on “monitoring and assessing how well a project,
program, or policy is being executed, and it often links the implementation to a particular unit
of responsibility”. The latter “provides feedback on the actual outcomes and goals of
government actions” (Kusek and Rist, 2004:15). For the purpose of this paper we will focus
on results-based M&E systems, understanding by this “a special public management tool
governments can use to measure and evaluate outcomes, and then feed this information back
into the ongoing processes of governing and decisionmaking” (Kusek and Rist, 2004:12).
An important aspect to highlight from results-based M&E systems is that the process of
measuring, evaluating and feeding-back the government with information is very complex; it
entails an “entire results chain that links the various elements” (Bedi et. al., 2006:XV) of the
system, mostly when a government has a large variety of individual monitoring instruments.
A wide range of independent M&E tools always generates “fragmentation, lack of
coordination, lack of demand, unclear mandates and responsibilities” (Bedi et. al., 2006:XV).
Therefore, a result-based M&E system requires not only a set of management tools, but also a
“range of institutional functions, which include mechanisms to coordinate among data
producers, develop common technical standards and platforms, build monitoring capacity,
organize information flows, compile and analyze data from various sectors, analyze
monitoring data and evaluate […] programs, generate annual progress reports and other
outputs, disseminate outputs across government and to the public, provide advice and support
to policy makers, and organize the participation of civil society” (Bedi et. al., 2006:XV-XVI).
As it is evident now, results-based M&E systems have two main purposes: Feedback –
learning process- and Accountability – internal and external-; which at the same time will
determine the objectives on which the system will focus more.If anM&E system is more
inclined to accomplish an accountability objective, it will seek for independence, impartiality
and credibility of the M&E activities. External validity of evaluations will be very important
as well as the selection of representative samples and activities (Holvoet, 2012b:19). The
design of the system will need to provide a “strict division (of) the evaluation and
16
implementation levels” (Holvoet, 2012b: 21) of government, which at the same time implies a
“minimal involvement of (the) implementation level” (Holvoet, 2012b: 21) in the M&E
processes, providing a reason to adopt an external evaluation strategy.
In contrast, if an M&E system is more inclined towards a learning goal, the principle that will
rule its performance would be the internal use of the information. It will zoom into some
specific issues, with the purpose to allow governmental agencies to identify the specific
aspects of their performance that are not working well -the outliers-, and it will mainly try to
increase internal validity. This type of M&E system will tend to have a “less strict division of
‘implementation’ and ‘evaluation’ function” (Holvoet, 2012b: 21) allowing a mayor
involvement of the implementation level in the M&E process, after taking “specific measures
to safeguard ‘independence’” (Holvoet, 2012b: 21).
Within this context, it is important to differentiate the two activities that constitute this public
sector management tool: monitoring and evaluation. Box 1 provides the definitions of both
concepts. What is important to appreciate here is the way in which these two elements
distinguish from each other and complement the other as well. The importance of
understanding them as two separate processes emergences from a widely recognized
weakness of M&E systems: that is, their two components normally get intermingled.
“Monitoring outcomes is often presented as also somehow constituting evaluation, almost
as an afterthought. That one ought to be measuring ‘changes in outcome’ rather than
‘levels of outcome’, and further control for confounding factors to arrive at some
measure of ‘impact’, is hardly ever mentioned” (Holvoet and Renard, 2007:71).
Box 1: M & E
Source: (OECD, 2002:21)
Monitoring: “A continuing function that uses systematic collection of data on specified
indicators to provide management and the main stakeholders of an ongoing
development intervention with indications of the extent of progress and achievement of
objectives and progress in the use of allocated funds”.
Evaluation: “The systematic and objective assessment of an on-going or completed
project, programme or policy, its design, implementation and results. The aim is to
determine the relevance and fulfillment of objectives, development efficiency,
effectiveness, impact and sustainability. An evaluation should provide information that is
credible and useful, enabling the incorporation of lessons learned into the decision–
making process of both recipients and donors. Evaluation also refers to the process of
determining the worth or significance of an activity, policy or program. An assessment,
17
Following Holvoet and Renard (2007:71) as a consequence of this, M&E systems end up
focusing more on keeping track of the targets and indicators that were (or not) met, than on
identifying why those targets and indicators were (or were not) met, or “questioning the
relevance and usefulness of some of the targets themselves and/or of specific interventions
designed to arrive at them”.
Another observation that has been made with regards to M&E systems, is that there is an over
emphasis of the input side and the technical dimensions of the system, “at a detriment of the
broader institutional and M&E policy issues”(Holvoet and Rombouts, 2008:9).When it comes
to the more institutional dimension of M&E, it is important to understand that this process
involves not only the creation of a shared system with structures for oversight and
coordination, and articulation between governmental actors and stakeholders4 involved in the
process as providers of the information. It also implies encouraging the use of the information
produced by the system and its dissemination among abroad set of stakeholders and potential
users of M&E information, in order to fulfill its learning and accountability functions. In other
words, an M&E system must always have a supply and a demand side; and one should always
take into account the political issues that determine both sides.
2.1 The Supply Side
When organizing the supply side of an M&E system it is frequent to find fragmented
monitoring mechanisms that normally generate “duplication and redundancies in data
collection, gaps or imbalances in monitoring, lack of data compatibility and poor information
flows” (Bedi et. al., 2006:xvii). Therefore, the first step that needs to be develop when
establishing the institutional design of the supply side is to rationalize the “existing
monitoring activities rather than introducing new ones” (Bedi et. al., 2006:20), and coordinate
the “relationships among the various actors in the monitoring field” (Bedi et. al., 2006:21);
including not only line ministries and sector level agencies but also the national statistics
institute, all project level systems, and local and regional ones.
This last action does not mean that the system needs to compact “all activities within a central
agency or under a single superstructure” (Bedi et. al., 2006:21), rather that it needs to create
network between the different supply actors, with:
4 Stakeholders of M&E systems are line ministries, sector agencies, the national statistics center, decentralized governmental
entities, the civil society and parliament amongst others.
18
“A clear allocation of responsibilities (…) increasing transparency and enabling the
various agencies to be held to account for their performance. It should help foster
stronger working relationships between the actors both inside and outside government. It
should map and organize information flows to ensure that data are available to
appropriate people at the proper time. It should develop modalities for consultation and
cooperation and mechanisms for agreement on common needs and standards” (Bedi et.
al., 2006:21).
It involves not only the identification of the details of the system – roles, responsibilities and
information flows require for the adequate functioning of the system-, but also the creation of
a complementary effect between them in order to seek cooperation or partnership in the
delivery of information.
Horizontal and vertical integration are also fundamental for the institutional design of M&E
systems. According to Kusek and Rist (2004:104-105) the importance of horizontal
integration relies on circulating knowledge capital5 within and among organizations, as well
as rationalizing the establishment of information systems: “all concerned organizations and
agencies need to coordinate and collaborate in sharing performance information, especially in
those instances where there are intra-institutional partnerships developed to achieve specific
targets”. Specifically, it relates to the link between central planning, line and sector ministries
and the national statistical agency. They also highlight the importance of vertical integration
referring to the relationship between geographical levels, so that the local and regional ones
can “feed into the larger national data base in determining progress toward the desired
outcomes” (Kusek and Rist, 2004:152-153).
In relation with this concept, Bedi et. al. (2006:32) mention two approaches that can be
adopted when designing this kind of integration: the centralized approach where “the central
government retains control of the process by centrally monitoring local governments”, with
the consideration that decentralization might enhance regional inequalities; or the
decentralized approach in which “local governments […] develop their own monitoring
arrangements so as to define and meet their own monitoring needs, as well as supply
administrative and financial data to the center”.
In order to institutionalize and sustain an M&E system, both Bedi et. al. (2006:26) and Kusek
and Rist (2004:47) mention the need of leadership. Bedi et al. (2006) refer in this respect to
the choice of institutional lead, recommending that a single agency should be empowered
5 “Good M&E systems are also a source of knowledge capital. They enable governments and organizations to develop a
knowledge base of the types of projects, programs, and policies that are successful, and, more generally, what works, what
does not, and why” (Kusek and Rist, 2004:20).
19
with this role and that it must be situated either close to the center of government or to the
budget; and it needs to correspond with the institutional, political and developmental realities
of the country. Kusek and Rist (2004) rather hint at strong and political leadership –in the
form of a political champion-. Basically, the argument goes, “[g]overnments willing to use
performance information to make policy generally have achieved some level of democracy
and openness. But even in these countries, there is often a reluctance to measure and monitor
because of fears that the process will bring bad news to leadership and stakeholders alike.
There are real political limitations to be recognized in building such systems”(Kusek and Rist,
2004:47).Therefore, leadership is closely related with the politics of M&E, which we will
discuss later in the last section of this chapter.
The final aspect that Bediet. al. (2006) consider important to examine is the role of non-
governmental actors in the supply side. Here, the concern is whether or not and to what extent
actors outside the government should participate in the production of M&E information.
According to the authors,the specific details of the involvement of outside government actors
depends on the political dynamics of each country but in general it may includecarrying out
monitoring activities, participation in the system’s institutional structures and providing
analysis and policy advice. In practice though, the role of non-governmental actors in M&E
systems is normally limited to ad-hoc, consultative participation practices (Gildemyn
2011:15).
2.2 The Demand Side
At the same time the government of a country is working toward the establishment of a well-
functioning M&E system, demand for that system and for the information it produces must be
promoted, in order to make M&E activities sustainable (Kusek and Rist, 2004:53). This task
can be very complicated if countries do not support their systems with result-based policies
and with an evaluation culture spread throughout the public sector:
“a sustained source of demand for performance information should be encouraged and
supported, putting the government on notice that it will need to demonstrate results—that
is, governments will need to demonstrate that the policies and programs being
implemented are meeting expectations. Governments need prodding to ensure that
reporting results becomes a regular and routine activity”.
20
The text of Bediet. al. (2006:35-36) mentions that the best way of doing so is by connecting
the M&E system with strategic points of the decision–making process –like the budget6-,
which ensures that the information produced by the system becomes fundamental for the
development of public policies. Examples of the promotion of M&E information outside the
government include involving the media, interest groups or the electorate in the dissemination
process. “The broader goal is to create a virtuous circle whereby government uses sound
analysis and information to formulate its policies, is open to informed discussion on its policy
choices, and, in turn, seeks to build legitimacy and electoral support through evidence of its
achievements”(Bedi et. al., 2006:35).
Both the analytical and the evaluation roles of M&E systems are largely determined by their
demand. To legitimate, stimulate and maintain those roles, the institutional design of the
systems must guarantee that the information - analysis and evaluations- produced from M&E
activities is “available in the proper form at the appropriate time” (Bedi et. al., 2006:36).
Consequently, to develop and preserve the demand for the system and its outcomes, the
analysis and evaluation unit must be as strong as the technical and data collection one, so that
the ‘culture’ of monitoring and evaluation for institutional learning and accountability gets
embedded in public institutions, and M&E information becomes relevant and helpful to
government managers in all phases of policies, programs, and projects, providing “clues to
problems that arise during the course of implementation, and […] [creating] opportunities to
consider improvements in implementation strategies” (Kusek and Rist, 2004:111).
Many problems are normally found when trying to accomplish so: 1) there is a general lack of
analytical capacity within the supply side of the system, which together with the weak
incentives inside public agencies in relation with performance findings, limit and lack the
demand for the system and its information, 2) analysis and evaluation units are normally
situated far away from the center of decision making. Therefore, their analysis become less
important and operational, and the ownership by the decision makers gets lost, 3) analysis and
evaluation units do not work well if they are big and have to develop other tasks at the same
time -like policy making and/or data collection-, mainly because they “become competitive
with other agencies and are likely to fail” (Bedi et. al., 2006:84).
6As Bedit et al. (2006: 86) put it“Creating a link between the PRS monitoring system and the budget process is a powerful
way of generating demand for monitoring. When agencies bid for public resources, this is an important opportunity to require
them to justify their policies and plans based on evidence provided by monitoring data”.
21
Under those circumstances leadership is fundamental: it must be driven by a central body
delegated to analyze and evaluate government performance. For the demand side of the
equation, leadership has to be even stronger than within the supply side, because analysis and
evaluation information may generate major reforms within and outside the government.
“Bringing results-based information into the public arena can change the dynamics of
institutional relations, budgeting and resource allocations, personal political agendas, and
public perceptions of governmental effectiveness” (Kusek and Rist, 2004:21) and therefore a
strong political champion is needed. Ones again, this factor is embedded into the politics of
M&E.
Another fundamental actor for the generationof M&E demand is the parliament because it
brings “legitimacy, country ownership, and the voice of constituencies into the process”(Bedi
et. al., 2006:47). According to Kusek and Rist (2004:184) “in many countries, the legislative
arm of government has also played an important leadership role, by enacting a reform
framework (…), key legislation (…), allowing flexibilities and incentives, or conducting
studies, audits or hearings on government performance”.
Two major roles of parliament can become fundamental in the institutionalization of M&E
systems from the demand point of view; first, its collaborative role with the central
government to formulate and implement development policy (feedback). Second, its oversight
role with respect to the central government and particularly, in relation with policies,
objectives and public resources –authorizing budgets- (political accountability); which
suggest the necessity of adequate, updated and reliable information that can be provided by
the system.
To promote demand for M&E it is also fundamental to develop a dissemination process: “If
they are to have an impact on the policy process, monitoring information, analysis, and
evaluation must be compiled into outputs and distributed to actors inside and outside
government” (Bedi et. al., 2006:39). Following Bedi et. al. (2006), to achieve this
purposeM&E information, analysis and evaluations must fulfill some requirements: they need
to be appropriate for different audiences and uses; the information must be ready at the
appropriated time and with the appropriated quality; it should be circulated through civil
society and others stakeholders not only to inform them but also to take their comments into
account; they need to be related to the national policy and its progress, the dissemination
strategy should include local level governmental agencies, and the M&E outputs need to be
22
disaggregated regional and locally to help them improve their particular functions(Bedi et. al.,
2006).
2.3 Politics of M&E
M&E systems are embedded in the political and institutional settings of a country. These
settingsinfluence the way in which both the supply and the demand side of the systems are
constituted. Therefore, is it fundamental to understand, analyze and study M&E systems
under their political frame, in order to step aside from the narrow, technocratic and
depoliticized approach to the system, and move forward to a broader “organisational and
political” understanding of M&E (Holvoet, 2012c).
In general terms, politics of M&E relate to the power relations and stakeholders’ interests that
impact over the institutionalization and developmentof M&E systems –quality and
usefulness-. Specifically, “the overall socio-political and economic context determines the
power relations between (and within) the parties and institutions involved. Both power
relations and interests influence key evaluation issues” (Holvoet and Rombouts, 2008:10),
reporting and dissemination procedures, budget, staff quantity and quality, and the
identification of the evaluation scope, indicators and targets; amongst others (Holvoet,
2012c:12).
Furthermore, and due to the fact that “politics are part and parcel of M&E” systems, they are
fundamental to develop effectively the system´s feedback-into decision making- function.
Likewise, since “evaluations implicitly make political statements about legitimacy, utility and
appropriateness” (Holvoet and Rombouts, 2008:10) of policies, programs and projects, they
can be used to benefit or neglect different stakeholders’ interests, while developing
theaccountability functionof the system.
Both the feedback and the accountability functions of M&E are closely related with decision
making processes; the former relates to improving policy making by learning from successful
and/or inadequate interventions, and the latter with providing elements to check if results
were met and policies implemented properly (Holvoet and Rombouts, 2008:12). Policies,
programs and projects are consequences of political decisions and that constitutes a political
issue that needs to be acknowledged as animportant influential factor of M&E systems.
Moreover, the “denial of politics of M&E” may undermine and jeopardize “M&E double
function of accountability, feedback and learning” (Holvoet, 2012c:12).
23
In order to analyze the relationship between interests of different stakeholders and M&E
policy and practice, Holvoet and Rombouts (2008:11-15) developed an analytical framework
for the politics of M&E. Even though for the authors it is clear that “not only decision-making
is determined by power relations and interests, but also non-decision making”, and therefore
“[m]easuring influence in decision-making processes in extremely difficult”; the idea of the
framework is to describe those processes, power relations and positions of interests of
different stakeholders, in order to understand the nature of these dynamics within the M&E
environment. Their purpose is to demonstrate that “the persistent exclusion and denial of the
importance and presence of political issues in M&E eventually risk undermining both M&E’s
functions of ‘accountability’ and ‘feedback&learning’ (Holvoet and Rombouts, 2008:10).
The design of the framework was based on the influence that some fundamental aspects of
M&E systems exert over the Political Opportunity Structure (POS); which “refers to the
political context in which stakeholders operate” (Holvoet and Rombouts, 2008:11). Following
Holvoet and Rombouts (2008:11), POS has three main building blocks. The first two [stable
elements7 and volatile elements8] constitute structures that can influence the interests’ position
and power relations of the third element: Stakeholders and their interests in relation with the
learning and accountability functions of M&E. Both stable and volatile aspects of POS -
especially the latter-, are subject to change by the actions and/or omissions of all kinds of
stakeholders9 –governmental and non-governmental- in their path to defend their own
interests.
The three elements of POS are related in the assessment framework with those specific M&E
elements that are influenced by the political context: the M&E institutional set-up, which
includes the legal framework, mandates, information flows, coordination, leadership and
champions of the system, roles and responsibilities of the governmental and non-
governmental agencies involved and the relationship between them. The second element is
capacity and capacity building, followed up by targets and indicators [i.e. type, number and
quality of surveys, selection criteria, levels of disaggregation, etcetera]. The fourth M&E
7 Stable elements refer to administrative structures, institutional entities; e.g. traditions, culture, institutions. 8 Volatile elements refer to sudden events and/or social changes; e.g. national policies, discourse, turnover of power, shift in
election outcomes. 9 According to the authors, it is possible to identify four types of stakeholders in developing countries: 1) the national
authorities constitute by a hall set of different national stakeholders like the executive, the legislative, oversight agencies
among others. 2) The national civil society which comprehends also a very diverse group of stakeholders and interests; 3) the
international donor community, and 4) international NGO´s (Holvoet and Rombouts, 2008:16).
.
24
element included in the assessment framework is feedback, and the way in which the
information produced by the system is distribute and use for learning and accountability.
In general terms, any “sound M&E system (…) unveils all sorts of information, wanted and
unwanted” (Holvoet and Rombouts, 2008:13), generating all types of tensions between the
different stakeholders of the system. Therefore, the argument goes, to fully understand the
dynamics of M&E and to guarantee a more political dimension of the system, it is
fundamental to analyze who is making the decisions and what kind of decision are being
taken, influenced by different interests and power positions. Technocratic and political
approaches to M&E should be combined in order to achieve a sound, useful and supportable
M&E instrument for feedback and accountability.
25
3. EMPHASIZING THE ACCOUNTABILITY FUNCTION
The New Aid Approach (NAA) and its focus on aid effectiveness brought a shift in
developing countries’ decision-making processes and development discourse and practice.
Specifically, the reform agenda10 generated -or at least demanded- important changes in
recipient countries’M&E systems, under the assumption that only sound M&E systems “can
uphold the principles of results-orientation, iterative learning, evidence-based policy-making
and accountability” (Holvoet and Rombouts, 2008:12).
When adopting the Paris Declaration and the Accra Agenda for Action, recipient countries
faced the conversion of aid instruments “from project to more programme-oriented aid and
[…] the inclusion of ‘broad-based civil society participation’ as an aid conditionality”
(Holvoet and Renard; 2007:66). They also had to work towards the enforcement of
ownership, partnership and mutual accountability; and the establishment of institutional and
political settings to do so, stimulating country-led M&E systems along with their feedback-
learning and accountability functions (Holvoet and Renard; 2007:67).
The process described above generated three important consequences. First of all, the fact that
effective M&E systems that perform well their learning and accountability functions are
“necessarily ‘uncomfortable’, highlighting both negative and positive experiences. […]When
results are not met or policies not implement as promised this may have implications for the
politicians that engaged to deliver”. (Holvoet and Rombouts, 2008:12).
Secondly, with national governments at the command of their M&E systems and development
strategies, and the translation from project to more sectorial programs and national policies;
“higher stakes and number of stakeholders” were generated (Holvoet and Rombouts, 2008:12)
and as a consequence, there was an increased in the influence that they exerted over national
politics and over all decision-making processes and specifically, among those related to M&E
(Holvoet and Rombouts, 2008:11). Thirdly, the new participatory approach11 to M&E; which
provides an enlargement of the role of civil society within these procedures is constantly
10 The reform agenda implies the commitments of donors and recipient countries that where incorporated in the Paris
Declaration on Aid Effectiveness (PD) (2005), and in the Accra Agenda for Action (2008). Specifically, these commitments
seek to promote development through aid effectiveness in the former, and to accelerate and deepen in the implementation of
the PD in the latter. They focus on Ownership, Harmonization, Alignment, Results and Mutual Accountability within the new
aid modalities. (OECD; 2008b) 11 A broad-based participatory approach is promoted as a mean to improve domestic accountability, comparative advantage -
due to their closeness to the community-, and to complement the dominant quantitative approach to M&E with CSOs
experience in participatory and qualitative approach to M&E tools,
26
blocked or limited in practice to the formulation stage of the policy cycle, since they depend
of all kinds of formal and informal arrangements (Gildemyn, 2011:8).
Therefore literature, strategies, discourses and practices about M&E implementation need to
include not only all the technical aspects related with the supply and the demand side of the
system, and the solutions to the challenges found during that process; they also need to focus
on the politics of M&E, on the informal and formal arrangements that determine this arena,
especially when it comes to the accountability function of M&E.
3.1 Public Accountability
The concept of accountability is applicable to many fields and it has being defined in very
different ways. For the purposes of this docuemnt we will refer only to the concept of public
accountability, exploring its different types and dimensions. In general terms, public
accountability denotes “answerability and responsibility of public officials” (Mainwaring,
2005: 7)12; the details of the concept and its variations will be introduced in the following
parts of this subtitle.
Having in mind that we will define accountability with the purpose to include the concept in a
more precise way within the analysis of Colombia’s M&E system, the following delimitation
of the concept to be applied specifically to those accountabilities agencies that are present, at
the national level, in the case of Colombia.
The contemporary notion of public accountability based on “strong promises of fair and
equitable governance” (Bovens, 2007:448, 449), emerged, according to Bovens’ (2007), from
the introduction of New Public Management [NPM] in the United Kingdom and the United
States. The insertion of “private sector management styles and instruments into the public
sector, including contract management within and outside the public sector, the use of
performance indicators and benchmarks to evaluate and compare the effectiveness and
efficiency of public agencies” was determinant to shape the transformation of the concept
(Bovens, 2007:3).
12 Mainwaring (2005) named this type of accountability as ‘Political Accountability’ but in order to avoid confusion with the
different dimensions of the concept that we will introduced later, we will rather identify it as public accountability, in line
with Bovens’ (2007) denomination of the concept.
27
Following Bovens (2007), under the NPM, accountability was conceived as an instrument to
achieve public governance, but nowadays it has a value on its own, it is a goal in itself, and it
constitutes an “icon for good governance in both the public and in the private sector”
(Bovens, 2007:4). But regardless the importance of the concept in both the development
discourse and practice, it remains fussy and malleable; its definition “varies widely from one
author to the next” (Mainwaring, 2005: 3), and the debate about the proper way to set its
boundaries continue. According to Gildemyn (2011:10), “[s]o far, the existence of more than
100 different definitions and types of accountability has been reported in the literature.
Despite this variety, the ultimate goal of accountability remains the same: to keep power
under control and prevent abuses”.
3.1.1 Dimensions and Institutional Arrangements
Mark Bovens (2007:2) presents an interesting analytical framework for the empirical study of
accountability arrangements in the public domain. This framework can help us to understand
the different ways in which it is possible to classify public accountability. The framework also
allows us to introduce some other aspects, definitions and perspectives that have being
developed by other authors with respect to the issue at stake. This framework allow us to
place boundaries to the concept of accountability, in order to be able to empirically determine
“when a certain practice or arrangement qualifies as a form of accountability” (Bovens
2007:5), and to “establish more systematically whether organisations or officials, exercising
public authority, are subject to public accountability at all” (Bovens 2007:4,5).
The author adopted a concise description of accountability, which includes “concrete
practices of account giving” (Bovens, 2007: 6). In this narrow sense, accountability would be:
“[T]he obligation to explain and justify conduct. This implies a relationship between and
actor, the accountor, and a forum, the account-holder. […] This usually involves not just
the provision of information about performance, but also the possibility of debate, of
questions by the forum and answers by the actor, and eventually of judgments of the actor
by the forum. Judgment also implies the imposition of formal or informal sanctions on the
actor in case of malperformance or, for that matter, of rewards in case of adequate
performance” (Bovens, 2007: 6).
Here, the accountability process examines ex post facto the actors’ conduct, and it entails a
“specific set of social relations” (Bovens, 2007: 7).With public accountability, the actor will
often be a public institution or a government agency”; and the forum “can be a specific
person, such as a superior, a minister, or a journalist, or it can be an agency, such as
28
parliament, a court, or the audit office13, but it can also be a more virtual entity, such as, in the
case of public accountability, the general public” (Bovens, 2007: 7).
The next three element of Bovens’ accountability notion are the ones that present more
variations across authors. The first one refers to the nature of the relationship between the
actor and the forum and if one should limit the concept of accountability only to those
relationships that can be classified as principal-agent interactions. According to Bovens
(2007:7), “[t]he relationship between the forum and the actor often will have the nature of a
principal-agent relation - the forum being the principal, e.g. parliament, who has delegated
authority to a minister, the agent, who is held to account himself regularly about his
performance in office”; but he also admits the existence of accountability relations that don’t
follow the principal-agent scheme, e.g. the one develop by courts and all other judicial bodies.
Along with the latter argument, Mainwaring (2005) presents a group of authors that support
the existence of accountability relations that do not fit under a principle-agent structure. To
quote from his article, “Kenney and O´Donnell […] argue that not all relationships of
accountability involve a principal that can dismiss an agent” (Mainwaring, 2005:14); they go
beyond principal-agent interactions. Mainwaring himself shares this definition, his concept of
accountability includes control agencies and the judiciary as key parts “of the web of
accountability because they are formally charged with overseeing and/or sanctioning public
officials for the discharge of their public duties” (Mainwaring, 2005:15); even though they are
not principals.
The second source of disagreement between authors is the degree by which the answerability
dimension of accountability has to be institutionalized, i.e. if the obligation has been
formalized or not. Following Bovens (2007:7), “[t]he obligation that lies upon the actor can
be formal or informal. Public officials often will be under a formal obligation to render
account on a regular basis to specific forums, such as supervisory agencies, courts, or
auditors. […] But the obligation can also be informal, as in the case of press conferences and
informal briefings or even self imposed, as in the case of voluntary audits”.
This dimension of accountability –answerability- is based on more than the just the obligation
of an actor to provide information about its performance –tasks, outcome, procedures-.
13The contemporary notion of public accountability goes beyond the pure concept of financial auditing based on the external
monitoring of inputs and activities, in compliance of legal standards; it takes into account the results of governmental actions
and therefore, actors are hold to account ex post facto. Consequently, for the purpose of this paper we will include within the
accountability forums only those audit offices that develop performance auditing; the Controllers General –Contraloría- for
the Colombian case.
29
“Often, particularly in the case of failures or incidents, this also involves the provision of
explanations and justifications […] and the possibility for the forum to interrogate the actor
and to question the adequacy of the information or the legitimacy of the conduct” (Bovens,
2007:7, 8).
Mainwaring (2005:10) deepens a little more on this issue by claiming that “intrastate
answerability and sanctioning are not limited to perceptions about the legality of a public
official’s or agency’s actions. Public official´s and agencies must provide political as well as
juridical accounting of the discharge of their public duties” (Mainwaring, 2005:11).The
argument refers directly to the traditional concept of accountability; electoral accountability14,
where “elected political representatives are accountable to voters […] for their political
actions regardless of whether they are perceived to have transgressed any constitutional
norms” (Mainwaring, 2005:12). As a result, the argument goes; intrastate accountability
should also include “judgments about political matters”, not only legal ones (Mainwaring,
2005:12).
Finally, the third disputed matter is the act of passing judgment to an actor and imposing
sanctions in case of negative judgements; in other words, the enforceability dimension of the
concept. Bovens’ (2007:8) position about this argument is that “the possibility of sanctions of
some kind is a constitutive element of narrow accountability and that it should be included in
the definition ”. He adds that it is not necessary that the forum itself becomes the punishment
performer, since not all accountability agencies and/or actors have the power to do so, which
is the case of oversight agencies, Civil Society Organizations (CSOs)15 and the media.
Mainwaring (2005:12-13) agrees with Boven’s position and argues that there is a need to
differentiate “between direct legally ascribed sanctioning power and other sanctions”. The
key aspect here is that all accountability agencies need to have “some capacity to redress
wrongdoing by referring a case to other venues (especially the justice system)” and therefore,
“accountability does not require direct, legally ascribed sanctioning power”.
Having in mind that this sanctioning aspect has a lot to do with the inclusion or not of social
actors as forums of accountability, we will also introduce Gildemyn’s (2011) explanation of
the issue, since she focuses on the contribution of CSOs to accountability processes -
specifically to M&E systems-. For the author, the enforceability dimension of the concept is
14 What O’Donnell calls vertical accountability (O’Donnell,1998:112) 15With Civil Society Organizations we denote “a wide variety of organisations, beyond non-governmental organisations
(NGOs), such as trade unions, think tanks, community-based organisations and others” (Gildemyn, 2011:8)
30
fundamental to talk about “full accountability”. But to develop it, it is necessary to count with
“the presence of accounting actors that have enough power and/or autonomy to impose
sanctions” (Gildemyn, 2011:11); and usually CSOs don’t have enough power or legitimacy to
sanction governments.
As we have already mentioned before, the role of social actors as accountability forums “has
been located in the vertical axis” traditionally16; but this cannot be consider a strong
mechanism of accountability because of “information asymmetry and the fact that voters are
punishing or rewarding government´s performance as one single package”.
As a consequence, the introduction of the New Aid Approach conditionalities regarding “CSO
participation in a variety of policy spaces, including M&E” (Gildemyn, 2011:15), reinvented
their role as fundamental actors to improve domestic accountability”–at least in the
development discourse- (Gildemyn, 2011:15). Now it is not only about performing the
traditional electoral –vertical - accountability forum; social actors can also develop now
intrastate accountability functions –O´Donnell’s horizontal accountability-; but given that
they are not governmental actor, this new accountability dimension “has been referred to as
diagonal or hybrid accountability”17(Gildemyn, 2011:16).
3.1.2 Types of Accountability
In relation with the section above and according to the purpose of this document, we will
consider public accountability as a public relation, adopting Bovens’ (2007:9)
conceptualization:
“A relationship qualifies as a case of accountability when:
1. There is a relation between an actor and a forum
2. in which the actor is obliged
3. to explain and justify
4. his conduct,
5. the forum can pose questions,
6. pass judgment,
7. and the actor can be sanctioned”
16Social actors developed the vertical dimension of accountability “[t]hrough elections, lobbying and other mechanisms” […]
to hold governments accountable for their past performance” (Gildemyn, 2011:15). 17According to Gildemyn (2011) “for hybrid accountability to work effectively the following five conditions should ideally
be present: 1) citizen participation should be institutionalized, 2) CSOs should have a continuous presence within the
diagonal accountability space, 3) the dialogue between state and non-state actors should be institutionalized, 4) CSOs should
have free access to information and 5) they should be able to disseminate their findings and results directly to the legislative
bodies.
31
As we clarified before, points 6 and 7 can be developed by a different forum with sufficient
power to do so. In relation with this last point Bovens (2007:12) presents two definitions that
complement his argument: 1) Accountability arrangements referring to “accountability
relationship that has taken on an institutional character; and 2)accountability regime defined
as a “coherent complex of arrangements and relationships. […] [It implies] a system of
interconnected, standardized forms of accountability”
To define the types of public accountability, the author (Bovens 2007: 13 -22) identifies four
ways in which it is possible to classify this kind of relationships: 1) by the type of forum, 2)
by the actor that should render account, 3) by the aspect of the conduct that is require to
render account for; and 4) the nature of the accountability relationship between the actor and
the forum, and specifically, the nature of the obligation. For the purpose of this paper and in
order to avoid the constant confusion present in the literature between O’Donnell’s (1998;
2004; 2005) classification of vertical and horizontal accountability, and its inconsistency
when examining such classification under “the nature of the obligation” (Bovens 2007: 13)
point of view; we will adopt the first way of classifying public accountability.
Under this parameter, Bovens (2007: 14) recognizes “five different types of forums and hence
at least five different kinds of public accountability”; which we summarise in the table
above18. Despite the different accountabilities dimensions, it is important to keep in mind that
all of them “can mutually reinforce each other” (Gildemyn, 2011:14).
The interaction between electoral –vertical- accountability, intrastate –horizontal-
accountability and social oversight –diagonal and societal accountability” generate “strong
mutual effects”; they are “mutually interactive” (Mainwaring, 2005:21). Intrastate
accountably can be improved through electoral accountability; furthermore, “social
mobilization, especially in conjunction with media coverage, can trigger improved intrastate
accountability” (Smulovits and Peruzzotti in Mainwaring, 2005:21).
18 We will leave aside the “Professional accountability: professional peers” since it goes beyond the
scope of this paper (Bovens, 2007: 16).
32
Table 1. Classification of Public Accountability by Type of forum (Bovens, 2007: 14-17)
Political
Accountability (Electoral)
Legal Accountability Administrative
Accountability Social Accountability
Electoral
representatives, political
parties, voters, media
Courts Auditors, inspectors, and
controllers
Interest groups, charities
and other stakeholders
- Extremely important
within democracies.
- Along the chain of
principal-agent relationships.
- The people’s
representatives render
account to the voters at
election time.
- Each of the links in the
chain is not only principal
and agent, but also forum
and actor.
- The mechanism of
political accountability
operates precisely in the
opposite direction to that
of delegation.
- Only two ends of the
chain – the voters and the
elected public servants.
- Political parties and
party barons: important,
informal political forums.
- Media: informal forums
for political
accountability.
- Can be develop by
‘ordinary’ courts or
also specialized
administrative courts
- Legal accountability
will usually be based
on specific
responsibilities,
formally or legally
conferred upon
authorities.
- Legal accountability is
the most unambiguous
type of accountability,
as the legal scrutiny
will be based on
detailed legal
standards.
- Wide range of quasi-
legal forums exercising
independent and
external administrative
and financial
supervision and control.
- These new
administrative
forums: local
ombudsmen and audit
offices, to independent
supervisory authorities,
inspector generals, anti-
fraud offices, and
chartered accountants.
- Auditing offices: have
been broadened to
secure not only the
probity and legality of
public spending, but
also its efficiency and
effectiveness.
- These administrative
forums exercise
regular financial and
administrative
scrutiny, often on the
basis of specific statutes
and prescribed norms.
- In reaction to a
perceived lack of trust
in government, more
direct and explicit
accountability relations
between public agencies
and clients, citizens and
civil society.
- Based on the role of
NGOs, interest groups and others as relevant
‘stakeholders’ not only
in determining policy,
but also in rendering
account.
- Agencies or individual
public managers should
feel obliged to account
for their performance to
the public at large or, at
least, to civil interest
groups among others.
- Instruments: the
institution of public
reporting and the
establishment of public
panels. The internet has
given a new dimension
to this form of public
accountability.
- The possibility of
judgment and
sanctioning are often
lacking.
33
4. DIAGNOSTIC OF COLOMBIA´S M&E SYSTEM
“Setting up and consolidating a whole-of-government M&E system is not just a technical or
policy initiative because, in actuality, it is an ongoing public sector reform, which involves all
administrative areas, sets out to change public officials’ behaviors, and has to be in line with the
broadest political objectives (e.g., democratization, governance, accountability, etc.)” (Castro,
2009:35).
The aim of this chapter is to present Colombia´s M&E diagnostic between 2001 and 2011.
The relevance of this period was mentioned before, but it will be made clear throughout the
rest of the document that during Alvaro Uribe’s administrations (2002-2006 and from 2006-
2010) SINERGIA undergone an enormous development, compared with what the system had
accomplished before, and in relation with the majority of M&E exercises developed in the
region -with the exception of Chile and Mexico (Burdescu et. al. 2005). We will try to assess
the quality of Colombia´s M&E system by identifying its strengths and pitfalls, in order to
understand how it is helping or hampering governmental effectiveness and accountability
processes.
The assessment will be based on an instrument developed by Holvoet and Renard (2007) and
adapted later by Holvoet and Inberg (2012) to evaluate sector M&E systems. The latter
application of the methodology (Holvoet and Inberg, 2012) was based on a checklist that
contained 34 questions19, developed to reinforce the diagnostic instrument to assess the
supply [Questions 7,8,19-26] and the demand sides [Questions 17,18, 30-34] of M&E
systems. The checklist also varied in its six dimensions: policy; indicators, data collection and
methodology; organization [structure and linkages], capacity, participation of actors outside
government, and use of M&E outputs.
One of the most interesting aspects about this approach is that the authors provide a “point
scoring system” (Holvoet and Renard, 2007:69); creating a combination of quantitative and
qualitative analytical tools within the assessment. The latest version of the assessment tool
encloses a five-point scoring system: weak (1), partially satisfactory (2), satisfactory (3), good
(4) and excellent (5).
19 Some of the questions where removed or modify either because they don’t apply for the Colombian case or because they
were sector oriented.
34
For Colombia’s M&E assessment we will also focus on the demand side of the system, which
is generally considered a major issue for many reasons. First of all, it implies the use of the
information produced by the system about the “results of government actions and policies”
(Kusek and Rist, 2002: 152) to accomplish the two main objectives of M and E: learning and
accountability; which implies “solving problems in program implementation; accountability
within the government; provision of public information (for accountability, legitimacy, or
public choice); improving program design; and prioritizing (and possibly coordinating) across
programs” (Shepherd, 2011: 4).
Secondly, it suggests a complex process that involves and requires incentives, an appropriate
environment and governmental willingness –institutional arrangements and power relations
between providers and users- to ensure that the information obtained through a well-
established M&E system is effectively used for policy action (Holvoet and Renard, 2007: 73,
76). Thirdly, in most of the countries adopting or strengthening M&E systems,“achieving
strong demand (…) is not easy” (Mackay, 2008: 179). The literature considered for this study
suggests that most of the developing countries do not succeed in creating demand for M&E or
at least find it very difficult: “M&E findings generally remain underutilised and this deficient
M&E demand side affects M&E supply and sustainability of the system” (Holvoet and
Inberg, 2012: 13).
Also, within the demand side, the M&E assessment will focus on the accountability function
of the system, introducing the concept of public accountability that was presented in chapter
3, by proposing a set of question within the Holvoet and Inberg (2012: 13) checklist that
relate to Bovens’ (2007) analytical and evaluative framework for public accountability.
Specifically, for this assessment the checklist maintains its six dimensions but we added two
more stakeholders (oversight agencies and the judiciary) to the ‘use of M&E outputs’
dimension. This dimension was also divided into two sub dimensions: feedback function and
accountability function. What we did was to re grouped the already existing question into the
two sub dimensions and we added some specific questions to differentiate between the
different types of public accountability.
35
4.1 A Technocratic Government with a Well-organized System
In general terms, the ‘Policy’ dimension of Colombia´s M&E system has an excellent
performance when it comes to separate its’ monitoring and evaluation components, and in the
establishment of an M&E plan. SINERGIA scored satisfactory in its performance of the
feedback function, basically because even though its approach to report, dissemination and
integration of information is clear, it is also quite limited. M&E alignment with planning and
budgeting processes it is partially satisfactory; hampering, along with the lack of autonomy
and impartiality [scored weak], the desirable functioning of the system.
4.1.1 Colombian M&E Context
It is important to remember that for the assessment of Colombia’s M&E system we will take
into account the process, evolution and changes that occurred between 2001 and 2011. The
reason for this is that even though the creation and consolidation of SINERGIA occurred in
three stages: from 1990 to 1996 the first one; from 1996 to 2002 the second one, and from
2002 until 2010 the third one (Mackay et al., 2007:2)20; it was just from the last stage onwards
that the system started to acquire different achievements. As we have already mentioned
before, the obligation to create a National M&E system was established by a Constitutional
mandate since 1991. But the system was not formally instituted until 1994 (Mackay et al.,
2007:2) when the design and organization of it was designated by an organic law -the 152 of
1994- to the National Planning Department [DNP by its Spanish acronym].
As a result of this period, the system was conceived to improve effectiveness in the
formulation and implementation of the National Development Plan [PND by its Spanish
acronym], by monitoring and evaluating the management and results of public policies and
projects; in other words, to measure the results of public management. Therefore, it conceived
an essential tool for decision making and accountability.
Unfortunately, this first drive was overshadowed by its second phase of implementation,
“when the standing of SINERGIA within the government […] decline, partly due to a
perception of difficulties with the management of the system” (Mackay et al., 2007:2),
generating doubts about its relevance and usefulness for the public sector. During this crises
period the institutional maintenance of SINERGIA was due to its Constitutional mandate, and
this is a very important aspect to highlight because this will occur constantly when analyzing
20 The publication year of the article is 2007 and therefore the authors marked the third stage from 2002 until 2007. We
consider that the third phase went until 2010.
36
the institutionalization of Colombia’s M&E system. According to Castro (2009:25), “there is
a regulatory tradition in Colombia that prioritizes what is established in the legal framework,
to ensure continuity from one administration to the next”. This regulatory tradition has been
very beneficial to guarantee the sustained development of SINERGIA; its “incorporation into
Colombia’s Constitution and laws has thus allowed the system to survive during times when
support for it was low or there was [sic] lack of interest in M&E activities”(Castro, 2009:25).
But at the same time this regulatory tradition has brought the greatest obstacles when it comes
to align the system with the planning and budgeting procedures, as we will see later on.
The third phase of the system’s implementation started in 2002 with the election of President
Alvaro Uribe Velez; who, according to Mackay et al. (2007:2), had a “strong desire for a new
culture of public administration, based strongly on social accountability – ‘social control’.
Thus, he introduced a system for monitoring and reporting progress vis-à-vis presidential
goals and the country’s development goals”. Supporting president Uribe’s desire, in 2004 the
National Economic and Social Policy Council (CONPES by its Spanish acronym)21
determined, by its 3294 document, the new definition and scope of action of SINERGIA,
including: 1) Monitoring the PND, 2) Evaluating strategic policies and 3) An accountability
function. Since then, SINERGIA was divided into two components or subsystems: i) the
monitoring system of government goals [known at first as SIGOB, then since 2009 as
SISMEG22, and now since 2012 as monitoring-SINERGIA], and the ii) National Evaluation
System [known at first as SISDEVAL and now since 2012 as evaluation-SINERGIA] (DNP;
2012: 3).
The CONPES document is actually the one that constitutes the M&E plan. Since Colombia
does not have a PRSP document, and the two National Strategies for International
21 The National Economic and Social Policy Council is the highest government body in charge of the coordination of
economic policy in Colombia. It does not issue decrees, but provides orientation for the macroeconomic policy of the
country. 22In 2002, "the system’s components were redesigned to incorporated modern tools for managing for results like SIGOB
[Government Results Information System]"(DNP 2012: 5), which is the monitoring system for the Government goals initially
developed by the UNDP. The idea of the system was "to produce short-term information on the results of those policies
reflected in the PND, hopping that High level Government and citizens could track the main commitments of the national
government" (DNP 2012: 5). In 2008 the DNP obtained the intellectual property rights of the system and introduced some
technological reforms for the "establishment of monitoring methodologies for public management, analysis techniques and
the use of information, roles and responsibilities towards the achievement of goals, and regular training exercises for public
officials” (DNP 2012: 5). These reforms were developed along with the institutional changes incorporated within the Public
Policy Monitoring and Evaluation Directorate (DSEPP by its Spanish acronym); which is the Directorate in charge of
SINERGIA inside the DNP. In 2009 a new reengineering process began, not only inside SIGOB but also within the processes
associated with development assessments; which lead to the emergence of SISMEG and SISDEVAL (These quotes were
translated by the author). Nowadays, the two components are called SINERGIA-monitoring and SINERGIA-evaluation but
they follow the same logic as SISMEG and SISDEVAL. The main difference between SIGOB and SISMEG is that the
former focused on processes and sectorial objectives, while SISMEG focuses on products and results and added transversal
objectives.
37
Cooperation developed during the period studied [2003 – 2006 and 2007-2010] did not
included an M&E plan, the institutionalization of the system’s reform required a document for
its formalization. This document was based on Uribe´s first administration National
Development Plan 2002 - 2006 ‘Towards a Communitarian State’, where on its fourth chapter
‘Increase Transparency and State Efficiency”, the government compromised with the renewal
of public administration and therefore, with the implementation of a state that focuses on
promoting effectiveness, efficiency, productivity and result-based management (DNP, 2003:
260 – 268).
Therefore, CONPES’s document 3294 justified the implementation of a Public
Administration Renewal Program [PRAP - by its Spanish acronym] and specifically, the need
to reform the National Evaluation System. This was basically due to the lack of information
about the efficiency and effectiveness of programs and policies implemented, as well as the
investment and allocation of public resources. As the main argument goes: “…despite the
magnitude of public resources invested over the past decade, the actual effects of these
investments are unknown […]. This makes it hard to determine whether the policy direction
has been the most appropriate […]. As a result, the decision to spread, suspend, replace or
modify programs and policies as well as the allocation of resources […] have been taking
with insufficient knowledge of the real factors of success or failure” (CONPES-DNP,
2004:5)23.
Also and due to the fact that this document only established the general guidelines for the
implementation of the system’s reform, the DNP elaborated in 2012 a Methodological Guide
for Monitoring the National Development Plan and the Evaluation of Strategic Policies24
(DNP 2012). This methodological guide actually supports the materialization of the processes
which are built upon on the CONPES’s document and therefore, can be consider a
fundamental part of the M&E plan.
The main aspects related to the M&E plan [what to evaluate, why, how and for whom] as well
as the relationship and differences between the two concepts [monitoring and evaluation] are
well defined in the documents mentioned and are easily identified in the Colombian case.
23This quote was translated by the author. 24Although this document was produced and published in 2012, far beyond the period established for the M&E assessment, it
contains a retrospective view of the Colombian system, including all aspects of its implementation and consolidation process.
All quotes and references from the methodological guide were translated by the author.
38
4.1.2 Assessing M&E Policy
CONPES document 3294 describes the difference between monitoring and evaluation as well
as their complementary relationship; “[w]hile monitoring should be continuous and
comprehensive of the priorities of a government, not all programs require evaluation and it
should not be permanent. The need to evaluate a program can respond to its strategic
importance, the amount of resources invested on it, its contribution to fill gaps in knowledge
or in response to an innovative design, among other criteria” (CONPES – DNP, 2004:10).
The majority of the documents regarding Colombia´s M&E system highlight how the
monitoring and the evaluation functions of the system complement each other. Within
SINERGIA’s model, the monitoring function determines what to evaluate. The Evaluation
function emphasizes what to monitor, feeding back the policy cycle (DNP 2012:6). Basically,
“indicators used to oversee management of the goals in the National Development Plan permit
the analyst to identify the stage within the delivery chain that is not working. The scope of the
evaluation can then be shifted to clarify what is happening at that stage”. In line with this
logic, “SINERGIA has identified a typed of evaluation for each stage of the logical
framework” (UNDP, 2011:51), as we in explain with more detail in the next section.
Following CONPES track, the DNP’s methodological guide is very explicit in detailing and
presenting what is going to be monitored and how (DNP 2012: 6-18). The same applies for
what is going to be evaluated, how and why to classify specific policies as strategic for
evaluation25 (DNP 2012:19-54). According to the document, SISMEG has to monitor
progress of the National Development Plan goals of the former government, annually and for
the whole period [four years]. Within the frame of results-based management, the first version
of it -SIGOB- sought to produce short-term information on the results of the
policies reflected in the PND. The aim was to provide the high level government and the
citizens with adequate information to follow up the main commitments of the administration.
After 2009, a reengineering of the subsystem led to a broader dimension of it –SISMEG-
allowing a more systemic and articulated monitoring process, including the way in which the
different institutions and actors facilitate the accomplishment of the commitments made
by the national government (DNP 2012:5).
25“Policies, programs and entities subject of assessments due to their strategic classification will be selected based on the
following criteria, among others: i) the amount of public resources invested, ii) the size and characteristics of the affected
population, iii) the importance to the sector, iv) the innovative nature of the program and/or policy, and v) the potential
applicability and replicability of the lessons learned from the evaluation”.
39
For the evaluation part, the DNP methodological guide claims that SISDEVAL and its
evaluation processes provide a value judgment on the design, implementation and effects on
the beneficiaries of a governmental intervention, in order to provide decision makers with
sufficient information to improve public policies (DNP 2012:3). Here, evaluations prepared
by external national and international consultancy firms, as well as universities and think
tanks are clearly decision-making oriented. The aim of these evaluations is to improve public
sector performance by assessing those public policies that are strategic for the national
government and that are reflected in the PND. In that way, the identification of bottlenecks
within those policies provides answers to decision maker’s questions about those aspects that
are hampering the good performance of government. As a whole, SINERGIA’s evaluation
side effectiveness is about learning and the use of the information for decision making (DNP
2012:19).
Last but not least, the theoretical framework of the 3294 CONPES document highlights how
the quality, relevance and availability of information affect the success of both the evaluation
and monitoring processes of the system. On the same line, Castro claims that “[i]f the system
does not have good information it cannot be used as a tool for quality evaluation” (Castro,
2006:30). In the Colombian context this has been a huge problem due to the low quality of the
monitoring information produced by sector entities. Also, the lack of connection between
SINERGIA and the National Statistics Administration Department – DANE (by its Spanish
acronym) has increased the cost of external evaluations, creating a clear “need for more
detailed information […] [from] either the regular household surveys conducted by the
national statistical office or from the administrative data produced by entities” (Mackay,
2007: 124).
4.1.2.1 Reporting and Dissemination
Despite the clarity within the system and the M&E plan about the differences between
monitoring and evaluation, as well as the links that must exist between them; when we talk
about its approach to reporting, dissemination and integration of M&E information within and
throughout government agencies and external actors, the system remains quite limited.
It is true that the system was extensively used by the President for political and social control,
and for many, this constitutes the reason why Colombia’s M&E system is a good practice in
Latin America. SINERGIA was very effective in feeding back to the planning process of the
40
executive branch at the national level. There was a clear integration of both monitoring and
evaluation information in the formulation process of the PND, and in the debate with respect
public policies and programs (Burdescu, 2005: 2). During the period studied SISMEG became
the most important information tool of the president. From the beginning, this monitoring
instrument and its information content was defined and adapted to fulfill the presidential
informational needs. It was also useful to define “roles and responsibilities among the DNP,
the President´s Office, ministries, and agencies” (Castro 2009, 11); this is, the national level
executive bodies, which were the ones in charge to input into the system the performance
information related to the PND targets and indicators that measure them.
Many reports documenting this case study highlight how SISMEG's information outputs were
intensively used by Uribe´s national government. Specifically, they mentioned how the
president personally used this information tool for two main purposes: to developed the
learning and feedback function within the executive branch, throughout his “administrative
monitoring monthly meetings with individual ministers”26 (Mackay, 2007:13); and to promote
social control processes by developing “weekly meetings in various municipalities” (Ibid) of
the country. Also, this information was extensively used by the “President´s Office in press
releases and reports to congress, in development plan evaluations and reports by the DNP, in
Sector Ministers’ meetings and reports, and in Mof budget documents submitted to Congress”
(Castro, 2009: 12).
Additionally, the database of performance information consolidated by SIGOB27 was
available to the public online and when the performance targets were not met, the “person
responsible for the information related to those specific targets28 […] [was] obliged to prepare
a report giving the explanations of the case” (Mackay, 2007:12, 13)29.
The evaluation component of the system also supported the feedback function of the
executive branch. SISDEVAL developed a process to define an action plan between the DNP
and the government agency in charge of the implementation of the policy evaluated. The idea
was to apply those relevant recommendations derived from the evaluations, that were viable
26This quote was translated by the author. 27 During this period, SIGOB´s database contained around 500 indicators to follow “government performance in terms of the
320 goals of the National Development Plan. For each performance indicator, the database […] [published] the target, the
strategy to achieve it, the base performance, the annual goals and the amount spent by the government” (Mackay, 2007:12,
13). 28 From now on this person will be known as the target administrator.
29This quote was translated by the author.
41
from a legal, economic and operative point of view, taking into account the current
policy context (DNP 2012:19).
SINERGIA’s feedback approach is clear; the information produced by both its monitoring
and its evaluation components “provides feedback on the actual outcomes and goals of
government actions” for the learning and planning processes (Kusek and Rist, 2004:15). But
this approach is limited. The distribution and usage of the output information relies only on
executive branch institutions at the national level, including line ministries and those agencies
that are considered head of sectors. Within the same line, top-down feeding back processes of
data production and analysis is almost non-existent. Furthermore, when we talk about the
implementation of the M&E system and the dissemination of its results at the local levels, we
found that its performance is very weak; as we will see in more detail in the section dealing
with vertical upward and downward integration. This issue is very problematic if we take into
account that “[f]or an M&E system to be considered successful and sustainable, the
information and findings of M&E have to be utilized intensively by all stakeholders,
including […], depending on how the system has been devised, civil society” (Burdescu,
2005: 3,4).
As Mackay (2007) notes, “there does not appear to have been a widespread adoption of M&E
practices by ministries and subnational governments” (Mackay, 2007:33). Also, and
according to Castro (2006:30), this could be due to a neglect of “incentives to stimulate
demand for evaluation from the different institutions” which also lead to a lack of
“dissemination of results, which undermined accountability” (Castro, 2006:30). As a result,
and taking into account the tight link identified between the monitoring and the evaluation
components of the system, the narrow approach with respect to distribution, dissemination,
integration and the use of information, is not only limiting the feedback function but also the
accountability –internal and external-one.
It seems that for the Colombian case the problem was derived from a lack of definition of the
scope of the system, and the extent to which it was supposed to involve different levels of
government. As Castro (2006) asserts, “[w]hile there were efforts to monitor the other levels
of government from the center, this created problems in a decentralized country such as
Colombia. Thus we had not clearly defined the different linkages, tasks and responsibilities of
the system, which up until 2002 created complex institutional problems” (Castro, 2006:30).
42
To support the score given to the feedback function and its reporting, dissemination and
integration approach, we will recall some of the results of a study developed by the National
Consultancy Center30 in 2010 for the DNP. The aim of the study was to “establish a baseline
to evaluate the achievements and progress of those actions aimed to strengthen the
Monitoring, Evaluation and Accountability activities” of Colombia’s M&E system. In order
to do so the study divided the universe into three groups “Group 1: National government
agencies which maintain strong relationships with SINERGIA”. This group was identified as
‘Cercanos’ [close] throughout the report. “Group 2: National government agencies which
maintain casual or indirect relationships with SINERGIA”. This group was identified as
‘Ocasionales’ [occasional] within the document. And “Group 3: Civil Society Organizations,
academy and NGOs”. This group was identified as ‘Externos’ [external] throughout the report
(National Consultancy Center, 2010:3). Table 2 shows the number of people surveyed for
each group.
Table 2. Number of surveys by group
Group Number of Surveys
1. Close 39
2. Occasional 24
3. External 58
TOTAL 12131
Source: National Consultancy Center, 2010: 57 - 62. Chart elaborated and translated by the author.
In general terms and according to the report, 100% of respondents from group 1 know about
SINERGIA; while in the case of group 2 and group 3 this percentage varies to 83% and 88%
respectively (National Consultancy Center, 2010: 8-9). In the case of group 2, the majority of
the people who know about SINERGIA argued reasons related to their previous jobs in
different governmental agencies, due to the M&E training programs that DNP organizes.
Others from the same group mentioned that their knowledge about SINERGIA was due to a
legal requirement that imposed the formulation of PND targets and indicators which should
be reported periodically (National Consultancy Center, 2010:9). People from group 3 argued
reasons related to their jobs as well, but this time their claims were attached to the use of the
M&E information as input for their own studies and projects (National Consultancy Center,
2010:9). Additionally, in line with these results and according to the study, “perception about
30 The National Consultancy Center is one of the most important and recognized consultancy firms of the country. 31 In addition they were 30 interviews, 10 from each group (National Consultancy Center, 2010: 3).
43
the quality of SINERGIA improves with the proximity of the group to the system (see Graph
1), and this is a tendency along the report” projects (National Consultancy Center, 2010:9).
Graph 1. Percentage of people that scored SINERGIA’s General Quality as excellent or
very good (Total: 30%)
Source: National Consultancy Center, 2010a: 10. Modified and translated by the author.
SINERGIA’s component: ‘monitoring PND results’ is well known by the three groups of
respondents. In total, 84% of the people that constitute the universe of the evaluation know
this component, while only 35% of them know the ‘targeted evaluations’ one (National
Consultancy Center, 2010: 11 and 16). The greatest unawareness about the evaluation
component was found in the second group [occasional], where only 20% of their respondents
claimed to know about SINERGIA (National Consultancy Center, 2010: 99). The third group
considered that the quality of this component is very low, due to the big gap that exists
between the beginning of the evaluation and the actual presentation and publication of the
results (National Consultancy Center, 2010a: 17).
The results of the study support our claim that SINERGIA’s outputs actually promote a
feedback and learning function within the national government, but this function is very weak
in its reporting, dissemination and integration approach. Both the monitoring, and to a greater
extent, the evaluation components of SINERGIA are reaching, for the most part, only national
executive government agencies which are closer to the system -basically line ministries and
the office of the President. Other agencies from different sectors and with different interests
are being excluded from SINERGIA’s output information. As a consequence, the perception
of the quality of the system decreases the further the agency is from it. In average, the lowest
scores provided by the three groups for the two components, were given to clarity and
relevance [see graphs 2 and 3]. Also, the average higher scores were provided to the utility
and current importance of the system information.
42
29
23
Group 1
Group 2
Group 3
SINERGIA's General Quality
44
Graphs 2 and 3. Percentage of people that scored SINERGIA’s quality in terms of
credibility, utility, clarity, current importance and relevance as excellent or very good
Source: National Consultancy Center, 2010: 103 - 109. Graphs elaborated and translated by the author.
4.1.2.2 Autonomy and Impartiality
Despite the clarity within the system about the differences and complementarities between
monitoring and evaluation, it is possible to recognize some deficiencies that generate gaps in
SINERGIA’s autonomy and impartiality and therefore, this part of the policy dimension of
the Colombian M&E system was assessed as weak.
First of all, there is a loss of balance between the learning and the accountability functions of
the system. Even though the two documents that constitute the M&E plan talk about
accountability, SINERGIA relies more on its monitoring component and therefore on its
learning and feedback functions, as well as on the internal accountability function. The
external accountability purpose of the system, which is the one that ensures a greater amount
of autonomy and impartiality, is the weakest since it is based on the evaluation component of
it. As we have already mentioned before, according to the results of SINERGIA’s evaluation
document presented by the National Consultancy Center (2010: 103 – 109), the monitoring
component of SINERGIA is the best known, and is the one with the highest perception of
quality on average for the three groups. In contrast, the document presents claims for greater
dissemination and publicity of evaluations. This is hampering not only the autonomy and
impartiality of the system but also the credibility of it, as was shown in graphs 2 and 3.
0
10
20
30
40
50
60
GRP 1 GRP2 GRP3
Monitoring PND results - Component
Credibility Utility
Clarity Current importance
Relevance
0
10
20
30
40
50
60
70
GRP 1 GRP2 GRP3
Targeted Evaluation
Credibility Utility
Clarity Current importance
Relevance
45
Second of all, among the documents that constitute the M&E plan, the need for autonomy and
impartiality of the system is not mentioned explicitly, which complicates the analysis of tough
issues. Moreover this certainly holds if we take into account that from both the monitoring
and the evaluation perspectives, the lack of autonomy and impartiality comes from two
sources: 1) an organization-structure issue and 2) a budget issue.
The lack of autonomy and impartiality from the organization-structure point of view is related
with the location of SINERGIA within the DNP. The DNP is the national government think
tank, its back office, and presidency is one of the system’s administrators. As a consequence,
it is not only questionable that the government is monitoring its own performance but also,
and from the evaluation point of view, that the resources to pay for the targeted evaluations in
many cases come from the implementing government agency or sector, although evaluations
are carried out by consultancy firms outside the government.
The academy, NGO’s, the private sector, among others constantly criticized Uribe´s
government for their PND results balance, elaborated with the output information of SIGOB:
“[T]he idea of measuring specific goals and objectives is, in principle, an appropriate focus on
macroeconomic planning. But it may happen that those goals are set in unambitious ways; just to get
good grades in advance ¿Is there an objective assessment by a third party about whether or not this
was the case of the PND 2002 - 2006? An even a more basic question ¿who should monitor and assess
development plans? The most obvious answer is that it should not be the implementing government”
(ANIF, 2007:2)32.
Furthermore, Uribe’s administrations were also criticized by its targeted evaluations, not only
because of the source of the resources to pay for them, but also because the implementing
government agencies or sectors were the ones that sometimes even hired the evaluation, and
they were also part of the ‘Evaluation Monitoring Committee’, which is responsible to discuss
and approve or denial the products delivered by the consultancy firms during the evaluation
processes (DNP 2012:46). As ANIF33 mentioned in august 2008, “the credibility problem
prevails in these ‘self-assessments’. It is still necessary to respond appropriately to the
concern that ANIF has risen time back: ¿independent evaluations with money of whom and
for whom?” (ANIF, 2008:2)34.
It is clear now that the limited inclusion of actors outside the government in the monitoring
process hampers its impartiality and credibility. In the case of the evaluation, the absence of
32 This quote was translated by the author. 33 ANIF is a think tank of the private sector, specialized in economic studies. 34 This quote was translated by the author.
46
an independent budget and the shortage of financial sources is also a reason adding to the lack
of credibility. The “practical dilemma that arises is that evaluation of public policies must be
done (at least partially) with public funds, usually channeled by the approval of the central
government. As a consequence, some governments may not be willing to hire truly
independent evaluators” (ANIF, 2007:2).35
Finally, when it comes to score the alignment of the system with the planning and budgeting
processes, SINERGIA scored partially satisfactory, basically because in the country there is
integration of the M&E results in planning but not in budgeting. At the beginning of this
section we have already described the inclusion of M&E information within the planning
process; now we are going to focus on the difficulties to include M&E information in the
budgeting process.
4.1.2.3 Alignment of Planning and Budgeting
As we have already mentioned before, since 2002 it was evident that Colombia had some
important issues related to the quality of the expenditure, the management of public resources
and the results that those invested resources were generating. This led to the development of
the PRAP, where the M&E system constituted an essential part since it was the appropriate
tool to improve budget allocations. But in Colombia the planning and the budgeting processes
are not linked and it had been very hard for the country to accomplish this. As a consequence,
although information produced by the M&E system has been very useful to develop the
feeding back of the planning processes at the national executive level, this have not been
possible when it comes to the implementation of a performance-based budgeting model.
There are three main issues that have being hampering this process: 1) the preparation of the
budget is divided between the MoF and the DNP, 2) there is a considerable budgetary
rigidness at the national level, and 3) the budget does not have a programmatic structure.
Figure 1 points out the first issue. The General National Budget [GNB] is divided in two main
parts: the investment budget and the current budget; the first one compromises about 15% of
the national budget and it “includes infrastructure and other investments, as well as
investments in human capital (such as education and training)” (Mackay, 2007: 117) and
everything related with social public expenditure. The second one, which comprehends the
rest of the GNB, includes payments to civil servants, the cost of “administrative activities,
35 Ibíd.
47
government debt servicing, pension payments and transfers to sub-national governments”
(Mackay, 2007: 117) among others, to guarantee the functioning of the state.
Figure 1. Colombia’s General National Budget
Source: Castro 2009: 4
Although the DNP is the one in charge of the formulation of the PND for each administration
at national level, it is only responsible for the preparation of the annual investment budget,
while MoF is the one in charge of the current one; which compromises about 85% of the
GNB. As a result, “[t]he budget and planning unit of the Department of National Planning and
the Ministry of Finance used very little M&E information” (Mackay, 2007:36), which
hampers the learning and feedback function of the system. As Castro mentioned, “[t]he MoF
did not consider plan results when preparing the budget, and, without performance
information, the President’s Office could not identify misalignments between government
priorities and annual budget allocations. In 2002, for example, almost 40 percent of the annual
budget did not reflect any government priority in the National Development Plan” (Castro,
2009:6).
This situation, and the lack of coordination and information exchange between DNP and the
MoF also generates problems around the use of M&E information for planning, decision
making and targeting evaluations inside the DNP. “The option of unifying the investment and
recurrent sides of the budget was recently considered within the government, but no
consensus was reached. At present, the structure of the national budget in Colombia
constrains the scope for use of M&E information” (Mackay 2007: 120).
Budgetary rigidness is the second aspect that obstructs the use of M&E information for
budgeting in the country, at least in the short-term. This rigidness in the allocation of the
public resources is mainly due to the constitutionally-mandated transfers from the national
48
government to territorial entities, “and a range of permanent entitlements and revenue
earmarks. As a result, as much as 95% of the budget is earmarked and is thus inflexible in the
short-run” (Mackay 2007: 117) which limits the potential of the monitoring information to
influence the budgeting process.
In the case of the evaluation information it is important to have in mind that they might be
some potential for it to influence the budgeting process in the medium and long term,
especially if there is "clear evidence about government performance in attaining Presidential
Goals and other government priorities”. This was the case of the targeted evaluation
developed about the program ‘Familias en Acción’, where the findings of the impact
evaluation induced Uribe’s government to maintain the program and to increase its capacity
(Mackay 2007: 117). But despite this case, in general, it is possible to say that M&E
information covers only partially government expenditures (Castro, 2009:6).
Finally, the third issue that limits the use of M&E information for the budgeting process is the
lack of a programmatic structure of the budget. Budget documents continue to be presented in
Colombia “on a line-item basis” and not as a performance – based budget36. Part of the
problem is related to the separation and lack of communication between the Public Policy
Evaluation Directorate (DEPP by its Spanish acronym)37 in charge of the formulation of the
PND and of SINERGIA and the Public Investment and Finance Directorate (DIFP),
responsible for the formulation of the investment budget. Even though both directorates are
part of the DNP, “[t]here does not appear to be any relationship between the level on which
SINERGIA focuses ―the Presidential Goals ― and the much more micro, project focus of
the work of DNP in preparing the annual investment budget” (WB, 2007:17). This is very
problematic if we take into account that the PND presented by each President for the approval
of the Congress actually contains the proposed investment for the four – year administration
and “constitute the ‘Programs and Projects Bank’ from which specific investments (and only
those) can be selected for inclusion in subsequent annual budgets” (WB, 2007:17).
Additionally and since “programs have not been rigorously constructed (following log-frame
or similar methodologies) and budget allocations are frequently estimates of the financial
support given to such a program under different budget lines” (WB, 2007:15), it is very
difficult to obtains M&E information about the financial performance of programs.
37 Today it is known as the Public Policy Monitoring and Evaluation Directorate (DSEPP by its Spanish acronym).
49
Furthermore the financial management information system [SIIF] on which the budget is
based, is not very well linked with SIGOB, making “it harder to link government spending on
particular activities to the outputs, outcomes and impacts produced by those activities” (WB
2007: 20).
As a whole, “[t]here is a need to further strengthen the relationship between evaluation
results, and planning and budget preparation” (Burdescu, 2005: 2).
4.1.3 Indicators, Data Collection and Methodology
In general terms indicators, data collection and methodology constitute the M&E dimension
with the best performance. The Methodological Guide for Monitoring the National
Development Plan and Evaluating Strategic Policies (DNP, 2012) established in a very
detailed way what to monitor and what to evaluate, and provided all the specific information
related to indicators, goals and methodologies to do so. Therefore, the Colombian M&E
system scored excellent in the selection of indicators, selection criteria, priority setting,
causality chain, methodologies used and data collection; good in the quality of the indicators,
and weak in the disaggregation of them.
When we were describing the policy dimension of the Colombian M&E system we mentioned
that the documents that constituted the M&E plan specify very clearly what to monitor and
what to evaluate, being SISMEG the one in charge to monitor the PND, while SISDEVAL
applied different evaluation methodologies to strategic policies. In order to accomplish so,
from the formulation of the PND the government must ensure the inclusion of some elements
that would allow and facilitate the monitoring process. The PND must be supported in a
structure that is scalable, with pillars constitute by objectives. At the same time objectives
need to be conformed by strategies and strategies by programs. The advantage of such type of
structure is that monitoring indicators can be introduced for each level of the structure. This
allows “measurement of outputs and outcomes that aggregated will reveal progress towards
those goals established for each level defined”. It will also harmonize the structure of the
PND with the monitoring process (DNP, 2012:19).
In addition, this structure allows the linkage between the different levels of the PND and
therefore, the aggregation of information between different actors from all sectors of
government. Also, under this structure the formulation of pillars and objectives is transversal
50
throughout sectors and is linked with presidential priorities. Strategies are formulated at sector
or ministerial level, while programs are design for each entity (DNP, 2012:20).
In line with this and due to the heterogeneous nature of the information, a pyramidal scheme
[Figure 2] was designed to facilitate the use of the information, responding “to the strategic
needs of the system users” […]. The scheme facilitates the organization of the information
according to the country’s priorities” (DNP, 2012:21). As Figure 2 reveals, each level of the
scheme has its own emphasis and its own instruments: “the top part of the pyramid focuses on
transversal maters, those that need high levels of coordination between sectors”. For this
level, during Uribe’s administrations, objectives across sectors were monitored and
periodically analyzed at Ministers councils.
Figure 2. Monitoring Pyramidal Scheme
Source: Dorado, 2011: 5.
The center of the pyramid contains all the sectoral issues. It monitors the performance of each
sector in relation to its own established outputs and outcomes. Given that this level only
involves those matters that concerned sectoral entities, and that the responsibility over the
information provided relies only on the head of the sector, objectives are analyzed in bilateral
meetings between the president and the corresponding minister. The bottom part of the
pyramid monitors the institutional performance of each entity regarding those priorities
recognized by the government, allowing comparisons between them.
So far it is already possible to identify the existence of indicators for the different levels of the
pyramid. Even though the documents that constitute the M&E plan do not present a list of
Transversal
Sectoral
Administrative
Management
51
indicators, those indicators are available online at SINERGIA’s website38. From one
administration to another, the number of indicators changes. During Uribe´s administrations
the DNP utilized about 500 indicators to monitor PND results. Nowadays, the list has
increased to 1000 indicators approximately, basically due to the inclusion of additional
transversal matters that were not being monitored before, and the addition of control boards
from all sectors [24] to SISMEG between August 2010 and December 2011 (SINERGIA’s
website). But indicators are not only classified by the levels of the pyramid and the structure
of the PND, they are also categorized by the logic of the causality chain.
SINERGIA is founded on a results-based model that not only focuses on inputs, activities,
and outputs but that is also oriented towards outcomes and impacts. Under this model,
monitoring and evaluation processes are articulated with the public policies structure,
specifically the public policy value chain. The value chain is understood as “a logic and
sequential relationship between inputs, activities, outputs, outcomes and impacts” (DNP
2012:14). As a consequence, the monitoring component of the system focuses on providing
short term information about governmental goals, taking into account procedure, output and
outcome indicators. The evaluation part of SINERGIA identifies its effects over the
beneficiary population, based on all the parts of the value chain, generating impact indicators
(DNP 2012:11), as shown in Figure 3. The idea is to produce integral analysis of public
policies and the performance of public management.
Figure 3. Value Chain Structure – First Version
Source: Dorado, 2011: 4.
This structure allows the understanding of the connections that exist between the initial results
and the contribution of each part of the value chain to the midterm and long-term results of
the intervention programs and policies (DNP 2012:22). During Uribe´s administrations the
38 https://sinergia.dnp.gov.co/PortalDNP/default.aspx. Last time consulted: march 2013.
52
INPUTS PROCEDURES OUTPUTS OUTCOMES IMPACTS
value chain was analyzed under a supply [inputs, activities and outputs] and demand
[outcomes and impacts] structure [Figure 3]. Nowadays the value chain is analyzed from a
different perspective, showing transversal relations between the different parts of the chain:
“the M&E system aims to measure specifically efficiency, effectiveness and clarity on public
management” and the accomplishment of PND goals (DNP 2012:14) [Figure 4].
Figure 4. Value Chain Structure – Second Version
Source: DNP 2012:15, own translation.
Thus, for both the monitoring and the evaluation processes, the first step established by
SINERGIA was to recognize and elaborate the value chain, with the aim to understand the
causal sequence of the development interventions that the government considered necessary
to implement, in order to achieve the desired objectives (DNP 2012:25). After the
construction of the value chain and specifically for the evaluation processes, it was also
necessary to create a delivery chain, in order to provide a better understanding of the delivery
route of goods and services implemented by the policy evaluated. The aim was to identify
those bottlenecks and critical points of the process that need to be part of the evaluation (DNP
2012a:27). Finally, these two chains were taken into account to define the assumptions, scope
and type of evaluation that was going to be done, accordingly to the parts of the process
[value chain] where the majority of the problems were concentrated (DNP 2012a:11).
At this point we have already introduced some aspects of SINERGIA’s M&E methodologies,
which are extremely well identified, and detailed step by step in the M&E plan documents, as
well as mutually integrated (DNP 2012: 6-18 and 19 -54). The coordination between them is
Clarity
Cost - Effectiveness
Efficacy
POLICY
OBJECTIVE
S
Effectiveness
Produced by SINERGIA Produceed by other actors
Efficiency
53
evident: the self-monitoring approach provides short-term and timely information about “how
are we performing”; information that is articulated with the evaluation process, which seeks to
provide deepen information about the causality of the actions developed within the policy
frame, and their impact over the beneficiary population (DNP 2012: 6). As a result, the
monitoring process indicates what to evaluate, and the evaluation results enables SINERGIA
to know where to reinforce the monitoring process (Dorado, 2011: 4).
Due to the value chain structure mentioned before, SINERGIA’s monitoring methodology is
based on procedure, output and outcome indicators, baselines and goals. Therefore, it is
important to analyze for this part of the document the quality of indicators, its disaggregation,
selection criteria and priority setting, as well as the sources of data collection. In comparison,
evaluation methodologies include some other aspects related to the policy cycle and the rest
of the components of the value chain, in order to determine the type of evaluation that must be
implemented, as we will explain in the next paragraphs.
Let’s start with indicators. SINERGIA uses three types of indicators, in order to “achieve a
better organization of the way in which prioritized public policy performance is presented”
(DNP 2012: 32): tracer indicators, sub-indicators and proxy indicators. The first type of them
reflects the objective of the policy as a whole, showing progress towards its goal. When these
indicators are too wide it is necessary to formulate sub-indicators, which complement and
explain tracer ones. Proxy indicators are substitutes of tracer ones, when information about
them is not availed in the short term (DNP 2012: 32).
In addition, there are two parameters that must be followed for the formulation of indicators
to guarantee their quality: 1) Indicators need to be CREAMS: the three types of monitoring
indicators [performance, output, and outcome] are formulated following the ‘CREAM’
criteria, with a slight change: indicators not only need to be clear, relevant, economic,
adequate and monitorable; they also need to be sensible, meaning that they need to provide
short term information (DNP 2012:22) Also, for the formulation of policies and programs,
goals need to be SMART: specific, measurable, accessible, realistic and time-bound (DNP
2012:21).
As a consequence, and due to the value chain structure, SINERGIA not only has good quality
indicators but also baselines and targets defined. It also has: an indicators’ selection criterion,
a definition of actors involved in the process, and priorities to limit the amount of indicators.
54
As we will see in Figure 5, there are some previous steps that need to be completed before the
implementation of the system, in order to guarantee its quality. The formulation of indicators
derived from the value chain, and the establishment of baselines and targets, and the
methodology to aggregate indicators’ results over time [accumulation] [steps 3 and 4]
constitute a technical exercise developed by DNP and presidency (DNP 2012:23-27).
To guarantee an objective selection of indicators, with priorities and limits in the number of
them and the actors involved, sectoral Strategic Plans are taken as the key input documents
for the selection of indicators (step 1 – Figure 5). These documents contain all Government
Action Programs (PAG) with objectives, strategies, activities, indicators and targets, as well
as those indicators that where prioritized under the National Development Plan (DNP
2012:23). PAG also specify the importance of defining roles, responsibilities and key actors
for the monitoring process; especially in relation with the data collection and the political and
technical coordination of the system and the quality of indicators.
Figure 5. Monitoring Previous Steps
Source: DNP 2012:23, own translation
Figure 5 shows that after steps 3 and 4 are applied, consultation processes of the selected
indicators and targets are implemented within those sectoral actors involved in the policy
implementation. The aim of these processes is that indicators and targets for each year and for
the four year period are legitimized by each sector ministry. Finally, the definition of roles
among the system and the creation of indicators data sheets are in place.
Within the monitoring methodology and in relation with the quality of indicators, it is
important to mention that in general, indicators are not disaggregated by region, sex or socio-
economic status, according to the information of the M&E plan documents, and the list of
indicators available at the website.
Value
Chain
Sectoral
strategic
Plans
Indicators
Formulation
Indicators
accumulation,
baselines and
targets
Consultation Indicators
data sheets
Roles’
Definition
1 2 3 4 5 6 7
55
On the basis of our experience and in relation to the first one, the absence of a regionalization
of indicators is due to the nonexistence of a territorial intervention strategy. Colombia is a
decentralized country where the national level cannot ensure/enforce outcomes in the
territories or intervene in the way in which they spend their own resources. This also
generates competing interests between levels, corporations and political affiliations,
obstructing the connection of actions and goals between the national level and the territories.
In addition, the lack of territorial information at the national level limits its knowledge about
the real problems and situation of the territories, blockading the formulation of indicators
disaggregated by regions. As Castro mentions, “[w]hile there were efforts to monitor the other
levels of government from the center, this created problems in a decentralized country such as
Colombia. Thus we had not clearly defined the different linkages, tasks and responsibilities of
the system, which up until 2002 created complex institutional problems.” (Castro, 2006:30).
In relation with indicators’ disaggregation by sex and/or social-economic status, it is possible
to claim that since the aim of the system is to monitor the performance of national
government priorities, indicators are formulated from a much more macro dimension and
therefore, information is not disaggregated to this level of detail. The only indicators that refer
to specific groups of the population (women, age groups, displacement people etc.) are those
related to targeted policies, like those formulated to beneficiate armed conflict victims, or
early childhood, childhood, adolescence and youth, or to reduce child labor or benefit the
elderly.
There is also an extremely well defined evaluation methodology with 5 standardized phases,
as Figure 6 shows: the selection of the policy to be evaluated [portfolio], the evaluation
design, and the adjudication of the evaluation, its execution and the use of the results.
Within the DNP, evaluations are understood as an “objective and systematic investigation,
applied to one of the different parts of the value chain, with the purpose to improve the
design, implementation, execution and impacts of a plan, policy, program or project” (DNP
2012:39). The idea is to complete the policy cycle and to “provide pertinent inputs for
formulation of government programmes.[…] The information produced is more detailed than
that emerging from monitoring indicators, which enables the evaluation to provide
recommendations oriented to improving the program and designing more efficient
government interventions (UNDP, 2009: 51). Evaluations, to be effective, need to “include
56
the participation of all stakeholders, answer to decision makers’ questions, and respond to
government agenda” (Dorado, 2011: 9).
Figure 6. Phases for the Evaluation of Strategic Policies
Source: Dorado, 2011: 9
SINERGIA identifies a different type of evaluation for each part of the value chain, in order
to rigorously “study the causalities derived from the […] intervention”, depending on the
stage of the plan, policy, program or project, and the “type of questions the policymaker
wants to assess” (UNDP, 2009: 51). Consequently, institutional evaluations take into account
processes and outputs; executive evaluations consider inputs, processes and results;
operational evaluations include inputs and processes; results evaluations study outputs and
results, and impact evaluations embrace results and the impact of the intervention, according
to the value chain (UNDP, 2009: 50).
When it comes to assess the way in which policies are selected as being strategic for
evaluation, it is important to mention that the consolidation of the SINERGIA’s evaluation
portfolio denotes a participative concentration process between the DSEPP and the rest of the
technical divisions of the DNP. The definition of policies is based on their level of importance
with respect to the achievement of the country’s strategic objectives (DNP, 2012: 40).
Also, and since “[t]here is an implicit debate on how to prioritize evaluations – to focus on
‘problem’ programs, pilot projects, high-expenditure or high-visibility programs, or on
systematic research to respond to questions of program effectiveness” (Burdescu, 2005: 4),
SINERGIA’s evaluation methodological guide mentions that the decision to evaluate a
specific policy, program or project relies on the design of the policy, the requirement to justify
External
consultancy
firms
57
spending, the need to determine its effectiveness, the necessity to improve the delivery of
goods and services and/or the option to extend or replicate it (DNP, 2012a: 13).
In addition, CONPES document 3294 mentions the size and characteristics of the affected
population, the importance of the policy or program for the sector and its innovative character
(CONPES-DNP, 2004: 15).
4.1.4 The Importance of the System’s Organization and its Capacity Building
The organizational dimension of the system, related to its structural components, is very well
arranged. In general terms, SINERGIA has an appropriated structure for coordinating,
supporting, overseeing, analyzing and feeding back information into the policy cycle; which
has a lot to do with its location at a central agency, the DNP, Colombia´s national government
think tank. Furthermore, the demand for M&E information comes from Presidency, being the
President himself and the Director of the DNP, i.e. the “champions” of the system. As Figure
7 shows, SINERGIA works very closely with presidency, line ministries and department
directors. For the monitoring process, there are three types of related actors: the sources of
information, basically planning offices at sector level entities; the administrators of the
information constituted by the DNP and Presidency; and the users of the information, i.e.
government, congress and civil society39. The first group has a continuous and stable relation
with the system. The coordination role between the sector entities and the system, to
guarantee the provision of coherent and updated information, is the responsibility of the head
of the sector.
39 CONPES document 3294 includes among the users of the M&E information control entities or oversight agencies for the
accountability process (CONPES, 2004: 16). Also, the published version of the monitoring methodological guide includes
within the organizational scheme of the monitoring system, oversight entities among the users of the information. As we will
see later on, in practice oversight agencies are not very much related with the M&E system.
58
Figure 7. SINERGIA: Roles and Responsibilities
Source: WB, 2007:6
Among the second group there are two types of administrators: the Presidency symbolizes the
political one, and its role is to set the country´s priorities for monitoring. DNP is the technical,
operative and technological one, and its task is to ensure consistency in information gathering
and analysis. The DSEPP of the DNP coordinates the monitoring process at two levels: from
the setting and articulation of the system to guarantee a successful monitoring process, to the
revision of the different results derived from it, in order to check their quality. Figure 5 shows
the steps previous to the monitoring process, while Figure 7 shows the steps related to the
monitoring of the PND.
Figure 7. Steps to Monitor de PND
Source: DNP, 2012:28, own translation
From the steps previous to the monitoring process, the definition of roles is very important for
the coordination and oversight functions within the system. During this step, the following
roles are defined: 1) Sectoral coordinator: this function is performed by the planning chief of
the entity head the sector, in charge of checking the data uploaded to the system by the rest of
1 2 3 4 5
Information
registration and
uploading
Monitoring
Routines
Complementary
Information
Overseeing and
adjustments
Link with
evaluation
59
the sector entities. 2) Program administrators: in charge to update the progress of those
programs that are being monitored. 3) Target administrator: updates each month the
information related to the progress and changes of the indicators associated to a specific
target. This role needs to be coherent with the indicator’s periodicity (DNP, 2012:26-27).
From the monitoring steps, the information registration and uploading is fundamental for the
functioning of the system. The monitoring group of the DSEPP is constituted by
‘sectorialistas’. Their role is to articulate the monitoring process between all the different
sectors and the presidency. They are also responsible for reviewing the information that is
being updated to the system. Another of their roles is to train program and target
administrators as well as sectorial coordinators, about the way in which monitoring process
works (DNP, 2012:28).
The establishment of routines is also very important for an adequate M&E institutional
structure. Routines determine the rules of the game for the program and target administrators,
in relation with their information updating role. Data update must be done each month, so that
it is possible to identify early warnings and to take corrective actions on time. ‘Sectorialistas’
also have routines for checking and approving information which is uploaded in the system.
The third group of actors, the users of the information, is constituted by the high level
government, the congress and the civil society. The first one of them uses the M&E
information in control boards for: 1) ‘Management dialogues’: sectoral meetings led by the
President and the corresponding sector Minister where sector strategies, goals, indicators and
their progress are examined; 2) ‘Workshops on good governance’: involving high level
government reviewing strategies, progress and challenges found on each sector and in
transversal matters prioritized; 3) ‘Bilateral dialogues’: between the President and each
Minister, aimed at reviewing in more detail sector strategies, goals and challenges; and 4)
‘Transversal meetings’: gathering all the different actors related to transversal matters in order
to discuss joint strategies, targeted population, action fields, activities and resources to move
towards transversal objectives (DNP, 2012: 34-35).
The DSEPP generates for the rest of the users of the information a set of monitoring products,
summarized in Box 2. Also, when implementing evidence based evaluations, DSEPP
incentivizes the participation of stakeholder from the first phase of the evaluation process and
onward, as Box 3 shows.
60
Boxes 2 & 3: Monitoring Products & Stakeholders Participation in Evaluations
But in spite of this, those stakeholders that are actually involved in the evaluation processes
are very few. Normally, the ones invited to participate are public policy formulators and
program executers, while stakeholders from other governmental agencies and from other
sectors are normally excluded. As a proof, the results of the study developed by the National
Consultancy Center in 2010 for DNP show that only 35% of the respondents knew about the
evaluation component of SINERGIA: “the greatest unawareness is found among the
“Ocasionales” group, where only 20% know this component (National Consultancy Center,
2010:15).
Also, when it comes to analyze the use of the monitoring information by different
stakeholders, it is possible to state that it is very low, basically because there is still a “lack of
basic knowledge of performance management issues among congressional members and the
media”, as well as among the general public. Also, “[l]ow stakeholder use of the information
is too strongly related to the fact that several key SINERGIA clients do not trust the content
of the report because they are prepared within the National Planning Department (where the
SINERGIA technical unit is located), and therefore the assumption exists that the report
cannot openly be [sic] critical of the government” (Castro, 2009:21).
Monitoring Products: 1) An
informative annual presidential report
to the Congress, which contains
progress towards goals set in the NDP;
2) A perception survey based on
citizens’ reactions about governmental
policies and programs and the way in
which they have affected positively or
negatively the citizenry; and 3) Online
performance information about the
evolution of different indicators. This
information contains: the sector and
entity responsible; the NDP pillar,
objective and program; the
quadrennial advance; the historical
evolution; baselines; the updating date;
some indicator related information; the
periodic evolution of the indicator, its
qualitative analysis and its data sheet.
This information is updated monthly.
(DNP, 2012: 32-34)
Stakeholders’ participation in
evaluation process: They participate
in the construction of the value and
delivery chains, the identification of
problems and information needs
associated within those chains, the
definition of the evaluation scope, and
the constitution of the “Technical
Evaluation Monitoring Committee”,
responsible for the discussing and
checking those evaluation products
delivered by the consultancy firm
during the execution of the
evaluation. Stakeholders’
participation is strategic for the
dissemination of evaluation
information among governmental
agencies, while it legitimates the
evaluation process and results (DNP,
2012: 42 - 51).
61
Other flaws related with the organizational dimension of the system have to do with the
absence of linkages at four levels: with the statistical office, with territorial levels, with other
directorates within DNP, and with projects.
The relationship between SINERGIA and the National Statistics Administration Department
[DANE by its Spanish acronym] is limited to the link that exists between the system and any
other head of sector as a source of the system’s information and therefore, DANE does not
have any role within the M&E system. It is true that DSEPP “was asked to prepare a national
information strategy […] [that aims to establish] better institutional arrangements for
improving coordination and harmonization of data collection and use at all levels, but
particularly among line ministries, the National Statistics Agency (DANE), the MOF, the
DNP, and the President’s Office (Castro, 2009:31).
However, according to the study developed by the National Consultancy Center (2010:14),
dialogue between SINERGIA and DANE is minimal, and there also exists different
approaches for the indicators required by the two entities. Many indicators utilized by them
are “complementary and sometimes even redundant; today indicators are difficult to link and
in many cases entities are not conscious of that” (National Consultancy Center, 2010:34).
This problem is not only found in relation with DANE’s indicators but also with the ones
utilized by the General Comptroller’s Office. Therefore, there most by a linkage process
between these three entities, in order to align indicators and methodologies for data collection.
Furthermore, according to MacKay (2007), this process must provide DANE “an important
role in ensuring the quality of the information used by SINERGIA (in National Consultancy
Center, 2010:43).
The relationship between the M&E system and the subnational entities is also week. One of
the main flaws of SINERGIA has been its lack of capacity to apply M&E models at regional
and local levels, mainly because they have not promoted an M&E culture within subnational
entities (National Consultancy Center, 2010:39). They have neither found the way to capture
local and regional data of public policy performance, mainly due to the decentralization
process that provided subnational entities with territorial autonomy in planning and public
spending. Therefore, even though since the amount of public transfers from the national to the
regional and local governments increased, the structure and the size of the national
government was not very well adjusted. The has lead to a lack of control of the national
62
government over those resources, losing at the same time the relation between the three levels
of government and their contribution in the achievements of national goals (DNP, 2010:61)
CONPES document 3294 mentions that the “monitoring of decentralized and territorial
entities will be oriented towards determining the situation of business, departments and
municipalities, in relation with their financial, fiscal and management goals, […] including
the fulfillment of law and development plans” (CONPES-DNP, 2004: 14). Along the same
line, SINERGIA’s methodological guide established that the document “was designed to
serve as input for those territorial entities wishing to adapt the successfully applied national
model to the municipal or departmental level” (DNP, 2012: 5). Since 2012, DSEPP has
developed a strategy for SINERGIA territorial intervention, constituted by two components:
subnational level technical assistance which aims to bring the M&E culture to the territories,
in order to allow them to monitor their development plans’ goals; and the territorial
monitoring, which implies the territorialization of sector information. The main goal of the
strategy is to generate a real linkage between SINERGIA and the subnational levels, and it is
projected to be done by 2015.
SINERGIA has found difficult to relate and link up with other territorial M&E activities
promote by different directorates within DNP, like the “Directorate for Sustainable Territorial
Development (DDTS), which is responsible for monitoring and evaluation of the work of sub-
national governments” (WB, 2007:18), and with the Directorate for Investment and Public
Finance (DIFP). The subnational level work of these three directorates has not been linked,
hampering not only the connection between the planning and budgeting processes, but also
the way in which the M&E technical assistance is brought to the territories. Finally,
SINERGIA is also separated from donors´ M&E mechanisms. Assessment covenants
incorporated within credit operations between the country and multilateral organizations are
taken into account when selecting those strategic policies that are going to be evaluated (DNP,
2012:40).
So far we have already shown the majority of the capacity strengths and weaknesses of the
Colombian M&E system. A DNP and WB document (2010:28, 29) resumes the main
challenges of the system, which we consider important to highlight:
63
- The need to develop new strategies and products that link financial sustainability,
ownership of analysis methodologies; and an effective regional perspective for the
implementation of M&E processes.
- The need to develop evaluation capacity in the public sector from the national
academic community and though knowledge transfer, in order to reduce the cost of
impact evaluations and to universalize the use of evaluation processes within public
entities.
- The need to incorporate the concept of accountability as a permanent attitude in the
management of public administration.
- The need to design evaluations from the very formulation of public policy,
diversifying the agenda of evaluations to include sectors that traditionally had never
participated.
- The necessity to promote policy M&E systems at regional and local governments.
- The importance of creating routines and relevant indicators for monitoring the PND in
a way that adequate budgetary aspects are linked with long-term development
perspectives.
Additionally, NPD 2011 mentions the necessity to improve the measurement of citizen`s
perception of the government performance and the promotion of citizen`s participation.
Nowadays, SINERGIA is working towards the remediation of weaknesses from three
perspectives. Firstly, they are developing SINERGIA territorial, a strategy that aims to insert
a regional perspective and to promote M&E systems at subnational level. This year
SINERGIA territorial is consolidating the work done last year with 51 municipalities and it’s
expanding its work to 12 departments and to 33 additional municipalities. Secondly, they
have hired a group of people to develop a proposal to link budgeting and planning through the
PND monitoring process.
Thirdly, every year DSEPP works to improve one of its monitoring products: the citizen´s
perception survey, by adapting the methodology used to correct flaws. This social
accountability mechanism is considered so strong that the participation citizens among the
M&E system is not considered as an important aspect for its accountability function.
Perception surveys and online information seems to be enough to promote and guarantee
social control. Fourthly, the accountability concept was introduced within the M&E context
in 2004 by CONPES document 3294, under the notion that public policy design and
64
implementation should be feedbacked by M&E information diffusion mechanisms, to provide
government institutions with incentives to continuously improve their performance, as well as
the citizenry with elements to execute social control (CONPES-DNP, 2004:16). But it was
just until 2010 when a new CONPES document was issued, the 3654, which contains the
accountability public policy of the executive branch to the citizen.
Finally, SINERGIA still has trouble to secure independent budget for its three main activities:
monitoring the PND, evaluating strategic policies and SINERGIA territorial. SINERGIA is
also having trouble in generating evaluation capacity, among DSEPP and any other
governmental institution; even thought in CONPES document 3294 this last aspect was
highlighted as a policy directive (CONPES-DNP, 2004: 13, 24).
4.2 Low Participation of Actors outside the Government and the External
Accountability Function
Participation of actors outside the government has a lot to do with the external accountability
function of the system. We have already mentioned before that SINERGIA is highly decision-
making oriented and therefore, it emphasizes more its learning and its internal feedback and
accountability functions, than its external accountability function. We have also highlighted
before some of the system’ limitations with regard to its organizational links with different
actors and levels of government. Therefore, in this part of the text we will concentrate only on
those additional aspects that are important to point out about the participation of actors
outside the government, in relation with SINERGIA’s external accountability function.
Since 2004, accountability became a pillar of SINERGIA, and it was understood as “public
management information generation and flow, with the aim to promote transparency and
control of public administration” (DNP, 2010: 25). Under this frame, M&E information
should be directed towards informing citizens, high level government, congress, control
agencies and sector stakeholders. Some of these actors need to be part of the system like
congress, civil society and donors; while others, like oversight agencies and [even though is
not listed] the judiciary, should not be part but users of the M&E information, since they must
guarantee their independence from the system. All of them together represent different
accountability perspectives.
As elaborated upon in the previous sections, the president is the champion of the system, the
political administrator of SINERGIA and the main user of the information. In line with this
65
and responding to the political control of Congress, one of SINERGIA’s monitoring products
is the presidential report to congress that contains the results of government interventions
between August and May of each year. Apart from this, Congress does not have any role
within the M&E system, this actor is not included in any of SINERGIA’s working groups,
and there is no feedback from Congress on the presidential report.
Therefore it is possible to say, regarding electoral accountability, that the M&E system
provides some information to democratically legitimized actors (Congress and voters) about
the conduct and actions of government bodies. There are accountability forums with limited
participation and debate from democratic legitimized actors, since they cannot rely on having
an adequate information position. Democratically legitimized actors vaguely use the M&E
information to hold government accountable, and the accountability arrangements indirectly
provide very few information about the propriety and effectiveness of the conduct and action
of government bodies. Administrative bodies have few incentives to engage in proactive and
sincere account giving, and to commit themselves to the agendas of their democratically
elected principals.
We have mentioned before that donors are taken into account for M&E activities only if there
are any established M and/or E obligations with multilateral agencies, in the context of credit
operations or international commitments, which allow the participation of this type of actors
within SINERGIA. Low participation of donors and international community in the M&E
system is also sustained by CONPES document 3654, when describing Colombia´s external
accountability. According to the document, this type of accountability is valid only when
states establish international commitments that institute accountability mechanisms towards
external organisms; which is the case of Millennium Development Goals, for example
(CONPES-DNP, 2010:22,23).
The document argues that even though external accountability is exercised by the Colombian
government, it is not clear if it is actually articulated under a unified standard notion; “[i]n
each case, responsible entities determine specific actions to render account to different forums
in relation to specific commitments, but there is not a defined structure about the country’s
position over this matter, that is systematically brought to different international scenarios”
(CONPES-DNP, 2010: 33,34). Therefore, it is possible to claim that upward accountability is
not articulated with SINERGIA’s M&E outputs. This may be due to the little participatory
space provided to donors within the M&E system, or to the very few linkages that exist
66
between M&E outputs and donors’ informational needs. But also, this can be a consequence
of the low donor-donor harmonization when it comes to demand government performance
information.
From the social control point of view, the citizenry participates much less than what CONPES
document 3294 and the accountability public policy from the executive branch to the citizens,
have set. In general terms, the accountability public policy document asserts that from the
wider accountability definition, it focuses on social accountability between the state and the
citizens, in order to create an accountability culture, to coordinate all accountability efforts
from different public entities, and to incentive an accountability attitude among public
servants and citizens (CONPES-DNP 2010:2). Certainly, the priority of the policy is to spread
M&E results for accountability among the citizenry, and the idea is to promote interaction
between government and the community, reinforcing transparency principles in public
administration (CONPES-DNP, 2004:16). Within this frame, accountability is understood as
both, citizens’ rights and public servants and entities’ obligation (CONPES-DNP 2010:7).
SINERGIA has a lot to do with this action. As the principal of many existing public sector
information systems40, SINERGIA is supposed not only to constitute a mechanism to deliver
information about the results of monitoring and evaluation of governmental actions and the
PND; it is also supposed to relate with other social control mechanisms41 established by the
constitution and the law, as their complement and their source of input information. To
accomplish this, the “government, through the National Planning Department” was supposed
to “seek partnerships with private sector and civil society organizations, seeking to ensure the
transparency, credibility and sustainability that a scheme of this nature requires” (CONPES-
DNP, 2004:17).
Despite of this, in reality the role of civil society within the M&E system is not very well
recognized or institutionalized. Experience shows that its participation is rather ad-hoc
without clear procedures. It is also limited to some very specific sector working groups,
involving few private actors and/or Civil Society Organizations [CSO] within the policy
evaluation processes, although the greater contribution comes from state stakeholders. In
addition, and even though the M&E information is available online, it is not displayed in a
comprehensive way, that could fit different or any type of public. Also, and albeit that some
40 To find the list of public sector information systems see CONPES 2010:27. 41 See CONPES 2010:70-72 for other social control mechanisms.
67
activities and actions are developed in order to socialize government results information to the
citizenry, in most cases participants are specialized private or academic public or state
stakeholders.
Likewise and even though perception surveys developed by DNP, based on citizens’
reactions about governmental policies and programs, probably constitute the most ‘sociable’
SINERGIA product, its publication and diffusion is at the discretion of the president, which of
course suppresses the ‘punitive’ power of the social accountability tool. Therefore, social
accountability and participation of citizens and CSOs with SINERGIA still constitutes an
important flaw of the system.
This limited participation is still a consequence of those problems highlighted in the
diagnostic of CONPES document 3654:
1) Information provided by public entities to citizens is limited, insufficient and/or
incomplete; it is also inappropriate [not understandable and far from citizens informational
needs], outdated and not delivered on time; it is also not differentiated towards specific
audiences, and is difficult to find.
2) There are limited accountability forums between citizens and the executive power, being
the principal ones Uribe’s ‘Consejos Comunales’ [Communal Councils]42 and the executive
branch Public Audiences. On the one hand, community councils constituted a unified
methodology, with the participation of attendees and a monitoring mechanism for the
commitments established between government and citizens. This accountability forum
constituted not only a social accountability scenario, but an internal and territorial one too
(CONPES, 2010:42). On the other hand, Public Audiences forums helped accountability to
citizens to become a goal within public administration, which had not existed before. But
despite of this, both types of accountability forums have difficulties in their implementation.
In general terms, both mechanisms end up being a unidirectional exercise instead of a
bidirectional one. Also, their attendance percentage is very low; “the total number of
participants is less than 2% of the adult population, [...] including participation by
teleconference and videoconference (CONPES-DNP, 2010:43); and the second type of
accountability forums “are perceived more as governmental propaganda mechanisms than as
42 Which in Santos administration became ‘Acuerdos para la Properidad Social’ [Arrangements for Social Posterity].
68
effective public management explanation exercises, in compliance with the principles of
participatory democracy” (CONPES-DNP, 2010:43).
Finally, the diagnostic of the accountability public policy from the executive branch to the
citizens mentions that problems found in the operation of accountability forums are due to the
difficulty of the explanations provided to the citizens, which are not comprehensible for all
types of publics. Accountability forums are not very pedagogic, government provides biased
or incomplete information and provides very few opportunities for citizens to provide
feedback and to discuss. Finally, calls for accountability forums are limited in advertising and
public encourage or invited to attend (CONPES-DNP, 2010:44, 45).
For both types of accountability forums, M&E information constitutes an important input, but
this two figures represent what Bovens (2007:7,8), Mainwating (2005:10) and many other
authors have call the “answerability” dimension of accountability, that is, the obligation to
provide information about government performance, to give explanations and justifications
about government decisions and actions related to the public domain, and “to respond to
probing questions, or conversely, the right to ask such questions” (Schedler in Dodson and
Jackson, 2005:230); leaving aside the enforceability dimension of the concept. Actually and
according to this notion, in Colombia answerability is even not highly developed, as it does
not promotes the right to ask questions and the obligation to respond to them.
This is related with the third problem highlighted by the diagnostic: 3) Insufficient incentives
for both citizens to ask for accountability, and for public entities to be accountable, generating
that social actors don’t know, don’t utilize and/or don’t trust accountability mechanisms. This
undermines their capacity to actually influence public action (CONPES-DNP, 2010: 35). In
relation with the latter, the document establishes difficulties connected with gaps in the
regulatory and institutional frameworks for social accountability in Colombia; which
encompasses: lack of public entities’ obligations in relation with the supply of information to
the public about public management results, the application of participatory governance and
the implementation of citizens’ participation spaces. Also, the lack of specific tools to
empower citizens and their organizations in their relationship with public servants and
governmental institutions is also mentioned (CONPES-DNP, 2010: 46).
Another aspect that limits incentives for public entities to be accountable is their inadequate
recognition of the scope and usefulness of social accountability. This is mainly due to their
69
fear for possible sanctions derived from exposure to critics, and the political convenience of
delivering complete performance information to citizenry, leaving aside the public character
of information and their right to be informed (CONPES-DNP, 2010: 47). Finally, public
institutional designs also hamper the proper utilization of resources for social accountability,
due to their “lack of inter administrative coordination systems and the establishment of clear
persons responsible for managing social accountability within public entities” (CONPES,
2010: 47, 48).
From the citizens’ point of view, their limited incentives to demand and promote social
accountability procedures and scenarios within government institutions, come from the low
recognition within public entities of their right to exercise social control. The reason for this is
that in Colombia social accountability has a negative connotation among public sector entities
and servants (CONPES-DNP, 2010: 48, 49). Citizens’ unawareness and unfamiliarity with
social control mechanisms, as well as the insufficient support provided to their accountability
requests have generated negative scenarios. All problems described above, which are related
to the implementation of social accountability, demonstrate unequal access to information.
They also reveal neglect of the right of social actors to information, and unequal power
relation between the accountability actor and the forum; and of course, the “lack of legitimacy
of CSOs’ engagement in M&E activities” (Gildemyn, 2011:19). Rules and laws alone cannot
guarantee effective information and accountability processes; “it requires both, political will
from government and servants, and the motivation and determination of citizens and civil
society to act claiming their rights” (CONPES-DNP, 2004:17).
In general terms it is possible to say that the national M&E system provides some information
to promote social control and participation. Accountability arrangements offer few incentives
to agents to commit themselves to social forums of control and participation within the public
policy cycle. M&E institutional arrangements hardly stimulate mechanisms to compare and
validate results information for accountability proposes, or evaluation capacities to promote
social accountability.
4.2.1 An Underutilized System: Assessing Intrastate (horizontal) Accountability
“My interest in horizontal accountability stems from its absence. Many countries, in Latin
America and elsewhere, have recently become […] "polyarchies," satisfying the criteria of fair
and free political competition […] but have weak or intermittent horizontal accountability. This
description fits almost every Latin American case except Costa Rica, Uruguay, and (perhaps)
70
Chile, and includes such long-established polyarchies as Colombia and Venezuela” (O’Donnell,
1998:112).
From the documents checked and from experience, it is possible to say that in Colombia the
use of the M&E outputs is very limited to the central level of government. Results of M&E
are a fundamental instrument of policy-making, policy-influencing and advocacy at central
level. M&E outcomes, dissemination strategies and institutional arrangements stimulate
greatly internal reflection, as well as the institutionalization and dissemination of lessons
learned. But, on the downside, the demand component of the system is still very weak, and
SINERGIA does not promote the use of the system’s information by donors, non-state
stakeholders, or even by state actors that are outside the executive and legislative branches of
government. The same goes for the local level; results of M&E activities are not used for
internal decision making and accountability purposes, they do not constitute an instrument of
policy-making at the local level. Some efforts are developed to reach citizen´s, as it was
described above; and even though it is true that there are social accountability tools and
forums somehow recognized, effective social accountability only comes when it is linked with
intrastate accountability, generating diagonal accountability scenarios; and this normally
happens outside the M&E system, driven by CSOs independent monitoring activities.
Before going deeper into this argument, it is important to analyze the last aspect of the M&E
assessment checklist: the effective use of M&E for intrastate accountability (oversight
agencies and the judiciary). According to CONPES document 3294 (2004:17,18), the users of
the M&E system are, from the political control point of view, the president and congress; and
from the social control perspective, congress, oversight agencies and citizenry. These actors
are also recognized as the users of the information within SINERGIA’s Methodological
Guide (SINERGIA, 2012:20), but links between the system and control agencies are justified
only by the availability of the M&E outcomes online, “facilitating the verification of progress
towards PND goals, […] [meeting] the principles of transparency and accountability”
(SINERGIA, 2012:21); and this is what describes more closely SINERGIA’s perception
about horizontal external accountability.
With respect to the same aspect, a study developed by the National Consultancy Center shows
that 44% of the respondents do not know the dissemination and accountability component of
the M&E system. 10% of the ones who know it consider that its quality is very poor (National
Consultancy Center, 2010: 19), and that the information provide by it “tends to exaggerate
71
government’s performance”, mainly because “evaluation indicators are produced by the same
agency that is being evaluate” (National Consultancy Center, 2010a: 14, 30). The lack of
credibility about SINERGIA’s information hampers its external accountability function. The
NCC claims that both ‘close’ and ‘external’ actors of the system strongly question its
credibility, due to its ‘judge and duty’ situation, “affecting the degree of credibility of the
information from potential users outside the Executive” (NCC, 2010a:33). Furthermore, and
as it was mentioned before, indicators and M&E processes are not linked between SINERGIA
and the General Comptroller’s Office, generating duplicity and disarticulation of
accountability information.
The other intrastate accountability actor is the judiciary, which it is not even taken into
account within SINERGIA’s users of the information. This probably has to do with the
approach adopted by the accountability component of the M&E system, and by the
accountability public policy from the executive branch to the citizens. In relation with the first
one, CONPES document 3294 states that the dissemination and the accountability component
of the M&E system aims to generate “regular results information dissemination to promote
interaction between the executive branch of government, other branches of government, and
the community; materializing transparency principles within public management. This
approach differs from the type of accountability exercised by legal entities, which seeks to
determine compliance with laws, procedures and standards, and its approach is to find guilty”
(CONPES-DNP, 2004: 16, 17).
In relation with the second one, the public policy theoretical framework recognizes the two
components of accountability: answerability and enforceability, and for the purposes of the
policy document the former is understood as information [accessibility and dissemination]
and dialog [explanations about government decisions]; while the latter refers to positive and
negative incentives [the existence of mechanisms to correct inappropriate actions and
encourage adequate ones] (CONPES-DNP 2010:14-16).
When the document describes the different types of accountability, it divides horizontal
accountability as ‘balanced accountability’ and ‘assigned accountability’; the former being “a
set of structures, practices and outcomes by which public branches of power -executive,
legislative and judicial- report, explain and reward or face sanctions to each other. [...] More
than strict accountability procedures, it recalls the creation of rules and procedures that allow
each power of government to react and implement corrective mechanisms, when other powers
72
interfere in its competence sphere" (CONPES-DNP 2010:20). The latter refers to “agencies
specifically created to control performance and conduct of the three branches of power. [...]
These agencies usually focus on the conducts that imply the violation of rules [...] in
disciplinary, corruption, human rights, social, economic, environmental and cultural,
prosecution and criminal matters” (CONPES-DNP 2010:20).
In Colombia, horizontal accountability is limited to the creation of standards, laws and
procedures for each branch of government to react and to apply corrective actions when other
powers interfere in their area of expertise. It also focuses on those conducts that imply
violation of rules. As a consequence, the notion of horizontal accountability adopted in the
country leaves aside for both ‘balanced accountability’ and ‘assigned accountability’, the
practices and forums that require the provision of information and explanations
[answerability] about government actions between entities from all branches of government;
as well as the application of sanction mechanisms [enforceability].
The policy document mentions the separation of powers that provides congress with the
political control of government, the judiciary with the legal and the constitutional43 control
over all public sector actions, and oversight agencies with a general control over all public
servants and with sanction mechanisms. But apart from government reports to congress;
judicial and oversight agencies’ accountability mechanisms are not articulated with the
demand side of the M&E system, as stakeholders of the information produced.
As a result, horizontal accountability forums do not have a sufficiently adequate information
position (availability of data, processing capacity). At the same time, administrative bodies
have no incentives to engage in proactive and sincere intrastate account giving, since it is
perceived only from its enforceability point of view. The role of oversight/control agencies
and the judiciary in relation with the M&E system is not recognized, and there is no
alignment with their oversight and constitutional control procedures.
Their informational needs are not taken into account for M&E and therefore, horizontal
accountability arrangements do not help to discourage corruption and improper governance on
time, since there is an “absence of networks within the horizontal accountability structures”,
and there is also a neglect of “another important element that can increase horizontal
accountability[:] […] generation of timely information, which is also one of the ‘tasks’ of
43 Constitutional control is developed specifically by the Supreme Court (CONPES-DNP, 2010: 31).
73
M&E” (Gildemyn, 2011: 13,14). Today, one of the most serious problems of the judiciary is
its inefficiency: by 2009 they were “3,045,884 files without being processed in the courts of
the country”. (Borrero and Ramírez, 2010: 73)44.
According to Gildemyn, “O’Donnell (1999:45) believes that CSOs (media, research and
dissemination institutions) can play an important role in providing timely and relevant
information” for horizontal accountability (Gildemyn, 2011:14), and that is what is happening
in Colombia, where CSOs are driving horizontal and diagonal accountability.
4.2.1.1 CSO´s-led M&E: Driving Diagonal and Horizontal Accountability
Accountability “incentives are the less developed component, as citizens have few channels to
demand more information and explanations, and even less to punish failure or inappropriate state
performance. Citizens usually go to oversight agencies and use judicial mechanisms for these
purposes. Positive incentives for accountability and proper behavior of public servants are
insufficient” (CONPES-DNP, 2010:46)
As elaborated upon in the previous section, in Colombia social actors who want to demand
accountability from government have very few mechanisms to do so [see CONPES, 2010:46-
49]. As a consequence, and since social actors do not have access to the answerability
dimension of accountability in their relationship with government, they use “established
expeditious and efficient” enforceability judicial mechanisms, created by the 1991
constitution to “demand respect for fundamental rights” (Borrero and Ramírez, 2010: 72)
[tutela45 and constitutional control46], to claim for the rule of law [enforcement action47], and
to ask for the guarantee of the citizens’ right to information [right to petition48].
But these mechanisms “wear down citizens and CSOs interested in learning about public
management, and that ultimately discourage and block efforts of social control or account
request. The main reason for this is that they were not created with the purpose of social
44 The details of the judicial crisis go beyond the scope of this document. We just want to highlight the inefficiency of the
system and its disarticulation from the M&E system, in order to show in the last part of the document the important role that
CSO’s are playing in promoting horizontal accountability. Own quote translation. 45 This mechanism seeks the protection of fundamental rights (life, liberty, etc.) that have been violated by act or omission by
any public or private authority responsible of providing a public service or by any persons to whom the applicant is in a
subordination and helplessness state (1991 Constitution). 46 Within this mechanism, a revision of ordinary rules is developed in order to ensure their compliance with constitutional
standards. In case of conflict with the Constitution, invalidation of rules is enforced. This was not created by the 1991
constitution but by the 1886 one, but it was reinforce in 1991 with the creation of the Constitutional Court (Cerra, 2001: 163). 47This mechanism, established by the 1991 Constitution, aims to ensure that laws are met in reality and do not remain on
paper. The difference with ‘tutela’ is that the enforcement action serves to guarantee the execution of laws, while ‘tutela’
ensures the protection of fundamental rights, when they are harmed or threatened by the act or omission of any authority
(Banco de la República, http://www.banrepcultural.org/blaavirtual/faunayflora/fen/texto/medio/partici.htm). 48 The right to petition is a right granted to citizens by the 1991 Constitution so that they can submit petitions to the
authorities to demand information on situations of general and / or particular interest (1991 Constitution).
74
control in mind. Citizens do not have institutions or expedite mechanisms ensuring their right
of information, participation and emerging conflicts solving” (CONPES-DNP, 2010:48).
Other diagonal mechanisms have been developed by the strongest CSOs, which are involved
in independent oversight and monitoring activities regarding government actions. After many
years of work, “using and creating information on the performance of government regarding
programmes and policies […] [they have achieved] answerability” (Gildemyn, 2011:16),
gained trust and credibility within the citizenry and horizontal enforceability actors, and
acquired influence not only within horizontal accountability actions, but also among planning
processes and formulation of public policies. As a consequence, CSOs have provided control
agencies and the judiciary with input information to ensure the rule of law and to improve the
quality of democracy, in exercise of the enforceability dimension of accountability, as we will
notice in the examples presented below.
But before going to the examples, there are some aspects that are important to highlight about
Gildemyn’s approach to the involvement of CSOs within M&E, since they match very much
the Colombian case. Firstly, “CSOs can embark on the M&E journey […] independently”,
and this ensures maintenance of their autonomy and legitimacy as “an independent source of
information” (Gildemyn, 2011:20).
Secondly, since diagonal accountability implies difficulties in its implementation, due to lack
of financial resources, a legal environment that does not guarantee the right to information,
unequal power relations within the accountability frame, and the lack of legitimacy of CSOs’
engagement in M&E activities” (Gildemyn, 2011:19), they are normally “supported by
donors who promote ‘social accountability- type’ initiatives” (Gildemyn, 2011:20). This is
also related to what some authors call ‘the rule of law revival’, enthusiastically embraced by
donor countries and financial institutions in order to counteract third wave democracies that
consolidated “electoral democracy, […] without establishing the rule of law to assure
individual freedom and civic pluralism” (Dodson and Jackson, 2005: 228).
Thirdly and as a consequence of the latter, “[i]n recent years, a strong subset of social
accountability initiatives has emerged “that emphasizes a solid evidence base and direct
dialogue and negotiation with government counterparts” (Gildemyn, 2011:20).
75
The examples presented below are not intended to be exhaustive but rather illustrative,
because the subject matter lends itself to further research.
4.2.1.1.1 CorporaciónTransparencia por Colombia
Transparencia por Colombia is the national chapter of Transparency International. This
widely recognized CSOs elaborates since 2002 a Government Transparency Index (NTI) that
measures the level of transparency and risk of corruption in public entities at the central level,
the three branches of government and control agencies; where higher levels of transparency
mean less risk of corruption. “The Index measures the existence of objective institutional
conditions to promote transparency and to control corruption risks; it does not assess
corruption facts” 49 (Transparencia por Colombia, 2012).
The NTI is not a perception index, or an aggregation of other measurement instruments. It is
composed by primary and secondary data; the former provided by the institution that are
being evaluated (online form, support documents and information recollected by
Transparencia por Colombia) and the latter provided by control agencies and administrative
departments on request. “It is a regular, systematic study that aims to strengthen public sector
institutions and encourage measures to control the risk of corruption in the public sector. It is
the first exercise to quantify risks of corruption in state institutions conducted by an
independent organization that monitors the risks of corruption from a civilian standpoint, and
strengthens the definition of public policies in combating corruption” (Ungar, 2010: 7).
The NTI is composed by three factors: Visibility, Sanction and Institutionalization. There are
also territorial version of the Index: “four editions of the Departmental Transparency Index,
and three editions of the Municipal Transparency Index” (Ungar, 2010: 7); and the financial
support to elaborate the different versions of them come “from the Inter-American
Development Bank, the Dutch Embassy, the German cooperation agency GTZ (CERCAPAZ
Program), the British Embassy, U.S. Agency for International Development, the European
Community”, and the CSO’s own resources (Ungar, 2010: 7).
So far we have already identified many of Gildemyn’s approach to the involvement of CSOs
within M&E: Transparencia por Colombia performs M&E activities independently, with the
financial support of donors who aim to consolidate the rule of law and to enhance the quality
of democracy through transparency and prevention of corruption. The Corporation also
49This quote was translated by the author.
76
generates solid evidence base and direct dialogue and negotiation with government agencies,
which participate voluntarily in the Index and provide Transparencia por Colombia with
primary and secondary data.
Ungar (2011) mentions that there are three main aspects in which the indexes influence: 1) In
terms of public opinion; since the different indexes not only offer a reference about the
evolution of government and public institutions, they also promote the involvement of
different stakeholders in the process. Additionally, indexes results are widely disseminated
thought the media, increasing not only visibility but also public pressure over governmental
institutions to improve their performance towards transparency, or to participate in the
indexes. In relation with the maintenance of CSOs’ autonomy and legitimacy as an
independent source of information, the author mentions that “publications of the index have
been well received because of the independent view it represents.” (Ungar, 2010: 11).
2) In terms of knowledge production; since “the index has established itself as the first
independent instrument of civil society that periodically measures the performance of public
entities in the prevention and control of their risks of corruption, and it enjoys great credibility
in public opinion and in the institutions” (Ungar, 2010: 9) at all levels of government and
throughout different branches of power. The importance of this is that the indexes have
become benchmarks for different government institutions, a yardstick for public opinion
regarding government performance, and a reference tool “for institutions to improve each
indicator, raise performance in the graded factors, and grow more self-confident because they
are monitoring their corruption risks and the effectiveness of the institutional steps to
minimize them” (Ungar, 2010: 9, 10)50
. 3) In terms of public policy; since the indexes provide standards to “analyze the impact of
public policy to combat corruption in the management of the entities and to generate
recommendations for public policies aimed at improving institutional control and
transparency” (Ungar, 2010: 11). Furthermore, recently the Constitutional Court approved a
bill about access to public information. This law was promoted by the Alliance Más
información, Más derechos, which is constitute by different CSOs among which
Transparencia por Colombia can be found (Ungar, 2013).
50 Examples that support this claim can be found at Ungar, 2010: 10.
77
4.2.1.1.2 CODHES - Consultancy for Human Rights and Displacement
CODHES was created in Colombia in 1992, with the aim to “promote the integral realization,
and to guarantee the validity of displaced people, refugees and migrants’ human rights; based
on the International Humanitarian Law, the International Refugees law and the guiding
principles on Internal Displacement” (CODHES website, 2013)51.
In 2005 CODHES created a Monitoring Committee, which constituted a civil society
initiative. This initiative stemmed from the “2004 ‘tutela’ judgment [T-025]52, issued by the
country’s Constitutional Court. Through T-025, the existence of a ‘Unconstitutional State of
Things’ (ECI by its Spanish acronym) was declared, due to the serious and repeated violation
of fundamental rights of millions of Colombians forcibly displaced. T-025 also recognized the
deep gap that exists between the rights recognized to people under this situation by Law 387
of 1997, and the real financial and institutional capacities and capabilities of the public policy
formulated to address this humanitarian crisis” (CODHES website, 2013). As a result, the
judgment ordered the attention of the basic rights of all displaced persons and established a
monitoring process involving national government, oversight agencies, international
community humanitarian agencies, and IDP’s organizations and human rights organizations.
At this point we can already start to identify the aspects highlighted from Gildemyn’s
approach to the involvement of CSOs within M&E. First of all, the autonomy and legitimacy
of the Monitoring Committee as an independent source of information is determined by the
way in which it was set up and its members. According to CODHES, the magnitude and
importance of the Constitutional Court decision pushed CODHES, another CSOs53, “the
Deanship of Law of Universidad de los Andes and important personalities from the national
arena to assume responsibility to Monitor IDP’s Public Policy” (CODHES website, 2013).
The Monitoring Committee had also between its members: journalists, church representatives,
many other academic actors from different universities, and many other human rights CSOs.
Internationally, the commission counted on the support of UNHCR and the Peace Nobel Prize
Rigoberta Menchu among others (CODHES website, 2013).
As a consequence, on the one hand the Commission is “respectful of the autonomy and does
not intend to develop any kind of representation of the interests of people and displaced
51 All quotes from CODHES were translated by the author. 52 “This judgment came in response to the growing organizational capacity of the displaced population, and after thousands of displaced
people employed the ‘tutela’ mechanism to appeal to justice, demanding effective access to their rights” (CODHES website, 2013). 53 Corporación Viva la Ciudadanía
78
communities, or the representation of civil society as a whole” (CODHES website, 2013). In
general terms, the Commission tried to support both, the monitoring process developed by the
Constitutional Court, and the claim processes faced by displaced people. On the other hand,
the Monitoring Commission was supported by donors who promote human rights and the
establishment of the rule of law to assure individual freedom and civic pluralism.
In relation with the third aspect: the construction of solid evidence and direct dialogue and
negotiation with government counterparts; the Monitoring Commission became a very
important technical and systematic instrument. It provided clear information inputs to
influence “the development of a comprehensive policy against the social catastrophe that
forced displacement represents” (CODHES website, 2013). Even though CODHES’
monitoring process is very independent, their criterion is to complement other actions, like
government ones: “the Commission concentrated its work on producing outcome indicators to
measure the Effective Enjoyment of Rights by IDPs [GED by its Spanish acronym]”. The
main reason to do so was that within the national level of government, institutions monitoring
displacement based their efforts only on activities and output indicators, leaving aside GED’s
result indicators.
79
5. CONCLUSIONS AND POLICY RECOMMENDATIONS
M&E systems are conceived as an excellent tool to improve government performance and
to construct good governance because of their decision-making guidance function based on
feedback and iterative learning processes, and their inside and outside accountability
function. The M&E assessment presented in this document showed that most of the
challenges regarding Colombia’s M&E system “are not from a technical nature. They refer
more to matters of system governance” (Cunill, 2010:86). In this country, the existence of
an internationally’ recognized M&E system contradicts the also recognized violation of
citizens’ rights, the government bad performance in transparency and corruption indicators;
as well as “institutional, political, and fiscal difficulties” (Castro, 2009: vi).
Throughout the document, it was possible to identify the main issues, problems and pitfalls
that are limiting the feedback-learning and accountability functions of Colombia’s M&E
system:
1) From the policy dimension:
- The system’s approach to reporting, dissemination and integration of M&E
information within and throughout government agencies and external actors is quite
limited. SINERGIA’s outputs actually promote the feedback and learning function
within the national government. The system’ information is use in the formulation
process of the PND, and in the debate with respect to public policies and programs.
But the distribution and usage of the output information relies only on executive
branch institutions at the national level, limiting the external accountability function of
the system. Both the monitoring, and to a greater extent, the evaluation components of
SINERGIA are reaching, for the most part, only national executive government
agencies which are closer to the system -basically line ministries and the office of the
President. Other agencies from different sectors and with different interests are being
excluded from SINERGIA’s output information. M&E systems should be something
more than just a public managing tool for the national government. They should be a
source of information for civil society, donors, oversight agencies, congress, courts,
interest groups, among others stakeholders; with the aim to enlighten accountability
processes and to inform the country’s web of accountability mechanisms.
- There is an evident lack of autonomy and impartiality when analyzing SINERGIA’s
organizational structure, which is hampering the credibility of the system from both
80
the monitoring and the evaluation perspectives. The lack of autonomy and impartiality
comes from two sources: 1) an organization-structure issue, related to the location of
SINERGIA within the DNP, and 2) a budget issue, based on the absence of an
independent budget and the shortage of financial sources.
- There is integration of M&E results in planning but not in budgeting processes.
This means that government performance information is not considered when
preparing the budget, lacking identification of misalignments between government
priorities and annual budget allocations. This also generates problems when it comes
to use M&E information for planning, decision making and targeting evaluations
inside the DNP.
- Other flaws related with the organizational structure of Colombia’s M&E system
relate with the limited participation of non-state stakeholders within both the
monitoring and the evaluation functions of the system. Normally, the ones invited to
participate are public policy formulators and program executers, while stakeholders
from other governmental agencies and from other sectors are normally excluded. Also
organization mistake related to the absence of linkages of the M&E system at four
levels: with the statistical office, with territorial levels, with other directorates within
DNP, and with projects.
It is important to take into account that policies, programs and projects are consequences of
political decisions and that constitutes a political issue that needs to be acknowledged as an
important influential factor of M&E systems. As a result, the “denial of politics of M&E”
may undermine and jeopardize “M&E double function of accountability, feedback and
learning” (Holvoet, 2012c:12).
2) From the external accountability function:
- SINERGIA is highly decision-making oriented and therefore, it emphasizes more its
learning and its internal feedback and accountability functions, than its external
accountability function.
- Regarding electoral accountability it is possible to state that Colombia’s M&E system
provides some information to democratically legitimized actors (Congress and voters)
about the conduct and actions of government bodies. There are accountability forums
with limited participation and debate from democratic legitimized actors, since they
cannot rely on having an adequate information position. Democratically legitimized
81
actors vaguely use the M&E information to hold government accountable, and the
accountability arrangements indirectly provide very few information about the propriety
and effectiveness of the conduct and action of government bodies.
- Upward accountability is not articulated with SINERGIA’s M&E outputs. This may be
due to the little participatory space provided to donors within the M&E system, and/or to
the very few linkages that exist between M&E outputs and donors’ informational needs.
But also, this can be a consequence of the low donor-donor harmonization when it comes
to demand government performance information.
- From the social control point of view, the citizenry participates much less than what
CONPES document 3294 and the accountability public policy from the executive branch
to the citizens, have set. In reality, the role of civil society within the M&E system is not
very well recognized or institutionalized. Experience shows that its participation is rather
ad-hoc without clear procedures. It is also limited to some very specific sector working
groups, involving few private actors and/or Civil Society Organizations [CSO] within the
policy evaluation processes, although the greater contribution comes from state
stakeholders. In addition, and even though the M&E information is available online, it is
not displayed in a comprehensive way, that could fit different or any type of public.
Also, and albeit that some activities and actions are developed in order to socialize
government results information to the citizenry, in most cases participants are specialized
private or academic public or state stakeholders. As a whole, there are insufficient
incentives for both citizens to ask for accountability, and for public entities to be
accountable.
3) From the demand side, horizontal and diagonal accountability:
- The demand component of the system is still very weak, and SINERGIA does not
promote the use of the system’s information by donors, non-state stakeholders, or even by
state actors that are outside the executive and legislative branches of government. The
same goes for the local level. Some efforts are developed to reach citizen´s, and even
though it is true that there are social accountability tools and forums somehow
recognized, effective social accountability only comes when it is linked with intrastate
accountability, generating diagonal accountability scenarios; and this normally
happens outside the M&E system, driven by CSOs independent monitoring activities.
- In Colombia, horizontal accountability is limited to the creation of standards, laws and
procedures for each branch of government to react and to apply corrective actions when
82
other powers interfere in their area of expertise. As a consequence, the notion of
horizontal accountability adopted in the country leaves aside for both ‘balanced
accountability’ and ‘assigned accountability’, the practices and forums that require the
provision of information and explanations [answerability] about government actions
between entities from all branches of government; as well as the application of sanction
mechanisms [enforceability].
- As a result, horizontal accountability forums do not have a sufficiently adequate
information position (availability of data, processing capacity). Administrative bodies
have no incentives to engage in proactive and sincere intrastate account giving, since it is
perceived only from its enforceability point of view. The role of oversight/control
agencies and the judiciary in relation with the M&E system is not recognized, and there is
no alignment with their oversight and constitutional control procedures.
In general terms, there is an “absence of networks within the horizontal accountability
structures”, and there is also a neglect of “another important element that can increase
horizontal accountability[:] […] generation of timely information, which is also one of the
‘tasks’ of M&E” (Gildemyn, 2011: 13,14).
M&E outcomes should inform not only political and social accountability, they should also
inform administrative and legal one; which includes not only donors, civil society and
congress among their stakeholders, but also high courts54 and control agencies as a
fundamental part of M&E networking. Under this statement it is possible to say that
Colombia’s M&E system focuses on the promotion of the feedback and learning function and
the internal accountability function [technocratic approach] leaving aside the more political or
institutional one, as well as the administrative and legal dimension of the concept. It is
necessary to construct a horizontal accountability network, where M&E information
constitutes and essential input. As O´Donnell claims: “Effective horizontal accountability is
not the product of isolated agencies, but of networks of agencies (up to and including high
courts) committed to upholding the rule of law” (1998:19).
Nonetheless, it is also necessary to find the right balance between the capacity to govern
effectively and to protect individual rights (Dobson and Jackson, 2003:229). The articulation
54 Mainly the Constitutional and the Supreme Courts, as well as the Council of state, since there are the judiciary entities responsible to defend citizen’s rights.
83
of M&E activities with external horizontal accountability process can hamper government
effectiveness and the legitimacy of the system among the national government.
In line with these arguments we present some policy recommendation:
- It is necessary to implement improving strategies to solve those M&E issues that are
hampering the effective performance of the system’s learning and accountability
functions, mainly related to the policy dimension of SINERGIA.
- A way to do so is by promoting a more open M&E system that includes participation of
external stakeholders, articulation of the system with subnational levels and with
budgeting processes. Also, the promotion of more dynamic electoral and social
accountability processes, deepen in the answerability and enforceability dimensions of
the concept, and the reinforced the demand side of the system, can help to accomplish so.
- Taking into account that Colombia has a legalistic tradition, it is essential to involve high
courts and control agencies as stakeholders of M&E information to ensure "the rule of
law". The recognition of the weak nature social accountability and the imbalance of
power when it comes to exercise sanctions upon leaders, generates a necessity to involve
this entities as strengtheners of social control; furthermore in a country with only 52.8%
of confidence in the electoral processes and 40.8% of confidence in the political parties.
- We have already highlighted the benefits of the implementation of accountability
networks and the way in which diagonal accountability processes can reinforce horizontal
and social accountability. Under this frame, a window of opportunity arises with the
Constitutional Court approval of ‘the access of public information’ bill last month. This
new law will provide a complete legal framework guaranteeing the right to access public
information, so that “those who are required to produce and deliver proactive public
information” (Ungar, 2013) are obliged to do so. Also, the law mentions the necessity to
promote ownership among those who have information rights; the citizenry.
- Within this scenario, a legislative faculty should be provided to the DNP to collect, report
and disseminate M&E information in coordination with a public information committee o
entity, like the Mexican ‘Access to Information Federal Institute’ or the Chilean
‘Transparency Council’ (Ungar, 2013).
84
BIBLIOGRAPHY
ANIF (2007) Evaluación del Plan Nacional de Desarrollo (2002-2006): Resultados y
credibilidad, http://anif.co/sites/default/files/uploads/105-01-20070000-OCT2-07.pdf.
(last consulted: 02 of june 2013)
ANIF (2008) Evaluación de Políticas Públicas por Objetivos: ¿Auto-Evaluación?,
Bedi, T., Coudouel, A., Cox, M., Goldstein, M. and Thornton, N. (2006) “Beyond the
numbers: understanding the institutions for monitoring poverty reduction strategies”,
The International Bank for Reconstruction and Development / The World Bank: 1-
223. http://anif.co/sites/default/files/uploads/17-01-20080000-ago14-08.pdf.(last
consulted: 02 of june 2013)
Burdescu, R. Villar, A. Mackay, K. Rojas, F. Saavedra, J. (2005) “Institutionalizing
Monitoring & Evaluation Systems: Five Experiences From Latin America”, en breve,
78: 1-4.
Bovens, M. (2007) “PUBLIC ACCOUNTABILITY: A framework for the analysis
and assessment of accountability arrangements in the public domain” draft, made for
CONNEX, research group 2: Democracy and Accountability in the EU, Utrecht
School of Governance, Utrecht University, The Netherlands.
Borrero and Ramírez (2010) “Vientos de Reforma a la Justicia” Revista Perspectiva,
24(2010):72-74.
Castro, M. F. (2006): “Colombia’s National System for Evaluation of Management
and Results” Towards the Institutionalization of Monitoring and Evaluation Systems
in Latin America and the Caribean, World Bank.
Castro, M. F. (2009) “Insider Insights: Building a Results-Based Management and
Evaluation System in Colombia”, Evaluation Capacity Development Working Paper
Series N° 18, The World Bank, ISBN 13: 978-1-60244-100-2, ISBN 10: 1-60244-100-
6.
Cepei (2010) Evaluación Conjunta de la Declaración de París, Fase 2.
CONPES – DNP, (2004) “Renovación de la Administración Pública: Gestión por
Resultados y Reforma del Sistema Nacional de Evaluación” Documento CONPES
3294, Presidencia de la Republica: Alta Consejería Presidencial, Ministerio de
Hacienda: DGPN, DNP: DEPP-DIFP.PRAP, Bogotá D.C.
CONPES – DNP, (2004) “Política de Rendición de Cuentas de la Rama Ejecutiva a
Los Ciudadanos”, Documento CONPES 3654, DNP: DEPP-DDTS-DJSG,
Departamento Administrativo de la Función Pública, Programa Presidencial para la
Modernización, Eficiencia, Transparencia y Lucha Contra la Corrupción, Bogotá D.C.
Centro Nacional de Consultoría (2010). “Línea de Base del Acceso, Uso y la Calidad
de la Información Que Produce Sinergia”.
Cunill, N. (2010) “Latin American Monitoring and Evaluation Systems and Their
Governance” Fifth Conference Of The Latin And Caribbean Monitoring And
Evaluation (M&E) Network”, Challenges In Monitoring And Evaluation: An
Opportunity To Institutionalize M&E Systems, World Bank.
DNP (2010) “15 años del Sistema Nacional de Evaluación de Gestión y Resultados –
Sinergia: Una mirada desde las evaluaciones de política pública más relevante”
Evolución de Sinergia y Evaluaciones en Administración del Estado, serie 1: 8-44.
DNP (2012) “Orientaciones para que un Alcalde o Gobernador Diseñe o implemente
un Sistema de Seguimiento a su Plan de Desarrollo” Guías Metodológicas Sinergia,
Bogotá D.C.
85
DNP (2012a) “Guía Metodológica para el Seguimiento al Plan Nacional de Desarrollo
y la Evaluación de Políticas Púbicas” Guías Metodológicas Sinergia, Bogotá D.C.
DNP (2012b) “Guía para la Evaluación de Políticas Públicas” Guías Metodológicas
Sinergia, Bogotá D.C.
Dorado, D. (2009) “Colombia: SINERGIA – Colombia’s System of Monitoring and
Evaluation” National Evaluation Capacities: Proceedings from the International
Conference, 15-17 of December 2009, UNDP.
Dorado, D. (2011) Monitoring and Evaluation System in Colombia: SINERIA.
Information for proper decision-making, Seoul-Corea.
Dodson and Jackson (2005) “Horizontal Accountability and the Rule of Law in
Central America”, in: Mainwaring, S. and Welna, C. (eds.) Democratic Accountability
in Latin America, Oxford, 228-265.
Gildemyn, (2011):8 “Towards an Understanding of Civil Society Organisations’
Involvement in Monitoring and Evaluation. Unpacking the Accountability and
Feedback Function M&E” Discussion Paper, 2011.03, Institute of Development
Policy and Management, University of Antwerp, Antwerp-Belgium.
Holvoet, N. (2012) “M&E in the Context of New Aid Modalities”, presentation of the
course Monitoring and Evaluation, 22/03/2012
Holvoet, N. (2012b) Basic notions and functions of M&E – Classification- History of
Development Evaluation, Module II: Evaluating Development Effectiveness, Unit 2:
Development Monitoring and Evaluation: Introducing the Landscape and Approaches,
Sessions 1&2, Institute of Development Policy and Management, University of
Antwerp, Antwerp-Belgium, 2011-2012.
Holvoet, N. (2012c) Organisational and political aspects of M&E, Module II:
Evaluating Development Effectiveness, Unit 2: Development Monitoring and
Evaluation: Introducing the Landscape and Approaches, Session3, Institute of
Development Policy and Management, University of Antwerp, Antwerp-Belgium,
2011-2012.
Holvoet, N. and Inberg, L. (2012) “Sector Monitoring and Evaluation Systems in the
context of Changing AID Modalities: The case of Uganda´s Health Sector”, O*
Platform Aid Effectiveness, Institute of Development Policy and Management,
University of Antwerp.
Holvoet, N. and Renard, R. (2007) “Monitoring and Evaluation Under the PRSP:
Solid Rock or Quicksand?”, Elservier 66(81).
Holvoet, N. and Rombouts, H. (2008) “The Denial of Politics in PRSP’s Monitoring
and Evaluation. Experiences from Rwanda” Discussion Paper, 2008.02, Institute of
Development Policy and Management, University of Antwerp, Antwerp-Belgium.
Kusek, J.Z., and Rist, R.C. (2002) “Building Results-Based Monitoring and
Evaluation Systems: Assessing Developing Countries Readiness”, Zeitschrift fur
Evaluation N° 1, 151-158.
______ (2004) “Ten steps to a results-based monitoring and evaluation system: a
handbook for development practitioners”, The World Bank, Washington, DC, 1 – 239.
Leftwich, A. 1994, ‘Governance, the State and the Politics of Development’,
Development and Change, vol. 25, no. 2, pp. 363-386. Mackay, K. (1999) “A
Diagnostic Guide and Action Framework”, ECD Working Paper Series N° 6, World
Bank Operations Evaluation Department, 1-28.
_____ (2008) “Building Monitoring and Evaluation Systems to Improve Government
Performance” in Country-led Monitoring and Evaluation Systems. Better Evidence,
86
Better Policies, Better Development Results, 169 -187. Editor Marco Segone, Senior
Regional Advisor, Monitoring and Evaluation, UNICEF Regional Office for
CEE/CIS, and former Vice President, International Organization for the Cooperation
in Evaluation (IOCE).
Mainwaring, 2005: 4
McGee, Rosemary and Heredia, Irma García, Paris in Bogotá: The Aid Effectiveness
Agenda and Aid Relations in Colombia (March 2012). Development Policy Review,
Vol. 30, Issue 2, pp. 115-131, 2012. Available at SSRN:
http://ssrn.com/abstract=2016689 or http://dx.doi.org/10.1111/j.1467-
7679.2012.00567.x
O´Donnell, G. (1998) “Horizontal Accountability in New Democracies”, Journal of
Democracy 9.3, National Endowment for Democracy and the Johns Hopkins
University Press, pp. 112-126.
OECD (2002), Evaluation and Aid Effectiveness, Glossary of Key Terms in
Evaluation and Results Based Management, 1 – 37.
OECD (2008a) “2008 Survey on Monitoring the Paris Declaration - Effective Aid By
2010? What It Will Take, OECD Publishing. ISBN 9789264050822 (PDF)
OECD (2008b) “The Paris Declaration on Aid Effectiveness and the Accra Agenda for
Action”, OECD
http://hdrnet.org/522/1/Paris_declaration_and_Accra_agenda_for_action.pdf
OECD (2011) “Aid Effectiveness 2005–10: Progress in implementing the Paris
Declaration”, OECD Publishing. ISBN 9789264125490 (PDF)
World Bank (2006) “Towards the Institutionalization of Monitoring and Evaluation
Systems in Latin America and the Caribbean”, World Bank Latin American and
Caribbean Studies.
World Bank (2007) “A diagnosis of Colombia’s National M&E System, SINERGIA”
Evaluation Capacity Development, ECD Working Paper Series N° 17: February 2007,
Washington, D.C.
Ungar, E. (2010) “Transparency Index of Public Entities: Experience in Evaluating
Risk of Corruption in Government Administration at National, Departmental, and
municipal Level” Fifth Conference Of The Latin And Caribbean Monitoring And
Evaluation (M&E) Network”, Challenges In Monitoring And Evaluation: An
Opportunity To Institutionalize M&E Systems, World Bank
Ungar, E (2013) “El Derecho de Acceso a la Información Pública” El Espectador, 15
de mayo 2013, columna de opinión.
Websites:
Transparency for Colombia: http://www.transparenciacolombia.org.co/
CODHES: http://www.codhes.org/
87
Appendix A: Assessing Public Accountability
Besides the accountability mapping exercise, Bovens (2007) also presents two possible levels
for the evaluation of accountability arrangements in the public domain: 1) a procedural
(internal) level “of a particular accountability mechanism or of a specific, concrete
accountability process”; and 2) a systemic (external) level which focuses on the “effects of the
accountability processes (…) based on the functions that accountability arrangements fulfill in
the political and administrative systems” (Bovens, 2007: 22).
The accountability notion adopted before allows us to be aware of particular characteristic of
the M&E external accountability function. If the outcomes of M&E systems are not handed to
other entities with the capacity to pass judgment and imposed formal and/or informal
sanctions (e.g. the congress, the judiciary, oversight agencies, the citizenry, the organized
civil society, the donor community, among others), it does not constitute a complete
accountability function. In other words, M&E systems development agencies need to have
some capacity to refer the outcomes of monitoring and evaluation processes to those
accountability institution and/or actors that have the power of sanctioning wrongdoings of
public officials and institutions; which relates directly with the demand side of the systems
and the use of the information produced by them. Under this perspective, M&E systems
should be understood as an accountability arrangement constitutive of the whole
accountability regime, or the web of accountability institutions.
Since the aim of this part of the document is to apply Bovens (2007) assessment framework to
adapt some parts of it to Holvoet and Renard’s (2007) and Holvoet and Inberg (2012) M&E
diagnostic instrument for the next chapter, we will focus on the second level of Bovens (2007)
evaluation framework; i.e. the systemic external level (see table 4).
Evaluating the External Effects of Accountability
For this level of accountability evaluation, Bovens (2007:24-31) presents three perspectives
that derivate from the analysis of the reasons why accountability is important (see table 4).
According to the author, “[i]n the academic literature and in policy publications about public
accountability, three answers are present. […] Public accountability is important to provide a
democratic means to monitor and control government conduct, for preventing the
development of concentrations of power, and to enhance the learning capacity and
effectiveness of public administration”. (Bovens, 2007:25). With this, the author is trying to
88
determine “what is the purpose of public accountability in a constitutional democratic state
and what are the evaluation principles for accountability arrangements?” (Bovens, 2007:24);
while at the same time develops three separate theoretical perspectives to evaluate public
accountability, based on reasons why it is important: The democratic perspective, the
constitutional perspective and the cybernetic perspective. They are summarized in Table 4.
Bovens (2007:30,31) highlights some issues that one must take into account for the
application of the evaluation framework: 1) “accountability arrangements may score well on
one perspective, but not on others”, 2) “these perspectives need not always point in the same
direction. What is considered beneficial from one perspective may very well be judged
detrimental from another perspective”. As we already mentioned before, excessive
accountability mechanism can compromised the effectiveness of public administration; which
lead us to the next issue;3) “within each evaluation perspective there always remains the
question of standards and levels of sufficiency”.
In the end, what is important is to put the evaluation framework in context, not only at a
country level -socio-economic, political and cultural context-; but also from a more
institutional perspective. Of course, for the application of any assessment tool there is always
“a theory, often implicit, about what constitutes sufficient democratic control, or adequate
checks and balances, […] satisfactory reflexivity, […] “or good enough governance”
(Bovens, 2007:31).
89
Three Perspectives to Evaluate the External Effects of Public Accountability (based on Bovens, 2007:24-31).
Evaluation
Perspective Based on Central idea Evaluation Criterion
Concrete Evaluation
Questions
The
democratic
perspective (Democratic
Tradition)
(Electoral
Accountability)
Popular Sovereignty: Public
accountability provides the
people’s representation and the
voters with the information
needed for judging the propriety
and effectiveness of the conduct
of the government.
- Accountability arrangements yield
relevant information about the
conduct of the government.
- Accountability offers actors with
democratic legitimacy
possibilities to control
administration, policy and
organisation.
The degree to which
accountability arrangements
or regimes directly or
indirectly contribute to the
possibilities for actors with
democratic legitimacy to
monitor, evaluate and
adjust the propriety and
effectiveness of government
conduct.
a. Are there any accountability forums in
which actors with democratic legitimacy
participate and can the latter rely on having
an adequate information position and
enforceable sanction options at their
disposal?
b. To what extent do the accountability
arrangements indirectly provide information
to democratically legitimised actors about the
propriety and the effectiveness of the conduct
and actions of government bodies?
c. To what extent does the accountability
arrangement itself allow for the adjustment of
the conduct of government bodies in the
direction desired by the actors with
democratic legitimacy?
d. Do the accountability arrangements offer
enough incentives to agents to commit
themselves to the agendas of their
democratically elected principals?
The
constitutional
perspective (Republican and
Liberal
Traditions)
(Intrastate
Accountability)
Prevention of Corruption and
Abuse of Power: Other public
institutions (e.g. independent
judicial power or a Chamber of
Audit) are put in place next to
the voter, parliament, and
political officials, and given the
power to request that account be
rendered over particular aspects.
- Accountability is essential
in order to withstand the
ever-present tendency
toward power
concentration in the
executive power.
- Public accountability
forums be visible, tangible
and powerful, in order to
be able to withstand both
the inherent tendency of
those in public office to
evade control and the
autonomous expansion of
The extent to which
accountability forums are
able to contribute to the
prevention of corruption
and the abuse of powers.
a. Do the accountability forums have a
sufficiently adequate information position
(availability of data, processing capacity)?
b. Do the accountability forums have enough
inquisitive powers to reveal corruption or
mismanagement?
c. Do the accountability forums have
incentives to engage in proactive and alert
account holding?
d. Do the administrative bodies have
incentives to engage in proactive and sincere
account giving?
e. Are the available sanctions strong enough
to have preventive effects?
90
power of the all-
encompassing bureaucracy.
- Major issue: whether
accountability arrangements
offer enough incentives for
officials and agencies to
refrain from abuse of
authority.
f. Does the accountability arrangement help
to discourage corruption and improper
governance?
The cybernetic
perspective (`intelligence of
democracy’ –
pluralist
tradition)
(Prevention;
learning in the
process of
policymaking)
Enhancing the Learning
Capacity: Public accountability
offers a regular mechanism to
confront administrators with
information about their own
functioning and forces them to
reflect on the successes and
failures of their past policy.
- Accountability is an
essential condition for
learning by administrative
bodies and holders of
executive positions.
- Accountability arrangement
enhances the learning
capacity and effectiveness
of the public
administration.
- Accountability
arrangements and other
feedback mechanisms to be
successful if they generate
feedback information and
stimulate elite groups to
reflect and to debate about
the significance of this
information with others
The degree to which
accountability
arrangements stimulate
administrative bodies and
officials to achieve a
higher awareness of the
environment, increase self-
reflection and induce the
ability to change.
a. Does the accountability arrangement
contribute to the availability of information
about former and current administrative
actions for the administrative body involved
and a wider range of administrative bodies?
b. Does the accountability arrangement
stimulate internal reflection and the ensuing
learning conduct in administrative bodies and
those holding public office?
c. Does the accountability arrangement
stimulate the accountability forums and the
administrative actors to (supervising) the
institutionalisation and dissemination of
lessons learned?
91
Appendix B: Assessment Benchmark
Policy Indicators, Data Collection and Methodology
Organization: Structure Organization:
Linkages Capacity
Participation of Actors Outside the Government
Use of M&E outputs: Feedback/Learning
Function
Use of M&E outputs: Accountability Function
Weak (1)
· There is a M&E where what to evaluate, why,
how and for whom is not clearly
stated
· What to monitor and to
evaluate is unclear and/or
very broad
· There is an unclear institutional structure for coordination, support, oversight, analysis of
data and feedback, with no stakeholders involved
· The M&E system and the statistical office are not linked
· The present capacity of the M&E unit
at central, sector, sub-sector and decentralized levels (e.g. fte, skills, financial
resources) is very limited
· The role of Congress is not recognized and
there is no alignment with
control and oversight
procedures.
· M&E results are not presented.
· Donors are not using very much the outputs of the M&E
system for their information needs. The demand for M&E data from donors is not really coordinated
· The difference and the
relationship between M and E
is vague or not defined
· There is a list of indicators but they are nor
SMART (specific, measurable, achievable,
relevant, time-bound)
· There is not an stable composition of
sector working groups or meetings held constantly
· There are not M&E units in
line ministries and semi-
governmental institutions or they are not linked to the central unit
· The current
weaknesses in the system are not identified
· The role of civil society is not recognized. They don't participate in the system or in
any sector working groups.
· Results of M&E activities are not used for internal purposes at the
national level. They do not constitute an instrument of
policy-making and/or policy-influencing and advocacy at
central level. M&E outcomes, dissemination strategies and institutional arrangements do
not stimulate internal reflection and conduct in those holding public office (at the national
level),nor the institutionalization and
dissemination of lessons learned. Institutional
arrangements do not allow adjustments of the conduct of
government bodies.
· The M&E system do not provide information to
democratically legitimized actors (Congress and voters) about the
conduct and actions of government bodies. Accountability
forums in which actors with democratic legitimacy participate
and debate do not exist. The democratically legitimized actors don't use the information to hold
government accountable, and the accountability arrangements do
not provide information about the propriety and the effectiveness of
the conduct and action of government bodies. The
administrative bodies have no incentives to engage in proactive and sincere account giving, and
to commit themselves to the agenda's of their democratically
elected principals.
92
· The need for autonomy and impartiality its not
mentioned, the M&E plan does not allows for
tough issues to be analyzed, there is
not an independent
budget
· Indicators are not
disaggregated by sex, region,
socio-economic status
· The demand for (strengthening of the)
M&E system comes from external actors (e.g.
donors)? A "champion" who advocates for the (strengthening of the)
M&E system is absent.
· There are not sector M&E units or they are not linked to the central M&E unit
· There are no plans/activities
for remediation
· The role of donors is not
recognized. Their participation is ad-hoc without clear
procedures. Donors do not
participate in sector working groups.
· Results of M&E activities are not used for
internal purposes at the local level. They do not constitute
an instrument of policy-making and/or policy-influencing and
advocacy at local level
· Intrastate (horizontal) accountability forums do not have a sufficiently adequate information
position (availability of data, processing capacity).
Administrative bodies have no incentives to engage in proactive
and sincere account giving. Accountability arrangements don't help to discourage corruption and improper governance. discourage
corruption and improper governance. The role of
oversight/control agencies and the judiciary is not recognized and there is no alignment with their
oversight and constitutional control procedures. Their
informational needs not taken into account for M&E.
· The+B4 approach to reporting,
dissemination, and integration is
vague and/or very broad
· The criteria for the selection of indicators are not established
· There are not Incentives (at central and local level) that stimulate data collection and data
use
· There are not M&E units at
subnational levels or there
are not linked to sector/national
M&E units
·
· The national M&E system do not provide information to promote social control and participation. Accountability
arrangements offer no incentives to agents to commit themselves to
social forum´s of control and participation within the public policy cycle. M&E institutional arrangements don't stimulate mechanisms to compare and validate results information for accountability proposes, nor
evaluation capacities to promote social accountability.
93
· M&E results are not integrated in planning and
budgeting
· The need to set priorities and to limit the
number of indicators to be
monitored is not acknowledged
· There is no effort to
coordinate with donor M&E
mechanism for projects and
vertical funds at sector level
· Different levels of
indicators (input-output-outcome-impact) are not linked (program theory) (vertical
logic)
· It is not clear how to monitor and
evaluate: Methodologies
are not identified and/or integrated
· Indicators are not linked to their sources of data collection, which are not even identified
(horizontal logic)
Partially Satisfactory
(2)
· There is a M&E plan which roughly indicates what to evaluate, why, how and for
whom
· Although what to monitor and to evaluate is define and there is a list
of indicator, sector indicators are only vaguely harmonized with
the NDP indicators
· There is a not very good institutional
structure for coordination, support, oversight,
analysis of data and feedback, with very few stakeholders involved and it is situated at a
sector ministry
· There is a weak linkage between the
M&E system and the statistical office and the
role of the statistical office within the M&E system is not
very clear
· The present capacity of the M&E unit
at central, sector, sub-sector and decentralized levels (e.g. fte, skills, financial resources) is
limited
· The role of Congress is not
very well recognized
although there is some alignment with control and
oversight procedures.
Congress does not participate in sector
working groups
· Not many M&E results are presented and there are not very well compared to
targets. Discrepancies are not analyzed. M&E outputs are not differentiated towards
different audiences.
· Donors use somehow the outputs of the M&E system for their information needs. The
demand for M&E data from donors is a little coordinated.
94
· The difference and the
relationship between M and E
is not clearly stated
· There is a list of indicators
that are somehow
SMART (specific, measurable, achievable,
relevant, time-bound)
· Sector working meetings are held occasionally for monitoring, their
composition is not very stable and very fez stakeholders are
represented
· There are some M&E units in line ministries
and semi-governmental
institutions which are partially linked to the central unit
· The current
weaknesses in the system are not very well
identified
· The role of civil society is not
very well recognized or
institutionalized. Its participation is rather ad-hoc without clear
procedures. Civil society does not
participate in sector working groups.
· Results of M&E activities are not very much used for internal purposes at
the national level. They rarely constitute an instrument of
policy-making and/or policy-influencing and advocacy at
central level. M&E outcomes, dissemination strategies and
institutional arrangements rarely stimulate internal
reflection and conduct in those holding public office (at the
national level), or the institutionalization and
dissemination of lessons learned. Institutional
arrangements hardly ever allow adjustments of the conduct of government
bodies.
· The M&E system provides some information to
democratically legitimized actors (Congress and voters) about the
conduct and actions of government bodies. There are
accountability forums with limited participation and debate from democratic legitimized actors,
since they cannot rely on having an adequate information position. Democratically legitimized actors
use vaguely the information to hold government accountable, and
the accountability arrangements indirectly provide very few
information about the propriety and the effectiveness of the
conduct and action of government bodies. The administrative bodies have few incentives to engage in
proactive and sincere account giving, and to commit themselves
to the agenda's of their democratically elected principals.
95
· The need for autonomy and impartiality can
hardly be perceived, the M&E plan does not allows for
important issues to be analyzed, an important part of
the budget is independent
· An essential part of Indicators are
disaggregated by sex, region,
socio-economic status
· The demand for (strengthening of the)
M&E system comes from the a sector ministry? There is a "champion" who advocates for the (strengthening of the)
M&E system
· There are some sector
M&E units which are partially linked to the
central M&E unit
· Plans/activities for remediation,
including training, appropriate
salaries, etc. are not very clear
· The role of donors is not very well recognized.
Their participation is rather ad-hoc
without clear procedures.
Donors sometimes participate in sector
working groups.
· Results of M&E activities are not very much used for internal purposes at the local level. They rarely constituted an instrument of policy-making and/or policy-influencing and advocacy at
local level
· Intrastate (horizontal) accountability forums have a
sufficiently adequate information position (availability of data,
processing capacity). Administrative bodies have few
incentives to engage in proactive and sincere account giving. Accountability arrangements
hardly help to discourage corruption and improper
governance. discourage corruption and improper governance. The role of
oversight/control agencies and the judiciary is not very well
recognized and there is low alignment with their oversight and constitutional control procedures. Their informational needs are not taken into account sufficiently for
M&E.
· The approach to reporting,
dissemination, and integration is not
very clear
· The criteria for the selection of indicators are spelled out but not sufficiently
detailed
· Few Incentives (at central and local level) are used to stimulate
data collection and data use
· There are some M&E units at subnational
levels which are partially linked to sector/national
M&E units
·
· The national M&E system provides some information to promote social control and participation. Accountability
arrangements offer few incentives to agents to commit themselves to
social forum´s of control and participation within the public policy cycle. M&E institutional arrangements hardly stimulate mechanisms to compare and validate results information for
accountability proposes, or evaluation capacities to promote
social accountability.
96
· There is integration of M&E results in planning
but not in budgeting
· The need to set priorities and to limit the
number of indicators to be monitored is
acknowledged somehow
· There is some effort to
coordinate with donor M&E
mechanism for projects and
vertical funds at sector level
· Different levels of
indicators (input-output-outcome-
impact) are slightly linked
(program theory) (vertical logic)
· It is not very clear how to
monitor and evaluate:
Methodologies are identified but not sufficiently spelled out or
integrated
· Indicators are partially
linked to those sources of data
collection that are identified
(horizontal logic)
Satisfactory (3)
· There is a clear M&E plan indicating the majority of the
following: what to evaluate, why,
how and for whom
· It is clear what to monitor and evaluate,
there is a list of indicators and the majority of
sector indicators are harmonized
with the NDP
· There is a 'good enough' institutional
structure for coordination, support, oversight,
analysis of data and feedback, with few
stakeholders involved and it is situated at a
central ministry
· There is a linkage between the M&E system and the statistical office but the role of the statistical office within the M&E system is not very clear
· The present capacity of the M&E unit
at central, sector, sub-sector and decentralized levels (e.g. fte, skills, financial resources) is
· The role of Congress is
recognized and there is alignment with control and
oversight procedures. Congress
participates
· Some M&E results are presented. Results are
compared to targets and there is an analysis of
discrepancies. M&E outputs are somehow but not very well differentiated towards different
audiences.
· Donors are using the outputs of the M&E system for their information needs. The
demand for M&E data from donors is coordinated
97
indicators somehow limited sometimes in sector working groups
· The difference and the
relationship between M and E
is sufficiently spelled out
· There is a list of SMART
(specific, measurable, achievable,
relevant, time-bound)
indicators but baselines and targets are not
clearly attached
· Frequent sector working meetings are
held for monitoring, few stakeholders are
represented
· There are M&E units in line
ministries and semi-
governmental institutions which are linked to the
central unit
· The current
weaknesses in the system are
identified
· The role of civil society is institutionally
arranged and it involves procedures
. Civil society participates
somehow in sector working groups.
· Results of M&E activities are used for internal purposes at the national level.
They are an instrument of policy-making and/or policy-influencing and advocacy at
central level. M&E outcomes, dissemination strategies and
institutional arrangements stimulate somehow internal
reflection and conduct in those holding public office (at the
national level), as well as the institutionalization and
dissemination of lessons learned. Institutional
arrangements allow in some way the adjustment of the
conduct of government bodies.
· The M&E system provides information to democratically
legitimized actors (Congress and voters) about the conduct and actions of government bodies.
There are accountability forums in which actors with democratic
legitimacy participate and debate somehow, and rely on having an
adequate information position. Democratically legitimized actors use some of the information to
hold government accountable, and the accountability arrangements
indirectly provide information about the propriety and the
effectiveness of the conduct and action of government bodies. The administrative bodies have some incentives to engage in proactive and sincere account giving, and
to commit themselves to the agenda's of their democratically
elected principals.
98
· The need for autonomy and impartiality can
clearly be perceived, the
M&E plan allows for some important
issues to be analyzed, the majority of the
budget is independent
· The majority of
Indicators are disaggregated by
sex, region, socio-economic
status
· The demand for (strengthening of the)
M&E system comes from the a sector ministry? There is a "champion" who advocates for the (strengthening of the)
M&E system
· There are sector M&E units which are linked
to the central M&E unit
· There are plans/activities for remediation,
including training, appropriate salaries, etc.
· The role of donors is
recognized, there are procedures for their participation. Donors participate in sector working
groups.
· Results of M&E activities are used for internal
purposes at the local level. They are an important
instrument of policy-making and/or policy-influencing and
advocacy at local level
· Intrastate (horizontal) accountability forums have an adequate information position
(availability of data, processing capacity). Administrative bodies have some incentives to engage in proactive and sincere account
giving. Accountability arrangements help somehow to
discourage corruption and improper governance. The role of oversight/control agencies and the judiciary is recognized and there
is alignment with oversight and constitutional control procedures. Their informational needs are in someway taken into account for
M&E.
· There is a clear approach to
reporting, dissemination,
integration
· The criteria for the selection of indicators are spelled out but not sufficiently
detailed
· Incentives (at central and local level) are used to stimulate
data collection and data use
· There are M&E units at subnational
levels which are linked to
sector/national M&E units
·
· The national M&E system provides information to promote social control and participation.
Accountability arrangements offer some incentives to agents to commit themselves to social
forum´s of control and participation within the public policy cycle. M&E institutional
arrangements somehow stimulate mechanisms to compare and validate results information for
accountability proposes, as well as some evaluation capacities to
promote social accountability.
99
· There is integration of M&E results in planning
and budgeting
· The need to set priorities and to limit the
number of indicators to be monitored is sufficiently
acknowledged
· There is some effort to
coordinate with donor M&E
mechanism for projects and
vertical funds at sector level
· Different levels of
indicators (input-output-outcome-
impact) are linked (program theory) (vertical
logic)
· It is clear how to monitor and evaluate: Methodologies
are identified and mutually
integrated
· Indicators are linked to
those sources of data collection
that are identified (horizontal logic)
Good (4)
· There is a comprehensive
M&E plan, indicating what to
evaluate, why, how and for whom
· It is clear what to monitor and evaluate,
there is a list of indicators and the majority of
sector indicators are harmonized
with the NDP indicators
· There is a very good institutional
structure for coordination, support, oversight,
analyses of data and feedback, with the
majority of stakeholders involved, and it is situated
at a central ministry
· There is a linkage between the M&E system and the statistical office. The role of
the statistical office within the M&E system is
clear
· The present capacity of the M&E unit
at central, sector, sub-sector and decentralized levels (e.g. fte, skills, financial resources) is
significant
· The role of Congress is
properly recognized and there is
alignment with control and oversight
procedures. Congress
participates in sector working
· Several M&E results are presented. Results are
compared to targets and there is an analysis of
discrepancies. M&E outputs are differentiated towards
different audiences.
· Donors are constantly using the outputs of the M&E system for
their information needs. The demand for M&E data from donors
is clearly coordinated
100
groups
· The difference and the
relationship between M and E is clearly spelled
out
· There is a list of SMART
(specific, measurable, achievable,
relevant, time-bound)
indicators with baselines and
targets attached
· Sector working groups participate in
monitoring, their composition is stable and
the majority of stakeholders are
represented
· There are M&E units in line
ministries and semi-
governmental institutions which
are properly linked to the central unit
· The current
weaknesses in the system are
clearly identified
· The role of civil society is
properly recognized. Its participation is institutionally arranged and
encompasses clear procedures. Civil
society participates very often in sector
working groups.
· Results of M&E activities are constantly used for internal purposes at the national level. There are an
importantly instrument of policy-making and/or policy-influencing and advocacy at
central level. M&E outcomes, dissemination strategies and
institutional arrangements stimulate internal reflection
and conduct in those holding public office (at the national
level), as well as the institutionalization and
dissemination of lessons learned. Institutional
arrangements allow for the adjustment of the conduct of
government bodies.
· The M&E system provides clear, consistent and adequate information to democratically
legitimized actors (Congress and voters) about the conduct and actions of government bodies.
There are accountability forums in which actors with democratic
legitimacy participate and can rely on having a good information
position. Democratically legitimized actors use the
information to hold government accountable, and the
accountability arrangements indirectly provide useful
information about the propriety and the effectiveness of the
conduct and action of government bodies. The administrative bodies
have incentives to engage in proactive and sincere account
giving and to commit themselves to the agenda's of their
democratically elected principals.
101
· The need for autonomy and
impartiality its mentioned
somehow, the M&E plan allows
for tough issues to be analyzed, There is an independent
budget
· Indicators are
disaggregated by sex, region,
socio-economic status
· The demand for (strengthening of the)
M&E system comes from the a central ministry
(e.g. ministry of planning or finance)? There is a
"champion" who advocates for the
(strengthening of the) M&E system
· There are sector M&E units
which are properly linked to the central M&E
unit
· There are good
plans/activities for remediation,
including training, appropriate salaries, etc.
· The role of donors is properly recognized, there
are clear procedures for their
participation. Donors participate
often in sector working groups.
· Results of M&E activities are constantly used for internal purposes at the local level. They are an
important instrument of policy-making and/or policy-
influencing and advocacy at local level
· Intrastate (horizontal) accountability forums have a good information position (availability of
data, processing capacity). Administrative bodies have
incentives to engage in proactive and sincere account giving.
Accountability arrangements help to discourage corruption and
improper governance. The role of oversight/control agencies and the
judiciary is properly recognized and there is alignment with oversight and constitutional control procedures. Their
informational needs are taken into account for M&E.
· There is a very clear
approach to reporting,
dissemination, integration
· The criteria for the selection of indicators are clear as well as
who is involved in the selection
· Many Incentives (at central and local level) are used to stimulate
data collection and data use
· There are M&E units at subnational
levels which are properly linked to sector/national
M&E units
·
· The national M&E system provides clear, consistent and
adequate information to promote social control and participation.
Accountability arrangements offer incentives to agents to commit themselves to social forum´s of
control and participation within the public policy cycle. M&E
institutional arrangements stimulates mechanisms to
compare and validate results information for accountability
proposes, as well as evaluation capacities to promote social
accountability.
102
· There is integration of M&E results in planning
and budgeting
· The need to set priorities and to limit the
number of indicators to be monitored is
acknowledged
· There is a clear effort to
coordinate with donor M&E
mechanism for projects and
vertical funds at sector level
· Different levels of
indicators (input-output-outcome-
impact) are clearly linked
(program theory) (vertical logic)
· It is clear how to monitor and evaluate: Methodologies
are well identified and mutually
integrated
· Indicators are linked to their sources of data collection which
are well identified (horizontal logic)
Excellent (5)
· There is an exceptionally
comprehensive M&E plan,
indicating what to evaluate, why,
how and for whom
· It is very clear what to monitor and
evaluate, there is a list of indicators
and sector indicators are
harmonized with the NDP indicators
· There is an excellent institutional
structure for coordination, support, oversight,
analysis of data and feedback, with many different stakeholders
involved and it is situated at a central ministry
· There is an explicit linkage between the
M&E system and the statistical
office. The role of the statistical
office within the M&E system is
very clear
· The present capacity of the M&E unit
at central, sector, sub-sector and decentralized levels (e.g. fte, skills, financial
resources) is very significant
· The role of Congress is very
well recognized and there is alignment with control and
oversight procedures. Congress
participates actively in sector working
groups
· Relevant M&E results are presented. Results are
compared to targets and there is an analysis of
discrepancies. M&E outputs are differentiated towards many different audiences.
· Donors are using very much the outputs of the M&E system for
their information needs. The demand for M&E data from donors
is perfectly coordinated
103
· The difference and the
relationship between M and E
is very clearly spelled out
· There is a list of SMART
(specific, measurable, achievable,
relevant, time-bound)
indicators with baselines and
targets attached
· Sector working groups are very active in
monitoring, their composition is stable and various stakeholders are
represented
· There are M&E units in line
ministries and semi-
governmental institutions which
are perfectly linked to the central unit
· The current
weaknesses in the system are
perfectly identified
· The role of civil society is very well recognized. Its
participation is institutionally arranged and encompasses
exceptionally clear procedures. Civil
society participates actively in sector working groups.
· Results of M&E activities are very much used for internal purposes at the national level. They are a fundamental instrument of
policy-making and/or policy-influencing and advocacy at
central level. M&E outcomes, dissemination strategies and
institutional arrangements stimulate greatly internal
reflection and conduct in those holding public office (at the
national level), as well as the institutionalization and
dissemination of lessons learned. Institutional arrangements allow
significantly for the adjustment of the conduct of government
bodies.
· The M&E system provides very clear, consistent and adequate information to
democratically legitimized actors (Congress and voters) about the
conduct and actions of government bodies. There are accountability forums in which
actors with democratic legitimacy participate and can rely on having
very good information position. Democratically legitimized actors use very much the information to
hold government accountable, and the accountability arrangements
indirectly provide very useful information about the propriety
and the effectiveness of the conduct and action of government
bodies. Agents have many incentives to engage in proactive and sincere account giving and to
commit themselves to the agenda's of their democratically
elected principals.
· The need for autonomy and
impartiality is explicitly
mentioned, the M&E plan allows
for tough issues to be analyzed, There is an independent
budget
· Indicators are
disaggregated by sex, region,
socio-economic status
· The demand for (strengthening of the)
M&E system comes from the a central ministry
(e.g. ministry of planning or finance)? There is a
highly placed "champion" who advocates for the (strengthening of the)
M&E system
· There are sector M&E units
which are perfectly linked to the central M&E
unit
· There are relevant and consistent
plans/activities for remediation,
including training, appropriate salaries, etc.
· The role of donors is very well recognized, there are exceptionally
clear procedures for their participation. Donors participate
actively sector working groups.
· Results of M&E activities are very much used for internal purposes at the
local level. They are a fundamental instrument of
policy-making and/or policy-influencing and advocacy at
local level
· Intrastate (horizontal) accountability forums have an excellent information position
(availability of data, processing capacity). Administrative bodies have many incentives to engage in proactive and sincere account
giving. Accountability arrangements help significantly to
discourage corruption and improper governance. The role of oversight/control agencies and the
judiciary is very well recognized and there is alignment with oversight and constitutional control procedures. Their
informational needs are widely taken into account for M&E.
104
· There is an explicit and consistent
approach to reporting,
dissemination, integration
· The criteria for the selection of indicators are very clear as well
as who is involved in the
selection
· Strong and numerous Incentives (at central and local level) are used to stimulate
data collection and data use
· There are M&E units at subnational
levels which are perfectly linked to
sector/national M&E units
·
· The national M&E system provides very clear, consistent and adequate information to promote social control and participation. Accountability arrangements offer many
incentives to agents to commit themselves to social forum´s of
control and participation within the public policy cycle. M&E
institutional arrangements stimulates significantly
mechanisms to compare and validate results information for
accountability proposes, as well as evaluation capacities to
promote social accountability.
· There is a very complete
integration of M&E results in planning
and budgeting
· The need to set priorities and to limit the
number of indicators to be monitored is
widely acknowledged
· There is an evident and
constant effort to coordinate with
donor M&E mechanism for projects and
vertical funds at sector level
· Different levels of indicators (input-output-
outcome-impact) are very clearly linked (program theory) (vertical
logic)
105
· It is very clear how to monitor and evaluate: Methodologies are extremely well identified and mutually
integrated
· Indicators are very well linked to their
sources of data collection which are exceptionally
well identified (horizontal logic)
109
Appendix C: Quantitative Analysis
Topics Score
Score Table
I. Policy 3,2
Weak 1
M&E plan 5
Partially Satisfactory
2
M versus E 5
Satisfactory 3
Autonomy & impartiality (accountability) 1
Good 4
Feedback 3
Excellent 5
Alignment planning & budgeting 2
II. Indicators, Data Collection and Methodology 4,4 Selection of indicators 5
Quality of indicators 4
Disaggregation 1
Selection criteria 5
Priority setting 5
Causality chain 5
Methodologies used 5
Data collection 5
III. Organization: a) Structure 4,3 Coordination and oversight 4
Sector Working groups 4
Ownership 5
Incentives 4
III. Organization: b) Linktages 2,2 Linkage with Statistical office 1
"Horizontal" integration 2
"Vertical‘ upward integration 5
"Vertical‘ downward integration 1
Link with projects 2
IV. Capacity 3,3 Present capacity 4
Problem acknowledged 4
Capacity building plan 2
V. Participation of Actors Outside the Government 1,67 Congress 2
Citizenry, Civil Society and Private Sector 2
Donors 1
110
VI. Use of M&E outputs: a) Feedback/Learning Function
3,00
M&E outputs - Presentation 3
Effective use of M&E at central level 5
Effective use of M&E at local level 1
VI. Use of M&E outputs: b) Accountability Function 1,50 Effective use of M&E outputs for Upward
Accountability 1
Effective use of M&E outputs for Electoral Accountability
2
Effective use of M&E outputs for Social Accountability 2
Effective use of M&E for Intrastate Accountability 1
