Upload
rachaelbeaven
View
535
Download
0
Embed Size (px)
Citation preview
Development of a Basket of Conflict, Security and Justice Indicators
Eric Scheye, Independent Consultant Diana Chigas, CDA Collaborative Learning Projects
May 29, 2009
i
Table of Contents
I. Introduction II. HMG Evaluation Needs
III. Conflict, Justice and Security Indicators:
Summary and Explanation A. Uses and Limitations of the Indicators B. Categories of Conflict, Justice and Security Indicators
B.I. Differences between conflict and justice and security indicators B.II. The need for reformulation of conflict indicator categories
B.III. Framework for conflict, justice and security indicators C. Criteria for Selection of Indicators
IV. Strengths and Weaknesses of the Indicators
A. Undertracking of Important Factors B. Misleading Indicators
V. Strengths and Weaknesses of the Data VI. Matrix of Conflict, Justice and Security Indicators
A. Assumptions Underlying the Choice of Indicators B. Using the Indicators C. Matrix of Indicators
A. Conflict Intensity B. Institutions/Mechanisms that Handle Grievances
B.I Capacity of Government to Deal with Grievances B.II Government Legitimacy and Accountability B.III Human Rights and Civil Liberties
C. Community Initiatives for Peace D. Military and Militarisation E. Justice
E.I Criminal Justice Capacity and Effectiveness E.II Rule of Law and Administration of Justice E.III Access to Justice E.IV Corruption
F. Security Sector F.I Prisons F.II Policing F.III Oversight of Justice and Security Sector
G. Personal Safety and Security
ii
Table of Contents, cont. VII. Conclusion and Recommendations
VIII. Appendices
A. Example: Application of Indicators to Burundi and Nepal, 2006-2008 B. List of Data Sources Used in the Matrix C. Matrix of Countries Covered by Indices/Data Sources D. List of Interviewees
1
Report:
Proposed Basket of Conflict, Security and Justice Indicators Eric Scheye and Diana Chigas
May 29, 2008
I. Introduction
HMG and DFID acknowledge their need “to be better at monitoring, evaluating, demonstrating and communicating the impact and value of its conflict, security and justice activities and programming”1 The need to measure more systematically the effect of programming resides at a number of different levels: global, regional, country, and programmatic. Each of these levels, however, targets a different audience and addresses different needs, which often are disparate and do not necessarily coincide or overlap. Consequently, it is not possible to construct a single, comprehensive set of measures for evaluation that answer all of HMG and DFID’s needs and are grounded upon “a short list of indicators that can be backed up by credible, [existing] datasets.”2 As a result, this report concentrates only on those conflict, justice, and security indicators that are “measurable at country level”3 so as “to help staff monitor trends”4 and the overall direction in which a country is moving.5
More specifically, this report recommends baskets of indicators that are derived from the “most commonly available global and regional datasets,” which reflect a selected number of conflict, justice, and security categories, including, but not limited to:
• conflict and armed violence; • levels of exclusion, participation, representation; • criminal justice; • access to justice; • oversight of the security sector; • qualitative perceptions of safety and security; and • violence against women.6
In developing the baskets of indicators, the authors interviewed a number of DFID and HMG staff concerning their needs for and experience with indicators. The purpose of the interviews
1 Terms of Reference, 30 January 2009, p. 1. 2 Terms of Reference, 30 January 2009, p. 1. 3 Terms of Reference, 30 January 2009, p. 2. 4 Terms of Reference, 30 January 2009, p. 1. 5 The recommended indicators do not directly measure state fragility, though it may be possible to extrapolate from the indicators and reach certain conclusions as to a country’s fragility. 6 Terms of Reference, 30 January 2009, pp. 2-3.
2
was to understand better the target audiences for the indicators, as well as needed uses and abilities to interpret indicators. We also examined a range of existing datasets to develop the baskets of indicators to determine the validity, timeliness, and accuracy of the data available. Throughout the report, the baskets of indicators have been constructed by extracting selected measures from pre-existing datasets. The resulting baskets, therefore, are composed of discrete indicators, which when combined provide multiple perspectives on a particular category. The original intent was to assemble baskets that would measure outcome, process, financial considerations, perceptions, and activity. That objective, however, could not be attained because of an overall paucity of data from existing databases. In a number of cases, individual indicators are, at best, proxies to measure the stated phenomenon, which is noted when appropriate in the comments section of the category/indicator matrix.
II. HMG Evaluation Needs A small number of DFID, MOD and FCO staff, both in headquarters and in the field, were interviewed to elicit their needs, experience with indicators, and ability to interpret them. The interviews revealed a variety of views regarding the organization’s priority needs for conflict, justice and security indicators. Depending upon the responsibility and perspective of the interviewee, the stated need was global, regional, country, and/or programmatic. It is safe to conclude that there is no consensus on either HMG or DFID’s priorities. In the opinion of the authors it is also unlikely that such a consensus is attainable or desirable. It may be most prudent to develop different evaluation products to address the wishes and needs of the varying audiences. Most interviewees emphasized the need to “measure impact,” though how impact was defined varied depending upon the stated purpose. The purposes ranged from, inter alia:
• programmatic outcomes for country offices; • sustainability of programmes; • submissions to thematic or regional pools of funding; • demonstration of impact on the ground to the British public; • establishment of baseline against which normal programming trends in fragile or
conflict situations can be ascertained; • measurements of mandatory DFID processes such as PSAs and CGAs; to • compilation of reports for parliamentary oversight.
A significant group of interviewees identified the need for country offices to measure progress and monitor trends at a higher level than at the programme or sectoral level. In addition, these interviewees noted that once programmes are designed, and, in particular, once goals are well-
3
defined, it is not veryexceedingly difficult to specify indicators to help evaluate their effectiveness. Moreover, as one person noted, the existing datasets do not have enough “granularity” to be useful at a country programme level. Those who identified the country office level as a priority also stated a desire to have a menu of indicators, from which they could choose in the design and implementation of actual programmes and which could give country offices an idea of what a good indicator would look like. There was general recognition that a single set of indicators cannot meet everyone’s needs. Choices will have to be made concerning the level and uses of these indicators. It was suggested that those choices could be reached, reflecting efforts of other donors and organisations, such as the Oslo Democracy Centre/UNDP effort to provide guidance on indicators of fragility and conflict and DFID/HMG’s own initiative to develop indicators for accountability, transparency, fragility, etc. All agreed that it would be wise to avoid a proliferation of indicators and important to build consensus with other organisations. III. Matrix of Conflict, Justice and Security Indicators: A Summary and
Explanation The following section outlines an approach to the development of baskets of indicators for conflict, justice and security, summarizes the categories and indicators propose, and identifies their most appropriate uses. There are also a number of challenges to the usefulness and robustness of the recommended baskets of indicators as measures of effectiveness. Attempts have been made to address these challenges in the criteria and process for selecting the discrete indicators, in the groupings of indicators into baskets, and in the variety of perspectives and methodologies included for measurement.
A. Uses and Limitations of the Indicators In the conflict and justice and security, Tthe range, level and nature of indicators provided in existing datasets suggest that they may be appropriate as high level indicators to monitor broad trends and the direction in which movementing is occurring, such as deaths in conflicts, fractionalization, overcrowding of prisons, days a court case requires, etc. Valid uses of the indicators, therefore, would include:
• To assess broad trends and directions of a country and region; and • To assess the impact of DFID and HMG strategies, policies and assistance on the
situation (peace, security, justice) “writ large.”
4
Caution should be exercised in using the indicators for the following:
• Country comparisons. There is limited utility in comparing these gross numbers across countries and regions, given the incommensurability of the data and the requisite need to interpret the data. Individuals may be displaced because of conflict or typhoons and a gross statistic cannot discriminate between causes. Prisons may be overcrowded due to a police service’s misguided arrest regime, an inadequate bail system, and/or an overloaded prosecutorial agency. Moreover, many of the datasets rank countries one against another. This may have some ‘quick glance’ usefulness to determine which countries are the most violent, conflictual, and fragile, but rankings are not prudent methods of understanding the country context and the direction in which a country is progressing or deteriorating. The advancement of a country up a ranking list may indicate an improvement in that country’s situation. It is equally likely, however, that it silently bemoans the deterioration of the situation in the country or countries that had been ahead of it the list the year(s) before.
• Programme or project evaluation. Existing datasets are not useful for use at the programme or project level, for a number of reasons;
i. The indicators are highly aggregated, especially in the conflict area. Many different
indicators are combined into one score, and because there is no consensus both within the field or across countries on which observable traits combine for measuring a concept, use of the indicators at the programmatic level is problematic.
ii. There is also aggregation with regard to ethnic group, sex/gender, race, etc., which makes discrimination and interpretation challenging, at best.
iii. There are significant gaps in data and thematic areas covered, as explained later in this report. These include: the activities of non-state actors, private security actors and civil society and community-based actors and local government. Many of these activities receive significant donor support, and thus could not be covered by the indicators. Even for programmes focused on governance or government capacity to provide justice and security, the lack of data in these areas makes it difficult to understand the impact of those activities on justice, security or peace “writ large.”
iv. Public perception surveys exist, but their coverage is scanty and spotty. There is little consistency in the countries covered or the questions probed. Timeliness is an addition challenge. As a result, the existing datasets over-rely on expert opinion, and have little data that could help measure impacts of programmes on the intended beneficiaries.
v. Proxy indicators are often used, making it difficult to assess the relevance and validity of the indicators in specific contexts. They may be valid in some contexts, but not in others.
5
vi. The indicators used in existing datasets do not permit attribution of progress to specific programmes or projects, as they measure the elements of the justice, security or conflict system, but not causes of improvements.
vii. The time delays in availability of data may make the existing datasets less useful for programmes of short duration, lasting two to three years.
viii. The degree of contextualisation needed to determine relevance and significance of the indicators makes these baskets less useful at the programmatic or project level.
This does not mean that the baskets of indicators presented here are irrelevant for programmes or projects. They can be used to assist in monitoring and evaluation of programme and project impacts, but only as supplemental evidence and then only when carefully done. When deployed to evaluate a programme, these baskets of indicators must be bolstered by specific country-level information that is directly related to the initiative under examination. A country-wide pre-trial detention number may hold little relevance, for example, unless reinforced by statistics that trace, over a number of years, the length of days in jail for those who have not yet been sentenced. The baskets need to be understood and used in light of guidance outlined in the OECD DAC Guidelines for Evaluation of Conflict Prevention and Peacebuilding Activities (working draft):7
• Conflict and context analysis. Conflict/context analyses do not need to be of the depth required by the SCA. An analysis of the specific drivers of conflict, as well as those affecting justice and security in the country, will help country offices and programme staff identify which indicators are most relevant and whether other indicators are needed or data gathered. For example, in Georgia, an analysis of the conflicts in that region would suggest that indicators related to fairness of elections or governance effectiveness would not be very relevant to assessing impact on conflict. By contrast, levels of grievances would be particularly useful. Conflict and context analysis would also assist in the determining attribution of impacts to DFID/HMG programmes by permitting the identification of various factors that may contribute to escalation or mitigation of conflict.
• Examination of theories of change. The OECD DAC Guidelines recommend that
evaluations of conflict prevention and peacebuilding activities identify and test the theory(ies) of change and implementation of the intervention—the the programme’s assumptions and logic about how and why it will work to change the conflict or situation and how the program activities will lead to the desired outcomes and impacts. This is important for selection of the appropriate indicators that will allow for testing of that theory in the particular context, as well as for consideration of whether any progress (or
7 OECD DAC Network on Conflict, Peace and Development Co-operation and the DAC Network on Development Evaluation, Guidance on Evaluating Conflict Prevention and Peacebuilding Activities (Working Document for Application Period) (Paris: OECD, 2007).
6
lack thereof) can reasonably attributed to DFID/HMG’s programmes. For example, a capacity-building programme for the courts in a post-conflict country may be based on the theory that improved rule of law will reduce tensions by building a minority’s confidence in the government and by providing a viable avenue for addressing grievances. The programme would want not only to examine indicators of criminal justice effectiveness and access to justice, with a focus on minority perceptions, but to look at indicators related to grievances and legitimacy of the state, also disaggregated by demographic group. This would likely require additional information related an examination of the kinds of cases being resolved in the courts (i.e. are they addressing issues at the heart of grievances that drive conflict?).
• As country indicators and not as a means to evaluate programme outputs and outcomes. As mentioned above, the indicators in the baskets outlined here do not directly refer to programme outputs or outcomes. They are country level indicators. In the example above, the programme would need to establish and monitor indicators related to the quality and outcomes of the capacity-building (e.g., are the beneficiaries gaining and using knowledge and skills; have behaviors changed; are new structures and processes being put into place and utilized; etc.). The next step would be to assess whether those changes instigated by the programme have contributed to improvements in the overall situation. The indicators supported by the datasets can be helpful to the latter question, but the diversity of contexts, needs and programmes make it impossible to establish relevant programme indicators.
The indicators presented below and in the Matrix should also be used and interpreted with care in light of the aforementioned problems and weaknesses inherent in the existing datasets. They therefore should be used with caution as guides for policy.
B. Categories for Conflict, Justice and Security Indicators The terms of reference proposed four categories of conflict prevention and peacebuilding indicators, in addition to measurement of “death/injury rates” as a result of conflict and armed violence:
• formal peacebuilding structures (including elections, parliament, human rights, media); • informal peacebuilding structures (traditional mechanisms, peace committees, mediation
capability, etc.); • qualitative peacebuilding (attitudes/ perceptions between groups, of state/non-state
structures, etc.); and • social cohesion. In the justice and security field.
7
For justice and security, the TORs requested indicators in: • rule of law; • security sector; and • personal safety and security.
The full framework and indicators for conflict, justice and security follows in Section VI.C. of this report. This section provides an explanation of the framework of categories of indicators and the criteria for selecting indicators and grouping them into baskets. (i) Differences between conflict and justice and security indicators. Differences in the fields, respectively, of conflict and peacebuilding and justice and security necessitate different approaches to the development of indicators. In the justice and security field, a limited number of indicators can be used to gauge the overall “health” of the system. The characteristics of a functioning system and the impacts on the population are relatively well-defined in number, especially if conceived as measuring processes across the individual institutions, agencies, and organizations of the sector, though how the provision of justice and security is ‘experienced’ by citizens and residents cannot be overlooked. Indicators relating activities and resources (by courts, police, prisons) to results (objective and perceived) can reveal salient features of the system. For example, the rate of conviction per arrest may highlight relevant information as to the effectiveness and efficiency of the criminal justice system, especially if correlated to various public perception data. While the indicators may not disclose the causes of any strengths and weaknesses or may hide possible systemic ethnic discrimination if not disaggregated, they still provide a set of standards for tracking basic progress. Interpretation is still required as to what specifically the collected data means, but the indicator is a good starting point, will raise pertinent questions, and is highly suggestive of what a country office should explore further .
In the conflict field, by contrast, there is no generalized theory of conflict. There is also little consensus on a standard set of variables and drivers of conflict. In any particular context, different variables can indicate strength or weakness. For example, successfully holding free and fair elections are often viewed as improvement in a conflict situation and a step toward building or strengthening democracy. However, in transitions to democracy, elections can, in some cases, exacerbate conflict. Similarly, government provision of basic services to the population and the strength of the economy are often associated with reduced conflict. In a situation in which the public expects little from its government, however, the lack of effective service delivery by the government may bear little to no relation to conflict. In other words, the indicators must be directly tied to an analysis of the conflict analysis to ensure their relevance. It is also likely that the indicators will change depending on what factors are driving the particular conflict.
The conflict indicators proposed here are designed to track elements of a transition to peace, not necessarily the presence (or absence) of structures and processes characteristic of a “healthy” system, such as elections, checks and balances, government services, GDP per capita, etc. By
8
contrast, the justice and security indicators track the “health” of the system and can be used throughout the programme or policy cycle – from needs assessment, implementation monitoring through to mid-term and end of programme evaluation. Under any and all circumstances, conflict and country analyses are critical for assessment of the characteristics of a “healthy” transition to peace. Different conflicts have different causes, different drivers, and different dynamics. Failure to undertake a conflict analysis may result in the selection of indicators that are not relevant to the conflict. Similarly, it is good practice once indicators are selected, to test the indicators at the time of conducting the baseline to confirm their validity and reliability and to ensure there are not other more valid indicators.
(ii) The need for reformulation of conflict indicator categories. The authors have reformulated the categories and the indicators for conflict from those suggested in the TORs to ensure that issues of impact of interventions can be tracked. The authors deemed this necessary for several reasons. First, several of the indicators/categories in the TORs refer to structures or outcomes, rather than impact. In other words, they do not indicate whether and how these elements reduce conflict, and, therefore, may not be as good indicators of reduction of conflict as they are of the establishment of mechanisms that we assume are good regulators of conflict. Second, many of the indices from which conflict indicators can be drawn do not make conceptual distinctions between statebuilding, peacebuilding, governance and development. Clearly all of these phenomena are related, and activities in all domains—socio-economic, governance, justice and security, and reconciliation and culture8—are needed. They are, however, not all the same. State weakness is not the same as conflict, nor its only cause, even when it may be a contributor to its escalation. Similarly, conflict can be seen as a result, a symptom or a cause of fragility.9
The authors, therefore, have reformulated the categories of conflict indicators to reflect the CDA Criteria of Effectiveness developed in the Reflecting on Peace Practice Project of CDA Collaborative Learning Projects. The Criteria summarize what the Reflecting on Peace Practice Project found to be effective intermediate-level benchmarks for assessing impact on “Peace Writ Large,”
Consequently, the categories of indicators of fragility cannot be equated with those of conflict.
10
8 See OECD DAC Network on Conflict, Peace and Development Co-operation and the DAC Network on Development Evaluation, Guidance on Evaluating Conflict Prevention and Peacebuilding Activities (Working Document for Application Period) (Paris: OECD, 2007), at 18.
or the broader societal peace:
9 Fabra-Mata, Javier & Ziaja, S. 2009 (forthcoming). Users’ Guide on Sources Measuring Fragility and Conflict (Draft 1.0) Oslo: UNDP and German Development Institute. 10 The CoE emerged from an intensive four-year process of collaborative learning and reflection about what makes peace process effective vis-à-vis “Peace Writ Large,” or societal-level peace. The process involved 26 case studies of projects, programmes and groups of programmes and over thirty consultations and feedback workshops with over a thousand practitioners, policy makers, donors and academics to analyse the cases in comparative perspective, identify issues and provide feedback and new experience on tentative conclusions. The initial findings are presented
9
1. Institutions that handle grievances underlying conflict are created or reformed; 2. Beneficiaries and communities develop their own peace initiatives; 3. People (citizens) increasingly resist violence and provocations to violence; 4. People’s security and their sense of security are increased; 5. Inter-group relations, reflected in, for example, changes in group attitudes, public
opinion, social norms and public behaviours, improve meaningfully.
(iii) Framework for Conflict, Justice and Security Indicators. We believe the reformulated categories preserve the essence of the elements suggested in the TORs. Following is an explanation of the reformulated categories and types of indicators. As mentioned above, the conflict indicators are organized around the RPP Criteria of Effectiveness. The justice and security indicators are organized largely by sector or area of activity: criminal justice, civil and administrative justice, access to justice criminal and civil justice, security sector (prisons, policing, oversight), and personal safety. A number of indicators are useful for assessment of effectiveness in peacebuilding and justice and security development. Comments are included in the full matrix of indicators in section regarding how to use the indicators as conflict or justice and security indicators. A. Conflict intensity: Includes deaths, but also other consequences of conflict, including
displacement, which may be source of conflict and instability, as well as non-physical conflict intensity
B. Institutions, mechanisms and process that can and do address grievances, inequalities,
injustices, etc. that cause and fuel conflict.
B.I. Capacity of government to deal with grievances. These indicators measure: • government “effectiveness” (can the government can develop and implement
policies that serve the needs of the population?); • leadership culture and process (how do the values, relationships and interactions
of leadership fuel or mitigate conflict?). It should be noted that these are proxies, and look at capacity, not results. In a fragilel state, there is likely to be little capacity. There also may be limited expectations of service provision from such a state. Consequently, a “low” score on effectiveness may not signify a worsening of conflict. Similarly, a “high” score on capacity does not necessarily mean that conflict is improved.
in Mary Anderson and Lara Olson, Confronting War: Critical Lessons for Peace Practitioners (Cambridge, MA: CDA, 2003).
10
B.II. Grievances. This category measures the results of government policies and practices. It also looks at the degree to which group grievances exist that may fuel conflict, including data, to the extent possible, on actual inequalities and perceptions of unfair treatment and inequality.
B.III. Legitimacy and accountability of government. This category also relates to the government’s capacity to deal with grievances, focusing on the relationship between citizens and the state to provide a proxy for the availability of non-violent means to address conflict. It includes measures of:
• elections to assess the representativeness and legitimacy of the government; and
• perceptions of the legitimacy of the state. This latter variable is a proxy for the degree to which people are likely to look to or accept the state to handle grievances and promote their well-being. Combined with the indicators under B.I., these measures could be an assessment of a state’s fragility.
B.IV. Human rights and civil liberties. These are indicators of the availability of peaceful
means to deal with grievances. They are also basic indicators of rule of law, one of the ways in which conflict and justice and security measurements can be somewhat knitted together. While B.III measures structural (elections) and perceptual (legitimacy of the state) aspects of this element, the degree of respect/repression of human rights and civil liberties assesses government behavior.
C. Community/Popular initiatives for peace. These indicators are intended to measure the extent to which there are local capacities for peace in the non-governmental sphere and to what extent do communities, citizens groups and others develop their own initiatives for peace?
D. Resistance to violence or provocations to violence. Military capacity and societal
militarization indicators look at the degree of militarization of society at governmental level, but are best considered to be proxies of violence and the tendency to utilise violent means. Because it is difficult to obtain data on non-state armed groups, we have used the accessibility of small arms and the number of deaths by small arms as proxies for the level of activity of non-state actors.
E. Rule of Law and Access to Justice
E.I. Criminal Justice Capacity and Effectiveness. Capacity and effectiveness are
measured from three perspectives:
11
• government activity and results in criminal justice, including data on whether the system is capable of dealing with the level of crime present;
• popular perceptions of effectiveness of criminal justice; and • resources (financial) dedicated to criminal justice, as a process indicator of
government commitment and action.
E.II. Civil and Administrative Justice. It is insufficient to examine rule of law with reference only to the criminal justice system. Consequently, this section looks at a set of wider issues revolving around judicial independence, perceptions of fairness, impartiality of decisions and judgments, and the police operating under civilian control. It should be noted that there is a dearth of reliable indicators that reach beyond the criminal justice system.
E.III. Access to Justice: State and Non-State Systems. Access to justice indicators include:
• capacity (does the government have the capacity to provide adequate access?); • access (is access being provided, or are costs, government restrictions of rights or
provision of services undermining access?); • perception (popular perceptions and experience of frequency with which state and
non-state systems are accessed); and • women’s access and capacity.
E.IV. Corruption. It is necessary to measure corruption as a phenomenon that can impede
the delivery of justice and security. At the same time, corruption can be a cause of conflict, as a grievance, and one of its many by-products. There are many ways of measuring corruption as a perception, an activity, and as a sign of governmental ineffectiveness.
F. Security Sector
F.I. Prisons. Measurements on prison capacity are ascertainable, but they do not indicate
the treatment received by those who have been incarcerated. Pre-trial detention is a challenging issue, but there is no method of directing assessing and pinpointing its causes or ways of measuring that will suggest how to reduce the severity of the challenge.
F.II. Policing. As with criminal and civil and administrative justice, the basket of indicators provided is intended to provide three perspectives on policing: a) capacity; b) behavior and effectiveness (homicides, extrajudicial killings), as well as perceptions of the police; and c) financial resources/commitment.
12
F.III. Oversight of Justice and security sector. Appropriate oversight of the actions and
institutions of the sector is a key component of democratization of the justice and security system. It is a means by which good and effective governance can be identified. It is, however, difficult to evaluate given existing datasets. The institutionalization of corruption is one proxy by which to measure oversight, given the premise that embedded corruption circumscribes the government’s ability to conduct adequate oversight. Similarly, appropriate oversight may be a means by which conflict can be addressed before its onset or thereafter.
G. Personal safety and security. Measures perceptions of the effectiveness of justice and
security sectors broadly, and of efforts to mitigate conflict. If the data could be disaggregated by conflict group, these would serve as more precise conflict indicators, as, in their current form, they do not distinguish between sources of insecurity nor which populations feel more or less secure. Most appear not to do so, except perhaps in the case of the perceptions surveys, where at least the Afrobarometer asks respondents about their identity/group, and thus in theory disaggregation by ethnicity, religion or group (as well as gender, etc.) would be possible. An adequate proxy for increase in security/sense of security related to the conflict might be obtained by looking at conflict intensity, state behavior violating physical integrity and perceptions of security, in particular concerning fears of political violence, fears of reprisals for political involvement.
C. Criteria for Selection of Indicators Within the categories, indicators were selected based on a number of criteria. Some were identified in the TORs, including:
• Geographical coverage, i.e. data is available for as many countries as possible; • Time coverage and update, i.e. data is available over time and is regularly updated; • Validity and reliability, i.e. the indicator measures what it is supposed to measure as
much as possible, and is replicable; • Sources of measurements that are sensitive to short-term change (to the extent possible);
and • Consistency with other international indicators, to the extent possible and valid.
The primary criterion for constructing the baskets was to provide a multi-dimensional picture of each phenomenon or category. The objective of these indicators is to move beyond tracking discrete institutional activities and evaluate the process and “the impact of the system as a whole
13
on public safety or the rule of law.”11
For conflict, justice and security, this implies that at least three distinct elements – activity/process, outcome, and beneficiary perception -- need to be taken into account to determine the impact of a project or programme. Consequently, clusters or baskets of indicators are required. The issue is not only what each individual indicator may be, but how the basket as a whole functions and is balanced. As much as the existing datasets permitted, the baskets were constructed based on the following considerations:
• To measure structure/capacity (e.g., legal frameworks, rights, human resources relative to the scale of the problem, etc.;
• To measure process (efforts to achieve outcomes, e.g. budgets dedicated to criminal
justice, military expenditure as % of GDP, etc.); • To measure outcome (changes in the conditions in the country with respect to conflict,
justice or security).
• To balance more objective (e.g. # homicides) and subjective (perceptions of insecurity) measures of the phenomenon. For conflict indicators, data on perceptions of people inside the country is especially important as a measure of impact, in that it reflects how changes in institutional, structural or political factors are viewed and provide information on whether these changes may lead to less conflict behavior.
• To minimize explicit or implicit adoption of a particular ideology or theory of change for conflict indicators.
IV. Strengths and Weaknesses of the Indicators
The existing datasets of conflict, justice and security indicators have several limitations that should be taken into account as they are used. Any DFID follow-up on these indicators should take their limitations into account as well.
A. Undertracking of important factors Several important categories of indicators are not tracked sufficiently or at all by existing datasets. This significantly affects the usefulness of the existing datasets for measurement of conflict, justice and security. The gaps include:
11 Todd Foglesong and Christopher Stone, Measuring the Contribution of Criminal Justice Systems to the Control of Crime and Violence: Lessons from Jamaica and the Dominican Republic. John F. Kennedy School of Government - Harvard University, April 2007, p. 5.
14
• Non-state capacities, processes and mechanisms for conflict transformation, justice and security. The conflict indices and justice and security indicators focus primarily on state mechanisms. The activities of non-state actors are rarely measured in existing datasets. There is little data on civil society initiatives for peace and equally little information on non-state mechanisms for provision of justice and security. Similarly, there is little to no measurement of private security actors, which suggests that another significant slice of the security arena escapes evaluation. This profoundly affects the ability to measure justice and security, given that a lack of state-provided access to justice is not equivalent to an absence of justice. Similarly, given the amount of service delivery, conflict resolution and peacebuilding work that is done by traditional structures and other non-governmental actors, and the amount of financial and other support given to them (youth, women, CSO and NGOs, media, etc.), this lack of data significantly undermines the ability to understand conflict and conflict resolution and the impact of assistance on conflict, in a country. What information that does exist on these phenomena is found in anthropological, sociological and other more qualitative research, is not readily measurable (or available), and, therefore, cannot be converted into a form consistent with the datasets. It is a significant missing element that would bear on measures of conflict, justice and security.
• Perceptions. Most of the datasets rely on expert perceptions of a particular phenomena –
from issues of inclusion (how inclusive is the governance system) to perceptions of the electoral process to questions of trust and confidence between groups. There are a few existing sources that try to gauge public perceptions, such as the Gallup World Poll, Afrobarometer, Arab Barometer, among others. However, these instruments often do not ask questions that are entirely relevant (valid), or, as in the case of the Afrobarometer, they do not ask the same questions repeatedly over time. Data regarding perceptions of a representative sampling of sub-groups is also scarce because what survey information that does exist may not consistently disaggregate the data. As a result, there is an over-reliance on expert opinion and under-reliance on public perceptions in the existing datasets. To the extent expert opinions are flawed, biased, or reflect a particular world view, the data may not be accurate. Greater measurement of public perceptions would provide direct data on perceptions of actors in and beneficiaries of conflict, justice and security policies and programmes, and permit more accurate assessment of impacts.
• Inter-group relations. Some indicators regarding factionalization (among leadership) and the existence of inter-group tensions (expert opinions) do exist and are useful. However, little data concerning inter-group trust, perceptions, or perceptions is collected systematically. The indicators focus more on generalized trust (do you trust other people?). Moreover, information about how different policies and actions affect differing groups differentially (i.e. data about horizontal inequalities) is not collected
15
systematically. This could be gathered to a large extent by disaggregating data on other indicators (e.g., perceptions of justice and security, conviction rates, inequalities of income, etc.) by ethnic/conflict group. Doing so may be quite sensitive for partner governments in post-conflict situations. However, without robust data on the state of inter-group relations and the ways in which policies affect groups differentially (horizontal inequalities), assessment of impacts of policies, availability of justice and security for vulnerable groups, and programmes addressing conflict will be difficult to evaluate.
• Conflict-specific dimensions of justice and security. Conflict indices have integrated existing human rights, civil liberties, justice and security indicators that were developed to track progress in these sectors to measure peace and conflict. The same is true for socio-economic indicators.12
However, these indicators, by themselves, do not necessarily measure conflict impacts. While the justice and security indicators available (and provided in the matrix below) are applicable to justice and security development, their lack of disaggregating the data makes them imprecise for the conflict impacts of justice and security. For example, as mentioned earlier, an improvement in the rate of convictions per arrest may suggest an improvement in the effectiveness and efficiency of the criminal justice system, yet if the majority of those arrested and convicted are from one ethnic group, this “improvement” may be the source of grievance and conflict. The sectoral indicators can measure underlying structural-institutional capacities for conflict resolution and peace, but without disaggregation by demographic group, or other further information about how they impact conflict actors, they can provide misleading information about conflict impacts.
B. Misleading Indicators. Several aspects of existing datasets can provide misleading information about trends in conflict, justice and security in a country. This is, in part, a consequence of the fact that in most of the categories, the indicators are proxies, rather than direct measures. A number of patterns emerged regarding ways in which the indicators in existing datasets can be misleading: • Ideological/theoretical or normative biases. Most of the peace and conflict datasets are based
on well-defined (although not necessarily explicit) theories of change, or assumptions about the mechanisms by which peace and stability will come about.13
12 Despite contributions of human rights, civil liberties, justice and security to peacebuilding, many of the indices actually are dominated by socio-economic and other governance indicators See F. Barton and K. von Hippel, “Early Warning? A Review of Conflict Prediction Models and Systems” (Washington, DC: Center for Strategic and International Studies, 2008).
In cases where the theory of
13 See Cheyanne Church and Julie Shouldice, The Evaluation of Conflict Resolution Interventions, Part II: Emerging Practice and Theory (Ulster: INCORE, 2003), at 33, for a fuller discussion of theories of conflict, peacebuilding and change. See also DAC Guidance on Evaluation of Conflict Prevention and Peacebuilding at 35
16
change is quite strong (e.g., Index of State Weakness in the Developing World, which defines fragility as the cause of conflict), the indicators potentially measure whether the elements of the theory of change are present, but may not reflect what is occurring in a country with respect to conflict. This is why we chose not to adopt any one of the peace, conflict and fragility indices available, but rather included indicators from several of these indices.14
A similar normative or theoretical bias can also result from the selection of indicators. For example, a country peacebuilding strategy that focused on improving economic and social policy capability or government service delivery might choose indicators related to government effectiveness. This selection of indicators might track whether the government overall is becoming more effective in service delivery, but would not track whether the implicit theory of change – that improved service delivery will reduce conflict – is valid in that context. In this case, additional indicators related for example to group grievances or the legitimacy of the state would be useful to add to mitigate the theoretical bias.
• Comparisons between countries. The TORs indicate cross-country comparability as a
criterion for indicator selections. A number of problems with cross-country comparison for conflict and justice and security indicators have been discussed earlier in this report. Most of the peace and conflict indices compare and rank countries relative to each other. However, the comparison, especially in light of with the inclusion of highly developed democratic countries such as Norway or Sweden included in the dataset, skew ratings and are of limited use. Moreover, comparisons potentially introduce a strong normative bias by defining a country as an underperformer because it does not have the institutional and social characteristics of the developed democracies.15
Finally, as already noted, comparisons between countries may mislead the user if a country rises in rankings not due to its own progress, but rather because of the decline of countries previously ranked above it.
In the justice and security sectors, where statistical information is more prevalent, cross-country comparisons are dangerous. For example, European data suggests that the ratio of prosecutors/100,000 persons in Switzerland is 5.5; in Sweden, 9.9; and in the Ukraine, 21.0. Similarly, the total judicial budget/per capita in Switzerland is .28; in Sweden, .23; and in the Ukraine is .49. Although these statistics may raise important questions about the individual countries’ judicial systems (when combined with a raft of additional data), in and of themselves they provide little valid insight into actual judicial performance and, for
for a discussion of uses of theory of change and implementation theories that elaborate the logic and assumptions of how programmes make a contribution to the larger peace. 14 A useful analysis of the different indices, and their strengths and weaknesses as indices, is being developed by UNDP (the Oslo Governance Center) and the German Development Institute, in a User’s Guide on Sources Measuring Fragility and Conflict. A first draft has been made available for comment. It will be a useful supplement and interpretive guide to these indicators. 15 See Fabra-Mata & Ziaja (2009 forthcoming.)., User’s Guide on Sources Measuring Fragility and Conflict (1st draft) (2009) for more discussion of this point.
17
comparative purposes, are misleading. Cross-country comparability is particularly difficult with respect to criminal justice data, given wide differences in police accessibility, known variability in what constitutes a criminal act, inconsistent legal definitions in how crimes are recorded, and vast discrepancies in the rates at which crime is reported. It may be more possible to compare victimization rates across countries, but even these measures remain susceptible to some of the aforementioned challenges.
For justice and security development, the establishment of broad guidelines and standards could, at best, be used as generic goals for countries might be more useful. For instance, the UN has suggested a ratio of approximately one police officer/450-500 persons. While that statistic may be a rough and perhaps idealized goalpost, its applicability to any one country would need significant modification to account for differences in geography, communication and transportation systems, population demographics, education, GDP, urbanization, etc. The same can be said for similar justice and security guidelines, such as:
• personnel costs comprising 70-80% of total police budgets (US); • patrol units comprising 60-65% of total police budgets (US); • training costs comprising 5% of total police budgets (US); • personnel costs of judge and court staff comprising 65% of total judicial budgets
(Europe); • court buildings and premises capital costs comprising 15% of total judicial budgets
(Europe); and • court fees contributing 10% to total judicial budgets (Europe).
The most suitable indicators for cross-country comparison may be those based upon perceptions and institutional/systemic processes, each indicator coinciding with the process and outcome of an activity as it is experienced by the intended beneficiaries. For these reasons, the baskets of indicators chosen for justice and security development try to incorporate as often as possible such data.
• Use of Proxy Indicators: Over-inclusiveness and Misleading Conclusions. The difficulty in
direct observation of improvements of conflict, justice and security (especially conflict) has meant that many indicators in existing datasets and in the proposed baskets are proxies.16
16 For example, leadership acceptance of democratic institutions is a proxy indicator for legitimacy of democracy and for the potential of elections to support peaceful transition rather than conflict. The GNI Coefficient/Uneven development is a proxy for concentration of wealth and horizontal inequalities. In the rule of law, the number of days required to resolve a dispute is a proxy for possibility of corruption, and attorney/court/enforcement costs a proxy for access to state justice systems.
In addition, many of the datasets identify conditions that may provide a good environment for conflict management and prevention and justice and security development, e.g., high GDP per capita, economic performance, governance according to democratic structures and
18
processes, improvement of government services, reduction in poverty, etc. However, while these factors may correlate highly with peace, they are not necessarily indicators of improved conflict prevention or peacebuilding. Furthermore, it cannot be assumed that conflict has been reduced if there have been general improvements in the economic, governance and justice/security indicators, even if these are important structural factors. In a number of instances—including elections, increased GDP, increased openness of political competition, etc.—improvements may also bring about increased conflict in the short- to medium-term, which could undermine long-term development and governance. Similarly, some of the indicators may indeed be good warnings of a worsening situation, but not of an improving situation. For example, deteriorating provision of public services, included in the Failed States Index, may indeed be an indicator of conflict and state failure. However, the reverse may not indicate improvement of the conflict situation. Caution should be exercised in interpreting baskets of indicators.
The justice and security indicators, as other sectoral indicators, are important supplements to the more general conflict indicators provided here. They can help country offices and headquarters hone in on particular dimensions of the peacebuilding process. However, the indicators for justice and security development, similar to those for the socio-economic dimensions of conflict, are not necessarily “conflict-sensitive.” In other words, one cannot measure the impact of justice and security on conflict using these indicators alone. Indicators on human rights and civil liberties also lack this conflict sensitivity. For example, if conviction rates in a particular country increase, this may be interpreted as improvement in the justice system. If, on the other hand, a significantly greater number of one group is convicted than another, this reputed general improvement may have negative repercussions for conflict management. Similarly, numbers of homicides may go down, indicating an improvement in personal security of the population. However, if one group remains disproportionately affected by homicides (or other crimes), then an improvement in this indicator could not be interpreted as an improvement in conflict. Disaggregation of crime, justice and security indicators by social/ethnic/religious group, therefore, increases the validity of these indicators as conflict indicators. Disaggregation also increases the utility for determining how to improve justice and security service delivery. For both conflict and justice and security development, the broader picture can be obtained by including perceptions of vulnerable groups about their treatment by their government, group inequalities and other potential grievances.
• Aggregation. A number of datasets, such as the World Governance Indicators, and a number
of the peace and conflict indices, aggregate indicators from a number of different sources. Aggregate indicators have a number of advantages. They may be more accurate in that they provide more precise evaluations of a situation than individual indicators. However, for the aggregate governance indicators, the margin of error appears to be large, so that small
19
changes in the indicators over time may not be statistically significant. Therefore, it is important to balance the use of aggregate and individual indicators, as has been attempted.
• A General Caution Regarding Interpretation. For all the indicators, whether conflict, justice
or security, caution needs to be exercised in determining what conclusions should be drawn from movements in any one indicator, a series of indicators, and/or in any category of indicators. This is particularly important if these indicators are to be used in country offices for programming purposes, which, as already discussed, is not recommended unless supplemented by a contextual analysis and other data. The indicators have been blended into baskets for a number of reasons, only one of which is to tackle a category from multiple perspectives. Equally importantly, broad conclusions should not be derived from the movement of any single indicator. For instance, an increase in trust in the police is, frequently, associated with an increase in crime rates as more victims of alleged crimes may lodge police reports. This rise in reported crime may not occur, however, if the reason the police have not been alerted to alleged crimes is due to the public’s generalized belief that the police are ineffective. Similarly, pre-trial detention statistics are notoriously difficult to interpret. A reduction in detention numbers, which may at first blush be considered a progressive movement, but may, upon further research, indicate that police have changed their arresting policy. Raw movements in re-trial numbers, therefore, are not be able to shed light upon whether that change was for the good. Reductions in pre-trial detentions may also be the result of increased bond provisions, which, in turn, if not coordinated with the local neighborhoods, may trigger various forms of extra-judicial killings and lynching. The dangers of misinterpretations of movements in indicators are equally great in the conflict arena. For example, a decrease in inter-ethnic physical violence may reflect better security, an improvement in the conflict situation, or that fewer acts of violence are sufficient to sustain. The latter conclusion was reached in a study of violence in Kosovo prior to the 2004 riots, when intimidation tactics had shifted from personal assaults to destruction of property intended to persuade minorities to leave. The same study also found that minorities’ increased movement outside their enclaves, which was interpreted by most as a sign of increased feelings of security on the part of minorities and improved inter-group relationships, was, in fact, a reflection of minorities’ confidence in peacekeepers and their increased knowledge of where they would be protected as they moved.17
Interpretations of movements in the indicators should be tested and verified against other indicators and based on a rigorous analysis of the conflict, justice and security situation.
17 D. Chigas et al., What Difference Has Peacebuilding Made in Kosovo? (Cambridge, MA and Prishtine/Pristina: CDA and CARE, 2006).
20
V. Strengths and Weaknesses of the Data
A number of issues regarding the types of data and measurement of indicators from the datasets should be highlighted:
1. Availability of data. It should be noted that for many of the datasets, information is not readily available and thus may be of limited short- to medium-term use in guiding country programmes. While some datasets are updated yearly, many are not, and often it takes up to a year for the collected data to become publicly available. This implies a significant time lag in the availability of data for assessment, severely limiting the utility of indicators for monitoring programmes and adjusting them to events.
2. Consistency. For perceptions data in particular – such as the Afrobarometer, the Arab
Barometer, and the Asian Barometer – there is also an issue of consistency. The same questions are not asked in each country or each round of a particular country’s surveys. Given the paucity of perceptions data generally, this is of particular concern.
VI. Matrix of Conflict, Justice and Security Indicators
The matrix presents baskets of indicators designed to assess country trends in conflict, justice and security so that DFID/HMG can better evaluate the impact and value of its strategies and assistance. The indicators were developed using existing datasets. The source of the data and scale for each indicator is contained in the second column of the matrix. More information, including geographic and time coverage, methodology and sources of the data for the indicators, is available in the Appendices B and C. The matrix includes brief comments on appropriate and inappropriate uses and on strengths and weaknesses of most indicators as well.
A. Assumptions underlying the choice of indicators The conflict, justice and security indicators were developed based on several assumptions and principles, in order to enhance their accuracy and usefulness. • No single theory or perspective of conflict. Most of the peace and conflict datasets are based
on well-defined, although not necessarily explicit, theories of change, peace and stability In cases where the theory of change is quite strong (e.g., Index of State Weakness in the Developing World, which defines fragility as the cause of conflict), the indicators measure whether the elements of the theory of change are present. Such indices and indicators, however, may not reflect what is occurring in a country with respect to conflict. Consequently, the baskets of indicators do not adopt or rely on any single index, but rather include indicators from several of these indices, in order to avoid implicit or explicit adoption of a particular theory of peace or conflict.
21
• Distinction of fragility and conflict. Underlying the choice of indicators is a distinction
between concepts of state fragility and conflict. While fragility and conflict are clearly related, they are not the same thing. There are many definitions of fragility, but the OECD’s characterization may best capture the essence of the concept -- state structures lacking political will and/or capacity to provide the basic functions needed to reduce poverty, stimulate development, and safeguard security and the human rights of their populations (OECD 2007). Conflict, by contrast, refers to real or perceived incompatibilities between needs, values and interests, primarily at the societal level, which have the potential to become or have become violent. In this sense, fragility can be the result, the cause, or a symptom of conflict. Therefore, some indicators related to fragility are included as indicators. However, they cannot be used alone as indicators of conflict or potential conflict.
• The importance of viewing phenomena from variety of perspectives. The baskets of indicators
represent a variety of perspectives on an issue, such as criminal justice, government effectiveness or grievances. They include indicators on structure, dynamics and outcome. They attempt to balance objective (# homicides) and subjective (perceptions of insecurity) measures. This ought to provide a more accurate picture of the phenomenon and underlines the fact that no one measure is adequate. It should also be noted that many indicators are proxies for the phenomenon they are intended to measure and this is highlighted in the uses and comments columns.
B. Using the Indicators
As noted above, the matrix concentrates on conflict, justice and security indicators that are measurable at country level to help staff monitor trends and the overall direction in which a country is moving. The indicators should be used in light of the cautions outlined in section III.A. above (Uses and Limitations of the Indicators). For purposes of interpretation, it is useful to presume a relationship between and among the indicators. It is partially for this reason that the baskets of indicators have been developed and organized to minimize the time required to collect data. At the same time, DFID and HMG’s strategy or assistance programme may focus on a particular sector, set of issues, or dimension of conflict, justice or security. A country programme may provide budget support and wish to assess whether the resources contributed to progress. In this example, use of the entire set of indicators would make sense. Care also needs to be exercised to avoid normative or theoretical bias that can result from the selection of too narrow a set of indicators. For example, a country peacebuilding strategy that focused on improving economic and social policy capability or government service delivery might choose indicators related to government effectiveness. This selection of indicators might
22
help to track whether the government is becoming more effective in service delivery. It would not track whether that improved service delivery reduces conflict and promotes peacebuilding. Indicators related to group grievances or the legitimacy of the state would be useful to add to mitigate the theoretical assumption that improved service delivery reduces conflict. Similarly, if a country wishes to assess trends in justice and security development as they relate to conflict, it would insufficient to focus only on justice indicators. If data disaggregated by group were not available, conflict indicators on grievances, inclusion, perceptions of unfair treatment, etc. would need to be examined. In this case, the combination of justice and conflict indicators would not directly assess a reduction in conflict due to improvements in justice. Rather, they could provide a proxy for determining whether general improvements in criminal justice provision also ameliorate the conflict situation.
(i) Selecting “baskets”: some criteria
a. Use a range of indicators. Too many may be burdensome and reduce focus; too few will provide insufficient or misleading picture of what is going on.
b. Include measurements of different aspects and different perspectives. Given the paucity
of perceptual indicators and data, perception surveys should be included wherever possible. They provide an important check of whether actions, policies and strategies are having desired impacts and whether outsider observations or measures correspond to people’s perceptions.
c. Avoid overreliance on one kind of source. The indicators rely on a number of sources: expert/narrative reports, events/numbers, surveys, media analysis, among others. Although it may not be possible to diversify sources, overreliance on one source will skew the analysis and introduce potential biases.
(ii) Interpreting the indicators
For all the indicators, as has been previously discussed, caution needs to be exercised in determining what conclusions should be drawn from movements in any one or several indicators, or in any one category of indicators. A few simple questions can serve as a guide to interpreting the indicators and deriving a coherent analysis of trends.
a. What is the story the movement of indicators tells? The story should be about trends and progress in the country under examination. It is not about comparisons with other countries or about the state of the country at a particular point in time. Deciphering the direction in which a country is moving may be difficult, particularly where indicators
23
have large margins of error; small changes in any one given year are statistically insignificant; and time delays or dataset coverage makes timely data unavailable. Understand what the indicator can tell you and what it cannot. For example,
government effectiveness can portray what the government may be capable of doing. It does not reveal what the government actually does. GINI provides information about the distribution of wealth, but does not disclose the relevance of that distribution to conflict. The homicide rate bespeaks of the number of people killed, but does not indicate whether lowering the murder rate is the priority need voiced by the local population.
Be careful about attributing meaning to what the indicator says. Can be misleading.
b. Is the story accurate and relevant? Relevance should refer to the analysis of the country
context, conflict and/or justice and security situation. Accuracy takes account of the degree to which the indicators faithfully represent the phenomenon they are supposed to measure.
c. What are gaps and weaknesses in the story that the indicators tell? Inevitably there will
be gaps in the story indicators tell, particularly given the lack of reliable data on non-state justice and security mechanisms and the service provided by private security companies. Other gaps may exist because of poor perception data. Still others may arise due to the normative biases of the indicators. Still others may become apparent because data is aggregated and analysis cannot be conducted into the services provided to different groups, which is crucial for a conflict analysis. The issue is to be aware of these gaps and seek ways to address them. It will be important to analysis what the absence of data implies and how the story may change if data were to become available.
d. Are there contradictions in the story that the indicators tell? Under each and every
circumstance, the contradiction(s) requires explanation. Does the contradiction reveal a lack of internal consistency within a category of indicators or between categories? Does the contraction disclose a lack of consistency between individual indicators? Or is the apparent contradiction disguising an issue for which more analysis is needed? For example, when government effectiveness improves, grievances may also increase. Launching a programme to address violence against women will, very likely, increase the reported incidents of such violence. When trust in the police strengthens, complaints about alleged police misbehavior and malfeasance may spike up. An initially more accessible state justice system could lower the satisfaction the citizenry have in the performance of the state justice system because the demand for justice outstrips its supply.
24
e. What is the minimum story that needs to be told? In justice and security development, the minimum story should outline the priority needs of the citizens and residents of the country and, if possible, their perceptions of the services delivered to them and by whom those services are delivered. In conflict, the minimum story should consider conflict intensity, leadership culture/process and grievances. The other factors are clearly relevant, but focus on socio-economic, governance, electoral or security factors will vary depending interest and relevance; they should be interpreted, however, in conjunction with the three “minimum story” elements.
f. What does the story leave out? Existing datasets have significant gaps in important kinds
of indicators. Specifically, perceptions data is often missing or there are large gaps in coverage. Country offices can fairly easily obtain this data through surveys or, if resources are an issue, staff (DFID or others) can go out to listen and ask questions to people, making sure to travel outside the capital. In addition, data on non-state mechanisms for conflict resolution, peacebuilding and provision of justice and security is missing, as is data on private security. These too would need to be collected in-country. Finally, the indicators that exist for socio-economic dimensions of peacebuilding, for governance, development and for justice and security, are largely not conflict sensitive. It is difficult to understand the conflict impact of improvements in justice systems, for example, from the indicators and the data that is currently available, or from the indicators of personal security. Measures of social capital (“trust”) also usually refer to generalized trust rather than amongst conflict groups. To the extent that disaggregated data can be disaggregated by group, this can provide conflict-sensitivity. A much less valid alternative is to interpret that data alongside indicators of grievances and conflict intensity.
UNDP, in its User’s Guide for governance indicators, advises that indicators should be among the first questions asked; they should not the last. For additional questions on indicators, there are several useful resources that can be consulted, in addition to DFID’s own, including:
• UN User’s Guide for Governance Indicators • Forthcoming User’s Guide on Fragility and Conflict Indices • DFID Guidance on Governance, and forthcoming on Voice and Accountability
25
Matrix of Indicators
A. Conflict Intensity Indicator Source Use Cautions
1. # deaths from armed conflict
Uppsala Conflict Data Project
Direct measure of level of civil war-type violence. Uppsala defines armed conflict as “contested incompatibility that concerns government and/or territory where the use of armed force between two parties, of which at least one is the government of a state, results in at least 25 battle-related deaths in one calendar year.”
There is a difference between deaths attributed to conflict and fatalities from other forms of violence, mainly criminal. The number of deaths from non-conflict causes far exceeds deaths from war. Consequently, very violent societies may be low on this indicator.
2. # displaced as % of population
UNHCR http://www.unhcr.org/ statistics/45c063a82.html
This should supplement # of deaths in order to capture the severity of the consequences and the level of fighting.
Persons displaced may have had to leave their homes for reasons other than conflict, such as hurricanes, drought, typhoons, etc.
3. Territory affected by conflict
Political Instability Task Force
Measures the scope of conflict. If a conflict is confined to a particular geographical area of the country, its severity may be considered to be lower, even if its intensity in that area is high.
4. Seriousness of ethnic, religious, social conflicts
Bertelsmann Transformation Index (BTI) Question 13.3 http://www.bertelsmann-transformation-index.de/ fileadmin/pdf/Anlagen_BTI_2008/ Alternate: World Governance Indicators, “Political Stability and Absence of Violence,” http://info.worldbank.org/ governance/wgi/index.asp
Serious conflict may exist without deaths. This indicator captures the severity of polarization, mobilization of people, and the potential for violence. The WGI “Political Stability and Absence of Violence” reflects a much broader set of stability concerns – from the possibility of military coup to political assassination and terrorism – and is primarily concerned with the stability of the government. BTI is a more narrow measurement of the intensity of social cleavages. Choice of the Bertelsmann vs. WGI indicator should be made based on the context. In countries where political, criminal or terrorism-based violence are of primary concern, WGI may be more appropriate, as well as if short–term stability concerns are paramount.
BTI is a less reliable indicator, in that experts’ assessment of seriousness or “irreconcilable” cleavages may differ. However, it is a more valid measure of the seriousness of polarization. WGI is less up-to-date than BTI. Additionally, while the WGI includes the Bertelsmann measure, it may be too broad to measure conflict intensity. The WGI could be useful to as a cross-check and verification. If the two measures are not similar, then further analysis may be warranted.
26
Indicator Source Use Cautions
5. Neighborhood War
Peace and Conflict Instability Ledger, University of Maryland
Not a direct indicator of conflict intensity in a country. It represents an exacerbating factor that can prolong and intensify conflict, as well as compound conflict consequences (e.g., displaced).
Most useful in regions in which there are ongoing and/or simmering conflicts, such as in the Great Lakes region. Can also provide data regarding risks of flows of refugees triggering movements in underlying societal cleavages.
B. Institutions/Mechanisms/Processes that Handle Grievances B.I. Capacity of Government to Deal with Grievances The capacity includes capacity to develop and implement policies that address grievances Effectiveness
Indicator Source Use Cautions
1. Governance Effectiveness World Governance Indicators http://info.worldbank.org/ governance/wgi/index.asp
This measures the overall quality of public service provision, bureaucracy, competence and independence of the civil service, inter alia. It reflects the quality of the “inputs” for the government to be able to produce and implement policies that would be needed to address (or prevent) grievances.
Aggregate indicator that includes many dimensions of effectiveness. Increases or decreases in the indicator will not provide sufficient information from which to draw concrete conclusions about the impact of interventions. In addition, small changes are likely to be statistically insignificant, unless viewed over a number of years.
2. Efficiency of revenue mobilisation
World Bank Country Policy and Institutional Assessment (CPIA) Criterion 14
Measures tax policy and tax administration. A proxy for the availability of state resources to implement policies and provide services that might address grievances.
While tax mobilization is a common measure of state capacity, it is also problematic and should be relied upon with caution. There is little reliable tax data in fragile states and the validity of this measure outside the developed countries is limited. States can rely on alternative sources of revenue (e.g., natural resources or other national industries).
27
Indicator Source Use Cautions
3. Extent to which government can coordinate conflicting objectives into coherent policy
Bertelsmann 15.2 http://www.bertelsmann-transformation-index.de/ fileadmin/pdf/Anlagen_BTI_2008/ Detailed_Results_BTI2008.xls
The capacity of those in government to formulate policies and develop consensus. It implies as well the extent to which the government is or is not affected by factionalization and/or polarization, which are important elements of a country’s capacity to handle grievances.
A low capacity here may reflect stalemate over how to deal with grievances or an inability to put into place policies and mechanisms to address problems.
4. Access to improved water sources
Alternatives: Human Development Index Primary school enrollment (% gross) Youth unemployment
UNICEF/WHO Global Water and Sanitation Report; World Development Indicators Human Development Index: UNDP Human Development Reports http://hdr.undp.org/en/ statistics/data/ Primary school enrollment and youth unemployment: World Development Indicators, WDI Online.
We have chosen access to improved water because it is a key government service, one which requires concerted effort and planning, important for human development (not just economic growth) and often destroyed during conflict. If this could be disaggregated by geographic region, it would be helpful to assess inequalities as well. Updated data may be difficult to obtain on this indicator. Consequently, the composite human development index, or primary school enrollment might serve as alternative proxies. Outside assistance may contribute to increases in these indicators, so they are not fully valid as indicators of government effectiveness. Where youth unemployment data is available, employment would be a useful indicator, in light of evidence
Many socio-economic indicators have been proposed and are used in the conflict indices. Infant mortality has been preferred by several that have found a correlation with conflict. However, improvements in infant mortality may result from outside intervention, even during conflict. Education would be an alternative indicator (mean years of schooling, or primary school enrolment), but, if there is significant displacement, education may be provided by churches, non-governmental organizations or in refugee camps. In both cases, there may be a correlation with conflict, not because of issues of government effectiveness, but because conflict affects the government’s ability to provide services and also affects access by humanitarian and development actors to provide the service. These measures could be used as indicators of conflict, but caution should be exercised in attributing
28
that youth unemployment enhances risks of conflict.
improvements to an improved conflict situation. These indicators should be viewed in conjunction with deterioration of public services (E.II.9), which has more consistent and recent scoring.
Leadership culture/ process This is an aspect of capacity, but has been separated out because of its importance to tracking conflict. Additionally, leadership is an oft-forgotten component of development policy, which needs to be more systematically taken into account. These indicators look at the degree to which the values and interactions of leaders/elites in governance: a) may or may not escalate political conflict; and b) may or may not support the government’s ability to acknowledge and deal effectively with grievances that may underlie conflict. Three aspects of leadership are assessed: degree of polarization/factionalization, degree of commitment or belief of leadership in democratic processes (a proxy for likelihood of abiding by democratic rules), and degree of inclusiveness of leadership/governance.
Indicator Source Use Cautions/Comments
5. Factionalized Elites Failed States Index/Fund for Peace (1.11) http://www.fundforpeace.org/web/ index.php?option=com_content&task =view&id=99&Itemid=146
Measures fragmentation of elites along group lines and degree of polarizing behavior of elites and leadership. A direct indicator of elite relationships amongst elite groups. It is also a proxy for a power struggle within the elite, polarization, and/or manipulation by elites, all of which has the potential to escalate conflict.
6. To what extent can political leadership manage political cleavages?
Bertelsmann Question 16.3
Relevant where there are deep societal cleavages along ethnic, class, regional or religious lines. Measures the capacity of leadership to bridge these cleavages. If B.I.5 is high, indicator is redundant; use if factionalization is not high.
Although relevant and valid as one measure of bridging social capital, its reliability is problematic. Terms such as “depolarization” and “consensus across the dividing lines” can be interpreted differently.
7. Regime/governance inclusion (Political legitimacy score)
State Failure Index, Center for Systemic Peace, George Mason University based on Polity IV
Measures inclusiveness of the governance process -- the degree to which minority groups (social, ethnic or religious) are included in governance.
A composite drawing on 4 indicators: Factionalism (Polity IV), Ethnic group political discrimination (Discrimination 2007), Political salience of elite ethnicity (Elite Leadership
29
characteristics), polity fragmentation. As an aggregate, it has all the problems with such types of indicators.
8. Extent to which democratic institutions are accepted or supported by relevant actors
Bertelsmann Question 4.2
Use in conjunction with indicators on elections and governance. Provides some triangulation for the elections’ capacity to support peaceful transition rather than greater conflict. Intended to measure the strength of “veto powers” or spoilers of democracy, it can also provide data on the degree of key actor “buy-in” to democratic processes.
Concepts of “relevant actors” are vague and may affect reliability of indicator.
Grievances These indicators attempt to look at the effects of government policies and practices on the grievances that can cause conflict. Several indicators of potential inequalities that may be sources of grievance and evidence of unwillingness to address them are included here. These should be seen in conjunction with indicators measuring perceptions of unfairness and grievance. In conflict situations, it is the perception of grievance that matters more than the objective measure of the grievance itself. The the nature and severity of grievances, and their potential to lead to violence, will naturally differ country to country; consequently, these indicators should be selected and viewed in connection with an analysis of conflict, if they are to be relied upon in a country setting. If grievances are worsening, it is likely that the government does not have mechanisms in place to deal with them.
Indicator Source Use Cautions/Comments
9. Group grievances
Failed States Index, Fund for Peace (I.3)
Mostly measures behavior vis-à-vis communal groups that could lead to grievances around which these groups could mobilize, including atrocities against groups, political exclusion, public scapegoating, “hate” radio, etc.
Because FSI relies on media material, data related to institutionalized political exclusion, atrocities or persecution may be under- or overweighted. The potential measurement errors should be balanced by the relevance of the indicator, one of the few in existing datasets, measuring this dimension of conflict.
10. Minority access to education, commerce, professions, etc.
Minorities at Risk, University of Maryland http://www.cidcm.umd.edu/ mar/data.asp
This and the following two refer to specific minority rights that are often areas around which grievances are mobilized. For use if relevant to the context.
They may or may not be relevant to a particular country situation. Caution should be exercised in reading the data, as the data is coded by minority group; where there are minority-led governments, the “majority” group should be considered.
30
Indicator Source Use Cautions
11. Restrictions on religion Minorities at Risk, University of Maryland
12. Restrictions on use of language or language instruction
Minorities at Risk, University of Maryland
13. GINI Coefficient
World Development Indicators Should be viewed in conjunction with uneven development (B.14) or group grievances (B.9) to provide a picture of the effects of inequalities of wealth on conflict.
14. Uneven Development Failed States Index/Fund for Peace (1.5)
A measure of horizontal inequalities that could fuel conflict.
Combined with the GINI coefficient could provide a proxy for the degree to which concentration of wealth is group-based and may impact on group relations.
15. How often has your group been treated unfairly by the government?
Afrobarometer Ghana 2008 survey, Question 8218
Experiential measure of sense of exclusion/oppression. Complements more “objective” measures of inequalities and group grievances.
If used in conjunction with human rights and civil liberties indicators, this public perception data can provide some indication of whether human rights/civil liberties restrictions may be a source of group-based grievance.
16. Your group has same, worse, or better economic conditions than others
Afrobarometer Ghana 2008 survey, Question 11
Experiential/perceptual measure of “horizontal inequalities.” Complements more “objective” measures of inequalities.
Used in conjunction with more standard development indicators (GNP per capita, e.g.), this public perception data can provide perspective on whether economic growth is likely to promote or mitigate conflict.
18 All Afrobarometer sources refer to the 2008 Ghana survey, unless otherwise noted.
31
B.II. Governance Legitimacy and Accountability Elections free and fair Is the government representative, does it reflect the voice of the people, and is it legitimate in the eyes of the population? The better the electoral process and the fairer it is in the eyes of the population, the more likely it is, theoretically, to have popular confidence to discuss and develop adequate solutions to problems. Strong caution should be exercised in relying only on data related to elections, as free and fair elections can also exacerbate societal and political cleavages. This indicator should be used, at a minimum, alongside indicators of political cleavage and governance inclusion.
Indicator Source Use Cautions/Comments
17. Electoral Process and Pluralism
Economist Intelligence Unit Democracy Index after 2006 Freedom House Freedom in the World, Political Rights A (Electoral Process) and B (Political Pluralism and Participation) http://freedomhouse.org/ template.cfm?page=15 (Tables and Charts)
Measures how free and fair elections are, at the national and municipal levels, as well as whether citizens can vote fairly. Should be triangulated with perceptual data from B.II.18. EIU Democracy Index was created in 2006. Freedom House has earlier data and permits separate examination of electoral process-related trends and trends in political participation and pluralism; EIU combines both.
EIU index is scored on 1-10 based on expert assessment of questions cross-checked with the World Values survey when feasible.
18. Confidence in honesty of elections Alternative for Africa/Arab: “Fairness of last national election?”
Gallup World Poll Afrobarometer Ghana 2008 survey, Question 71, Arab Barometer Q 211
Measures popular perception of the fairness of elections. For African and Arab countries covered, Barometer questions are more specific and recent. These surveys should be substituted when available.
Current year’s data available; historical data available with subscription.
32
Legitimacy of the State Perceptions of the legitimacy of the state are an important supplement to data and perceptions of elections. Important as it measures the extent to which people are likely to look to the state and have confidence in the state to handle grievances and promote their well-being.
Indicator Source Use Cautions
19. Delegitimization of state
Failed States Index/Fund for Peace (I.7) Alternative: State Fragility Index, Political legitimacy and Social legitimacy scores
Direct indicator of popular confidence in the government. Proxy for the capacity of government to respond constructively to grievances and popular confidence in the government’s ability to deal with grievances. Looks at the degree of delegitimization of the state, including elite corruption, resistance of ruling elites to political representation, loss of popular confidence in state institutions (e.g., election boy-cotts, public demonstrations, inability of state to collect taxes, etc.) and linkage of crime syndicates to ruling elites.
Delegitimization of the state does not necessarily lead to conflict, as the “revolutions” in Georgia and Ukraine suggest. It may, however, reflect background conditions that, in combination with other factors, could result in conflict, such as in Timor-Leste and the 2006 crisis.
20. Relevant groups agree about citizenship and accept the nation state as legitimate?
Bertelsmann 1.2 Relevant in the event of possible separatist conflict, where the unity of the nation state is in question. Also can be used in conjunction with group grievances, as a component of the indicator looks at the extent of discrimination against groups.
“De facto” exclusion of groups from political citizenship may be susceptible to differing interpretations by experts and, consequently, the indicator may not be entirely reliable. The fact that the indicator covers two important concepts – support for the nation state and discrimination – makes it very relevant, but the inclusion of two related by separate concepts makes it imprecise.
33
Indicator Source Use Cautions
21. Public support for government restrictions on civil liberties a. Agree or disagree with statement: "Government should be able to ban any organization that goes against its policies." And the reverse statement
b. Agree or disagree with statement: "Government should be able to close newspapers that print stories it does not like." And the reverse statement
c. Agree or disagree with statement: "Government should not allow the expression of political views that are fundamentally different from the views of the majority." And the reverse statement
Afrobarometer Ghana 2008 survey, Question 19 Afrobarometer Ghana 2008 survey, Question 20 Afrobarometer Ghana 2008 survey, Question 21
Use in conjunction with indicators on human rights and civil liberties and/or deligitimization of the state. Provides interpretive data on the significance for conflict of human rights and civil liberties restrictions and delegitimization of the state: is there public support for repressive actions by government? If so, the repressive actions or corruption in the government may not be a source of conflict and improvements in those indicators may not signify any impact on conflict.
Limited coverage, as only Afrobarometer includes these questions. To be an accurate interpretive measure, these indicators should be disaggregated by group. Afrobarometer does ask about group identity in its surveys. Although the published survey results are not disaggregated, disaggregation should be possible.
22. Control of Corruption World Governance Indicators Corruption reflects on the legitimacy of the state, but there is no direct relation to conflict. If there is serious concentration of wealth, and factionalization of elites (B.I.5. or 7), this indicator may be relevant to assessing the degree to which capture of state resources is a driver of conflict.
34
B.III. Human rights & Civil liberties These are indicators of rule of law, as well as a proxy for the potential for the extent to which peaceful mechanisms for handling grievances exist. We have specifically excluded physical issues here; these have been placed under perception of safety and security.
Indicator Source Use Cautions/Comments
23. Civil liberties EIU Democracy Index Freedom House Freedom in the World D (freedom of expression/ belief), E (associational/ organizational rights), and G (personal autonomy) for greater disaggregation and/or data prior to 2006.
Mostly freedom of media-related and freedom of expression data, but also includes questions about mechanisms to redress grievances, security, and whether property rights are protected.
This may be over inclusive as a conflict indicator, and should be interpreted in conjunction with other indicators of horizontal (group-based) socio-economic, political inequality, and/or grievances.
24. Suspension or Arbitrary Application of the Rule of Law and Widespread Violation of Human Rights
Failed States Index, Fund for Peace (9)
This is an aggregate of four descriptive sentences, ranging from emergence of authoritarian/ dictatorial rule to rising number of political prisoners.
Alternative is State Repression Index from Political Terror Index; FSI is more comprehensive.
25. Allowable advocacy activities CIVICUS Civil Society Index 2.5.2 Looks at extent to which CSOs are free to engage in advocacy and criticize government. A proxy for the availability of non-violent means for civil society actors to have voice and address grievances. In combination with data under C. below, provides an impression of the feasibility and existence of civil society initiatives to promote peace.
This indicator does not provide any data on existing local capacities for peace, only the existence of space for action by local, non-state actors. It does not include traditional mechanisms or processes for conflict resolution, which may, in some contexts, be significant. It is also very limited in its coverage of countries.
35
C. Community Initiatives For Peace Community/Civil society initiatives for peace This looks at civil society initiatives for peace: to what extent are there local capacities for peace? In general, there is very little data on local, non-governmental initiatives or capacities for peace that is systematic and covers a wide geographical area. The existing datasets provide very little, the indicators below have very limited geographic coverage, and do not address the range of issues for which indicators are needed.
Indicator Source Use Cautions/Comments
1. Civil society actions to promote non-violence and peace
CIVICUS 3.4.2 Measures extent of civil society activity and advocacy for peace/non-violence.
Note there are similar indicators for gender equity and democracy. Alternative or additional indicators would be CIVICUS 3.3.2 on how much civil society promotes tolerance at the societal level. Qualitative data, unlike most of the other indicators in conflict, justice and security.
2. Activity and success of civil society in influencing public policy in human rights
CIVICUS 4.1.1 Approximates potential influence and effectiveness of civil society on peace issues, premised on the assumption that human rights, like peace, is a sensitive issue.
36
D. Capacity/Willingness to Resist or to Escalate Violence Military and societal militarization The indicators in this category reflect opportunity to fight/military feasibility of war and justice and security development
Indicator Source Use Cautions/Comments 1. Military expenses as
% of GDP The Military Balance, Stockholm International Policy Research Institute
Expenses, numbers of armed forces, and security officers/police give an indication of the resources dedicated and available to the government to engage in conflict. In light of the difficulty in determining the resources available to non-state actors, these are used as proxies for the opportunity and willingness to fight as it is assumed that a government will increase its capability to respond to threats of armed violence.
Many security expenditures -- military, intelligence, police -- may be off-budget. Consequently, this may not be an accurate reflection of expenses.
2. # armed forces per 100,000 people
The Military Balance, SIPRI
A proxy for resources available to pursue conflict. In conjunction with D.3 and D. 1, assesses the level of resources available for conflict.
To the extent that international and/or regional (i.e. cross-border) conflict is an issue in a region, this may not be a good indicator of resources available for internal conflict. Caution should be exercised in interpreting trends in D.1-D.3. Upward trends may reflect intensifying conflict, but may also reflect greater outside threats to security, as well as threats from crime.
3. # security officers and police per 100,000
10th Crime Trends Survey, UN Office on Drug and Crime
Similar to D.2, it is a proxy for resources available to pursue conflict.
Have been unable to obtain definition of “security officers” from UNODC.
4. Ease of access to small arms
Global Peace Index (collected by EIU)
The availability of small arms is a proxy for the ability of non-state groups to engage in violence and conflict. Ease of access to small arms, however, also pertains to violence that can be committed that has nothing to do with conflict.
This is an expert assessment and, as a proxy, cannot be considered reliable. However, as there are no other measures of capacity of non-state groups to engage in violence and conflict, this is a default measure.
37
Indicator Source Use Cautions
5. Intentional homicide committed by firearm, per 100,000
UNODC 2.6 Measure of level of violence. More valid for security development than as a measure of conflict, given the distinction between violence pertaining to conflict and other, usually criminal, activities.
Have been unable to obtain a meaningful definition of “intentional” from UNODC. Even if a consistent definition were to have been obtained, there will be little consistency across countries in how “intentional” is determined by the responsible authorities. Nevertheless, it is the best measure available.
6. Non-violence within the civil society arena
CIVICUS 3.4.1 Measures use of nonviolent means by civil society to express their interests in the public sphere.
Does not fully measure the propensity to use violent/non-violence means in the non-governmental sphere, as it is limited to civil society organizations. However, in the absence of other data, it can provide an indication, which should be interpreted in light of other contextual knowledge. The limited coverage of the data, however, further erodes the indicator’s utility.
38
E. Justice E.I. Criminal justice capacity and effectiveness The indicators in this category reflect the overall structure, capacity, and effectiveness of the criminal justice system and its component parts. Activities and results
Indicator Source Use Cautions
1. Conviction rate (total # of persons brought before the court/# total number of persons convicted)
UNODC 9.1 This is a process indicator, measuring the effectiveness of the overall criminal justice system, from police through prosecution and courts. It may be the best single non-perception statistic, although interpreting what the data means will require additional analysis and investigation.
Almost all UNODC statistics can be broken down by age, gender, type of crime. The organization’s indicators, however, cannot be broken down by group. Consequently, these indicators are useful as the starting points, but would need further specification if they are to be valuable for conflict analysis. UNODC statistics are compiled from reports filed by governments with the UNODC. There statistics, therefore, are as reliable as the original government database, which may be questionable. Coverage of UNODC’s statistics varies too according to if a government files a report.
2. # of recorded crimes per 100,000 people
UNODC 2.0 Is the standard measure of crime rates around the world, though admittedly a notoriously unreliable one.
Reported crime is a dubious indicator, given that a preponderance of crime goes unreported. At the very least, reported crime rates need to be balanced by victimization studies. It is preferable if reported crime rates are also interpreted in light of trust/confidence in the police and indicators that assess the effectiveness of the entire criminal justice process, such as E.1.
39
Indicator Source Use Cautions
3. # convictions/# police UNODC Q11/UNODC 1.1
Measures the efficiency of the police. A low score should lead to questions about the overall management and police effectiveness. A low score may also be indicative of a poor relation between the police and the prosecutorial service. Conversely, a high score may suggest that the police are addressing only low level crimes, one for which obtaining a conviction is relatively easy.
Cannot be used without reference to other indicators, namely E.1 and E.4. Also best used in conjunction with police satisfaction and victimization surveys.
4. # convictions/# prosecutors UNODC Q11/UNODC 5.0
Measures the efficiency of the prosecutors. All these efficiency statistics can and should be used together.
As with E.3., this indicator needs to be placed within the context of the overall criminal justice system and cannot be used as a stand-alone measure.
5. # prosecutions/# prosecutors UNODC Q7/UNODC 5
Measures the workload of the prosecutors and may be indicative of their capacity, particularly suggestive of their being underresourced. Needs to be analyzed alongside E.4.
Care should be used with this indicator because it is possible that prosecutors have work rules that determine the number of cases they handle.
6. # convictions/# prosecutions UNODC Q11 / UNODC Q7
This is another measure of the effectiveness of the criminal justice system.
The significance of this number can be interpreted only in conjunction with the other criminal justice indicators, in particular E.I.1. E.!. 3, 4, and 6 are complementary.
7. # prosecutions/# judges UNODCQ7/ UNODCQ8
Measures the workload of judges. To be used in conjunction with E.5.
Criminal justice is only one type of case heard by the judges. This capacity statistic, therefore, is best used when the judicial system conducts its work in specialized courts.
8. Average # months of criminal trials
UNODC Q 9.2 Measures the efficiency of the judicial system. If the average # of months appears to be excessive, it would be a leading indicator from which to probe further.
There should be no assumption that the speed with which a criminal trial is dispatched implies that it is more or less fair and equitable. Adherence to rule of law principles cannot be measured by length of a trial. Therefore, it must be used in relation to the other indicators.
40
Perceptions
Indicator Source Use Cautions
9. How well or badly would you say the current government is handling the following matters, or haven’t you heard enough to say?... F. Reducing Crime (1 = very badly… 4 = very well)
Afrobarometer 57F
Perception question useful to reveal the attitude of the population to government policy. It may be suggestive of the population’s confidence in the government and criminal justice system.
It suffers from the same weaknesses as all other indicators derived from the Afrobarometer – consistency, timeliness, etc.
10. Who do you think has primary responsibility for managing each of the following tasks - national government? local government? traditional leaders? Members of community? None of them? Don't know.
Afrobarometer 58 E and H
Perception question meant to begin to assess the relationship between state-provided and non-state provided justice and security. It is only a proxy because there may be a discrepancy between who is believed to have primary responsibility and to whom individuals go when they have a dispute.
There are very few indicators that address the range of non-state justice and security issues. Hesitancy should be used in its interpretation with respect to the distinctions between local government, traditional leaders, and members of the community, as there may be significant overlap between and among these levels.
11. In your opinion, how often… do officials who commit crimes go unpunished?
Afrobarometer 45 E and D
Measure of perceived impunity of government officials and is one measure of adherence to rule of law principles.
Perceived impunity may be an important indicator for conflict if it were susceptible to disaggregation.
12. In your opinion, how often… do ordinary people who break the law go unpunished?
Afrobarometer 45.E
Measure of perceived impunity of the citizenry and is one measure of adherence to rule of law principles.
Should be used in conjunction with E.10 to evaluate the perceived impunity of government vis-à-vis citizenry.
41
E.II. Rule of Law: civil and administrative justice
Indicator Source Use Cautions
1. # days to resolve dispute World Bank Doing Business
Though an increase in number can suggest an increased possibility of corruption, this indicator is, at best, a proxy. An increase in the number may also be suggestive of overall government capacity. World Bank Doing Business are perception surveys of 181 economies and are administered to local business leaders, lawyers, professionals involved in regulatory activities, and government officials. Because the surveys have been conducted since 2003 and are administered annually, they are among the most reliable and consistent perception surveys.
In general, World Bank Doing Business surveys are not meant to be comprehensive measurements of corruption, governance, or adherence to rule of law. Nevertheless, because they are administered to selected segments of the population, they are suggestive and should be used in conjunction with other indicators to provide a more rounded assessment. It should also be noted that the segment of the population to which these surveys are administered is a vital group, even if it is not vulnerable or marginal one.
2. # steps to claim, obtain, enforce contracts
World Bank Doing Business
As with E.11.1, an increase in number can suggest increased possibility of corruption. This indicator, however, has the added value of being a proxy for adherence to rule of law, given that one of its components is the enforceability of contracts.
This indicator is adherence to the rule of law from the perspective of those engaged in regulation and business in the country’s most populace city.
3. To what extent does an independent judiciary exist?
Bertelsmann 3.2 An independent judiciary is one of the principal criteria of rule of law. Bertelsmann uses country experts to write and evaluate so that this indicator may not be as useful as perception surveys.
It is important to note that judicial independence is only one principle among many by which rule of law is evaluated. There is discussion among justice development practitioners whether judicial independence is the judicial paramount principle, but it remains an open question.
42
Indicator Source Use Cautions
4. Court system fair/impartial? Law and regulations predictable and consistent
World Bank Enterprises Survey
Perception from business leaders, which, albeit a limited constituency, is an important one.
5. Should the President always obey the law?
Afrobarometer 37 This is a perception question whether it is believed that the President is beholden to the law. In this sense, it is a proxy for popular belief in the rule of law, one of major principles of which is that no one is above the law. Again, however, the difficulty is the phrasing of the question as “should” cannot be equated to “does.”
Should be used in relation to questions of impunity above.
6. How much do you trust… Courts of law?
Afrobarometer (49H), alternatively, see Gallup World Poll for Asia
In addition to being about “trust” per se, it is also presumed that “trust in the courts” implies a belief that the decisions rendered by the courts are fair, impartial, and independent. Hence, this is also a proxy indicator.
If it could be broken down by group (which for Afrobarometer may be possible), could give an indication of group grievance/perceptions of exclusion.
7. Does the rule of law prevail in civil and criminal matters? Are police under direct civilian control?
Freedom House, Freedom in the World F2
Freedom House indicators are based upon the opinion of country experts. This indicator is an aggregate of seven separate questions, ranging from the political independence of prosecutors, detainees access to legal counsel, to law enforcement officials independent from non-state actors (criminal enterprises, influential players and others).
This indicator may be best used in relation to E.II.3, as both are administered by country experts. Whether the police are under civilian control is a primary principle of democratic governance.
8. Do laws, policies, and practices guarantee equal treatment of various segments of the population?
Freedom House. Freedom in the World F4
This indicator is an aggregate of six separate questions, ranging from different groups’ ability to exercise their human rights to whether violence against such groups widespread.
This is one of the indicators that bridges justice and security measures and conflict indicator.
43
In addition to measuring an important criteria of rule of law (equality before the law), this indicator is a proxy for exclusion, possibly factionalization. In conjunction with other indicators, may help to understand conflict dimension and impacts of rule of law.
E.III. Access to Justice: state and non-state systems
Capacity
Indicator Source Use Cautions
1. # police per 100,000 UNODC 1.1 (Rate) Standard measure of police capacity used around the world.
The UN has promulgated that a rough standard would be roughly in the range of 1 police officer per 450 persons. However, the standard does not take into account education levels; difficulty of terrain roads, and communication; equipment/logistics possessed by the police; poverty rates, and age distribution of the population, all of which will affect the needed police manpower.
2. # prosecutors per 100,000 UNODC 5.0 (Rate) Measure of prosecutorial capacity. Should be used in conjunction with other capacity and efficiency indicators. The capacity of the prosecutorial service is suggestive of whether a state-provided court system can meet a country’s short- to intermediate-term needs or whether there ought to be donor support directed to non-state/local justice networks.
As with other capacity indicators, this does not measure activities or outcomes. It may be suggestive, however, of whether the prosecutorial service is understaffed. It can be eliminated, if process data is ascertainable – E. 1, 4, 5, and 6.
44
Indicator Source Use Cautions
3. # lawyers per 100,000 Law Schools / Bar Associations
The number of lawyers in a country is an important indicator of whether a state-provided court system can meet a country’s short- to intermediate-term needs or whether there ought to be donor support directed to non-state/local justice networks.
There is no dataset that contains this statistic, but it ought to be readily obtainable by consulting the national Bar Association or through the country’s law faculties. Caution must be used in analyzing the gross number because there is no certainty that each and every lawyer practices the law.
4. # hours to pay taxes, # payments per year
World Bank Doing Business This is a proxy indicator under the assumption that if both increase, businesses will pay fewer taxes and funding for state-provided justice will be limited, compelling people to resort to non-state system.
To be used with caution and mainly as a supplemental or supporting indicator.
Access
Indicator Source Use Cautions
5. Attorney, court, enforcement costs as % of claim value
World Bank Doing Business
The high cost to obtain access to state system will compel people to resort to non-state system.
This is not a direct measure and should be used with caution, but, once again, it is an attempt to uncover the relationship between state and non-state justice.
6. Government repression of groups
Minorities at Risk If government engages in repression, equitable access will, by definition, be limited. This indicator is an aggregate of three questions on repression.
This is not a perception survey, but is scored by graduate students and reviewed by experts.
7. Deterioration of public services
Failed States Index/Fund for Peace (8)
This indicator is a proxy, for it is presumed that if public services deteriorate justice and security services do as well. Also an indicator of government effectiveness and grievances; can be used in conjunction with B.I.1, B.I.4 and B.I.14 to gauge conflict potential of socio-economic indicators.
The indicator is estimated based upon readings of open source media from the country under examination.
45
Women’s access & capacity
Indicator Source Use Cautions
8. # women police/total police UNODC 1.2/UNODC 1.1
This is an estimate of gender equity in the police. However, as a measure of representation, this indicator does not record equity in power and authority.
Access to justice for women is not limited to their employment representation, but no other data is available.
9. # women prosecutors/total prosecutors
UNODC Q5.1/UNODC Q5.0
Same as above. Save as above.
10. # women judges/total judges UNODC Q8.1/UNODC Q8.0
Save as above. Save as above.
11. # women prison staff/total prison staff
UNODC (14.1+14.4)/UNODC (14.3+14.0)
Save as above. Should be comparable to # women prisoners/total prisoners (available from UNODC or King’s College).
12. Women as % of legislators, senior officials, managers
IMD World Competitive Yearbook 2.5.10
This indicator is a survey of business executives, which is thereafter supplemented by statistical data, when available. Though the question is one of representation (gross number), it is suggestive of access and equality.
This source is extremely limited (only 55 countries), but it is the only dataset found that estimates % of women in positions of authority and influence.
Perception
Indicator Source Use Cautions
13. Trust in courts vs. trust in traditional leaders (how much do you trust each?)
Afrobarometer (Compare 49H vs. 49I)
All below are perception surveys, measuring to whom people turn to for justice.
The comparison of trust in courts vs. traditional leaders is suggestive of which mechanism the citizenry turns toward to address their disputes. Should be used in conjunction with other perception data to disclose discrepancies, if any.
14. Use of wasta (clout)(During the past five years, have you ever used wasta to achieve something personal, family-related, or a neighborhood problem?)
Arab Barometer 228 This is a proxy question and not a direct one
15. Reference to traditional leaders (asks to list the types of wasta – choice 4 is
Arab Barometer 227
46
traditional leaders)
16. Go to court to resolve disputes? (If you were to have a dispute with another citizen, would you try to resolve it in a court or in another government institution?)
Arab Barometer 228
17. Who would you go to resolve a dispute? Arab Barometer 229
18. Women are/should be subject to traditional laws19
Afro Barometer 23.B This question may or may not be in each country survey and the # of the question depends on which survey it is.
E.IV. Corruption
Indicator Source Use Cautions
1. Do you think the government is doing enough to fight corruption or not?
Gallup World Poll This is not a perception of corruption, but of the government response to it.
May be best used in connection with other impunity indicators. If there are variations between and among them, this would be an issue for further investigation.
2. Corruption perceptions Transparency Int’l - Corruption Perceptions Index
This is a country’s overall score, which is derived from thirteen separate sources, each of which is a different measure of corruption.
3. Registering property - # of procedures, # of days
World Bank Doing Business
The longer it takes and the more complicated it is, the greater the opportunity for corruption and the more likely corruption may occur (especially because of the involvement of property).
As this is a proxy, its use must be judicious.
19 In the 2005 survey, respondents were asked: “In our country, women should have equal rights and receive the same treatment as men do.” B. “Women have always been subject to traditional laws and customs, and should remain so.” Respondents could agree with A, B, or neither.
47
Indicator Source Use Cautions
4. Gifts given to receive public services
World Bank Enterprise Survey
Includes selected questions about electricity, construction, inspections, % government contracts given away, % total annual sales for business, import license, operating license.
This survey is completed by business leaders and is an estimate of their perception of a country’s governance structure and system and the degree to which government services have been effectively privatized.
F. Security Sector F.I. Prisons
Indicator Source Use Cautions
1. # prisoners/# beds King’s College This is a measure of how overcrowded the prison system is.
Overcrowding is a primary way of evaluating the performance of a prison system.
2. # juveniles/# beds for juveniles King’s College/ UNODC Measure of how overcrowding affects juveniles.
3. # pre-trial detention prisoners prior to disposition of case/# convicted prisoners
UNODC/King’s College This is a gross number recording the depth of the pre-trial detention challenge. More useful data would be the average number of months in prison an incarcerated person languishes prior to initial disposition of his/her case, but that data is unavailable.
As a gross number, it does not disclose the causes or reasons why there is a pre-trial detention problem.
4. # persons with AIDS, mental illness, TB/# prisoners
UNODC Q (18.2+18.3+18.4)/ UNODC 15.0
Health is a prime issue in prison and a prime determinant of prison quality. It would be expected that overcrowding and health would, in most circumstances, be closely correlated.
The problem with the indicator is that that data is unavailable on other illnesses, or on how much is pre-existing conditions, and therefore not the result of prison mismanagement.
48
F.II. Policing
Capacity
Indicator Source Use Cautions
1. # police per 100,000 UNODC Q 1.1 (Rate)
Behavior & Effectiveness
Indicator Source Use Cautions
2. # of intentional homicides completed per 100,000 people
UNODC 2.3 (rate) This is the standard measure of homicide.
There is no consistent or workable definition of “intentional,” and none provided by UNODC.
3. In the last 12 months, were you assaulted, mugged, property or money stolen?
Gallup World Poll This is the best victimization proxy available.
Victimization surveys are crucial to balance crime rate data and give a depth of understanding to perception surveys that measure fear and insecurity. Victimization surveys are also important means by which to evaluate questionnaires that look at public satisfaction in police performance.
4. How much trust do you have in… the Police
Arab (201.4), Asian (13) and Afro-barometers (49G)
Trust in the police is a crucial variable and one that ought to be measured, if at all possible.
If (5)-(8) could be broken down by group and income, it would give good indication of the feelings of security among conflict parties. It should be noted that in the first few years after the cessation of a conflict and with the rebuilding of the national police, public trust and confidence in the police often rises. This increase, however, may be more aspirational than real and, often, trust in the police declines sharply thereafter.
49
Indicator Source Use Cautions
5. Get help from police when needed? 20
Afro Barometer 71 (2005)
If police are not forthcoming with help or assistance, then individuals might resort to the non-state system.
In many fragile countries, individuals and groups do not appeal, first and foremost, to the police for help. Instead, they opt to consult their local leaders. In some countries, the police are not accessible, so that it would be inconceivable to request help from the police.
6. Difficulty in getting help from the police?21
Arab Barometer 213
Same as above.
7. Extrajudicial killings CIRI Index Important in itself. Killings may be by police or other government bodies. When the death is caused by other actors, the overall issue remains one that still concerns policing.
A proxy by which to evaluate the effectiveness of policing. There may be a correlation between extrajudicial killings and the entire criminal justice system, but the relationship is not contained within the CIRI Index’s raw score. It will have to be ferreted out.
8. Criminalization/ delegitimization of the state
Failed States Index, Fund for Peace (I.7)
Broader than policing, but does raise policing issues. Looks at acts of resistance to government (boycott elections, resist taxes, protests, etc.)
The extent to which a state is criminalized and/or has privatized the provision of its services is a proxy for the effectiveness of the criminal justice system.
20 This question was asked in the 2005 survey: “Based on your experience, how difficult is it to obtain the following services? Or do you never try to get these services from government? C: help from the police when you need it. (4=very easy, 1=very difficult). 21 The question asked: “Based on your experience, how difficult is it to obtain the following administrative or social services from the government?” One option was 4.” Get help from the police when you need it.”
50
F.III. Oversight of justice and security sector
Indicator Source Use Cautions
1. Security sector operating as a state within a state
Failed States Index, Fund for Peace (I.10)
Looks at emergence of elite guards that operate with impunity, state-sponsored private militias, army within the army.
2. Corruption mechanisms institutionalized
Bertelsmann 15.3 This is a proxy for oversight, given the assumption that institutionalization of corruption implies the absence of effective oversight.
Should be used in conjunction with E.VII.1. This is presumed that they will be correlated. If there is a discrepancy, that would be grounds for further investigation.
G. Personal Safety and Security Perception of personal safety and security In conjunction with indicators from A (conflict intensity), intentional homicide and ease of access to small arms (D. 4, 5; F.II.3), recorded crimes (E.I.2), and policing, these can give an idea of safety and security. Without disaggregation by demographic group, the indicators cannot provide information about conflict-related vs. non-conflict-related security concerns.
Indicator Source Use Cautions
1. State behavior violating physical integrity
Political Terror Scale This is a broader measure than extrajudicial killings.
If these indicators could be disaggregated by group, they would be more helpful as conflict indicators.
2. Feel safe walking alone at night in your community?
Gallup World Poll A perception question measuring insecurity.
Gender and age disaggregation available. Caution must be used, however, in that the term ‘community’ is undefined. One could, for example, feel safe walking within one’s own community, but not in the neighboring one.
3. Experienced fear, stress, worry a lot of the day yesterday?
Gallup World Poll This indicator measures security writ large. It is not specifically measuring physical security.
This indicator borders on human security and should be treated with care.
4. Over the past year, how often, if ever, have you or anyone in your family… B. had something stolen from your house, C. been physically attacked?
Afrobarometer 9B, C This indicator is a victimization question.
51
Indicator Source Use Cautions
5. In this country, how often do people have to be careful of what they say about politics?
Afrobarometer 47 A proxy for psychological security and for conflict-related security fears.
6. In your opinion, what are the most important problems facing this country that the government should address? (one choice is political violence)
Afrobarometer 56 Provides a sense of how important issues of political violence are to people.
7. # reported rapes per 100,000 women
UNODC Violence against women is an underreported crime and rape is a proxy question.
There is a lack of data on violence against women.
52
VII. Conclusion and Recommendations The existing datasets provide a number of indicators that can be useful in monitoring and evaluating the impacts of conflict, justice and security activities at the country level. While not perfect, they provide a means of monitoring and assessing the general direction of a country, and thus can provide a supplemental perspective on assessing the impact of DFID/HMG strategies and activities. They can help answer a number of questions relevant to assessment of impact, including: How has the situation changed over time? Which changes in structures, attitudes, behaviours, relationships or practices (of how many people/classified according to horizontal divisions) can be ascertained? 22
These indicators are not a comprehensive treatment of indicators. Nor do they provide a complete guide for assessment of impact. For more complete guidance on the selection and interpretation of indicators, it is advisable to refer to good practices in selecting indicators, the UNDP User’s Guide for Governance Indicators, and the forthcoming Users’ Guide on Sources Measuring Fragility and Conflict. In addition, the OECD/DAC Guidance on Evaluation of Conflict Prevention and Peacebuilding Activities lays out a useful framework within which these indicators may be used to measure the impact of DFID/HMG activities and strategies. The OECD/DAC guidance recommends several preconditions that help staff and policy makers prepare for, support and learn from evaluations.
Summary: Key steps for policy makers, programme planners and managers 23
Establish relevant, clear and measureable objectives State an explicit theory of change and programme logic Complete and monitor a conflict analysis Develop and monitor relevant indicators Focus on strategy and policy coherence Conduct systematic, rigorous evaluation
Within this framework, there are a number of things DFID/HMG might do to improve the availability of useful and robust conflict, justice and security indicators.
1. Invest in local capacity to manage performance of the “system,” in particular peacebuilding, justice and security. Donor effectiveness may be best measured by assessing the performance of the institutions, agencies, and organizations programmes are designed to support. To the extent that the sustainability of any intervention for justice and security development or conflict prevention/peacebuilding depends on local ownership, it is important that justice and security sector institutions, as well as institutions related to peacebuilding, improve their own systems for assessing performance. Moreover, development and monitoring of indicators that are relevant
22 OECD DAC Guidance on Evaluating Conflict Prevention and Peacebuilding Activities (Working Draft), p. . 23 OECD DAC Guidance on Evaluating Conflict Prevention and Peacebuilding Activities (Working Draft), p. 20.
53
and appropriate to the specific context could supplement and correct for the gaps and deficiencies in global-level indicators covered in existing datasets and would allow for more accurate assessment of the impact of interventions, policies and actions. It is, therefore, recommended that a percentage of assistance to a country be earmarked for the development performance management capacity, including appropriate indicators and data collection methods and systems.
2. Development of indicators and datasets for non-state mechanisms and processes for justice,
provision of security and peacebuilding. This should be an area for significant investment—either at country level or globally. Similarly, there is need to design and support methods of measuring the performance of private security companies, whose business is booming around the world, but particularly in fragile countries.
3. Strengthening of indicators and data on perceptions. The Afro- and other barometers can provide valuable data on public perceptions of conflict, justice and security. However, the questions are not entirely consistent, the frequency of collection of data not sufficient, and the coverage also relatively limited. Gallup World Poll has more extensive coverage, but few of the questions are relevant to the elements of conflict, justice and security that are useful to monitor in order to assess impact. We would recommend that DFID/HMG invest in Afrobarometer (and other barometers) to further systematize and expand the collection of data. In addition, at relatively low cost for DFID/HMG, questions might be proposed or added to the barometers or to the Gallup polls to gather missing information about perceptions.
4. Disaggregation by group (ethnic/religious/social/ideological). In order for sectoral indicators to become more conflict-sensitive and provide data on conflict impact, disaggregation by relevant conflict group would be useful. Some perceptions surveys – such as the Afrobarometer – as about respondents’ identity, and thus might easily be able to provide disaggregated data. In some post-conflict situations, such as in Kosovo, information on crime, justice and security has been disaggregated by ethnicity, and has provided valuable information on distributional impacts of policies and actions. In other situations, this may be sensitive, but would be useful for DFID/HMG to take up with host governments.
Appendix A
Example: Application of Indicators to Burundi and Nepal, 2006-2008
A. Conflict Intensity
Indicator Source/Scale Nepal Burundi Comments
1. # deaths from armed conflict
Uppsala Conflict Data Project Absolute number
2006 2007 2008 2006 2007 2008
Burundi: There is a postive reduction in most indicators, reflecting the momentum in the peace process and the negotiations with the last remaining rebel group. The conflict here would no longer be defined as an “armed conflict” by UCDP, and one sees also a significant reduction in “total population of concern” (includes IDPs and refugees), especially in the IDP population. Tensions have decreased significantly, but still are not low.
Nepal: Nepal has also seen a significant reduction in conflict intensity – particularly in deaths (it too would no longer be classified as “armed conflict”). The stable level of displaced includes returns of about 50,000 IDPs who are counted in the “total population of concern” because UNHCR is following them. Nonetheless, the level of tension, although decreasing, remains extremely high. The World Governance Indicators also suggest high levels of political instability and violence (a rating of -2.13 in 2007, from -2.09 in 2006 – a trend within the margin of error).
457 0 0-24 137 0-24 missing
2. # and increase in IDPs/refugees
UNHCR (in 000) http://www.unhcr.org/ statistics/45c063a82.html
105 105.5 464 423
3. Territory affected by conflict
Political Instability Task Force 0 = best, 4 = worst
4 Ended ended ended ended ended
4. Seriousness of ethnic, religious, social conflicts
Bertelsmann Transformation Index (BTI) Question 13.3 1 = best, 10= worst http://www.bertelsmann-transformation-index.de/fileadmin/pdf/Anlagen_BTI_2008/ Detailed_Results_BTI2008.xls
Alternate: World Governance Indicators, “Political Stability and Absence of Violence.” http://info.worldbank.org/governance/wgi/index.asp
9 Unpub-lished
8 10 Unpub-lished
5
5. Neighborhood War - # neighboring states experiencing armed conflict
Peace and Conflict Instability Ledger, University of Maryland red = high risk (2 or more) yellow = medium (1) green = low risk (0)
medium risk
Unpub-lished
n/a Unpub-lished
B. Institutions/Mechanisms/Processes that Handle Grievances B.I. Capacity of Government to Deal with Grievances The capacity includes capacity to develop and implement policies that address grievances Effectiveness
Indicator Source/Scale Nepal Burundi Comments
1. Governance Effectiveness
World Governance Indicators 2.5= best -2.5= worst
2006 2007 2008 2006 2007 2008
In both countries, government effectiveness is rated as low, and has not changed in the 2006-2008 time period. One would not, however, be able to detect small changes in an aggregate indicator such as WGI’s governance effectiveness, and, unfortunately, data is unavailable on access to improved water sources. We turned to HDI and primary school enrollment to gather some information about provision of services. The only indicator with up-to-date data is primary school enrollment, which increased in Burundi and remained the same in Nepal. It might be noted that infant mortality (also available through the World Development Indicators) decreased slightly in both.
The general picture thus appears to be one of slight improvement, but with significant challenges, particularly in light of the slightly deteriorating score on B.I.3 (government coordinating conflicting objectives) in Nepal, and the low, even if improving, score for Burundi.
-0.82 -0.81 Unpub-lished
-1.26 -1.34 Unpub-lished
2. Efficiency of revenue mobilisation
World Bank Country Policy and Institutional Assessment (CPIA) Criterion 14 1=low, 6=best
3.5 3.5 Unpub-lished
3 3 Unpub-lished
3. Extent to which government can coordinate conflicting objectives into coherent policy
Bertelsmann 15.2 1 = worst 10= best
5 Unpub-lished
4 2 Unpub-lished
3
4. Access to improved water sources
Human Development Index
Primary school enrollment
World Development Indicators
89
.530
126
Unpub-lished
126
Unpub-lished
124
71
.382
103
Unpub-lished
114
Unpub-lished
Unpub-lished
Leadership culture/ process
Indicator Source/Scale Nepal Burundi Comments
5. To what extent can political leadership manage political cleavages?
Bertelsmann Question 16.3 10 = best 1= worst
2006 2007 2008 2006 2007 2008
In both countries, the ability of the leadership to manage the political cleavages has improved, which is to be expected given the progress in the peace processes in both Nepal and Burundi during this period. The increase in both countries in the acceptance of democratic institutions is also a positive sign of commitment to the peace process.
However, the high scores on “factionalized elites” – which is a much more short-term-sensitive indicator of political polarization, would suggest that the upward trend is very vulnerable. There have been improvements in Nepal, but Burundian politics remains quite factionalized despite progress in the peace process.
3 Unpub-lished
6 4 Unpub-lished
7
6. Factionalized Elites Failed States Index/Fund for Peace (1.11) 0=best, 10=worst
9 8.5 8.3 7.8 7.5 7.8
7. Extent to which democratic institutions are accepted or supported by relevant actors
Bertelsmann Question 4.2 10 = best 1= worst
2 Unpub-lished
4 2 Unpub-lished
5
8. Regime/governance inclusion (Political legitimacy score)
State Failure Index, Center for Systemic Peace, George Mason University based on Polity IV 0 = best (stable) 3 = worse (fragile)
2 0
Grievances
Indicator Source/Scale Nepal Burundi Comments
9. Group grievances
Failed States Index, Fund for Peace (I.3) 0=best, 10=worst
2006 2007 2008 2006 2007 2008
Several of the indicators (B.I.11-12) are not relevant in these situations, and consequently can be ignored.
Burundi: There appears to be improvement in behavior that generates group grievances (B.I.9), but little movement on the socio-economic factors that may underlie group grievances. Viewed in conjunction with the indicators on leadership process and relationships, there appears to be less progress on dealing with underlying drivers of conflict. Regrettably, the Afrobarometer does not include Burundi in its latest polls; data on popular perceptions of the government and inter-group relations would be valuable here.
Nepal: Uneven development and group grievances are very high, and seem to have remained stable or become slightly worse, despite the peace agreement and a reduction in conflict-related violence. This may merely reflect the challenges of implementing the peace accord – both the difficulties encountered in implementation, and the time it will take to address some of the underlying issues.
9.2 8.9 9 7 6.7 6.7
10. Minority access to education, commerce, professions, etc.
Minorities at Risk, University of Maryland, Column AF 0=no restrictions 4=worst restrictions
n/a Unpub-lished
Unpub-lished
1 Unpub-lished
Unpub-lished
11. Restrictions on religion
Minorities at Risk, University of Maryland 0=no restrictions 3=sharply restricted
na Unpub-lished
Unpub-lished
0 Unpub-lished
Unpub-lished
12. Restrictions on use of language or language instruction
Minorities at Risk, University of Maryland 0=no restrictions 3=sharply restricted
na Unpub-lished
Unpub-lished
0 Unpub-lished
Unpub-lished
13. GINI Coefficient World Development Indicators
Unpub-lished
Unpub-lished
Unpub-lished
33 Unpub-lished
Unpub-lished
14. Uneven Development Failed States Index/Fund for Peace (1.5) 0=best, 10=worst
8.5 9.2 9.2 8.8 8.8 8.8
15. How often has your group been treated unfairly by the government?
Afro Barometer Ghana 2008 survey, Question 82
Inap-plicable
Inap-plicable
Inap-plicable
na na na
16. Your group has same, worse, or better economic conditions than others
Afrobarometer
Ghana 2008 survey, Question 11
Inap-plicable
Inap-plicable
Inap-plicable
na na na
B.II. Governance Legitimacy and Accountability
Elections free and fair
Indicator Source/Scale Nepal Burundi Comments
17. Electoral Process and Pluralism
Economist Intelligence Unit Democracy Index 0=worst 10=best Freedom House (FH) 0 = smallest degree of rights
2006 2007 2008 2006 2007 2008 Nepal has experienced significant improvement in this area; this is consistent with the history of the peace process. This is also reflected in popular perceptions of the fairness of elections. Burundi, having experienced a successful 2005 election that was a watershed, has remained stable. While this is a positive sign of the development of democratic processes, the high degree of factionalization (leadership culture and process) suggest that the electoral process could as easily fuel as mitigate conflict.
EIU: .08
FH: A – 1 (of 12) B – 6 (of 16)
EIU: 1.33 FH: A – 4 B – 9
EIU: 4.42
FH: A – 9 B – 11
EIU: 4.42 FH: A – 9 B – 9
18. Perceptions of honesty of elections
Gallup, alternatives are Afrobarometer #71 or Arab Barometer 211
25% 60% (Oct 2008)
56% (July 2008)
Legitimacy of State
Indicator Source/Scale Nepal Burundi Comments
19. Delegitimization of state Failed States Index/Fund for Peace (I.7) 0=best, 10=worst
2006 2007 2008 2006 2007 2008 While the state appears to be gaining legitimacy in Nepal, there is an interesting discrepancy between this indicator and B.I.6 and B.I.8. The mediocre score, which has remained unchanged since 2006, on B.I.20 suggests that the danger of group-based challenges to the state, and conflict, remain present. Requires further analysis.
9.2 8.5 8.3 7.2 7.1 7.1
20. Relevant groups agree about citizenship and accept the nation state as legitimate?
Bertelsmann 1.2 1 = worst 10= best
6 Unpub- lished
6 6 Unpub- lished
8
21. Public support for government restrictions on civil liberties
Afro Barometer Ghana 2008 survey, Question 19
Inap-plicable
Inap-plicable
Inap-plicable
na na na
22. Control of Corruption World Governance Indicators best=2.5 worst= -2.5
-0.67 -0.66 Unpub-lished
-1.12 -1.06 Unpub-lished
B.III. Human rights & Civil liberties These are indicators of rule of law, as well as a proxy for the potential for the extent to which peaceful mechanisms for handling grievances exist. We have specifically excluded physical issues here; these have been placed under perception of safety and security.
Indicator Source/Scale Nepal Burundi Comments
23. Civil liberties EIU Democracy Index After 2006 0=worst, 10=best
Freedom House (FH) Freedom in the World D, E, G 0 = worst/smallest
2006 2007 2008 2006 2007 2008
There has been an improvement Nepal, given the results of 2008. Based on the Freedom House scores, the biggest improvement has been in freedom of expression and belief. Human rights and civil liberties remain at concerning level in both Nepal and Burundi, and are not improving in Burundi.
EIU: 5.59 FH: D – 6 (of 16) E – 3 (of 12) G – 6 (of 16)
EIU: 5.59 FH: D – 7 E – 6 G – 10
EIU: 4.71 FH: D – 6 (of 16) E – 5 (of 12) G – 9 (of 16)
EIU: 4.71 FH: D – 6 E – 5 G – 8
24. Suspension of Rule of Law and Human Rights
Failed States Index, Fund for Peace 0=best, 10=worst
9.1 8.8 8.8 7.5 7.5 7.5
25. Allowable advocacy activities
CIVICUS Civil Society Index 2.5.2
2 Unpub-lished
Unpub-lished
na Unpub-lished
Unpub-lished
C. Community Initiatives For Peace Community/Civil society initiatives for peace This looks at civil society initiatives for peace: to what extent are there local capacities for peace?
Indicator Source/Scale Nepal Burundi Comments
1. Civil society actions to promote non-violence and peace
CIVICUS 3.4.2 2006 2007 2008 2006 2007 2008 This is an important category of indicators for which there is nearly no data. CIVICUS does not collect data regularly, and thus comparisons over time, even for those countries that are covered, is difficult.
3 Unpub-lished
Unpub-lished
na Unpub-lished
Unpub-lished
2. Activity and success of civil society in influencing public policy in human rights
CIVICUS 4.1.1 1 Unpub-lished
na Unpub-lished
Unpub-lished
D. Capacity/Willingness to Resist or to Escalate Violence Military and societal militarization The indicators in this category reflect opportunity to fight/military feasibility of war and justice and security development
Indicator Source/Scale Nepal Burundi Comments
1. Military expenses as % of GDP
The Military Balance, Stockholm International Policy Research Institute www.milexdata.sipri.org
2006 2007 2008 2006 2007 2008
Data for UNODC is available only to 2006. We were unable to obtain the Global Peace Index data for individual factors, but it should be available through the Economist Intelligence Unit or through Vision of Humanity, which sponsors the GPI. Based on the current data, militarization trends cannot be assessed; it may be useful to go back further in time.
2% 1% 5% 5%
2. # armed forces per 100,000 people
The Military Balance, SIPRI
2.298850575
3. # security officers and police per 100,000
10th Crime Trends Survey, UN Office on Drug and Crime
202.83 Unpub-lished
Unpub-lished
Unpub-lished
Unpub-lished
Unpub-lished
4. Ease of access to small arms
Global Peace Index (collected by EIU)
1=very low, 5=very high
Unpub-lished
Unpub-lished
na Unpub-lished
na Unpub-lished
5. Intentional homicide committed by firearm, per 100,000
UNODC 2.6 0.3 Unpub-lished
Unpub-lished
na Unpubl-ished
Unpub-lished
6. Non-violence within the civil society arena
CIVICUS 3.4.1 2 Unpub-lished
Unpub-lished
na Unpub-lished
Unpub-lished
E. Criminal Justice E.I. Criminal justice capacity and effectiveness
Activities and results
Indicator Source/Scale Nepal Burundi Comments
1. Conviction rate UNODC 9.1 2006 2007 2008 2006 2007 2008
14.29
15.93 (2005)
Unpub-lished
Unpub-lished
Unpub-lished
Unpub-lished
Unpub-lished
2. # of recorded crimes per 100,000 people
UNODC 2.0 14.96 15.07 (2005)
Unpub-lished
Unpub-lished
Unpub-lished
Unpub-lished
Unpub-lished
3. # convictions/# police UNODC Q11 / UNODC 1.1
0.052
0.049 (2005)
Unpub-lished
Unpub-lished
Unpub-lished
Unpub-lished
Unpub-lished
4. # convictions/# prosecutors
UNODC Q11/UNODC 5.0
13.5
12.6 (2005)
Unpub-lished
Unpub-lished
Unpub-lished
Unpub-lished
Unpub-lished
5. # prosecutions/# prosecutors
UNODC Q7/UNODC 5 6.02
6.45 (2005)
Unpub-lished
Unpub-lished
Unpub-lished
Unpub-lished
Unpub-lished
6. # convictions/# prosecutions
UNODC Q11 / UNODC Q 7
Unpub-lished
Unpub-lished
Unpub-lished
Unpub-lished
Unpub-lished
7. # prosecutions/# judges UNODCQ7/UNODCQ8
1307/232
1400/195 (2005)
Unpub-lished
Unpub-lished
Unpub-lished
Unpub-lished
Unpub-lished
8. Average # months of criminal trials
UNODC Q 9.2 n/a Unpub-lished
Unpub-lished
Unpub-lished
Unpub-lished
Unpub-lished
Perceptions
Indicator Source/Scale Nepal Burundi Comments
9. Government reducing crime?
Afrobarometer 57F
2006 2007 2008 2006 2007 2008
na Inap-plicable
Inap-plicable
Inap-plicable
na na
10. Who do you think has primary responsibility for managing each of the following tasks - national gov? local gov? traditional leaders? Members of community? None of them? Don't know.
Afrobarometer 58 E and H
na Inap-plicable
Inap-plicable
Inap-plicable
na na
11. In your opinion, how often… do officials who commit crimes go unpunished?
Afrobarometer 45 E and D
12. In your opinion, how often… do ordinary people who break the law go unpunished?
Afrobarometer 45.E
na Inap-plicable
Inap-plicable
Inap-plicable
na na
E.II. Rule of Law: civil and administrative justice
Indicator Source/Scale Nepal Burundi Comments
1. # days to resolve dispute World Bank Doing Business
2006 2007 2008 2006 2007 2008
2. # steps to claim, obtain, enforce contracts
World Bank Doing Business
39 39 39 44 44 44
3. Independence of Judiciary
Bertelsmann 3.2 1 = best, 10= worst
3 Unpub-lished
3 4 Unpub-lished
5
4. Court system fair/impartial? Law and regulations predictable and consistent
World Bank Enterprises Survey
Unpub-lished
Unpub-lished
Unpub-lished
n/a Unpub-lished
Unpub-lished
5. Should the President always obey the law?
Afrobarometer 37 Inap-plicable
Inap-plicable
Inap-plicable
na na na
6. How much do you trust… Courts of law?
Afrobarometer (49H), alternatively, see Gallup World Poll for Asia
47% na na na
7. Rule of law/police under civilian control?
Freedom House, Freedom in the World F2
n/a n/a n/a n/a n/a n/a
8. Equal treatment of demographic groups?
Freedom House. Freedom in the World F4
n/a n/a n/a n/a n/a n/a
E.III. Access to Justice: state and non-state systems
Capacity
Indicator Source/Scale Nepal Burundi Comments
1. # police per 100,000 UNODC 1.1 (Rate)
2006 2007 2008 2006 2007 2008
202.83
206.96 (2005)
Unpub-lished
Unpub-lished
Unpub-lished
Unpub-lished
Unpub-lished
2. # prosecutors per 100,000
UNODC 5.0 (Rate)
0.79
0.80 (2005)
Unpub-lished
Unpub-lished
Unpub-lished
Unpub-lished
Unpub-lished
3. # lawyers per 100,000 Law Schools / Bar Associations
4. # hours to pay taxes, # payments per year
World Bank Doing Business
408 hrs 34 payments
408 hrs 34
payments
408 hrs 34
payments
140 hours 32 payments
140 hours 32 payments
140 hours 32 payments
Access
Indicator Source/Scale Nepal Burundi Comments
5. Attorney, court, enforcement costs as % of claim value
World Bank Doing Business
2006 2007 2008 2006 2007 2008 There appears to be a deterioration of public services in both countries. This can be viewed in conjunction with B.I.6 and B.I.8. The question arises is whether higher tensions contributed to the deterioration, or whether the deterioration has exacerbated tensions. Has the inability of the governments to respond triggered unhappiness with the leadership and government agencies?
26.8 26.8 26.8 38.6 38.6 38.6
6. Government repression Minorities at Risk 0=no repression; 5=violent repression and killing
n/a Unpub-lished
Unpub-lished
0 Unpub-lished
Unpub-lished
7. Deterioration of public services
Failed States Index/Fund for Peace (1.8) 0=best, 10=worst
6.2 6.6 7 8.5 8.9 9.0
Women’s access & capacity
Indicator Source/Scale Nepal Burundi Comments
8. # women police/total police
UNODC 1.2/UNODC 1.1
2006 2007 2008 2006 2007 2008
1418/56064 same in 2005
Unpub-lished
Unpub-lished
Unpub-lished
Unpub-lished
Unpub-lished
9. # women prosecutors/total prosecutors
UNODC Q5.1/UNODC Q5.0
2 women/ 217 total
prosecution personnel
Unpub-lished
Unpub-lished
Unpub-lished
Unpub-lished
Unpub-lished
10. # women judges/total judges
UNODC Q8.1/UNODC Q8.0
3/232 Unpub-lished
Unpub-lished
Unpub-lished
Unpub-lished
Unpub-ished
11. # women prison staff/total prison staff
UNODC (14.1+14.4)/UNODC (14.3+14.0)
61/658
61/657 (2005)
Unpub-lished
Unpub-lished
Unpub-lished
Unpub-lished
Unpub-lished
12. Women as % of legislators, senior officials, managers
IMD World Competitive Yearbook 2.5.10
Perception
Indicator Source/Scale Nepal Burundi Comments
13. Trust in courts vs. trust in traditional leaders (how do you trust each?)
Afrobarometer (Compare 49H vs. 49I)
2006 2007 2008 2006 2007 2008
Inap-plicable
Inap-plicable
Inap-plicable
na na na
14. Use of wasta (clout) (During the past five years, have you ever used wasta to achieve something personal, family-related, or a neighborhood problem?)
Arab Barometer 226
Inap-plicable
Inap-plicable
Inap-plicable
Inap-plicable
Inap-plicable
Inap-plicable
15. Reference to traditional leaders (list the types of wasta – choice 4 is traditional leaders)
Arab Barometer 227
Inap-plicable
Inap-plicable
Inap-plicable
Inap-plicable
Inap-plicable
Inap-plicable
16. Go to court to resolve disputes? (If you were to have a dispute with another citizen, would you try to resolve it in a court or in another government institution?)
Arab Barometer 228
Inap-plicable
Inap-plicable
Inap-plicable
Inap-plicable
Inap-plicable
Inap-plicable
17. Who would you go to to resolve a dispute?
Arab Barometer 229
Inap-plicable
Inap-plicable
Inap-plicable
Inap-plicable
Inap-plicable
Inap-plicable
18. Women are/should be subject to traditional laws1
Afro Barometer 23.B
Inap-plicable
Inap-plicable
Inap-plicable
na na na
1 In the 2005 survey, respondents were asked: “In our country, women should have equal rights and receive the same treatment as men do.” B. “Women have always been subject to traditional laws and customs, and should remain so.” Respondents could agree with A, B, or neither.
E.IV. Corruption
Indicator Source/Scale Nepal Burundi Comments
1. Government doing enough to fight corruption?
Gallup World Poll 2006 2007 2008 2006 2007 2008
Corruption has significantly increased in Burundi. Once again, this could be seen in relation to B.I.6, B.I. 8, and E.III.9. Is this an emerging overall trend?
2. Corruption perceptions Transparency Int’l - Corruption Perceptions Index 1=most corrupt 10=least corrupt
2.5 2.5 2.7 2.4 2.5 1.9
3. Registering property - # of procedures, # of days
World Bank Doing Business
3 steps 5 days
3 steps 5 days
3 steps 5 days
5 steps 94 days
5 steps 94 days
5 steps 94 days
4. Gifts given to receive public services
World Bank Enterprise Survey
Unpub-lished
Unpub-lished
Unpub-lished
n/a Unpub-lished
Unpub-lished
5. How many of the following people do you think are involved in corruption?
Afro Barometer 50 (ALL)
Inap-plicable
Inap-plicable
Inap-plicable
n/a n/a n/a
6. Municipal/national officials involved in corruption?
Asian Barometer 114-115
Inap-plicable
Inap-plicable
Inap-plicable
F. Security Sector
F.I. Prisons
Indicator Source/Scale Nepal Burundi Comments
1. # prisoners/# beds King’s College 2006 2007 2008 2006 2007 2008
0.986450989 Unpub-lished
Unpub-lished
Unpub-lished
Unpub-lished
Unpub-lished
2. # juveniles/# beds for juveniles
King’s College/ UNODC
50 Unpub-lished
Unpub-lished
Unpub-lished
Unpub-lished
Unpub-lished
3. # pre-trial detention prisoners prior to disposition of case/# convicted prisoners
UNODC/King’s College
n/a Unpub-lished
Unpub-lished
Unpub-lished
Unpub-lished
Unpub-lished
4. # persons with AIDS, mental illness, TB/# prisoners
UNODC Q (18.2+18.3+18.4)/ UNODC 15.0
% of prisoners
with HIV: 0.6 % with mental
illness: 0.6
Unpub-lished
Unpub-lished
Unpub-lished
Unpub-lished
Unpub-lished
F.II. Policing
Capacity
Indicator Source/Scale Nepal Burundi Comments
1. # police per 100,000 UNODC Q 1.1 (Rate)
2006 2007 2008 2006 2007 2008
202.83 Unpub-lished
Unpub-lished
Unpub-lished
Unpub-lished
Unpub-lished
2. # cases discontinued/# cases referred to prosecutors
UNODC 9.0-7.0 n/a Unpub-lished
Unpub-lished
Unpub-lished
Unpub-lished
Unpub-lished
Behavior & Effectiveness
Indicator Source/Scale Nepal Burundi Comments
3. # of intentional homicides completed per 100,000 people
UNODC 2.3 (rate) 2006 2007 2008 2006 2007 2008
The Nepalese state has improved. How does this relate to the other indicators that have suggested deterioration of public services, ability of the leadership to address policies, etc. The surface discrepancy may have to do with the end of the insurgency vis-à-vis government services. Needs
1.84 Unpub-lished
Unpubl-ished
Unpub-lished
Unpub-lished
Unpub-lished
4. In the last 12 months, were you assaulted, mugged, property or money stolen?
Gallup World Poll 9% theft 4%
assaulted
5. How much trust do you have in… the Police
Arab (201.4), Asian (13) and Afro-barometers (49G)
n/a n/a n/a
6. Get help from police when needed? 2
Afro Barometer 71
Inap-plicable
Inap-plicable
Inap-plicable
n/a n/a n/a
2 This question was asked in the 2005 survey: “Based on your experience, how difficult is it to obtain the following services? Or do you never try to get these services from government? C: help from the police when you need it. (4=very easy, 1=very difficult).
7. Difficulty in getting help from the police?3
Arab Barometer 213
Inap-plicable
Inap-plicable
Inap-plicable
Inap-plicable
Inap-plicable
Inap-plicable
more analysis.
8. Extrajudicial killings CIRI Index 0=worst, practiced frequently; 1= practiced occasionally; 2=best, not practiced at all
0 0 Unpubl-ished
0 1 Unpub-lished
9. Criminalization/ delegitimization of the state
Failed States Index, Fund for Peace (I.7) 0=best, 10=worst
9.2 8.5 8.3 7.2 7.1 7.1
F.III. Oversight of justice and security sector
Indicator Source/Scale Nepal Burundi Comments
1. Security sector operating as a state within a state
Failed States Index, Fund for Peace (I.10) 0=best, 10=worst
2006 2007 2008 2006 2007 2008
In both countries, this indicator would suggest significant improvement, but the meaning of that improvement cannot be ascertained from this one indicator alone. In the case of Nepal, this movement correlates well with E.VI.9.
9 8.3 8.5 7.3 6.8 6.8
2. Corruption mechanisms institutionalized
Bertelsmann 15.3 1 = best, 10= worst
4 Unpub-ished
4 2 Unpub-lished
4
3. General government transparency
IMD World Competitive Yearbook 2.3.12
3 The question asked: “ Based on your experience, how difficult is it to obtain the following administrative or social services from the government?” One options was 4.” Get help from the police when you need it.”
G. Personal Safety and Security
Perception of personal safety and security
Indicator Source/Scale Nepal Burundi Comments
1. State behavior violating physical integrity
Political Terror Scale 1=lowest level of terror; 5=highest level of political terror
2006 2007 2008 2006 2007 2008
It would appear that Nepal has improved on the political terror scale, which the next data point in the series (2008?) would be able to verify. However, the lack of data about perceptions of safety, and the time lag in obtaining data on homicides, small arms and other violence makes it difficult to capture any deterioration of public security such as that which has occurred in Nepal over the last two years. In light of the absence of data, these figures could be vied in connection with B.II.19 (delegitimization of the state), and A. 4 (seriousness of social conflicts) as a proxy, although that too is not very satisfactory.l
There appears to be additional correlation with E.VI.9 and E.VII.1.
4.5 3.5 Unpubl-ished
4 4 Unpub-lished
2. Feel safe walking alone at night in your community?
Gallup World Poll 52%
3. Experienced fear, stress, worry a lot of the day yesterday?
Gallup World Poll
4. Over the past year, how often, if ever, have you or anyone in your family… B. had something stolen from your house, C. been physically attacked?
Afrobarometer 9B, C
Inap-plicable
Inap-plicable
Inap-plicable
n/a n/a n/a
5. In this country, how often do people have to be careful of what they say about politics?
Afrobarometer 47 Inap-plicable
Inap-plicable
Inap-plicable
n/a n/a n/a
6. In your opinion, what are the most important problems facing this country that the government should address? (one choice is political violence)
Afrobarometer 56 Inap-plicable
Inap-plicable
Inap-plicable
n/a n/a n/a
7. # reported rapes per 100,000 women
UNODC 0.76 Unpub-lished
Unpub-lished
Unpub-lished
Unpub-lished
Unpub-lished
Appendix B: List of Data Sources
Producer Index Geographic
CoverageTime Coverage Methodology Data Source Comments
Datasets Used for Indicators
1 Afrobarometer Afrobarometer
20 sub-Saharan African countries(No survey in Ethiopia, Sudan, DRC, Sierra Leone)
Every 2-3 years; earliest poll was 1993, but earlier surveys had very few countries
[Survey] Survey
Coverage is an issue. And only a few countries are covered in each round.Questions are good, but not all countries have exactly the same Questions, so comparability is questionable.
2 Arab Barometer Arab Barometer
10 Arab countries (five completed), data not yet available for UAE, Yemen, Oman, Iraq, Lebanon, Bahrain. None on Egypt and Saudi Arabia
First survey in 2006, in five countries; next wave will add six countries
[Survey] SurveyOnly five countries so farAlso, questions may not be the same in every countries.
3 Asianbarometer Asianbarometer13 Asian countries and 5 South Asian countries
First wave (2001-2003) for 8 countries; second wave (2005-2007) for 18 countries; third wave being planned
[Survey] Survey
All major countries are coveredBut like the others, different countries may not have the same questions
4Bertelsmann Transformation Index
Bertelsmann Transformation Index (BTI)
125 transformation countries. Does not include US, Canada, Western Europe, Australia, Japan.
Released every two years; goes back to 2003
[Survey] + [Expert]Survey w/Bertelsmann questionnaire of country experts
Survey w/Bertelsmann questionnaire of country experts
5CIDCM, University of Maryland
Peace and Conflict Instability Ledger (PCIL)
160+ countries 2001, 2003, 2005, 2008
[Statistical Model]: Based results of studies that identified factors strongly correlated with major incidents of instability in the past six decades, including: 1) Recent instability [PITF], 2) country size dummy var., 3) regime consistency [Polity IV], 4) infant mortality, 5) economic openness [(X+M)/GDP], 6) Militarization, 7) Neighborhood War (Upssala, IPRI)
Correlates of War Project, Gleditsch 2002, PITF data, Polity IV data, Penn World Table, Uppsala Conflict Data, International Peace Research Institute, World Bank World Development Indicators
Appendix C: List of Data Sources 2
Producer Index Geographic
CoverageTime Coverage Methodology Data Source Comments
6CIRI Human Rights Data Project
CIRI Human Rights Index
195 countriesannually from 1981 - 2007
[Expert]: At least two expert coders for each country meet with the CIRI staff, then score the indicators based on information from the State Dept annual reports on human rights practices, and country reports from Amnesty International
State Department annual reports on human rights practices, and Amnesty International's country reports, with the latter treated as authoritative when there are conflicts between the two
Relies on subjective coding, and coders only read two sources, so subjectivity and bias are big issues.
7 CIVICUS Civil Society Index
56 countries in the latest version. Asia and Eastern Europe are best covered. Very little coverage in Africa, Middle East and Latin America; North America and Central Asia are excluded
Every 3-4 years. Pilot phase in 2001-2002; Phase 2 in process 2008-2010
[Survey]: mainly surveys of experts, organizations, population - but combined with media monitoring and existing quantitative data
Mostly surveys; but also media reports
8Econmist Intelligence Unit (EIU)
Democracy Index
167 countries (missing a few like Somalia and Western Sahara)
2006, 2008[Expert]: Survey of experts on 60 multiple choice questions
Expert surveys
10 Freedom HouseFreedom in the World
193 countries Annually, since 1972
[Expert]: Research analysts and academic advisers produce country reports and propose ratings, which are then reviewed in a series of regional and then global meetings
foreign and domestic news reports, academic analyses, nongovernmental organizations, think tanks, individual professional contacts, and visits to the region
Scoring by experts are subjective, and it's unclear if different sets of experts would score the same way; the system enables cross-country comparisons
11 Fund for PeaceFailed States Index (FSI)
177 countries Annually since 2005
[Expert/Software]12 Indicators, each assessed by feeding English, web-accessible information (news, reports) through a software that looks for "key phrases" associated with each indicator. The more coverage, the higher the score. Each indicator is scored from 1 to 10.
Meltwater News Service
Assumption is that if key phrases appear more times in the English, electronic media, then a given issue is more severe or important.
Appendix C: List of Data Sources 3
Producer Index Geographic
CoverageTime Coverage Methodology Data Source Comments
12 Gallup Gallup World Poll
150 countries
Does not yet include Bahrain, Oman, Cote d'Ivoire, Libya, and Turkmenistan
Since 2005, continuously updated
[Survey]- random telephone interviews in countries where phones are common, and face to face interviews where they are not
SurveyThe most comprehensive global survey available. Full results are only accessible by subscription
13GMU / Center for Systemic Peace
State Fragility Index & Related Datasets
162 countries
Appears to be updated annually, data goes back to 1946 where available
[Expert] Assessments by academics based on available literature
Bienen/van de Walle, Center for Systemic Peace, Gurr/Harff, Minority aft Risk, Political Terror Scale, UNDP Human Development Report, UNDP Structure of Trade, US Census Bureau Intl Database, Polity IV, World Bank WDI
Good time and country coverage Does not report margin of errorWeighting can be suspect
14GMU / Center for Systemic Peace
Polity IV dataset
All major independent states from 1800 to 2007 - 162 countries at this moment
1800-2007; appears to be updated every few years
[Expert] Assessments by academics based on available literature
Published materials, unspecified
15 IMDWorld Competitiveness Yearbook
55 countries
Missing: all of Africa outside S. Africa; most of Middle East, East Europe, Balkans, all of Central Asia, much of SE Asia.
Annual, since 1989[Survey] of business executives + looking at available public data
Surveys and other publicly available hard data
16 Intl Institute of Strategic Studies
Military Balance 170 countries Annually, since 2004 [Primary data] National governments; NATO; UN; OSCE; IMF
17 SIPRIMilitary Expenditure Database
171 countries Annually, since 1988 [Primary data]National governments; NATO; UN; OSCE; IMF, secondary sources that cite primary data
Appendix C: List of Data Sources 4
Producer Index Geographic
CoverageTime Coverage Methodology Data Source Comments
18Transparency International
Corruption Perception Index
180 countries Annually since 1995[Composite] Combination of 13 sources that measure corruption
Asian Development Bank, African Development Bank, Bertelsmann Transformation Index, Country Policy and Institutional Assessment, Economist Intelligence Unit, Freedom House, Global Insight and Merchant International Group; IMD, Political and Economic Risk Consultancy, and the World Economic Forum
19UN High Commissioner on Refugees
Refugee statistics 150+ countries Annually since 1994 [Primary data]
20UN Office on Drug and Crime
Survey of Crime Trends and Operations of Criminal Justice Systems
86 countries
Missing: almost all of Africa; Brazil, Russia, China, Indonesia, Pakistan, Iran, Saudi, UK, etc.
Now every two years, used to be every five years, since 1970
[Survey]Data provided by national governments to a survey, which can be viewed online
Relies on countries to respond, so coverage is spotty. Even when countries do respond, they may leave out important stats; and there may be definitional issues that the country office may need to h k
23University of Maryland
Minority at Risk Project
284 groups and 118 ethnopolitical organizations
1945 - present; updated every 3 years or so
[Expert/Student]: Variables are coded by trained grad and undergrad students using open source info, then reviewed by senior staff. Inter-coder reliability has not been tested. Coders are "encouraged" to carry out their own reliability and validity tests
Unspecified open source information
Subjective coding, and system for reviewing coding decisions is weak. Also, updated only once every 3 years.
24University of North Carolina
Political Terror Scale
183 countries 1976-2007
[Expert/Student]: Staff and students assign scores to countries by reading State Department annual reports on human rights practices, and Amnesty International country reports
State Department annual reports on human rights practices, and Amnesty International's country reports
Relies on subjective coding, and coders only read two sources, so subjectivity and bias are big issues.
25 Uppsala UniversityUppsala Conflict Data Program
Global - all conflicts that match their definition
Annually updated, data goes back to 1946
[Primary data] Uppsala/PRIO researchers read primary and secondary sources and count or estimate the numbers
Unspecified primary and secondary sources
Appendix C: List of Data Sources 5
Producer Index Geographic
CoverageTime Coverage Methodology Data Source Comments
26 Vision of HumanityGlobal Peace Index (GPI)
140 countries
Missing: Pacific Islands, Sierra Leone, Liberia, Eritrea, Pacific Islands, but most major countries covered.
Annually since 2007
[Expert] A combination of existing quantitative data and qualitative scores done by Economist Intelligence Unit analysts
Economist Intelligence Unit, Expert Estimates, Uppsala Conflict Data Program, International Peace Research Institute in Oslo, UN Office of Drugs and Crime (UNODC), International Institute of Strategic Studies (IISS)
27 World BankWorldwide Governance Indicators
212 countries and territories
Annually since 1996[Composite] Composite indicators taken from a huge number of sources
See website, too many to list, including many sources listed here
28 World BankWorld Development Indicators
153 economies and 56 other smaller economies
Annually, since 1960
29 World BankCountry Policy and Institutional Assessment
76 countries Annually[Expert]: World Bank country staff assign the scores, which then get reviewed by higher levels
Lacks transparency
30 World Bank Doing Business 181 economies Annually, since 2003
[Survey]: Surveys of local experts including lawyers, business consultants, accountants, government officials etc
Surveys
31 World Bank Enterprise Survey 110 countries Every three years, since 2002
[Survey] Surveys
32World Economic Forum
Global Competitiveness Report
134 countries Annual, since 1979[Survey] of business executives + looking at available public data
Survey, IMF, World Bank, UN Population Fund, Economist Intelligence Unit, World Health Organization, UNESCO, etc. - see report for complete sources
Source Source Link Latest data
Jam
aica
Guy
ana
Bur
undi
CA
R
Cd'
l
DR
C
Eth
iopi
a
Cha
na
Gui
nea
Gui
nea
Bis
sau
Ken
ya
Libe
ria
Mal
awi
Moz
ambi
que
Nig
eria
Rw
anda
Sen
egal
Sie
rra
leon
e
Som
alia
Sou
th A
fric
a
Sud
an
Tanz
ania
Uga
nda
Zam
bia
Zim
babw
e
Kos
ovo
Ser
bia
BiH
Geo
rgia
Arm
enia
Aze
rbai
jan
Kyr
gyzs
tan
Tajik
ista
n
Leba
non
OP
Ts
Yem
en
Afg
hani
stan
Pak
ista
n
Ban
glad
esh
Indi
a
Nep
al
Sri
Lan
ka
Bur
ma
Uppsala/UCDPhttp://www.pcr.uu.se/research/UCDP/data_and_publications/datasets.htm
2008 X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
World Development Indicators
2009 X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
PITF http://globalpolicy.gmu.edu/pitf/pitfdata.htm
Years Vary X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
Bertelsmann
http://www.bertelsmann-transformation-index.de/fileadmin/pdf/Anlagen_BTI_2008/Detailed_Results_BTI2008.xls
2008 X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
World Governance Indicators
2.5=best-2.5=worst 2008 X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
World Bank CPIA Criterion
http://go.worldbank.org/S2THWI1X60 2008 X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
UNICEF/WHO Global Water and Sanitation Report; World Development Indicators
Latest Data varies 2000-2007
X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
Failed States Index/Fund for Peace
http://www.fundforpeace.org/web/index.php?option=com_content&task=view&id=99&Itemid=146
2009 X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
State Failure Index, Center for Systemic Peace based on Polity
http://www.systemicpeace.org/polity/polity06.htm 2007 X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
Afrobarometer http://www.afrobarometer.org/questionnaires.html 2008 X X X X X X X X X X
EIU Democracy Index Freedom House
http://graphics.eiu.com/PDF/Democracy%20Index%202008.pdf
2008 X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
Gallup World Poll 2009 X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
Failed States Index, Fund for Peace
http://www.fundforpeace.org/web/index.php?option=com_content&task=view&id=99&Itemid=141
2009 X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
CIVICUShttp://www.civicus.org/csi/phase-one/csi-country-reports
2003-2006 X X X X X X X X X X X
SIPRI, The Military Balance
http://www.sipri.org/databases/milex 2007 X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
SIPRI, The Military Balance
http://www.sipri.org/databases/milex 2007 X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
UNODC 2005-2006 X X X X X X X X X X X X X X
Global Peace Index (collected by EIU)
http://www.visionofhumanity.org/gpi/results/rankings/2008/
2009 X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
World Bank Doing Business
http://www.doingbusiness.org/ X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
World Bank Enterprises Survey
http://www.enterprisesurveys.org/
Years vary X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
Appendix C
Source Source Link Latest data
Jam
aica
Guy
ana
Bur
undi
CA
R
Cd'
l
DR
C
Eth
iopi
a
Cha
na
Gui
nea
Gui
nea
Bis
sau
Ken
ya
Libe
ria
Mal
awi
Moz
ambi
que
Nig
eria
Rw
anda
Sen
egal
Sie
rra
leon
e
Som
alia
Sou
th A
fric
a
Sud
an
Tanz
ania
Uga
nda
Zam
bia
Zim
babw
e
Kos
ovo
Ser
bia
BiH
Geo
rgia
Arm
enia
Aze
rbai
jan
Kyr
gyzs
tan
Tajik
ista
n
Leba
non
OP
Ts
Yem
en
Afg
hani
stan
Pak
ista
n
Ban
glad
esh
Indi
a
Nep
al
Sri
Lan
ka
Bur
ma
Freedom House, Freedom in the World
2008 X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
Minorities at Riskhttp://www.cidcm.umd.edu/mar/assessments.asp?regionId=99
2009 X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
Failed States Index/Fund for Peace
http://www.fundforpeace.org/web/index.php?option=com_content&task=view&id=99&Itemid=145
2009 X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
IMD World Competitive Yearbook
http://www.imd.ch/research/publications/wcy/list_of_countries.cfm
2009 X
Arab Barometer http://www.arabbarometer.org/reports/countryreports/comparisonresutls06.html
2006 X
Transparency Int’l - Corruption Perceptions Index
http://www.transparency.org/news_room/in_focus/2008/cpi2008/cpi_2008_table
2008 X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
Asian Barometerhttp://www.asianbarometer.org/newenglish/surveys/SurveyResults.htm
Years Vary X X X X X
CIRI Index http://ciri.binghamton.edu/myciri/my_ciri.asp 2006 X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
Political Terror Scale
http://www.politicalterrorscale.org/ 2007 X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
Source Source Link Latest data
Jam
aica
Guy
ana
Bur
undi
CA
R
Cd'
l
DR
C
Eth
iopi
a
Cha
na
Gui
nea
Gui
nea
Bis
sau
Ken
ya
Libe
ria
Mal
awi
Moz
ambi
que
Nig
eria
Rw
anda
Sen
egal
Sie
rra
leon
e
Som
alia
Sou
th A
fric
a
Sud
an
Tanz
ania
Uga
nda
Zam
bia
Zim
babw
e
Kos
ovo
Ser
bia
BiH
Geo
rgia
Arm
enia
Aze
rbai
jan
Kyr
gyzs
tan
Tajik
ista
n
Leba
non
OP
Ts
Yem
en
Afg
hani
stan
Pak
ista
n
Ban
glad
esh
Indi
a
Nep
al
Sri
Lan
ka
Bur
ma
Uppsala/UCDPhttp://www.pcr.uu.se/research/UCDP/data_and_publications/datasets.htm
2008 X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
World Development Indicators
2009 X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
PITF http://globalpolicy.gmu.edu/pitf/pitfdata.htm
Years Vary X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
Bertelsmann
http://www.bertelsmann-transformation-index.de/fileadmin/pdf/Anlagen_BTI_2008/Detailed_Results_BTI2008.xls
2008 X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
World Governance Indicators
2.5=best-2.5=worst 2008 X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
World Bank CPIA Criterion
http://go.worldbank.org/S2THWI1X60 2008 X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
UNICEF/WHO Global Water and Sanitation Report; World Development Indicators
Latest Data varies 2000-2007
X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
Failed States Index/Fund for Peace
http://www.fundforpeace.org/web/index.php?option=com_content&task=view&id=99&Itemid=146
2009 X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
State Failure Index, Center for Systemic Peace based on Polity
http://www.systemicpeace.org/polity/polity06.htm 2007 X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
Afrobarometer http://www.afrobarometer.org/questionnaires.html 2008 X X X X X X X X X X
EIU Democracy Index Freedom House
http://graphics.eiu.com/PDF/Democracy%20Index%202008.pdf
2008 X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
Gallup World Poll 2009 X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
Failed States Index, Fund for Peace
http://www.fundforpeace.org/web/index.php?option=com_content&task=view&id=99&Itemid=141
2009 X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
CIVICUShttp://www.civicus.org/csi/phase-one/csi-country-reports
2003-2006 X X X X X X X X X X X
SIPRI, The Military Balance
http://www.sipri.org/databases/milex 2007 X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
SIPRI, The Military Balance
http://www.sipri.org/databases/milex 2007 X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
UNODC 2005-2006 X X X X X X X X X X X X X X
Global Peace Index (collected by EIU)
http://www.visionofhumanity.org/gpi/results/rankings/2008/
2009 X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
World Bank Doing Business
http://www.doingbusiness.org/ X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
World Bank Enterprises Survey
http://www.enterprisesurveys.org/
Years vary X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
Appendix D
Source Source Link Latest data
Jam
aica
Guy
ana
Bur
undi
CA
R
Cd'
l
DR
C
Eth
iopi
a
Cha
na
Gui
nea
Gui
nea
Bis
sau
Ken
ya
Libe
ria
Mal
awi
Moz
ambi
que
Nig
eria
Rw
anda
Sen
egal
Sie
rra
leon
e
Som
alia
Sou
th A
fric
a
Sud
an
Tanz
ania
Uga
nda
Zam
bia
Zim
babw
e
Kos
ovo
Ser
bia
BiH
Geo
rgia
Arm
enia
Aze
rbai
jan
Kyr
gyzs
tan
Tajik
ista
n
Leba
non
OP
Ts
Yem
en
Afg
hani
stan
Pak
ista
n
Ban
glad
esh
Indi
a
Nep
al
Sri
Lan
ka
Bur
ma
Freedom House, Freedom in the World
2008 X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
Minorities at Riskhttp://www.cidcm.umd.edu/mar/assessments.asp?regionId=99
2009 X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
Failed States Index/Fund for Peace
http://www.fundforpeace.org/web/index.php?option=com_content&task=view&id=99&Itemid=145
2009 X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
IMD World Competitive Yearbook
http://www.imd.ch/research/publications/wcy/list_of_countries.cfm
2009 X
Arab Barometer http://www.arabbarometer.org/reports/countryreports/comparisonresutls06.html
2006 X
Transparency Int’l - Corruption Perceptions Index
http://www.transparency.org/news_room/in_focus/2008/cpi2008/cpi_2008_table
2008 X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
Asian Barometerhttp://www.asianbarometer.org/newenglish/surveys/SurveyResults.htm
Years Vary X X X X X
CIRI Index http://ciri.binghamton.edu/myciri/my_ciri.asp 2006 X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X
Political Terror Scale
http://www.politicalterrorscale.org/ 2007 X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X