Embed Size (px)
Citation preview
ACTIONABLE GOVERNANCE
INDICATORS (AGIs)
Application and Development
Stephanie Trapnell, PRMPS
April 29, 2010
AGI Initiative http://agi
Overview
What is, and is not, an AGI? AGIs on different governance
dimensions Application and development of AGIs
What are AGIs?Design
Organizational Capacities
Governance
system performa
nce
Exogenous factors
InputsOutput
s
Design (Institution
al Arrangem
ents)
Organizational Capacit
y
Governance
System Performa
nce
Outcomes
The Missing Middle
Resources invested in projects to deliver its outputs. Examples: Funding, contracts, materials
Goods and services produced by the project.Examples: Surveys and Trainings conducted, Laws revised, Agencies established
Desired state of well-being -- a set of conditions, experiences or behaviors – that is the goal for change or improvement.Examples: Maternal/infant mortality rates, Standardized test scores of K-8 students
Actionable Governance
Indicators (AGIs) focus on specific
and narrowly-defined aspects of governance.
What are AGIs?
AGIs provide information on the discrete elements of governance reforms.Specific Narrowly-defined
AGIs reflect actionability, i.e., they establish the link between actions and indicator values.
What AGIs cannot do…. AGIs do not provide a comprehensive picture of
governance in a country.
AGIs do not provide a general picture of the state of a particular governance system, i.e., business regulation, freedom of information, etc.
Individual AGIs are not used for ranking countries.
Except in cases of aggregation!
Revising targets into AGIsTarget AGI
Extension services reaching 10% of commercial farmers
% of extension services reaching commercial farmers
Base pay is at least 75% of real pay, up from 10%
Percentage of base pay that is real pay
Follow-up on at least 3 audits where a prima facie case is apparent
Number of audits conducted where a prima facie case is apparent
Clearly defined?
Improved participatory budgeting in 30 poor municipalities.
Capacity of authorities in transport management and project implementation strengthened
Greater use of flexibility in bankruptcy law
4. AGIs on governance dimensions
10
Political Accountability• Political competition, broad-based political parties• Transparency & regulation of party financing• Disclosure of parliamentary votes
Checks & Balances• Independent,
effective judiciary• Legislative oversight
(PACs, PECs)• Independent
oversight institutions (SAI)
• Global initiatives: UN, OECD Convention, anti-money laundering
Citizen
s/Firm
s
Citizens/Firms
Citi
zen
s/F
irms
Citizens/Firms
Decentralization and Local Participation• Decentralization with accountability• Community Driven Development (CDD)• Oversight by parent-teacher associations & user groups• Beneficiary participation in projects
Civil Society & Media• Freedom of press, FOI• Civil society watchdogs• Report cards, client surveys
Private Sector Interface• Streamlined regulation• Public-private dialogue• Extractive Industry
Transparency• Corporate governance• Collective business
associations
Effective Public Sector Management
• Ethical leadership: asset declaration, conflict of interest rules
• Cross-cutting public management systems: meritocracy, public finance, procurement
• Service delivery and regulatory agencies in sectors
Governance Dimensions
11
Political Accountability• Political competition, broad-based political parties• Transparency & regulation of party financing• Disclosure of parliamentary votes
Citizens/Firms
Public Accountability MechanismsGlobal IntegrityFreedom HouseAfro/Latino/AsiaBarometer
Different Dimensions of Good Governance Are Measured Differently (examples)
12
Different Dimensions of Good Governance Are Measured Differently (examples)
Doing Business sub-indicatorsBusiness Environment and EnterprisePerformance Surveys (BEEPS)Investment Climate Assessment (ICA)Business Risk ServiceTransparency International (TI) indexGlobal Competitiveness index
Citizen
s/Firm
s
Civil Society & Media• Freedom of press, FOI• Civil society watchdogs• Report cards, client surveys
Private Sector Interface• Streamlined regulation• Public-private dialogue• Extractive Industry
Transparency• Corporate governance• Collective business
associations
Public Accountability Mechanisms (PAM)Human rights databaseReporters Without BordersMedia sustainability index
13
Public Expenditure Review (PER)Public Expenditure Tracking Surveys (PETS)Public Expenditure and Financial Accountability (PEFA)Quality of Service Delivery Surveys (QSDS)Score cards
Decentralization and Local Participation• Decentralization with accountability• Community Driven Development (CDD)• Oversight by parent-teacher associations & user groups• Beneficiary participation in projects
Citizens/Firms
Different Dimensions of Good Governance Are Measured Differently (examples)
14
Judicial/Rule of Law indicatorsPEFATI indexGlobal Integrity indicatorsOpen Budget Initiative
Checks & Balances• Independent,
effective judiciary• Legislative oversight
(PACs, PECs)• Independent
oversight institutions (SAI)
• Global initiatives: UN, OECD Convention, anti-money launderingC
itize
ns/
Firm
s
Different Dimensions of Good Governance Are Measured Differently (examples)
15
Effective Public Sector Management
• Ethical leadership: asset declaration, conflict of interest rules
• Cross-cutting public management systems: meritocracy, public finance, procurement
• Service delivery and regulatory agencies in sectors
Public Accountability Mechanisms (PAM)PSG country-specific projectsScore cardsPEFAGlobal Integrity indexOECD procurement indicators
Different Dimensions of Good Governance Are Measured Differently (examples)
5.Application and Development of AGIs
AGI Initiative http://agi http://agidata [WB internal] https://www.agidata.org [external]
Tools
• Instruments (diagnostics, data collection)• Guidance materials • Data (collection and dissemination)• Analyses (papers, graphs, tables, etc.)
Data
• One stop shop access to existing AGI data (WB and external)
• Data analysis platform
AGI Product Lines
Incorporating AGIs into projects
aEstablishe
d Data Sources
bEstablished Data
Collection Instruments
c
Developm
ent of
New AGIs
Established Data Sources
http://agidata [WB internal] https://www.agidata.org [external]
AGI Sources
20
Content Summary:
6 thematic areas, 1182 indicators 13 sources
Note: The total includes the main indicators & sub-indicators.
CPIA DB ES
HRM (P&E)
PAMPEFA
SC (CSID
)
AFROBAR GI
IBPPD OB
Polity
IV
PFI (R
SF)0
50
100
150
200
250
300
350
2142
120
157
70
100
414
340
89110
9 2
No. of indicators
WB data sources External data sources
Data Sources on AGI Data Portal
AfroBarometer – Governance in Social Sectors (AFROBAR) Business Environment & Enterprise Performance Surveys
(BEEPS) Country Policy and Institutional Assessments (CPIA) Doing Business (DB) Enterprise Surveys (ES) Global Integrity Index (GI) Human Resource Management (HRM) International Budget Practices and Procedures Survey (IBPP) Open Budget Index (OB) Press Freedom Index by Reporters without Borders (RSF) Public Accountability Mechanisms (PAM) Public Expenditure & Financial Accountability (PEFA) Statistical Capacity (SC)
Thematic groups of AGIs Public Sector Management
Human Resource Management Public Finance Management Procurement Service Delivery Statistical Capacity
Political Accountability Electoral Competition Branches of Government Integrity Systems Budget Oversight
Checks and Balances Branches of Government Oversight Institutions Redress Institutions Other
Civil Society and Media Civil Society Organizations Media Press Freedom Index Public Access to Information Open Budget Public Access to Key Fiscal Information
Private Sector Interface Permits and licenses Regulation of Entry Property Rights Corruption in Business-Government
Transactions Infrastructure Trade Taxes
Decentralization and Local Participation Transparency of inter-governmental fiscal
relations Oversight of aggregate fiscal risk from
other public sector entities
AGI Dataset SnapshotsCategory Question being answered
Description What is the data source?
Source Is the data located on an external website?
Number of indicators on theAGI Data Portal
How many AGIs are available on the AGI Data Portal?
Country coverage What is the breakdown of country coverage by geographic region and income?
Years In which years has the data been collected?
Periodicity How often is the data collected?
Scope of indicators Are the AGIs aggregated?
Type of data Is the data based on facts or perceptions?
Object of measurement What is the data measuring?
Reliability of data How is the reliability of the data established?
Changes in methodology/coverage
Has the instrument, data collection process, or coverage of countries changed?
Limitations What are the limitations of the data regarding applicability and interpretation?
Guidance on Indicator Selection
What is the indicator measuring?
If not AGI, what are the characteristics of the underlying indicators?
How are the underlying indicators weighted?
What is the object of measurement?
Are the indicators based on facts or perception?
How was the data collected? By whom? When?
How many experts or expert assessments were used for each indicator?
How many households or individuals were surveyed?
Was the survey representative of the population as a whole?
Were the questions direct and easy to understand by respondents?
Guidance on Indicator Selection
Are comparisons over time appropriate?
Did the composition of countries change?
Did the composition of data sources change?
Did the methodology change?
Did the weighting of the indicators change?
Guidance on Indicator Selection
Are cross-country comparisons appropriate?
Is country coverage large enough to allow country rankings?
Are data sources large enough to be representative?
Is the indicator too context-specific to be compared across countries?
Guidance on Indicator Selection
Established Data Collection Instruments
Expansion of datasets in established AGI sources
Human Resource Management Public Accountability Mechanisms Public Expenditure Tracking Surveys/
Quality of Service Delivery (PETS/QSDS) Teacher Policies around the World
Considerations when using established instruments
Limitations Quality of data Reliability of data
Consistency Sampling methodology Survey instruments
Method of Data
CollectionSchedule
Data collection
agentCost
Repeated data collection
Developing new AGIs: Considerations
AGIs
Conceptual clarity
Exogenous factors
Sustainability
Availability of data
Reliability
Validity
Relevance
Conceptual Clarity Narrowly-defined indicators that reflect empirical
research
Making Services Work: Service Delivery in the Human Development Sectors
Forestry Outcomes: An Analytical Framework for Governance Reforms
Public Accountability Mechanisms (PAM) Methodology Paper
Relevance and Validity
RelevanceIs the indicator capturing data that is relevant
to the goals of the project?Does the data being captured help to
understand the design, capacity or performance of a system under study?
Validity (Construct)Is the indicator measuring what it claims to be
measuring, or is it measuring something else?
Reliability
Inter-rater or Inter-observer○ Two or more independent experts provide
data
Parallel forms & Internal consistency○ Different questions assess the same construct
Test-Retest○ Used for items that are stable over time, such
as personality
Statistical tests for reliability Average Inter-item Correlation, Average Item-total
Correlation, Split-Half Reliability, Cronbach's Alpha
Other considerations
Availability of dataCan the data be captured, given the
resources of the project and the context? Sustainability
Should data collection be conducted regularly for the data to be useful?
Exogenous factorsCannot always be accounted for in the
design of the indicator
Red flags on Aggregation Aggregation requires an understanding of
the relative importance of each AGI to the concept being measured in the aggregated indicator.
○ What are the underlying weights of the AGIs? Equal weighting (averages) or unequal weighting (assigning more importance to some AGIs but not others).
This understanding should be rooted in empirical research.
Thank you!
