ACTIONABLE GOVERNANCE

INDICATORS (AGIs)

Application and Development

Stephanie Trapnell, PRMPSApril 29, 2010

AGI Initiative http://agi

Overview What is, and is not, an AGI? AGIs on different governance

dimensions Application and development of AGIs

What are AGIs?Design

Organizational Capacities

Governance

system performa

nce

Exogenous factors

Inputs Outputs

Design (Institution

al Arrangem

ents)

Organizational Capacit

y

Governance System Performan

ce

Outcomes

The Missing Middle

Resources invested in projects to deliver its outputs. Examples: Funding, contracts, materials

Goods and services produced by the project.Examples: Surveys and Trainings conducted, Laws revised, Agencies established

Desired state of well-being -- a set of conditions, experiences or behaviors – that is the goal for change or improvement.Examples: Maternal/infant mortality rates, Standardized test scores of K-8 students

Actionable Governance

Indicators (AGIs) focus on specific

and narrowly-defined aspects of governance.

What are AGIs? AGIs provide information on the discrete

elements of governance reforms.Specific Narrowly-defined

AGIs reflect actionability, i.e., they establish the link between actions and indicator values.

What AGIs cannot do…. AGIs do not provide a comprehensive picture of

governance in a country.

AGIs do not provide a general picture of the state of a particular governance system, i.e., business regulation, freedom of information, etc.

Individual AGIs are not used for ranking countries.

Except in cases of aggregation!

Revising targets into AGIsTarget AGIExtension services reaching 10% of commercial farmers

% of extension services reaching commercial farmers

Base pay is at least 75% of real pay, up from 10%

Percentage of base pay that is real pay

Follow-up on at least 3 audits where a prima facie case is apparent

Number of audits conducted where a prima facie case is apparent

Clearly defined? Improved participatory budgeting in 30

poor municipalities.

Capacity of authorities in transport management and project implementation strengthened

Greater use of flexibility in bankruptcy law

4. AGIs on governance dimensions

10

Political Accountability• Political competition, broad-based political parties• Transparency & regulation of party financing• Disclosure of parliamentary votes

Checks & Balances• Independent,

effective judiciary• Legislative oversight

(PACs, PECs)• Independent

oversight institutions (SAI)

• Global initiatives: UN, OECD Convention, anti-money laundering

Citizens/Firm

s

Citizens/Firms

Citi

zens

/Firm

sCitizens/Firms

Decentralization and Local Participation• Decentralization with accountability• Community Driven Development (CDD)• Oversight by parent-teacher associations & user groups• Beneficiary participation in projects

Civil Society & Media• Freedom of press, FOI• Civil society watchdogs• Report cards, client surveys

Private Sector Interface• Streamlined regulation• Public-private dialogue• Extractive Industry

Transparency• Corporate governance• Collective business

associations

Effective Public Sector Management

• Ethical leadership: asset declaration, conflict of interest rules

• Cross-cutting public management systems: meritocracy, public finance, procurement

• Service delivery and regulatory agencies in sectors

Governance Dimensions

11

Political Accountability• Political competition, broad-based political parties• Transparency & regulation of party financing• Disclosure of parliamentary votes

Citizens/Firms

Public Accountability MechanismsGlobal IntegrityFreedom HouseAfro/Latino/AsiaBarometer

Different Dimensions of Good Governance Are Measured Differently (examples)

12

Different Dimensions of Good Governance Are Measured Differently (examples)

Doing Business sub-indicatorsBusiness Environment and EnterprisePerformance Surveys (BEEPS)Investment Climate Assessment (ICA)Business Risk ServiceTransparency International (TI) indexGlobal Competitiveness index

Citizens/Firm

s

Civil Society & Media• Freedom of press, FOI• Civil society watchdogs• Report cards, client surveys

Private Sector Interface• Streamlined regulation• Public-private dialogue• Extractive Industry

Transparency• Corporate governance• Collective business

associations

Public Accountability Mechanisms (PAM)Human rights databaseReporters Without BordersMedia sustainability index

13

Public Expenditure Review (PER)Public Expenditure Tracking Surveys (PETS)Public Expenditure and Financial Accountability (PEFA)Quality of Service Delivery Surveys (QSDS)Score cards

Decentralization and Local Participation• Decentralization with accountability• Community Driven Development (CDD)• Oversight by parent-teacher associations & user groups• Beneficiary participation in projects

Citizens/Firms

Different Dimensions of Good Governance Are Measured Differently (examples)

14

Judicial/Rule of Law indicatorsPEFATI indexGlobal Integrity indicatorsOpen Budget Initiative

Checks & Balances• Independent,

effective judiciary• Legislative oversight

(PACs, PECs)• Independent

oversight institutions (SAI)

• Global initiatives: UN, OECD Convention, anti-money launderingC

itize

ns/F

irms

Different Dimensions of Good Governance Are Measured Differently (examples)

15

Effective Public Sector Management

• Ethical leadership: asset declaration, conflict of interest rules

• Cross-cutting public management systems: meritocracy, public finance, procurement

• Service delivery and regulatory agencies in sectors

Public Accountability Mechanisms (PAM)PSG country-specific projectsScore cardsPEFAGlobal Integrity indexOECD procurement indicators

Different Dimensions of Good Governance Are Measured Differently (examples)

5.Application and Development of AGIs

AGI Initiative http://agi http://agidata [WB internal] https://www.agidata.org [external]

Tools• Instruments (diagnostics, data collection)• Guidance materials • Data (collection and dissemination)• Analyses (papers, graphs, tables, etc.)

Data

• One stop shop access to existing AGI data (WB and external)

• Data analysis platform

AGI Product Lines

Incorporating AGIs into projectsa

Established Data

Sources

bEstablished Data Collection

Instruments

c

Developm

ent of

New AGIs

Established Data Sources

http://agidata [WB internal] https://www.agidata.org [external]

AGI Sources

20

Content Summary: 6 thematic areas, 1182 indicators 13 sources

Note: The total includes the main indicators & sub-indicators.

CPIA DB ES

HRM (P&E)

PAMPEFA

SC (CSID

)

AFROBAR GI

IBPPD OB

Polity

IV

PFI (R

SF)0

50

100

150

200

250

300

350

2142

120

157

70

100

4 14

340

89110

9 2

No. of indicators

WB data sources External data sources

Data Sources on AGI Data Portal AfroBarometer – Governance in Social Sectors (AFROBAR) Business Environment & Enterprise Performance Surveys (BEEPS) Country Policy and Institutional Assessments (CPIA) Doing Business (DB) Enterprise Surveys (ES) Global Integrity Index (GI) Human Resource Management (HRM) International Budget Practices and Procedures Survey (IBPP) Open Budget Index (OB) Press Freedom Index by Reporters without Borders (RSF) Public Accountability Mechanisms (PAM) Public Expenditure & Financial Accountability (PEFA) Statistical Capacity (SC)

Thematic groups of AGIs Public Sector Management

Human Resource Management Public Finance Management Procurement Service Delivery Statistical Capacity

Political Accountability Electoral Competition Branches of Government Integrity Systems Budget Oversight

Checks and Balances Branches of Government Oversight Institutions Redress Institutions Other

Civil Society and Media Civil Society Organizations Media Press Freedom Index Public Access to Information Open Budget Public Access to Key Fiscal Information

Private Sector Interface Permits and licenses Regulation of Entry Property Rights Corruption in Business-Government

Transactions Infrastructure Trade Taxes

Decentralization and Local Participation Transparency of inter-governmental fiscal

relations Oversight of aggregate fiscal risk from

other public sector entities

AGI Dataset SnapshotsCategory Question being answered

Description What is the data source?

Source Is the data located on an external website?

Number of indicators on theAGI Data Portal

How many AGIs are available on the AGI Data Portal?

Country coverage What is the breakdown of country coverage by geographic region and income?

Years In which years has the data been collected?

Periodicity How often is the data collected?

Scope of indicators Are the AGIs aggregated?

Type of data Is the data based on facts or perceptions?

Object of measurement What is the data measuring?

Reliability of data How is the reliability of the data established?

Changes in methodology/coverage

Has the instrument, data collection process, or coverage of countries changed?

Limitations What are the limitations of the data regarding applicability and interpretation?

Guidance on Indicator Selection

What is the indicator measuring?

If not AGI, what are the characteristics of the underlying indicators?

How are the underlying indicators weighted?

What is the object of measurement?

Are the indicators based on facts or perception?

How was the data collected? By whom? When?

How many experts or expert assessments were used for each indicator?

How many households or individuals were surveyed?

Was the survey representative of the population as a whole?

Were the questions direct and easy to understand by respondents?

Guidance on Indicator Selection

Are comparisons over time appropriate?

Did the composition of countries change?

Did the composition of data sources change?

Did the methodology change?

Did the weighting of the indicators change?

Guidance on Indicator Selection

Are cross-country comparisons appropriate?

Is country coverage large enough to allow country rankings?

Are data sources large enough to be representative?

Is the indicator too context-specific to be compared across countries?

Guidance on Indicator Selection

Established Data Collection Instruments

Expansion of datasets in established AGI sources

Human Resource Management Public Accountability Mechanisms Public Expenditure Tracking Surveys/

Quality of Service Delivery (PETS/QSDS) Teacher Policies around the World

Considerations when using established instruments

Limitations Quality of data Reliability of data

Consistency Sampling methodology Survey instruments

Method of Data

CollectionSchedule

Data collection agent Cost

Repeated data collection

Developing new AGIs: Considerations

AGIs

Conceptual clarity

Exogenous factors

Sustainability

Availability of data

Reliability

Validity

Relevance

Conceptual Clarity Narrowly-defined indicators that reflect empirical

research

Making Services Work: Service Delivery in the Human Development Sectors

Forestry Outcomes: An Analytical Framework for Governance Reforms

Public Accountability Mechanisms (PAM) Methodology Paper

Relevance and Validity Relevance

Is the indicator capturing data that is relevant to the goals of the project?

Does the data being captured help to understand the design, capacity or performance of a system under study?

Validity (Construct)Is the indicator measuring what it claims to be

measuring, or is it measuring something else?

Reliability Inter-rater or Inter-observer

○ Two or more independent experts provide data

Parallel forms & Internal consistency○ Different questions assess the same construct

Test-Retest○ Used for items that are stable over time, such

as personality Statistical tests for reliability

Average Inter-item Correlation, Average Item-total Correlation, Split-Half Reliability, Cronbach's Alpha

Other considerations Availability of data

Can the data be captured, given the resources of the project and the context?

SustainabilityShould data collection be conducted

regularly for the data to be useful? Exogenous factors

Cannot always be accounted for in the design of the indicator

Red flags on Aggregation Aggregation requires an understanding of

the relative importance of each AGI to the concept being measured in the aggregated indicator.

○ What are the underlying weights of the AGIs? Equal weighting (averages) or unequal weighting (assigning more importance to some AGIs but not others).

This understanding should be rooted in empirical research.

Thank you!