View
147
Download
1
Tags:
Embed Size (px)
DESCRIPTION
Measuring the impact of capacity development: Pact’s Organizational Performance Index
Citation preview
March 19, 2013Jennifer Mulik, MA, MPH– Global Director [email protected]
Measuring Capacity Development Performance
The problem
“The monitoring and evaluation of capacity building is as much a challenge now as it was two decades ago.”
Simister & Smith, 2010, INTRAC
Theory of Change for Capacity Development
Outputs
Change in the internal systems, skills and policies of organizations, networks and systems
Outcomes
Improved external performance of organizations, networks and systems
Impact
Improved health, environment and/or livelihoods in the communities served by organizations, networks and systems
(a)Stronger local organizations;(b)Improve their performance; and (c)Have greater impact in their target communities.
• OCA• OD
Roadmap• ISP
Tracking
• OPI• Outcome
Mapping• Most Significant
Change
• Evaluations
Defining Performance - Framework
IDRC/Universalia CD Outcomes Framework
Efficiency
Relevance
Sustainability
Effectiveness
Leadership
Structure
Human resources
Program / services
Financial
Technology
Linkages
Administrative
Political
Social / cultural
Technological
Economic
Stakeholder
History
Mission Culture
Incentives Rewards
Infrastructure
Operationalizing Performance
Effectiveness:Results
Standards
Efficiency:DeliveryReach
Relevance:Target Population
Learning
Sustainability:Resources
Social Capital
Benchmarks
Level 1 Level 2 Level 3 Level 4
Effective
Results
The organization is in the process of developing outcome level targets for its programs & services.
The organization has set clearly defined outcome level targets for its programs & services.
The organization has met over 50% of outcome level targets for its programs & services.
The organization has met over 75% of outcome level targets for its programs & services.
Level 1Level 2
Level 3Level 4
Global OPI Scores by Length of Partnership
Results
Standards
Delivery
Reach
Target Population
Learning
Resources
Social Capital
1.0
2.0
3.0
4.0
Less than 1 year
1-3 Years
More than 3 years
Global OPI Scores Time Series
Results
Standards
Delivery
Reach
Target Population
Learning
Resources
Social Capital
1.0
2.0
3.0
4.0
2011
2012
Ethiopia OPI Scores by Length of Partnership
Effectiv
enes
s
Efficienc
y
Relev
ance
Sustai
nabi
lity
1.0
1.5
2.0
2.5
3.0
3.5
4.0
1.3
1.81.5
1.8
2.62.8 2.8
2.52.5
3.1 3.1
2.5
Less than 1 year1-3 YearsMore than 3 years
Organizational Performance Index (OPI) Testing Project
The purpose of the OPI Testing Project funded by the Rockefeller
Foundation is to check the OPI reliability, i.e. whether the Index generates the same set of scores when it’s reviewed by two people,
separately.
Kappa statistic inter-score reliability
Country Pact 1 v Pact 2
Pact 1 v Partner
Pact 2 v Partner
Vietnam .804 .498 .556
Ethiopia .706 .454 .576
South Sudan .605 .571 .437
Swaziland .504 .495 .583
Nigeria .415 .502 .299
Zimbabwe .357 .724 .587
Result
Our OPI tool and methodology of application
meets the reliability standard