ACDI/VOCAM&E Community of

Practice

AGENCY-LEVEL MEASUREMENT: Can it work, and does it matter?

ACDI/VOCA Vision • VisionA world in which people are empowered to succeed in the global economy.• MissionTo promote economic opportunities for cooperatives, enterprises and communities through the innovative application of sound business practice.• Our Work

– Financial Access – Enterprise Development – Community Development – Food Security– Agribusiness

Results Based Oriented• M&E systems at ACDI/VOCA have two primary purposes:

accountability to stakeholders and learning to facilitate results-oriented program design and knowledge-based management. – Accountability refers to measuring project efficiency, effectiveness,

relevance, and sustainability of projects, as well as transparently (disclosing) sharing findings from project evaluation to stakeholders, and using the findings to inform resource allocation and other decisions.

– Learning refers to the systematic generation of knowledge that can be used to refine designs and introduce improvements into future efforts with the goal of achieving meaningful and sustainable results. M&E systems should ultimately demonstrate that ACDI/VOCA’s activities are contributing to its vision and producing sustainable and equitable development results for the communities in which it works.

Results Based Guiding Principles • Integrated into program management and technical approaches; • Systematic so as to produce reliable, accurate and useful information, while

also • Flexible to the complex environments in which ACDI/VOCA operates; • Participatory of stakeholders and beneficiaries;• Ethical in the collection and reporting of information about and to beneficiaries,

partners and stakeholders (Confidentiality and anonymity-Do no harm); • Supportive of building local capacity of staff, partners and beneficiaries in M&E; • Foster inclusivity and equity of program interventions for both men and women

and traditionally marginalized groups; and.• Results-oriented: in addition to focusing on implementation processes;

ACDI/VOCA M&E systems will emphasize results by asking the “so what” questions.

Capturing Performance Indicators on a Statistical Canvas (CaPISC)

• CaPISC is an online database that captures the performance monitoring plans for all of ACDI/VOCA’s 70 active projects. To help ACDI/VOCA measure progress towards the goals at the project and practice area level to support our attention to accountability and learning. The system will provide ACDI/VOCA with agency-wide performance measures for:

• Reporting to stakeholders and donors • Communicating with the development community• Managing regional and country programs through comparative

performance measures• Measuring beneficiaries reached by AV intervention• Measuring integration of activities• Reporting on Companywide common indicators• Managing Practice Areas/ individual Project• Developing New Business (Marketing)

Why Global Performance Indicators?

• To assess the performance of its projects, A/V would like to use a comprehensive performance measurement framework.

• ACDI/VOCA will track this progression of its performance through results reporting against indicators from the project level (predominantly outputs), to the broader program level (predominantly outcomes).

• The indicators will be standardized to the greatest degree possible, thus allowing ACDI/VOCA and its partners to compare performance across the portfolio of projects.

A Starting Point• Global Level Indicators

– Per capita expenditures (as a proxy for income) of USG targeted beneficiaries

– Number of jobs attributed to A/V implementation – Number of hectares of natural resources showing

improved biophysical conditions as a result of USG assistance

– Percent of beneficiaries adopting xx number of improved practices/technologies

Goals of CaPISC

• CaPISC allows for current project reporting thus more responsive management and program adaptation.

• Reporting with CaPISC will allow for enhanced program and institutional learning.

Uses of CaPISC – as a Results Based Management Tool• Visually depict progress on project goals and indicators• Determine progress of key project indicators COP Dashboards• Provides tools to help you analyze your data• Create reports for donors

Use CaPISC to Help in New Business Development

• Illustrate results over time and across projects• Show past experience and results• Demonstrate capability using indicator data• Model PMPs after similar ones in the system

– Uniform terminology– Global indicators

• Indicator library

Challenges/Lessons Learned

• It has been a long path to persuade the management to invest in this database.

• It is taking some time to train project level M&E specialist on how to use and enter the data

• Data quality (especially data that comes from the COs).

• Recourses (human and financial)

InterAction Forum 2012

CRSBeneficiary and Service Delivery Data (BSDI)

Measuring Agency-Level Results

May 1, 2012

Beneficiary and Service Delivery Indicators (BSDI)

Output Level Indicators

Measure Agency-wide Performance

that

Beneficiary and Service Delivery Indicators (BSDI) Presentation

CRS Fundamental Program Outputs

on

Services Delivered

Beneficiaries Served

Illustrative Program Area and Services Catalogue: Water and Sanitation

Beneficiary and Service Delivery Indicators (BSDI) Presentation

BSDI Data Flow Map

Beneficiary and Service Delivery Indicators (BSDI) Presentation

Point of Service Delivery

Beneficiary and service delivery data are registered

Regional/Partner OfficeBeneficiary and service delivery data are recorded and records

are updated

Project Management OfficeBeneficiary and services delivery

data are stored, maintained, analyzed and reported

Data

Data

On-line AgencyDatabase (PIMS) Donor

Reports

Data

Typically data are recorded on paper by individual beneficiary

Typically data are transcribed to Excel files

Typically data are stored in a project

database

BSDI Performance Products

BSDI allows accurate, annual agency-wide tracking of CRS performance

“Number” and “% targeted” of direct beneficiaries served” by Sub-program area service categories Geographic location (GPS) point of service delivery Gender and other beneficiary demographic characteristics

BSDI allows integrated technical and financial performance analysis

BSDI will track “double counted” beneficiaries in a multi-sector projects

Beneficiary and Service Delivery Indicators (BSDI) Presentation

Mission Metrics

Reflect/ Discuss

Analyze

Revi

ew

Resu

lts/L

earn

Make

Decisions

Take Action

Incoming Data

Cycle of Learning

21

Save the Children

Measuring Agency Level Results

InterAction –Forum 2012

May 01, 2012

23Blank.potx

23

Purpose, key drivers and elements of Agency Level Measurement

Purpose• Measure progress towards multi-

year strategic plan—specifically to support an overarching objective of “accountability for results”

• Communicate w/staff and board and

on performance against our intended results for children within the context of our “Theory of Change”

Key Drivers • Accountability to ourselves and to

others (specifically children)

• Need to have a tool for management purposes from the senior leadership of the organization (including the board).

Key Elements • Operational Metrics (eg, budget

performance, revenue generation, brand recognition, media rating etc)

• Programmatic and Policy Metrics (eg. Reach, Global Indicators around our Theory of Change)

Audience and Use• A dashboard for Senior

Management Team/Board (Accountability)

• Staff at all levels (Global, Regional, CO staff) – Learning, new business development

• Public/Donors – public report (Accountability)

24Blank.potx

We will…

build partnershipscollaborate with children, civil society organizations, communities, governments and the private sector to share knowledge to ensure children’s rights are met

THEORY OF CHANGE:how we work to create impact for children

… be the voiceadvocate and campaign for better practices and policies to fulfill children's rights and to ensure that children’svoices are heard (particularly those children most marginalized or living in poverty)

… be the innovatordevelop and prove evidence-based, replicable breakthrough solutions to problems facing children

… achieve results at scalesupport effective implementation of best practices, programs and policies for children, leveraging our knowledge to ensure sustainable impact at scale

Theory of ChangeHow we work to create impact for children

25Blank.potx

Understanding Results

Input Output Impact

The products, capital goods and services which result from a development intervention; may also include changes resulting from the intervention which are relevant to the achievement of outcomes.

Positive and negative, primary and secondary long-term effect produced by a development intervention, directly or indirectly, intended or unintended

The financial, human, and material resource used for the development intervention.

Input Output Impact

E.g. $ spent, staff time e.g. # health workers trained, teachers trained,

e.g. children’s lives saved; under 5 mortality rate over district, region or country

Outcome

The likely or achieved short-term and medium-term effects of one or more interventions’ outputs.

Outcome

e.g. children accessing life saving interventions;Policy change implemented

Def

initi

onE

xam

ples

RESULTS

The output, outcome or impact (intended or unintended, positive and/or negative) of a development intervention.

GLOBAL INDICATORS EVALUATIONS REACH

26

Components of our program/policy results monitoring system (towards our strategic goals)

Reach • Measures the number of children and adults who have received

program input and/pr services directly and indirectly by Save the Children and its partners in the fiscal year disaggregated by theme and sub-theme area.

Global Indicators • A set of outcome indicators linked to Save the Children’s global

outcome statements which measure achievement of outcomes.Advocacy Measurement• A tool to measure progress against an advocacy objective aimed at

policy change and/or implementation

27

Example of Global Indicators

Intended Impact: Children learn and develop with age appropriate care and education

Outcome Statement: In support of MDG2, by 2015, Save the Children will have significantly contributed to getting 2 million of the hardest to reach children into school, raising the Quality of Learning Environments (QLE), and improving the learning outcomes of more children

Indicators: Quality of LE: % of Basic Education and Early Childhood Care

and Development schools / sites supported by SC that achieve 4 guiding principles

28

Lessons learned and key challenges

• Buy in and support from the senior leadership at all levels (global and country) is critical for the success of such a system.

• Agency dashboard is used to discuss results quarterly—informing decision making on a regular basis.

• Data quality and management can be overwhelming, capacity (at all levels) and resource constraints to institute a strong system at all levels.

• Shared global objectives has helped to establish shared metrics across the Save the Children Global Movement – however determining attribution is a challenge.

• Keeping the balance between a concise dashboard (with few metrics) while providing a comprehensive picture of our results for children and the complex nature of our work.

• Reach figures do not tell us about our program impact. But measuring outcome and impact level indicators are resource intensive.

ContactMaby PalmisanoSenior Director, Monitoring and Evaluation, ACDI/VOCAmpalmisano@acdivoca.org

Harry "Hap" CarrSenior Technical Advisor for Monitoring and Evaluation, CRSHarry.carr@crs.org

Barbara WillettSenior Technical Advisor – DM&E, Mercy Corpsbwillett@mercycorps.org

Muluemebet ChekolSenior Director, Monitoring & Evaluation, Save the Childrenmchekol@savechildren.org

29