1

Integrated Managing for Results

Managing for Results: USAID Nigeria & IPs

Day 3: Performance Information

2

Integrated Managing for Results

Task 5: Selecting indicators (“X” rejection criteria)

Results & Indicators Direct Objective Useful Practical Attribute Timely Adequate

1. Growth of AgribusinessAnnual sales of assisted enterprises

2. Growth of Agribusiness Annual sales attributed to USAID assistance

3. Growth of Agribusiness# customers for 1 year or moreAnnual customer sales attributed to USAID assistance

1. Effective Community-Based Resource Management Implemented

# hectares of target area using collaborative management model

2. Effective Community-Based Resource Management Implemented

# protected areas using collaborative management model

1. Improved Food Security of Vulnerable Groups% target group population with access to emergency relief supplies

2. Improved Food Security of Vulnerable Groups% target group population consuming minimum daily food requirement

3. Improved Food Security of Vulnerable Groups# person days of employment created

3

Integrated Managing for Results

Task 5: Selecting indicators (“X” rejection criteria)

Results & Indicators Direct Objective Useful Practical Attribute Timely Adequate

1. Family health improved Under 5 mortality rate in target areas

2. Family health improvedRatio of under 5 mortality rate in targeted areas to national average

3. Family health improved National under 5 mortality rate

1. Increased contraceptive security Average % warehouses with stock-outs of 1 or

more contraceptives

2. Increased contraceptive securityAverage % points-of-sale with stock-outs of 1 or more contraceptives

1. Increased private sector-led economic growth Cumulative dollar volume of private investment

in target area

2. Increased private sector-led economic growthPresent value of cumulative dollar volume of private investment in target area

3. Increased private sector-led economic growthValue of investment in target area

4

A few words about baselines A few words about baselines and targetsand targets

BaselineBaselineThe condition or level of performance that exists prior to implementation of the program or intervention

TargetTargetThe expected level of achievement of the result, as stated in the terms of the performance indicator, within a given period of time. Targets convey an understanding of the anticipated magnitude of change vis á vis USAID’s investment.

5

Integrated Managing for Results

Performance Baseline

Value of the performance indicator at the beginning of the planning period.

Baselines can/should be: Set just prior to the implementation of USAID-supported

activities that contribute to the achievement of the relevant SO or IR

Measured using the same data collection method that the SO team will use to assess progress

Changed if the data collection method changes (document!)

6

Target and Baseline: An Target and Baseline: An IllustrationIllustration

Intermediate Result Intermediate ResultIntermediate Result

Strategic Objective: A competitive,private financial sector that is more

responsive to the needs of amarket-oriented economy

Key Indicator: Value of credit/equityprovided to small and medium enterprises

by private financial institutionsBaseline:2001

$10 million

Target:2007

$50 million

7

Integrated Managing for Results

Performance Target

Commitments made by the SO team about the level and timing of results to be achieved in a specified time period.

Targets: Can be expressed in quantity, quality or efficiency May be determined by setting final target first, then interim targets May need to be set after activities or sites are selected Can be adjusted over time Should be realistic! Should be outside the margin of error of historical trend

If you don’t know where you’re going, you’ll end up somewhere else - Yogi Berra

8

Integrated Managing for Results

Baseline and Target Setting - Best Practices

Look at historical trends Consider partner and customer

expectations of performance Think about social norms and

cultural factors Consult experts/research findings Benchmark accomplishments

elsewhere Disaggregate where relevant and

possible

Contract AEP-C-00-99-00034-00

The client has determined that this document (please check one): ___ Is appropriate for public distribution on the Internet ___ Is not appropriate for public distribution on the Internet

Assessing Data QualityUSAID Performance Management Workshop

“There are three kinds of lies:

lies, damned lies, and statistics.”

--Mark Twain

10

Integrated Managing for Results

Issues

MANAGEMENT Can you make decisions based

on this data?

Better quality data leads to better informed management and planning.

REPORTING Is this data believable?

Audiences want to know how “credible” your data is so they can trust your analysis and conclusions.

11

Integrated Managing for Results

Five standards for quality of data

VALIDITY

RELIABILITY

PRECISION

TIMELINESS

INTEGRITY

12

Integrated Managing for Results

Validity

Key question: Do data clearly and directly measure what we intend?

Issue: Direct Result: Poverty of vulnerable

communities in conflict region reduced

Indicator: Number of people living in poverty

Source: government statistics office

The government doesn’t include internally displaced people (IDPs) in the poverty statistics

Issue: Bias Result: Modern sanitation

practices improved Indicator: Number of residents

in targeted villages who report using “clean household” practices

Source: door-to-door survey conducted three times a year

Most of the people in the targeted region work long hours in the fields during the harvest season

13

Integrated Managing for Results

Reliability

Key question: If you repeated the same measurement or collection process, would you get the same data?

Issue: Consistency or Repeatability Result: Employment opportunities for targeted sectors expanded Indicator: Number of people employed by USAID-assisted enterprises Source: Structured interviews with USAID-assisted enterprises, as

reported by implementing partner AAA, BBB, and CCC

The SO Team found out that the implementing partners were using these definitions: AAA – employees means receives wages from the enterprise BBB – employees means receives full-time wages from the enterprise CCC – employees means works at least 25 hours a week

14

Integrated Managing for Results

Timeliness

Key question: Are data available timely enough to inform management decisions?

Issue: How Frequent Result: Use of modern

contraceptives by targeted population increased

Indicator: Number of married women of reproductive age reporting using modern contraceptives (CPR)

Source: DHS Survey The DHS survey is conducted

approximately every 5 years

Issue: How Current Result: Primary school attrition

in targeted region reduced Indicator: Rate of student

attrition for years 1 and 2 at targeted schools

Source: Enrollment analysis report from Ministry of Education

In July 2002 the MOE published full enrollment analysis for school year August 2000 – June 2001

15

Integrated Managing for Results

Precision

Key question: Are the data precise enough to inform management decisions?

Issue: Enough detail Result: CSO representation of

citizen interests at national level increased

Indicator: Average score of USAID-assisted CSOs on the CSO Advocacy Index

Source: Ratings made by Partner XXX after interviews with each CSO

The SO team reported this data to the Mission Director:

1999 = 2.422000 = 32001 = 3.000

Issue: Margin of error Result: Primary school attrition

in targeted region reduced Indicator: Rate of student

attrition for years 1 and 2 at targeted schools

Source: Survey conducted by partner. Survey is informal and has a margin of error of +/- 10%

The the USAID intervention is expected to cause 5 more students (for every 100) to stay in school longer

16

Integrated Managing for Results

Integrity

Key question: Are there mechanisms in place to reduce the possibility that data are manipulated for political or personal gain?

Issue: Intentional manipulation Result: Financial sustainability of targeted CSOs improved Indicator: Dollars of funding raised from local sources per year Source: Structured interviews with targeted CSOs

When a SO Team member conducted spot checks with the CSOs, she found out that organizations CCC and GGG counted funds from other donors as part of the “locally raised” funds.

17

Integrated Managing for Results

Techniques to Assess Data Quality

WHY

Goal is to ensure SO team is aware of: Data strengths and weaknesses Extent to which data can be trusted when making

management decisions and reporting

All data reported to Washington must have had a data quality assessment at some time in the three years before submission.

ADS 203.3.5.2

18

Integrated Managing for Results

HOW? Steps to Conduct Assessment1. Review performance data

Examine data collection, maintenance, processing procedures and controls

2. Verify performance data against Agency data quality standards Reliability, precision, timeliness, validity, integrity

3. If data quality limitations are identified, take actions to address them Triangulate; Supplement with data from multiple sources Report the limitations Revise indicator

4. Document the assessment and the limitations in the Performance Indicator Reference Sheet

5. Retain supporting documentation in files Decisions and actions concerning data quality problems Approach for conducting data quality assessment

6. If data will be included in the annual report, disclose the DQA findings in the “data quality limitations” section of the Annual report

19

Integrated Managing for Results

Task 6: 1 hour DQA on Middleland Data

1. Collect the rating sheet from the back of the MEMS DQA form

2. Each person selects one indicator from the Middleland PMP in tab 1

3. Each person reviews the Indicator Reference Sheet for the indicator selected

4. Check off the yes/no column for the indicator and note any issues/recommendations in the appropriate column

5. List those indicators with problems on a flip chart

20

Integrated Managing for Results

Analyze data for a single indicator

Compare actual performance against target(s)

Compare current performance against prior year

Compare current performance to baseline(s)

21

Integrated Managing for Results

Analyze trends in performance

Analyze target trend(s) against actual trend(s)

Examine performance (met, exceeded or short of target) of lower results in relation to higher results

Examine trend data from critical assumption monitoring to help interpret results

22

Integrated Managing for Results

Assess USAID contribution

Examine timing of results in relation to timing of USAID efforts

Compare trends in results to trends in changes of level of effort

Compare performance to control group or benchmarks in similar environments

23

Integrated Managing for Results

What does the data tell you?

Do you trust your PMP data? Is the hypothesis working? Why and/or why not? Do you need more information?

24

Integrated Managing for Results

Performance monitoring and evaluationPerformance Monitoring Focuses on whether results are

being achieved or not Ongoing, routine Often quantitative A process that involves

identifying indicators, baselines and targets

collecting actual results data comparing performance

against target Contributes to management

decision making

Evaluation Focuses on why/how results are

achieved or not Occasional, selective Quantitative and qualitative A structured, analytical effort to

answer managers’ questions about validity of hypothesis unexpected progress customer needs sustainability unintended impacts lessons learned

Makes management recommendations

25

An Evaluation Statement of An Evaluation Statement of WorkWorkStates purpose, audience and

use of the evaluationClarifies the evaluationquestion(s)Identifies activity, program, or approach to be evaluated Provides a brief background on implementationIdentifies existing performance information sources

26

An Evaluation Aims to An Evaluation Aims to Produce…Produce…

RECOMMENDATIONS

PROPOSED ACTIONS FOR MANAGEMENT BASED ON THE CONCLUSIONS

CONCLUSIONS INTERPRETATIONS & JUDGMENTS BASED ON THE FINDINGS

FINDINGS FACTS & EVIDENCE COLLECTED DURING THE EVALUATION

27

Data analysis and useData analysis and useWhat is happening with respect to each individual result?

Were performance targets met?How did performance compare with last year’s?What is the trend fromthe baseline year?How does the trendcompare with expectations?

28

Rules of thumb…Rules of thumb…Results wereachieved as

intended

Program wasimplementedas planned

Implementationsuccess,strategysuccess

Program wasimplementedas planned

Results werenot achievedas intended

Implementationsuccess,strategyfailure

Programwas not

implementedas planned

It is not clear

whether results

would have been

achievedImplementation

failure,strategy

uncertainty

29

Implementation problems?Implementation problems?Are the activities on track and are outputs being produced on time?Are outputs being produced in sufficient quantity?Is the quality of the activities and outputs adequate?Are our partners achieving critical results and outputs for which they are responsible?

30

Data analysis and useData analysis and use

What are we going to do about it?

Do we change the...•Program?•Results framework?•Activities?•Indicators?•Targets?

31

Integrated Managing for Results

Task 7: 1 hour Performance Information and Use

1. Review the SOT analysis of performance information for Bangladesh.

2. Consider these questions: Do you trust the data? Does the Development Hypothesis work?

3. Record on a flipchart what decisions you might make, including decisions to evaluate questions that arise.