31
Focusing on Development Outcomes in Projects SARAR M&E Workshop 20-21 June 2006 New Delhi, India

Focusing on Development Outcomes in Projects SARAR M&E Workshop 20-21 June 2006 New Delhi, India

Embed Size (px)

Citation preview

Page 1: Focusing on Development Outcomes in Projects SARAR M&E Workshop 20-21 June 2006 New Delhi, India

Focusing on Development Outcomesin Projects

SARAR M&E Workshop 20-21 June 2006New Delhi, India

Page 2: Focusing on Development Outcomes in Projects SARAR M&E Workshop 20-21 June 2006 New Delhi, India

Outline

1. Management for results as an emerging practice

2. Processes to manage quality3. Methodology to manage for results

– outcome focused design and supervision

Page 3: Focusing on Development Outcomes in Projects SARAR M&E Workshop 20-21 June 2006 New Delhi, India

1 - Management for Results is an emerging practice within the development community

Compared to Financial Management and Procurement, Management for Results is an emerging practice (within the last 10 years), now rooted in a broader agenda of the development community (Paris Declaration)

Management for Results has been practiced in recent years under several titles (M&E, management by objectives, balanced scorecard, budget rationalization)

The emphasis on Management for Results grows out of a need to improve development effectiveness. It will help the Bank and it’s development partners to tell their results story in a more convincing way

Page 4: Focusing on Development Outcomes in Projects SARAR M&E Workshop 20-21 June 2006 New Delhi, India

2 - Quality Processes

Processes to manage quality Country level Portfolio level Project level

Page 5: Focusing on Development Outcomes in Projects SARAR M&E Workshop 20-21 June 2006 New Delhi, India

The Bank’s view of what needs to be monitored now includes the entire results hierarchy

ImpactsOutcomesOutputsActivitiesInputs

Procurement & Disbursements

Deliverables

Effectiveness

Efficiency

Longer-term benefits

Results

Results Hierarchy

Page 6: Focusing on Development Outcomes in Projects SARAR M&E Workshop 20-21 June 2006 New Delhi, India

Key Events in the Bank related to Results Management Jan ‘96 - Performance indicators become mandatory in all

new projects; Feb ‘97 - Strategic Compact commits the Bank to 100%

quality-at-entry by FY2001; June ‘97 - Revised supervision reporting system

introduced, including indicators monitoring; Aug. ‘97 - Revised project documentation introduced for

new operations (including the Logical Framework); 2002 - Results Secretariat unit formed within the Bank 2003 - Logical Framework ‘simplified’ to the Results

Framework 2004 - Task Force on Organizational Effectiveness

created 2005 – Paris Declaration on Aide effectiveness

Page 7: Focusing on Development Outcomes in Projects SARAR M&E Workshop 20-21 June 2006 New Delhi, India

Magnitude of the Results Management Task Bank-wide

6 Bank regions (AFR, EAP, ECA, LAC, MNA, SAR)

1282 new projects under preparation;2372 existing projects under

supervision;US$ 20-25 billion in new lending per

year; the Bank also manages US$ 9 billion in trust funds

Page 8: Focusing on Development Outcomes in Projects SARAR M&E Workshop 20-21 June 2006 New Delhi, India

ResultsBasedCAS

ResultsMonitoring &

LearningSystem

Results Based CAS Progress Reports

CountryPortfolio/ ProgramReviews

Design, prepare & implement projects

& AAA(PDO CAS)

Identify Mix of BankLending & AAA

aligned with CAS Outcomes

1

2

3

45

6

7CAS

Completion ReportPortfolio

Performance (ARPP)

National Development StrategyPoverty Reduction Strategy

Country level Results Cycle

Page 9: Focusing on Development Outcomes in Projects SARAR M&E Workshop 20-21 June 2006 New Delhi, India

SAR Portfolio level quality standard

Defines portfolio management as a management function that seeks to maximize the overall quality of the portfolio by (i) carefully screening new projects prior to entry, and (ii) identifying projects that are performing poorly for early closure or restructuring.

Page 10: Focusing on Development Outcomes in Projects SARAR M&E Workshop 20-21 June 2006 New Delhi, India

Project level standards for SAR work

a. Standard for Quality-at-Entry (Design) Project Concept Note (PCN) Project Appraisal Document (PAD)

b. Standard for Project Supervision Project status report (PSR) Project completion report (PCR)

Page 11: Focusing on Development Outcomes in Projects SARAR M&E Workshop 20-21 June 2006 New Delhi, India

a. Quality-at-Entry Standard

A fully satisfactory project contains a well conceived intervention model (i.e., storyline) which…

is designed to bring about specific, worthwhile and economic outcomes;

embodies a focused and realistic strategy;

can be expected to document and achieve results (outputs and outcomes) within the time frame and resources allocated to the project.

Page 12: Focusing on Development Outcomes in Projects SARAR M&E Workshop 20-21 June 2006 New Delhi, India

Typical Problems, Quality-at-Entry:

Bringing about specific, worthwhile, and economic outcomes: Many projects have difficulty bringing about identifiable outcomes because they are focused on other things: (i) the PDO is too high (i.e., CAS-level, long-term,

sweeping, non-specific, non-attributable, unattainable), or(ii) the PDO is too low (i.e., output level, supply-driven, internally controlled by the project, or trivial).

Embodying a focused and realistic strategy: Many projects lack sufficient focus, realism and clarity with respect to strategy;

Documenting and achieving results within the time and resources available: Many projects do not document results adequately, and then have difficulty convincing observers that the intended results were achieved.

Page 13: Focusing on Development Outcomes in Projects SARAR M&E Workshop 20-21 June 2006 New Delhi, India

b. Quality in Supervision Standard

For existing projects in the portfolio, high quality supervision is characterized by…

proactive management of emerging problems, and consistent, honest reporting focused on

documenting and achieving the expected development outcomes.

High quality supervision is accomplished without extension of the time frame or expansion of the funding available to the project.

Quality supervision also meets or exceeds the requirements of the Bank’s current guidelines and procedures for projects under supervision.

Page 14: Focusing on Development Outcomes in Projects SARAR M&E Workshop 20-21 June 2006 New Delhi, India

Typical problems in Supervision:

Supervision plans often lack a strategic vision and a necessary focus on outcomes;

Many projects fail to use key performance indicators as a management tool during supervision;

Too many projects lack a focus on outcomes in supervision reporting (ISR, MTR, ICR)

Page 15: Focusing on Development Outcomes in Projects SARAR M&E Workshop 20-21 June 2006 New Delhi, India

3 – Focus and methodology

It is important to focus on ‘outcomes’ What is a focus on outcomes?

Page 16: Focusing on Development Outcomes in Projects SARAR M&E Workshop 20-21 June 2006 New Delhi, India

Terminology:

“Results”

“Impacts”

“Outcomes”

“Outputs”

Page 17: Focusing on Development Outcomes in Projects SARAR M&E Workshop 20-21 June 2006 New Delhi, India

It is important to focus on ‘outcomes’

Results Outcomes Output, outcome or impact (intended or unintended, positive and/or negative) of a development intervention.

Specific concept: expected demand-side response needed to validate the causal chain

Success means progressive movement along the causal chain

Provides useful focus to the PDO

Page 18: Focusing on Development Outcomes in Projects SARAR M&E Workshop 20-21 June 2006 New Delhi, India

What is a focus on outcomes?

Provides a convincing storyline (outcome level);

Provides a clear and meaningful vision of success (objective statement);

Answers the question: What visible changes in behavior can be expected among end users as a result of the project, thus validating the causal chain?

Page 19: Focusing on Development Outcomes in Projects SARAR M&E Workshop 20-21 June 2006 New Delhi, India

Outcomes appear on the demand side of the project, validating the causal chain

Inputs

Activities

Outputs

Outcomes

Impacts

Demand

Supply

Project Funding ($)

Implementing Agency

(Internal)

(External)

Project implementation environment

Project Objective

Project Deliverables

Longer-term goals

+ -- ICR

Efficiency

Effectiveness

Project Components

Page 20: Focusing on Development Outcomes in Projects SARAR M&E Workshop 20-21 June 2006 New Delhi, India

The previous Logical Framework has been ‘simplified’ to the Result Framework

PDO

Hierarchy of Objectives

Key Performance

Indicators (KPIs)

Means of Verification

Critical Assumptions

Impact [IM]

Input[IP]

Level

Outcome [OC]

Output [OP]

External

InternalResult Framework

Page 21: Focusing on Development Outcomes in Projects SARAR M&E Workshop 20-21 June 2006 New Delhi, India

Outcome-focused projects utilize participatory design

Output focus Outcome Focus

Page 22: Focusing on Development Outcomes in Projects SARAR M&E Workshop 20-21 June 2006 New Delhi, India

Using a storyline to structure a design concept: Crossing the River of Uncertainties

PresentPresent

UnsatisfactoryUnsatisfactory

SituationSituation

FutureFuture

VisionVision

of Successof Success

ResultsResults

Page 23: Focusing on Development Outcomes in Projects SARAR M&E Workshop 20-21 June 2006 New Delhi, India

The parts of a storyline

Present Present unsatisfactorunsatisfactory situationy situation

Future vision Future vision of successof success

Page 24: Focusing on Development Outcomes in Projects SARAR M&E Workshop 20-21 June 2006 New Delhi, India

A dysfunctional storyline fails to provide a consistent approach for the delivery of results

ResultsResults

River of UncertaintiesPresent

UnsatisfactorySituation

FutureSatisfactorySituation

Page 25: Focusing on Development Outcomes in Projects SARAR M&E Workshop 20-21 June 2006 New Delhi, India

Project Storyline of PCN/PADProject Storyline of PCN/PAD

‘‘Crossing the River of Uncertainty’Crossing the River of Uncertainty’

A good storyline bridges key areas of uncertainty with a outcome-level approach

ResultsResults

PresentUnsatisfactory

Situation

FutureSatisfactorySituation

Specify Specify indicators to indicators to verify successverify success

IdentifyIdentify

ProblemProblem

DetermineDetermine

StrategyStrategySpecify Specify Vision of Vision of Success [OC]Success [OC]

CountryCountry

ContextContext

SectorSector

ContextContext

ProcessProcess

EnvironmentEnvironmentReportingReporting

EnvironmentEnvironment

Page 26: Focusing on Development Outcomes in Projects SARAR M&E Workshop 20-21 June 2006 New Delhi, India

Appendix:Project examples

• Rural water

• Rural health

• Transport sector

Page 27: Focusing on Development Outcomes in Projects SARAR M&E Workshop 20-21 June 2006 New Delhi, India

Rural water example (causal chain)

–Increase in rural productivity and decrease in incidents of poor health caused by water-related diseases–Significant increase in the adoption and use of improved water supply, sanitation and hygiene practices by participating households (objective achieved)–Participating households gain access to completed water supply and sanitation facilities (output delivered)–Target population selects among available options for water supply and sanitation services to meet their needs at affordable prices– Awareness raising activities to promote new integrated approach–Integrated water supply, sanitation and hygiene services are made available at the county level –Capacity building for integrated delivery of water supply, sanitation and hygiene services at the county level

Problem: Poor water supply, sanitation and hygiene practices limit the productivity of rural households and often adversely affect health

Supply

Demand

Page 28: Focusing on Development Outcomes in Projects SARAR M&E Workshop 20-21 June 2006 New Delhi, India

Rural water example storyline

Problems being

addressed

Strategy to address the problems

Vision of Success

(objective expressed as

expected outcome)

Evidence of Success(KPIs)

Poor water supply, sanitation and hygiene practices [OC] limit the productivity of rural households [IM] and often adversely affect health [IM].

The basic strategy is to expand access to improved water supply, sanitation and hygiene services in participating villages using an integrated three-in-one approach, participatory processes, and county-line agencies as service providers.

Total project costs:

US$ 75 M

Significant increase in the adoption and use of improved water supply, sanitation and hygiene practices by participating households.

-Evidence* that participating households have adopted the improved practices [OC], as expected.- Evidence* of reduced time spent in collection of water by households [OC].___* Via independent technical and social audit teams.

Page 29: Focusing on Development Outcomes in Projects SARAR M&E Workshop 20-21 June 2006 New Delhi, India

Rural health example storyline

Problems being

addressed

Strategy to address the problems

Vision of Success

(objective expressed as

expected outcome)

Evidence of Success(KPIs)

- Use of health services by poor rural households is decreasing in key provinces due to a number of factors [OC].

The basic strategy is to try to make health care more affordable at the point of use through specific initiatives related to health financing, the quality of service available, and public health more generally.

Total project costs:

US$ 72.5 M

To increase the use of health services by poor rural households in targeted provinces [OC].

- Evidence* of reduce percentage of poor households that cite the high out-of-pocket cost of health care as a reason for not accessing health services when needed [OC].___* Via independent technical and social audit teams.

Page 30: Focusing on Development Outcomes in Projects SARAR M&E Workshop 20-21 June 2006 New Delhi, India

Transport sector example storyline

Problem(s) being addressed

Strategy to address the problem(s)

Vision of Success

(objective expressed as

expected outcome)

Evidence of Success(KPIs)

Travel in the municipality is constrained by hilly terrain, traffic bottlenecks [OC] and associated slow travel times [OC].

- The project is guided by the municipal development strategy, which aims to emphasize the role of transport in improving core urban functions for the next 5 years.

- A total of five infrastructure and TA components are included in the project.

- Total project costs:

US$ 200 M

Traffic bottlenecks removed [OC] in order to improve core urban transport functions [IM] in the municipality.

Evidence* that travel times have been reduced and associated traffic bottlenecks eliminated [OC], as expected.___* Via independent technical and social audit teams.

Page 31: Focusing on Development Outcomes in Projects SARAR M&E Workshop 20-21 June 2006 New Delhi, India

The End

‘Focusing on Development Outcomesin Projects’

presented by Charles G. ChandlerAssumption Analysis, Inc.

www.AssumptionAnalysis.com