The ITAD team comprised: Kevin Tayler External Evaluation ... · WSP Council, appointed ITAD ~Water, in association with WEDC, to carry out an external evaluation of WSP’s work

REPORT

The ITAD team comprised:

Michael Snell

Robert Boydell

Kevin Tayler

Charles Chandler

Water and Sanitation Program1818 H Street, N.W. Washington, D.C. 20433, USA

Phone: (1-202) 473-9785; Fax: (1-202) 522-3313; E-mail: [email protected]

Africa (WSP-AF)World Bank, P.O. Box 30577, Nairobi, Kenya


Latin America and the Caribbean (WSP-LAC)Water and Sanitation Program, Banco Mundial, Mision Residente del Perú

Álvarez Calderón 185, San Isidro, Lima 27, PerúPhone: (51-1) 615-0685; Fax : (51-1) 615-0689; E-mail: [email protected]

East Asia and the Pacific (WSP-EAP)World Bank, P.O. Box 1324/JKT, Jakarta 12940, Indonesia


South Asia (WSP-SA)World Bank, 55 Lodi Estate, New Delhi 110 003, India

Phone: (91-11) 2469-0488, 2469-0489; Fax: (91-11) 2462-8250, 2461-9393; E-mail: [email protected]

Website: www.wsp.org

REPORTExternal Evaluationof the Water andSanitation Programfor the period1999 to 2003

External Evaluationof the Water andSanitation Programfor the period1999 to 2003

External Evaluation of theWater and Sanitation Program

for the period 1999 to 2003

Report submitted on June 2004

to the Water and Sanitation Program Council

ITAD~Water12 English Business Park, English Close,

Hove, BN3 7EE, United Kingdom

in association with

The Water, Engineering and Development Centre (WEDC)

Loughborough, United Kingdom

CONTENTS

GLOSSARY AND LIST OF ABBREVIATIONS 4

EXECUTIVE SUMMARY 8

WSP COUNCIL DECISIONS 13

1 INTRODUCTION 16

2 EVALUATION FRAMEWORK 17

3 THE PROGRAM FROM FY 1999 TO FY 2003 19

3.1 The Program’s Objectives and Field of Work

3.2 WSP’s Focus Countries

3.3 Main Activities in the Evaluation Period

3.4 Institutional Arrangements and Governance

3.5 Funding and Finance

3.6 Previous Evaluations

4 EVALUATION OF THE FIVE YEARS FY 1999 TO FY 2003 26

4.1 Introduction to this Chapter

4.2 Strategic and Planning Issues

4.2.1 Strategy and the Definition of the Program’s Role

4.2.2 Focus and Balance

4.2.3 Planning, Monitoring, Internal Evaluation

4.3 Evaluation against Our Framework

4.3.1 Relevance

4.3.2 Effectiveness

4.3.3 Efficiency

4.3.4 Sustainability and Replicability

4.3.5 Outcomes and Impact of Program Interventions

4.4 Evaluation against other Aspects

4.4.1 Institutional and Governance Arrangements

4.4.2 Responsiveness to Opportunity

4.4.3 Finance and Resources

4.4.4 Relationships between Actors

4.4.5 Cross-Cutting Issues

4.4.6 Knowledge, Communication, and Publications

4.4.7 Staffing and Staff Skills

4.4.8 Languages and Sub-Regions

4.4.9 Response to Evaluations and Self-Evaluation

4.5 WSP’s Comparative Advantage

5 FUTURE CONTEXT AND WORKING ENVIRONMENT 50

5.1 Trends in the Sector

5.2 Needs and Demands of WSP’s Partners and Clients

6 RECOMMENDATIONS 52

6.1 Continuing with the Program’s Work

6.2 Finance and Resources

6.3 Institutional Arrangements and Governance

6.4 Objectives and Strategy

6.5 Staffing and Staff Skills

6.6 WSP’s Ways of Working

6.7 Relationships with Other Global Organizations

Appendix 1 Terminology for this Evaluation 57

Appendix 2 Evolution of WSP’s Planning and Monitoring Methods, 1998-2003 59

Appendix 3 Previous Evaluations 61

Appendix 4 Finance 65

Appendix 5 People Met and Consulted 66

Appendix 6 Documents Seen and Consulted 70

Appendix 7 Terms of Reference for this Evaluation 77

Annexes concerning WSP’s four regions (each having its own contents list and appendices):

Annex AF Africa Region

Annex EAP East Asia and Pacific Region

Annex LAC Latin America and Caribbean Region

Annex SA South Asia RegionAnnexes available on request

4 External Evaluation Report

GLOSSARY AND LIST OF ABBREVIATIONS

AA or AAA Analytical and Advisory Activities (a type of funding by the World Bank)

ACDI Agencia Canadiense de Desarollo Internacional (= CIDA)

ADB Asian Development Bank

AF Africa Region

APL Adaptable Program Loan (a multi-phase lending instrument of the World Bank)

APPJ Proyecto Alimentación de Agua para Pueblos Jovenes (a project for water supply to newsettlements in Lima, Peru)

ASCID Agencia Sueca de Cooperación International para el Desarollo (= SIDA)

AusAID Australian Agency for International Development

BNWP Bank-Netherlands Water Partnership

BP Business Plan

BP-CY 1998 WSP’s Business Plan For Calendar Year 1998 (planning changed to fiscal years in 1999)

BP-FY… WSP’s Business Plan for Fiscal Year…

CARE Cooperative for American Relief Everywhere (an international NGO)

CAS Country Assistance Strategy (World Bank)

CEPIS/PAHO Pan-American Center for Sanitary Engineering and Environmental Sciences

CDD Community-Driven Development

CIDA Canadian International Development Agency

condominial This name (derived from Portuguese and Spanish terms) is given to a set of methods forapproach designing, constructing, and operating WSS services in urban and peri-urban areas. The

infrastructure for a suitable group of adjacent households or plots is designed as a unit, witha single connection to the main system (sewers or water mains). Within the unit the networkis often routed across plots or under pavements (sidewalks) instead of under roads, yieldingeconomies in the length, depth, and diameter of pipes. Secondly, social and engineeringwork is integrated: communities are involved from the planning stage onwards, and aregiven informed choices. Often hygiene education, and/or help with improvement of sanitaryfacilities inside dwellings, are included in the package.

CY Calendar Year

Danida Royal Danish Ministry of Foreign Affairs

DDA Demand-Driven Approach

DFID Department for International Development (UK Development Agency)

DGDC Directorate-General for Development Cooperation (Belgian Development Agency)

DRA Demand-Responsive Approach

EAP East Asia Pacific Region

ESA External Support Agencies

ESW Economic and Sector Work (World Bank)

EU European Union

FIS = SIF, in Spanish-speaking countries

FY Fiscal Year

External Evaluation Report 5

Go... Government of...

GTZ Deutsche Gesellschaft für Technische Zusammenarbeit (German Development Agency)

GWP Global Water Partnership

HASWAS Hygiene and Sanitation in Water Supply (Lao PDR)

HIPC Highly Indebted Poor Country

IDA International Development Association

IBRD International Bank for Reconstruction and Development (the World Bank)

IO Internal Order (a job code in the World Bank’s monitoring system)

IRC Resource Center for Water Supply and Sanitation, The Netherlands

KfW Kreditanstalt für Wiederaufbau (German aid organization)

LAC Latin America and Caribbean Region

M&E Monitoring and Evaluation

MDB Multi-lateral Development Bank

MDG(s) Millennium Development Goal(s)

MoH Ministry of Health

MoU Memorandum of Understanding

MPA Methodology for Participatory Assessment (resulting from the PLA initiative; for details seeSection 4.4.5)

n/a not applicable

NGO Non-Governmental Organization

O&M Operations and Maintenance

ODI Overseas Development Institute

OED Operations Evaluation Department (World Bank)

PAC Program Advisory Committee or, in Peru, a certain Peruvian urban WSS project

PEAP Poverty Eradication Action Plan

PLA Participatory Learning and Action Initiative (a WSP-IRA initiative in 1998-2000;for details see Section 4.4.5)

PPIAF Public-Private Infrastructure Advisory Facility

PPP Public-Private Partnership

PPPL Proyecto Piloto De Pequeñas Localidades, Peru (pilot project for small towns)

PRAGUAS RWSS project of the Government of Ecuador

PROAGUA W&S program in Chiclayo, Peru, of the German Technical Cooperation – GTZ

PRONASAR Programa Nacional de Agua y Saneamiento Rural (RWSS project of the Government ofPeru, covering small towns as well as rural areas)

PROREDES Programa de Redes Secondarias (de SEDAPAL); program of secondary networks, ofSEDAPAL in Lima

PROMESAL Proyecto de Mejoramiento Sanitario para areas marginales de Lima. Project to improve thesewerage systems in peri-urban Lima, implemented by SEDAPAL, funded by World Bankand JBIC

PROSABAR Programa de Saneamiento Básico Rural (Bolivia)

PRS Poverty Reduction Strategy

PRSC Poverty Reduction Support Credit


PRSP Poverty Reduction Strategy Paper

PSP Private Sector Participation

PSSP Private Sector Service Provider

RAC Regional Advisory Committee (of WSP; see Section 3.4 for description)

RBM Results-Based Management

RRAS-CA Red Regional de Agua y Saneamiento de Centro America (regional WSS network)

RWS Rural Water Supply

RWSS Rural Water Supply and Sanitation

S&H Sanitation and Hygiene

SA South Asia

SANBASUR Proyecto de Saneamiento Básico en la Sierra Sur (Cusco, Peru)

SDC Swiss Agency for Development and Cooperation

SEDAPAL Servicios de Agua Potable y Alcantarillado, Lima (Potable Water and Sewerage Service)

SIDA Swedish International Development Agency

SIF Social Investment Fund (usually a government-run funding instrument for channellingconsiderable investments into pro-poor interventions, often linked with debt reliefarrangements)

SSIP Small-Scale Independent Provider

SSPSS Small-Scale Providers of Sanitation Services

ST Small Towns

SWAp Sector-Wide Approach (World Bank)

TA Technical Assistance

TF Trust Fund: a World Bank mechanism whereby funds from bilateral donors are channelledto WSP regions and projects

ToR Terms of Reference; the ToR for this evaluation were issued by WSP in October 2003 as partof the World Bank’s Request for Proposals dated July 2003

UES Urban Environmental Sanitation

UNDP United Nations Development Program

UNICEF United Nations Children’s Fund

UWSS Urban Water Supply and Sanitation

WB World Bank

WHO World Health Organization

WPA Work Plan Agreement

WRM Water Resource Management

WSP Water and Sanitation Program

WSP-ESA Water and Sanitation Program-East and Southern Africa

WSP-WCA Water and Sanitation Program-West and Central Africa

WSS Water Supply and Sanitation

WSSCC Water Supply and Sanitation Collaborative Council

WSTF Water Services Trust Fund

WTP Willingness-to-Pay

WUP Water Utility Partnership



The Water and Sanitation Program (WSP) is aninternational partnership, set up in 1978, whosedeclared aim is ‘to help poor people gain sustainedaccess to improved water supply and sanitation’. Inlate 2003, the WSP management, as secretariat to theWSP Council, appointed ITAD~Water, in associationwith WEDC, to carry out an external evaluation ofWSP’s work in the five fiscal years, 1999 to 2003, and tomake recommendations for the future. This is the finalevaluation report, issued in June 2004. This mainreport, supported by four annexes, describes work inWSP’s regions, each with its own executive summary.1

Evaluation of WSP’s Work inFY 1999 to 2003Our overall finding is that WSP was effective andefficient in the service of valuable objectives duringthe review period. Various changes made duringthe five years have improved the Program’sgovernance and performance.

The evaluation analyzed the Program’s work in termsof the extent to which its own activities and outputsachieved outcomes and lasting impacts that werebeyond its control. As a partnership organizationworking in a difficult field, where many developmentinitiatives in the past have proved ineffective orunsustainable, most of the Program’s work involveschanging attitudes, spreading knowledge andunderstanding, and demonstrating the merits ofappropriate ways of providing services in particularcultures and situations. The achievement of outcomesto the benefit of poor people depends on the actionsof parties other than WSP itself. The Program is activein supporting the development of appropriatepolicies and effective institutions in the public andprivate sectors, and organizational arrangementswhere they can work well together.

WSP’s work is organized in four main areas thatinteract to strengthen each other:❚ Seeking to guide and influence the policies ofgovernments, national and local, on water andsanitation service provision for poor people;

❚ The generation, articulation, filtering, andpropagation of knowledge about improved ways of

providing such services, both organizationalmethods and technology;

❚ Capacity building to strengthen the institutions,of all sizes and kinds, which undertake the bulk ofthe service provision work; and

❚ Guiding and influencing the design of water andsanitation projects to be funded and implementedby others, which gives the Program valuableleverage to stimulate large-scale outcomes withlimited resources.

All these ways of working are applied in three typesof situations: rural, small towns, and urban (usuallyperi-urban), though many aspects cut across thispragmatic classification.

WSP focuses on a limited number of countries tomake good use of limited funds. The geographicfocus sharpened during the five years, reducing from27 to 22 countries, which are now distributed amongthe Program’s regions as follows:

Africa Region (AF): 9 countries

East Asia-PacificRegion (EAP): 5 countries

Latin America &Caribbean Region (LAC): 5 countries

South Asia Region (SA): 3 countries,including Indiawhose large size andfederal structure makeit a special case;several IndianStates are involved

Finance and ResourcesThe Program’s annual expenditure averaged US$ 13million during the last four years evaluated, rangingfrom 12 to 15 million. Global management costswere kept down to about nine percent of totalexpenditure. The current expenditure rate remainsat about US$ 14 million. Assured funding for the nextthree years is about US$ 12.5 million per year, orup to US$ 16 million, with some funds that arecurrently under negotiation. Most donors tend to

EXECUTIVE SUMMARY

1 These four annexes can be viewed and printed from www.wsp.org


commit funds for only about three years ahead,which is unfortunate because most of WSP’sinterventions need periods of the order of five yearsor more to achieve their full potential. Roughly two-thirds of the future funding is tied by donors toparticular countries, themes or projects. WSP’sability to respond to emerging opportunities, whichit generally does well, is constrained by the limitedproportion of core or untied funding. Such fundsare also valuable for smoothing out fluctuations infunding for particular regions, which otherwise canlead to loss of momentum and sometimes loss ofexperienced staff. A funding crisis in the LAC regionin 2001 did have such consequences, but theoperation was kept going at the initiative of SDCand with the help of the World Bank; that region’sfunding has now stabilized again. Funds aremanaged under the World Bank’s procedures; thisevaluation does not include financial auditing, butthe accountability appears satisfactory.

These resources currently enable WSP to deployabout 80 staff, of which less than 10 percent are inthe headquarters group. The Program also sharessome administrative staff time at headquarters, andmakes good use of interns and consultants.

Given the effectiveness of WSP and the large globalneed for sustainable WSS, there is scope for WSP tousefully expand the scale of operations; that scale iseffectively limited by donor funding and not by theProgram’s inherent capacity. We consider that thecurrent management structure, with four regions,could fruitfully handle an increase in turnover of upto 50 percent. This would, of course, depend ondonor funding, and on being able to attract extrastaff of the necessary caliber and experience. Anexpansion much beyond that level might requiresome restructuring.

Strategy, Planning, Monitoring,and Resource ManagementWe consider that, at the start of the five-yearevaluation period, there was an underlying lack ofclarity about ends and means, especially in WSP’swritten statements, and subsequent formulationshave not effectively replaced those statements. At apractical level, the consequences were not verysevere because the Program’s implied objectiveswere usually understood by its relatively smallnumber of key staff. In our view, the Program hasthus succeeded in maintaining a good focus on its

declared core objective, namely helping its clientsand partners to achieve sustainable access to WSSfor poor people. The focus has been sharpened overthe evaluation period, both geographically andthematically. Focusing on the poor is a complexmatter, and specific pro-poor interventions are notnecessarily always the best mean of achieving theobjectives; WSP must continue to adopt a broad andthoughtful approach. The continuity and henceeffectiveness of WSP’s focus would be helped if itsdonors could find ways to plan any change ofpriorities several years ahead.

Planning and monitoring were weak in the earlieryears of our evaluation period, with several changesin the format of Business Plans, but steps were takenand they have now been much improved. Thereremains a need to consolidate cost-effective waysof combining coherent, outcome-focused, multi-year planning with creativity and adequateflexibility, and to achieve consistent application ofsuch planning and monitoring procedures over allregions and all projects.

ImpactThe Program’s work has evidently achieved relevantoutcomes in diverse situations, with considerableimpact already observed and more considered likelyto follow. Some interventions have been lesssuccessful than others on these criteria, but that isto be expected in a difficult field and for aninstitution that is often leading the way rather thanfollowing proven recipes.

Institutional and Governance ArrangementsThe Program is an international partnershipreporting to a Council and administered by theWorld Bank. It works through four fairly autonomousregional offices and a small headquarters, mostlylocated in or near World Bank offices. We considerthat these fundamental institutional andgovernance arrangements are satisfactory, and forpractical purposes the best available, but that thedetails could be improved and clarified. Inparticular the degree to which WSP is in practicecontrolled by the World Bank should be discussedby the Council and an agreed statement shouldreplace the ambiguous provisions of the Charter. Weconsider that the provision for the Council to beautomatically and always chaired by a World Bankofficial could with advantage be changed. This refersnot to presiding at meetings but to the role of the


Chair between the annual Council meetings whenit is required to provide leadership and resolveconflicts, and to represent the WSPC in interactionswith other stakeholders.

Interactions with Other Actors in the WSS SectorThese interactions are, of course, crucial for arelatively small organization operating as apartnership and relying on others to generatelarge-scale impacts. WSP has many importantrelationships with other sector actors, and most ofthem are very satisfactory. There are some tensions,not necessarily of WSP’s making, which could bealleviated by good communication and byclear thinking and discussion about shared andseparate objectives.

Knowledge, Communication, and PublicationsWSP’s knowledge-related work is important andfruitful as a complement to its other activities, andis mostly well done. Quality and appropriatenessof publications are usually good, though inevitablyvariable in a difficult field where all staff are involved,including the less experienced ones. Standards haveimproved in the course of the evaluation period.

Staffing and Staff SkillsWSP has an excellent core of staff with relevant skillsand experience, but they are spread quite thinly andslightly weakened by recent losses. But the Program’swork poses big challenges, and we see the need forcontinuing regular review of future staffing,grounded in a very clear understanding of WSP’sspecial objectives and its comparative advantagerelative to other organizations.

Corporate Culture andResponse to CommentariesThe Program’s work was the subject of manyevaluations during the review period, and itresponded with several beneficial changes. Wenotice that the Program’s staff tends to respond toany criticism, even if mixed with much praise, in asensitive and defensive way. This reflects the cultureof the World Bank to which WSP staff are continuallyexposed, and also their dedication, professionalism,and pride in their work. It may also reflect a desireto achieve measurable targets, though we considerthat too much emphasis on such targets tends toundermine the value of a ‘learning organization’. Butthis sensitivity seems to us unnecessary and notentirely healthy. The Program works at the leadingedge of a field which is professionally and

intellectually difficult and challenging, and no oneshould expect perfection. Inevitably indevelopment work, some interventions fail, orsucceed only partly. It appears to us that a moreself-questioning attitude on WSP’s part, togetherwith a less defensive attitude to constructivecriticism, would strengthen rather than weakenits effectiveness.

WSP’s Comparative AdvantageWe conclude that the Program has significantcomparative advantages in relation to otherorganizations working in its field or overlapping toa greater or lesser extent with its functions. Itspartnership style enables it to collaborate andcomplement those other organizations whereappropriate. As analyzed in this report’s Section 4.5,there are several types of work, such as advocacy,policy advice, learning or knowledge management,which are also covered by others. But WSP’s globalreach, valuable contacts, long track record, highprofessional capacity, and ability to maintaincontinuing real-time contact with key decision-makers all give it particular advantage.

RecommendationsWe expect the current trends to continue towardsincreasing decentralization in relevant countries,and towards an increase in the need for sustainableWSS serving poor people in urban and peri-urbanareas without the reduction of the needs in ruralareas and small towns. Decentralization will placeheavy demands on local government bodies, whichoften have little relevant experience. TheMillennium Development Goals are an importantexpression of collective will, though as looselydefined numerical targets for access to services theyare not useful for planning and resource allocation.WSP’s work is and remains highly relevant to severalof the MDGs and their associated targets.

Our principal recommendation is that WSPshould continue with its valuable work, and shouldseek a significant funding increase for the next fiveto 10 years.

By ‘significant increase’ we mean something of theorder of 30 to 60 percent, rather than 10 percent or100 percent. This does not refer to any quantitativeanalysis; it merely reflects our views of the sector’sneeds, of the Program’s comparative advantages,and of the scale of work that could be handledwithout major restructuring of the Program. The next


five to 10 years will be crucial for the sector, not leastin relation to the MDGs, whose time-scale runs to2015. To attract a funding increase of this order,WSP would, of course, have to persuade existingdonors to give more, and/or persuade new donorsto start giving.

Our more detailed recommendations in Chapter 6suggest actions by WSP and its Council that weconsider would help to do this. We do not prejudgehow easy or difficult these actions may be. The mainpoints, not in order of importance or urgency, are:

a) The Program should remain a partnershiporganization directed by its main stakeholdersrepresented on its Council, while being hostedand administered by the World Bank, andoperating under World Bank financial andaccounting systems.

b) Within that structure, the Council shouldconsider taking on a more active role than atpresent, through a delegated executivecommittee or otherwise, in the direction andcontrol of the Program.

c) The Council should debate the extent to whichWSP’s thematic and technical work ought tobe controlled by the World Bank, and if possiblereplace the ambiguous provisions of theCharter with a clear and agreed statement.

d) The Council should consider changing its rulesso that it elects its Chair from time to timeinstead of being chaired automatically by anofficial of the World Bank.

e) The Council, the WSP, and the World Bankshould all take conscious steps to ensure thatother parties understand the distinct nature ofthe program and do not perceive it as asubservient agent or arm of the World Bank.

f) WSP central management should maintain itspresent balance between controlling theregions and allowing them more autonomy,and the regions should respect that balanceand comply with the necessary centraldiscipline regarding planning, monitoring, andresource management.

g) A way should be sought for reporting the viewsof the Regional Advisory Councils to the annualCouncil meetings.

h) WSP should seek to clarify its strategy bysharpening the definition of its objectives,emphasizing the need to address institutionalfailure and ineffectiveness.

i) WSP should both plan and monitor its resourceuse with a multi-year perspective, rather thanjust within fiscal years as it has tended to dountil recently.

j) Donors should seek ways to commit fundsseveral years in advance, typically five years.

k) WSP should make systematic and transparentarrangements for coping with the fact thatdonors often do not commit funds more thanthree years in advance.

l) The Program and the donors together shouldseek ways of achieving a joint flexibilityin resource allocation by increasing theproportion of untied, partially-tied orcore funding.

m) WSP should seek ways to improve the flexibilityof resource allocation in the course of eachproject’s life.

n) WSP should continue with regular reviews ofthe mix of staff skills, and seek ways ofproviding key staff with a degree of securityof employment over more than aboutthree years.

o) When designing its own projects, the Programshould strengthen its efforts to identify andspell out intended outcomes and impacts,rather than concentrating on outputs, andshould link this to its internal monitoring byfinding simple ways to record the perceivedachievement or otherwise of those outcomes.

p) When a WSP project is being designed, thiscareful formulation of the intended outcomesand impacts should lead to identification of theother actors whose attitudes and behavior needto change, and then to an analysis of theirconstraints, habits, motives, and priorities.

q) When asked to help others to design aninvestment project, WSP should take carealways to offer and apply its own distinctiveexperience and judgement, rather thancurrently fashionable solutions.


r) In view of the political sensitivity ofinstitutional reform, WSP should furtherstrengthen its efforts to build up close andtrusting relationships both with donors/funders and with government officials andlocal politicians.

s) WSP should consider using a wider range ofmedia, and should more often sharethe distribution of publications withother organizations.

t) WSP should seek more ways to exchangeinformation between its regions.

u) WSP should continue and strengthen itsefforts to overcome language barriers and

to foster the work of sub-regions such asWest Afr ica; donors should considerhelping with the costs of translation andinterpretation.

v) WSP staff should have enough confidence intheir own caliber and achievements to adopta more self-questioning culture, together witha less defensive attitude to constructivecriticism.

WSP should reconsider how it works with otherorganizations on the global WSS and related fields,building on its review of objectives and strategy, andon the comparative advantages of differentorganizations.


WSP COUNCIL DECISIONS

WSP Council Decisions of May 28, 2004,on the Response to the ExternalEvaluation and Adoption ofRecommendationsA draft version of this report was presented to theWSP Council at its meeting in Berlin on May 27,2004, and discussed. On May 28, the WSP

management presented its response to theevaluation, and in particular to therecommendations set out in the draft evaluationreport’s Executive Summary (which were almost thesame as those of this final report). The table belowrecords the recommendations, the managementresponses, and the Council decisions.

Evaluator Recommendation

The Council should consider changingits rules so that it elects its Chair fromtime to time instead of being chairedautomatically by an official ofthe World Bank.

The Council, WSP, and the World Bankshould all take conscious steps toensure that other parties understand thedistinct nature of the Program and donot perceive it as a subservient agent orarm of the World Bank.

WSP central management shouldmaintain its present balance betweencontrolling the regions and allowingthem more autonomy, and the regionsshould respect that balance and complywith the necessary central disciplineregarding planning, monitoring, andresource management.

A way should be sought for reporting theviews of the Regional Advisory Councilsto the annual Council meetings.

WSP should seek to clarify its strategyby sharpening the definition of itsobjectives, emphasizing the needto address institutional failureand ineffectiveness.

WSP should both plan and monitor itsresource use with a multi-yearperspective, rather than just withinfiscal years as it has tended todo until recently.

Program Response

No change recommended.

Agreed. Clear instructions to beissued by WSP management toProgram staff clarifying distinctnature of Program[by August 1, 2004].

Agreed.

Agreed. RAC reports/MoUs to bedistributed to Council[by August 1, 2004].

Agreed. WSP will issue a revisedstatement clarifying itsobjectives [by January 1, 2005].

Agreed. As acknowledged inreport, in FY04 WSP beganplanning on a multi-year basis,and will continue to improvethe process.

Council Response

No change.

Agree with WSPresponse.

Recognizing theregional nature of WSP,the global identity ofWSP should bemaintained andstrengthened.

Agree with Programresponse.

New Program Manager(PM) to issue revisedstatement to reflectMDG priorities byJanuary 1 , 2005.

Agreed. Council wouldlike to see modificationsreflected in the BPfor FY05.



Donors should seek ways to commitfunds several years in advance, typicallyfive years.

WSP should make systematic andtransparent arrangements for copingwith the fact that donors often do notcommit funds more than three yearsin advance.

The Program and the donors togethershould seek ways of achieving a jointflexibility in resource allocation byincreasing the proportion of untied,partially-tied or core funding.

WSP should seek ways to improveflexibility of resource allocation in thecourse of each project’s life.

WSP should continue with regularreviews of the mix of staff skills, and seekways of providing key staff with a degreeof security of employment over morethan about three years.

When designing its own projects, theProgram should strengthen its efforts toidentify and spell out intendedoutcomes and impacts, rather thanconcentrating on outputs, and shouldfind simple ways to systematicallyrecord the perceived achievement orotherwise of those outcomes.

When a WSP project is being designed,this careful formulation of the intendedoutcomes and impacts should lead toidentification of the other actors whoseattitudes and behavior need to change,and then to an analysis of theirconstraints, habits, motives,and priorities.

Program Response

WSP will develop a fundingstrategy and continue to engagewith donors on an individualbasis for future funding.Draft funding strategy sent todonors by December 1, 2004.

Agreed. As above, to be reflectedin the funding strategy.

Agreed. As above, to be reflectedin the funding strategy.

WSP already reviews and revisesallocations of resources at mid-year and start of following year.Starting in FY05, WSP will conductthe review at the project level.

WSP and WSS Sector Boardalready review staff skillsperiodically. By nature of itsfunding, WSP can only issuefixed-term contracts. Subjectto performance, theseare extended.

Agreed. As acknowledged in thereport, from FY04 onwards WSPhas begun outcome/impact-based planning, and willintroduce semi-annualproject-level monitoring in FY05.

Agreed. The outcome indicatorsWSP develops already take intoaccount the expected change inbehavior of the client. These willbe further strengthened.

Council Response

Agreed. Chair would liketo involve finance sub-committee (Dutch,SDC, DFID, Danida) inthis process.




Council asks Bank totake advantage ofexisting and newemployment conditionswithin the Bank tomaximize security ofemployment.

Agree. Council urgesgreater emphasison outcomes.

Agree. Council urgesgreater emphasison outcomes.



When asked to help others to design aninvestment project, WSP should takecare always to offer and apply its owndistinctive experience andjudgement, rather than currentlyfashionable solutions.

In view of the political sensitivity ofinstitutional reform, WSP should furtherstrengthen its efforts to build up closeand trusting relationships both withdonors/funders and with governmentofficials and local politicians.

WSP should consider using a widerrange of media, and should more oftenshare the distribution of publicationswith other organizations.

WSP should seek more ways toexchange information betweenits regions.

WSP staff should have enoughconfidence in their own caliber andachievements to adopt a moreself-questioning culture, togetherwith a less defensive attitude toconstructive criticism.

WSP should reconsider how it workswith other organizations on the globalWSS and related fields, building on itsreview of objectives and strategy, and onthe comparative advantages ofdifferent organizations.

Program Response

Depending on the context anddrawing on its global knowledgeand field experience, based onclient demand, WSP providesappropriate TA.

Agreed. WSP’s permanent fieldpresence and its role as a neutralbroker has enabled it to buildtrusting relationships with sectoractors and provide real timetechnical assistance on acontinuous basis. Continuousefforts will be made tostrengthen partnerships.

Agreed. WSP has begun using arange of media and willintegrate the approach into itsforthcoming communicationstrategy [by November 1, 2004].

Agreed. WSP regions already havecross-regional projects. RTLs willtake initiative to promotemore effective cross-regionalexchanges. One of the mainresponsibilities of the new globalthematic coordinator willbe promoting cross-regionalexchanges.

WSP welcomes constructivecriticism and healthy debate.

Agreed. WSP’s modus operandi isworking in partnerships. TheProgram will revisit its existingpartnership strategy andrevise it as necessary[by January 1, 2005].

Council Response

Council agrees withProgram, and urges WSPto bring its comparativeadvantage to bear,promoting innovationand new ideas.

Council agrees withProgram, and urges WSPto bring its comparativeadvantage to bear and tocollaborate with UNsystem and otherpartners.

Council agrees withProgram response.

Council agrees withProgram response, andwould like to see thisitem reflected in agendaat future meetings. Chairand WSP managementrequested to include inthe result agreement ofthe PM and the RTLspecific cross-regionalwork in global andregional work programs.

Council commends thequality of the WSP staff.

Council urges new PMto take lead in revisingthe strategy.Council reiterates theimportance it sees in thecoordination betweenWSP and other leadactors in the sector, suchas WSSCC/GWP.


INTRODUCTION

The Water and Sanitation Program (WSP) is aninternational partnership, set up in 1978, whosedeclared aim is ‘to help poor people gain sustainedaccess to improved water supply and sanitation’.Though it started as a series of separate projectssupported by the UNDP and implemented by theWorld Bank, the WSP has now evolved into a globalpartnership, active in some 30 countries andfinancially supported mainly by bi-lateraldevelopment agencies. Its governance wasredefined in a ‘Program Charter’ in 2001, which setup a Water and Sanitation Program Council andmade provision for that Council to commissionregular reviews of the performance of the WSP. InOctober 2003 the WSP management, as secretariatto the WSP Council, issued Terms of Reference (ToR)for an External Evaluation of WSP’s operations inthe five-year period, July 1998 to June 2003, withrecommendations for the future. In December 2003the British firm ITAD~Water, in association with theWater, Engineering and Development Center(WEDC), was appointed to carry out the evaluation.Work began on January 12, 2004 and was scheduledto be completed by July 2004. The evaluation wasconditioned by its budget and ToR. Only onemember of the evaluation team visited each of WSP’sfour regions, and one of these visits was delayed untilApril. The team is referred to in this report as ‘we’.

This is the final overall evaluation report. It is issuedin late June 2004.

1This evaluation has been guided by our discussionswith many stakeholders and WSP staff in the fourregions and the headquarters, and by responses fromothers to whom we sent a questionnaire. Names ofboth groups are listed in Appendix 5. It has also beenguided by our study of many documents, listed inAppendix 6. The evaluation remains an independentand external one, and all the conclusions andrecommendations are our own.

We acknowledge with thanks the assistance of theWSP management in Washington and the regions,and of all the WSP staff who have helped us withfinding and collating information and with makingrelevant contacts. We also thank the manystakeholders in various countries who have giventime to meet us and have responded to our questionsor showed us projects implemented with or by WSP.

This main report presents the evaluation at globallevel, for all regions and the center. Supported byappendices, it has five chapters: Chapter 2 explainsthe framework of this evaluation, and Chapter 3describes WSP’s work in the region. The evaluationof the five-year period is then presented inChapter 4, which covers the 28 topics required bythe ToR. Chapter 5 summarizes expected trends andneeds in the next few years, and recommendationsfor the future are in Chapter 6. Regional details arepresented in four annexes, each of which has its ownexecutive summary and appendices.


In order to give our evaluation a clear frameworkand nomenclature, Table 2.1 defines a hierarchy ofplanning and evaluation levels, from inputs toimpacts. It is worded in a general way to refer to anysort of project relative to its implementing agency,by which we mean the party who controls theactivities which use inputs to produce outputs.(Our table uses the same terms as the ‘Results-basedmanagement (RBM) framework’ which wasprepared for WSP in 2003,2 but defines themmore closely.3)

The important distinction, emphasized in theright-hand column of this table, is between thingsthat an implementing agency can control, throughits resources, activities, and outputs, and things thatit cannot control, though it often seeks to influencethem. The identification of outcomes as distinctfrom outputs therefore depends on the identity ofthe implementing agency. In many developmentprojects the implementing agency is a governmentdepartment, so any project-induced changes in that

EVALUATION FRAMEWORK2

2 Results-based management framework for WSP; Report from the initial phase; Per O. Bastoe, March 2003.

3 Some versions of RBM terminology use ‘outcomes’ as a wider term, including within it our ‘impacts’ under the name ‘final outcomes’.It is widely recognized that a particular RBM framework needs to be designed for each specific organization, especially one that works bypartnerships (as emphasized in the undated UNDP document, ‘Introduction to Results-Based Management; RBM in UNDP: Overviewand General Principles’, which was developed in cooperation with SIDA).

department’s attitudes or behavior are internal tothat project and are therefore not outcomes ofsuch projects.

In this evaluation we are dealing with the work andactivities of WSP, as an implementing agency of whatit and its subcontractors do. If WSP, and anysubcontractors under its control, perform activitiesand produces outputs which change the attitudesand behavior of other parties, those changes areoutcomes of WSP’s work. This holds even if thechanged party is the implementing agency of arelated and wider development project: suchchanges are outcomes of WSP’s work though onlyoutputs of that wider project; we are evaluating WSP,not the wider projects.

For example, suppose that a governmentdepartment responsible for developing rural watersupply schemes observes that most of theseschemes are not sustainable and the users are notsatisfied; the infrastructure breaks down and the

Hierarchy of Levels

Impacts

Outcomes

Outputs

Activities

Inputs

Explanation

Aggregated effects in the long term (several years at least) andon a wide scale (whole counties or regions), which can be partlyattributed to the implementing agency’s work.

Behavioral changes by target groups, or other external desiredresponses to the implementing agency’s outputs, in the shortterm and on a local scale.

Deliverables, products or services which stimulatedesired outcomes.

Discrete defined items in work plans or Business Plans, carriedout by the implementing agency, or by others under its control.

Resources, such as money, staff or time, controlled by theimplementing agency.

ControlDistinction

Outside thedirect controlof theimplementingagency.

Within thedirect controlof theimplementingagency.

Table 2.1: Hierarchy of Planning and Evaluation Levels


users do not have the skills and financial resourcesto repair and maintain it. The department asks WSPto help, and after analyzing the situation WSPsuggests adding new steps to the implementationprocess for future schemes (for example,participatory needs assessment, design guided byusers’ informed choice, setting up effective usergroups for maintenance). WSP persuades thegovernment people of the merit of these steps andtrains government staff to use them. Later schemesare implemented in this new way and provesustainable. For the government department, asimplementing agency for the water supplyschemes, the new steps are activities, the better-designed schemes and their capable user groups areoutputs, and the sustainable water supplies enjoyedby the users are outcomes. For WSP, as executingagency of the advisory and training functionembedded in this process, the analysis andpersuasion are activities, the workshops and trainingmanuals are outputs, and the more enlightened andsuccessful nature of the government department isan outcome. The sustainable water supplies enjoyedby the users is another, more indirect outcomewhich, along with the final impact (sustained watersupply over many years and beyond the location ofthe original project), is common to both, thegovernment department and WSP.

For WSP’s interventions or projects, the distinctionbetween things within or outside ‘the direct controlof WSP’ is a useful one, but it must be rememberedthat activities and outputs can be designed to makeoutcomes and impacts more likely to follow (that is,to be more effective). When an intervention is welldesigned in this way, certain matters are in effectshifted from outside its field of action (assumptions)to inside it (activities and outputs). This is oftendone by including persuasion and deliberateattitude-changing work among WSP’s activities, to

4 Most of the text of these working definitions was provided by WSP in February 2004.

increase the likelihood of other parties changingtheir priorities and methods.

Appendix 1 describes how we use five further termsspecified in our ToR, defining the following specificaspects for our evaluation:❚ relevance❚ effectiveness❚ efficiency❚ sustainability and replicability❚ responsiveness❚ strength of institutions

As a partnership organization, WSP works withmany other organizations in various ways. Itdistinguishes between different sorts of relationshipby the use of the following terms, which are notprecisely defined but which help to indicate the sortof relationship in particular cases4:

❚ Ultimate Clients: Poor WSS consumers served byWSP’s direct clients.

❚ Direct Clients: Primarily refers to developingcountry governments (especially with respect tocountry work), but also refers to private sectorservice providers, semi-autonomous utilities, etc.In addition, WSP loosely uses this term to refer toknowledge clients, such as NGOs and strategicpartners, when WSP provides information to them.

❚ Strategic Partners: These are a wide range ofagencies with whom WSP collaborates in someactivities, such as WSSCC, UNICEF, UN Agencies,WaterAid, WUP, and many NGOs.

❚ Financing Partners: Organizations who supportWSP financially have a special stakeholderrelationship to the Program. Most of them channelthe funds through the World Bank’s Trust Fundmechanism.


3.1 The Program’s Objectives and Field of WorkWSP’s work was defined over the five-year evaluationperiod in a series of Business Plans, whose evolutionis described and commented upon in Appendix 2and Section 4.2.3. This chapter summarizes whatWSP was aiming to do in the five years, while thefour annexes to this report (available on request)give details of what was being done in the four regions.

The Program’s work at the start of this period wasplanned within the framework of the 1999-2003

THE PROGRAM FROM FY 1999 TO FY 20033strategy document, whose main statements andcategories are summarized in Table 3.1.

For practical purposes that strategy statement wassuperseded by a 2001 document, ‘WSP Renewal –Strategy Update and Progress Report’, prepared forthe October 2001 Council meeting in The Hague,which re-stated the five ‘product lines’ and added‘sanitation and hygiene’ to the agenda. It alsoemphasized, as its central theme, the achievementof institutional change and reform on a wide scale.The Program Charter, also issued in 2001, used a

Label

‘missionstatement’

‘mutuallysupportingobjectives’

‘products’or‘productlines’

‘learningagenda’

‘keyfactors/buildingblocks’

Table 3.1: Summary of WSP’s Original Declared Strategy for 1999-2003

WSP’s Statements, in its ‘Program Strategy: 1999-2003’

‘to help poor people gain sustained access to improved WSS’‘…works with partners in the field to seek innovative solutionsto the obstacles faced by poor communities,and strives to be a valued source of advice to achieve widespread adoption ofthese solutions’

1a generate knowledge (‘learning’)1b communicate knowledge2 strengthen sector policies3 improve investment effectiveness

1 pilot and demonstration projects2 knowledge generation and management3 sectoral networking4 policy support5 support for large-scale investment projects

(Each product has ‘expected outputs’ defined in the 99-03 Strategy, Part 4)

rural agenda urban agenda small towns agenda

1 shared values2 diverse and highly skilled staff3 quality assurance processes4 partnering and networking5 decentralized operational structure6 governance arrangements7 sustainable financial base

Source: Water and Sanitation for the Poor: Innovation through Field Experience – Program Strategy:1999-2003; WSP


similar statement of objective to the 1999-03 strategydocument. There was, from 2000 onwards, adiagrammatic ‘Strategic Results Framework’, whichrestated the strategy in the form of a hierarchicaltree.5 Starting with the above five ‘product lines’, itnamed three ‘outputs’ corresponding to policy andstrategy support, investment support (improvingquality of the investments of other parties), andknowledge management. This led to statements ofhigher objectives. Formulations about strategyduring these years tended to interpret the conceptof ‘helping poor people gain sustained access…’ inthe mission statement in terms of helping the WSP’sdirect clients (usually government or serviceproviders) to help their clients, the poor. This andother concepts concerning the Program’s strategyand ways of working were in general internalizedby its staff, without much reference to the formalstatements quoted here.

The Business Plan for FY 2001 included a globallogframe, putting its emphasis on effectiveapproaches by WSS service providers. That logframeincluded time-related targets. To illustrate this,Table 3.2 quotes those at the three-year timescale(up to the end of FY 2003) within a 10-yearperspective. They were presented as objective-levelindicators under the heading ‘impact outcomes’.

These three-year targets are generally at the outcomelevel, since their achievement on a wide scaleimplies action by parties outside WSP. Theaccompanying 10-year targets involved large-scaleinstitutional reforms and hygiene and sanitation

5 One version of that framework was presented on page 6 of the mid-year review for FY 2000, we were shown another in April 2004.

Higher target ‘Partners have launched national sector reform programs and establishedappropriate frameworks for investments in WSS.’

Subordinate ‘At least 50 percent of focus countries have:targets ❚ adopted DRA-based investment strategies for rural and small town WSS;

❚ introduced improved institutional arrangements for management of WSS services for poor communities in small towns, urban, and peri-urban areas;❚ adopted procurement strategies and enhanced incentives for private sector provision of goods and services; and❚ integrated effective hygiene and sanitation promotion and gender-sensitive approaches into large programs for community-based water and sanitation.’

Table 3.2: Three-year Targets in WSP’s Logframe of 2001

promotion, as well as piloted models for extendingsustainable WSS services to the poor. Thus inFY 2001 the main features of the 1999-2003strategy document were still discernible, butdifferently expressed.

Since 1998 the ‘mission statement’ quoted inTable 3.1 has been replaced for some purposes by anew form of words, usually accompanied by WSP’sname so that it does not need to mention water andsanitation. This reads: ‘An international partnershipfor improving sector policies, practices, andcapacities to serve poor people.’ Statements aboutstrategy are issued from time to time, most recentlyin the Annual Review for FY 2003.

Although the 1999-2003 Strategy Document was notformally replaced, the strategy for the five years wasthus occasionally redefined in an ad hoc way, whilekeeping to a relatively constant and consistent core.The study of the detailed task sheets and tableswithin the successive Business Plans reveals anunderlying pattern which can be seen to representa constant implied policy or strategy. By their nature,nearly all the Program’s activities and projects servethe Millennium Development Goals (MDGs) in oneway or another.

In the spirit of helping direct clients to help theirclients (the poor), the Program’s main strands ofwork included helping other sector actors to improvepolicies, institutions, organizational arrangements,and projects. These applied to rural, small town,and peri-urban fields (the attempt in BP-FY 2001 to


add ‘sanitation and hygiene’ to those three agendasdoes not seem to have become a lasting feature,although there are some activities that attend tohygiene promotion, often and rightly as a necessarycomplement to sanitation projects). The emphasesshifted among these ‘agendas’ from time to time andfrom place, generally in response to needs andopportunities. The context to which WSP wasresponding, in most of the countries concerned,included government policies of shifting towardsdecentralization of responsibilities for serviceprovision to local bodies such as municipalities.This made capacity building and communicationparticularly important.

The help to other sector actors often involved thesetting-up of country or regional sector networks,promoting communication between actors wholacked it. The means by which support was given tovarious bodies and processes also included many

co-ordinating activities involving national andinternational bodies, as well as diagnostic studies,the transmission of ideas and methods from oneplace and time to another, and capacity buildingfor many sorts of sector actors. Details of theProgram’s aims and ways of working are in theregional annexes of this report (available on request).

WSP’s objectives are entirely consistent with theMillennium Development Goals, and in particulartheir Targets 10 and 11 which aim for a halving of theproportion of people without sustainable access tosafe water and basic sanitation. These MDGs are notmuch used by WSP as explicit planning objectives;planning is rightly done in terms of more subtle andplace-specific formulations of objectives.

3.2 WSP’s Focus CountriesTable 3.3 lists the focus countries in the first year ofthe evaluation period and the current year, showing

Fiscal Year

1998-99(FY 1999)

2003-04(FY 2004)

Table 3.3: WSP’s Focus Countries in FY 1999 and in FY 2004

Africa

BeninBurkina FasoCôte D’IvoireEritreaEthiopiaGhanaKenyaMalawiMaliMozambiqueNigerRwanda

BeninBurkina FasoEthiopiaKenyaMauritaniaMozambiqueSenegalUgandaYemen

East Asia-Pacific

CambodiaChinaIndonesiaLaosMongoliaThe PhilippinesVietnam

CambodiaIndonesiaLao PDRThe PhilippinesVietnam

Latin America-Caribbean

BoliviaEcuadorPeru

BoliviaEcuadorHondurasNicaraguaPeru

South Asia

BangladeshIndiaNepalPakistanSri Lanka

BangladeshIndiaPakistan

Source: Information from WSP, May 2004


that the geographical focus sharpened during thefive years from 27 countries to 22. WSP has a writtenset of criteria for guidance on the selection ofcountries in which it should work, which wasdiscussed with the Council in 2002.

3.3 Main Activities in the Evaluation PeriodTable 3.4 summarizes the main fields of work andthe partners involved.

In addition to these country-specific arrangements,central core funding is provided by DFID (UK),Danida, Norway, Netherlands, SDC (Switzerland),and DGDC (Belgium). Some other funds areprovided to WSP regions with only partialrestrictions on their use, notably from SDC,Netherlands, AusAID, Luxembourg, Norway,SIDA (Sweden).

3.4 Institutional Arrangements and GovernanceThe Program is an international partnershipreporting to a Council and administered by theWorld Bank. It works through four regional officesand a small headquarters, mostly located in or nearWorld Bank offices.

Having started out in 1978 as a program financedmainly by UNDP and the World Bank, WSP is nowfunded almost entirely by individual governments,referred to in this report as donors. Until 2000 WSPwas guided by a Program Advisory Committee(PAC), which did not have formal terms of referenceuntil 1996. It included a considerable number ofrepresentatives from governments and otherinstitutions, not all of them significant donors. In2000, a review of governance (described briefly inAppendix 3) was commissioned, and a PAC workinggroup developed a new arrangement which wasenshrined in a Program Charter and adopted inMarch 2001. It provides for a WSP Councilcomprising contributing donors, one member froman affected country, one member from a strategicpartner such as a leading NGO or global sectororganization, one sector expert, and representativesof UNDP and the World Bank. The Council is

required to meet formally at least once a year, andinformal consultations between some of itsmembers take place between formal meetings.It can and does appoint sub-committees, includingone for financial matters.

The Charter specifies that the Program ‘is directedby its participating partners and administered byand within the World Bank on behalf of its donors.The WSP manager and staff report through theDirector, Energy and Water, to the Vice President,Private Sector Development and Infrastructure VicePresidency’.6 The Chair of the Council is defined as aVice-President of the World Bank,7 although inpractice this role has always been delegated to arelevant Director in the Bank, who often asks anotherCouncil member to take the Chair for particularCouncil sessions. The Chair is required to ‘provideleadership and resolve conflicts, represent the WSPCin interactions with other WSP stakeholders, andgenerally seek to further the sense of partnership thathas characterized the operations of the WSP’.8

The institutional relationship with the World Bankis a special case, and important both for functionand for perceptions. The Program’s website states,‘Its special relationship with the World Bank – oneof the largest investors in water and sanitationservices for developing countries – increases theProgram’s access to national policymakers and itsability to influence the sector. WSP staff contributeto the World Bank by participating in World Bankprojects, working closely with Bank staff, andleading thematic discussions and thinking withinthe sector.’ This establishes the WSP’s identity as aninstitution distinct from the Bank, and thus havinga relationship with the Bank to the advantage ofboth, but not part of the Bank’s structure and notunder the Bank’s control. The relationship iscomplex and easily misunderstood because, as thewebsite also and correctly says, ‘the Program isadministered by the World Bank and follows theBank’s management and administrative processes.’That WSP website also goes on to say ‘[WSP]functions as an independent unit within theDepartment of Energy and Water in theInfrastructure Vice Presidency’, which, subject tothe interpretation of the vague term ‘independent’,

6 Program Charter, clause 7.




Country Main Fields of Work Main Partners

AFRICA

Benin Policy support in rural areas WB, DANIDA, GTZ, UNICEF, DGDC, and NGOs

Burkina Faso Urban – large-scale UNDP, AFD, and DANIDAinvestments

Ethiopia Rural water supply – recently WB, DGDC, BNWP, DFID, EU, NGOs, PPIAF,changed to sanitation UNDP, and UNICEF

Kenya Urban/Reform GTZ, KfW, AFD, UNICEF, WHO, SIDA, and JICA

Mauritania Urban/Small towns GTZ, KfW, AFD, UNICEF, WHO, SIDA, and JICA

Mozambique Rural water SDC, UNICEF, and WaterAid,

Senegal Large-scale investment projects AFD, WB, NGOs, and UNDP

Uganda Small towns water – recently DFID, DANIDA, SIDA, KFW, AUSTRIA,changed to sanitation NWSC, KCC, and UWASNET

Yemen Rural water The Netherlands

Note: In addition to the country-specific partnerships listed here, WSP-AF works with WUP, UNCHS, PPIAF,Cities Alliance, Urban Management Program (UMP), Municipal Development Program, Africities, HTN,UNICEF, WaterAid, WSSCC, PSO, Ecosan Network, SANDRES, SANDEC, ODI, PSI, PSEau, ITNs

EAST ASIA -PACIFIC

Cambodia Rural/Policy reform Local partners, World Bank, ADB, UNDP, SIDA,WHO, UNICEF, and selected NGOs

Indonesia Urban/Rural/Policy reform AusAID, World Bank, and UNICEF

Lao PDR Urban/Rural/Policy reform Sida, ADB, UNDP, JICA, WHO, UNICEF,World Bank, and selected NGOs

The Philippines Rural/Knowledge sharing AusAID, World Bank, ADB, and GTZ

Vietnam Rural Water and Sanitation AusAID, DANIDA, FINNIDA, and UNICEF

LATIN AMERICA & CARIBBEAN

Bolivia (up to Peri-urban WSS, some rural, SDC, WB operations, governments, water2001) policy support (standards, etc.) utility companies, SIDA, GtZ, local and

national government and private sector bodies,and NGOs

Peru Urban, rural, small towns, CIDA, SIDA, SDC, ACDI, national governmentpolicy support and private sector bodies and NGOs, WB,

USAID, PRONASAR, and PROPILAS

Ecuador Investment support for rural SDC, local NGOs, local government, and WB(until 2002) and small-town WSS operations

Central Policy advice (supporting SDC, GtZ, national government bodies, andAmerica: legislation), support for NGOsHonduras and design of SIF projectsNicaragua

SOUTH ASIA

Bangladesh WSS sector reform ADB, AusAID, DANIDA, DFID, SDC, SIDA, UNICEF,World Bank, and selected NGOs

India Rural water and sanitation ADB, AusAID, Cities Alliance, DANIDA, DFID, SDC,SIDA, UNICEF, World Bank, and selected NGOs

Pakistan Institutional reform ADB, UNICEF, World Bank, and selected NGOs,DFID, and SDC

Sources: Regional evaluations, and information from WSP headquarters in May 2004

Table 3.4: Main Activities and Partners in Particular Countries, as on July 2003


can be seen as indicating that the Program is a partof the Bank after all. Some small presentationalaspects give signals to the same effect: the visitingcards used by WSP staff bear the words‘administered by the World Bank’, in very smallprint but accompanied by a prominent World Banklogo; some publications refer to the Program as ‘theWater and Sanitation Program of the World Bank’.Consequences are discussed in the next chapter.

WSP has a decentralized structure and style, with aconsiderable degree of autonomy for the regionaloffices and their managers. Many of the donors alsooperate in a decentralized manner, with someautonomy for their offices in various countries andregions around the world. WSP thus relates to itsdonors at two levels – globally through the Councilor by direct contact, and regionally through contactbetween leaders of the Program and the donors inthe regions. This multi-level partnership style is wellanchored in the Charter’s clause 8. One of the maininstruments in WSP’s decentralized governance isthe use of Regional Advisory Committees (RACs).Active particularly in the Africa and East Asia Pacificregions, these bring together annually a wide groupof WSP’s partners, clients, and other sector actors todiscuss the Program’s annual work plan, to hearexplanations of its priorities and approach, and toprovide a forum for stakeholders’ needs. They helpto avoid overlapping activities, and to promote wide‘ownership’ of the partnership’s work in each region.At present, there is no formal mechanism for themto report to the Council.

3.5 Funding and FinanceWSP expended some US$ 52 million in the four yearsFY 2000 to FY 2003 inclusive, an average ofUS$ 13 million per year but varying from US$ 12 to15 million; details are summarized in Appendix 4.The relative expenditure in the four regions andglobal work were:❚ Africa, 28 percent❚ South Asia, 22 percent❚ East Asia Pacific, 20 percent❚ Latin America and Caribbean, 12 percent❚ Global thematic program, about 10 percent❚ Global management, about 9 percent

After recovering from a difficult period in FY02,when funding was at a low ebb, creating problemsof continuity, particularly in WSP-LAC, funding hasbecome more stable in FY04. As of December 2003,WSP has assured funding through FY07 ofover US$ 48 million9, and additional probablecontributions of a further US$ 14 million. Thesefunds are from bilateral donors, typically inthree-year agreements. Additionally, the World Bankpurchases WSP staff time (so-called ‘cross support’),which amounts to US$ 0.4-0.5 million per year. Thistranslates to approximately US$12.5 million peryear for FY04 to FY07 from the assured contributions,or over US$ 15.5 million per year if the additionalcontributions materialize. The current expenditurerate is about US$ 14 million per year.

A distinction is made between ‘tied’ and ‘core’funding. Tied funds are allocated by their donors tospecific regional, country, or project activities. Corefunding is not tied in this way, and is used to supportglobal initiatives and for management andadministrative costs. Additionally, the core moniesprovide funds for regions during periods of fiscalshortfalls, and also comprise the ‘contingency fund’,a source of money available for emergency situations.

Over the past few years, donor support hasincreasingly come from the donors’ regional andcountry budgets, and less from their headquarters.This has resulted in proportionately morecontributions being tied. Of the US$ 48 million,about three-quarters, or US$ 34 million, is tied. Ofthe US$ 14 million core funding, about US$8 million is core contributions from donors, andabout US$ 6 million comes from the 12 percentGlobal Program Management fee that is levied onevery trust fund. This level of core funding is justsufficient to maintain the HQ operations, supportregional operations, and build up by FY03 acontingency fund of some US$ 0.5 million.

A commentary about the balance between rural,urban, and small-town themes, also coveringcross-cutting and other categories, is shown inSection 4.2.2.

In recent years, the Program’s maximumexpenditure rate has been about US$ 15 million.

9 This funding is comprised of US$7.8 million carried over from FY03, and annual total contributions of US$ 17.8 million (FY04),US$ 11.8 million (FY05), US$ 8.1 million (FY06), and US$ 3.1 million (FY07).


We consider that, if it were justified on grounds ofneed and comparative advantage, an increase upto the order of 30 to 60 percent could be managedwithout the major restructuring of the institutionand its ways of working.

3.6 Previous EvaluationsWSP has been either the sole subject or one of thesubjects of more than 10 evaluations of various sortsin the last 10 years. These are listed and discussed inAppendix 3. An overall one was conduced for WBand UNDP in 1996, well before our review period,and in 1999 a series of evaluations of particularregions were done for various donors. In 2000, a

consultant reviewed WSP’s governance, focusingparticularly on the operations of the Program andits Regional Advisory Committees. More recently,several World Bank evaluations of aspects ofits work have covered some of WSP’s activitiesand outputs.

WSP has taken some important decisions based onthe evaluations. The most important of these relateto the Program’s planning systems and thereplacement of the PAC with the Council. However,it has not accepted all criticisms and has on occasionresponded to evaluations with detailed rebuttal ofmany of their points.


4.1 Introduction to this ChapterChapter 4 uses the framework, terminology, andcriteria set out in Chapter 2 and Appendix 1 toevaluate the Program’s work described in Chapter 3.Discussion of strategic and planning issues, togetherwith monitoring, is in Section 4.2. The evaluation forthe five-year period is then presented in Sections 4.3and 4.4, using the aspect titles explained inAppendix 1, plus some additional headings andsections. Section 4.5 brings together some generalconclusions of the evaluation in a discussion ofcomparative advantage. The nomenclature and listof sub-sections fulfil the requirements of thisevaluation’s Terms of Reference, while adding somefurther aspects where we consider it useful. Thischapter is based on the corresponding chapters ofthe four regional annexes, which provide somedetails. Most sections in this evaluation chapter havebrief statements of our conclusions, in bold type andusually at the end of the section.

4.2 Strategic and Planning Issues4.2.1 Strategy and theDefinition of the Program’s RoleThe written statements about WSP’s objectives andways of working have been described in Section 3.1.In our view, they show a certain degree of confusion,though it is debatable how serious this is. This isparticularly true of the 1999-2003 strategy document,which has not been much used anyway, but has notbeen replaced by a single clear strategic statement.Confusion is also found in subsequent statements,in some of the Business Plan formats that have beenused over the last six years, and in the ToR for thisevaluation. Table 3.1, which quotes that strategydocument, illustrates this. Its three central lists,called ‘mutually supporting objectives’, ‘productlines’, and ‘learning agenda’, though there areimplicit ends-and-means chains between the firsttwo, do not make such linkages clear and seem tohave been used by WSP as unstructured lists of ideas.The otherwise valuable 2001 strategy update added‘sanitation and hygiene’ to the list of learningagenda, which had until then been a spatialdistinction.10 Later there was a short-lived move to

EVALUATION OF THE FIVE YEARS FY 1999 TO FY 20034add policy reform to that list too, although that wasalready established as one of the objectives with itsexplicit product line.

Later statements indicate some improvement inclarity. The Strategic Results Framework of about2000 was drawn up as a hierarchical ends-and-meanstree, with higher and lower levels of objectives, thelatter corresponding to outcomes in our frameworkand clearly distinguished from outputs. The conceptof indirect work (‘we help our clients to help theirclients, the poor’) has been articulated from time totime but not clearly built into strategy statements.The Program’s ‘mission statement’ (which, as a briefslogan encapsulating what the organization aimsfor and does, should not be expected to distinguishlevels) has occasionally been supplanted orreplaced by others, but in our view this has not muchclarified the Program’s role.

Until recently, most WSP documents did not makea clear distinction between outputs, outcomes, andimpacts. We note that since about 2003, the Programhas, for planning purposes, been using anoutcome-oriented approach called Results BasedManagement (see Appendix 3 and Section 4.2.3below). The older strategic statements from 1998 and2001, described in Section 3.1, are not consistentwith this, and we feel there is a need for an updatedand clearly articulated statement of objectives andstrategy, to underpin the improved planningmethods and to communicate what WSP does, andwhy, to new staff and to outsiders. It can be arguedthat the diversity of the situations WSP faces in theregions and countries would make too rigid astandard strategy statement counter-productive.However, we feel that a well-structured statementcould be formulated to avoid this problem, forinstance by incorporating the fact that there maybe several alternative means of achieving a singleend, each appropriate in a different situation. Newslogans or ‘mission statements’ should grow out ofsuch a strategy statement, not substitute for it.

Any updated strategy formulation would need toreflect the emerging emphasis on what is now a very

10 WSP Renewal: Strategy Update and Progress Report; prepared for the October 2001 Council meeting at The Hague: Section 2.2.


pressing need in the sector, namely addressingmassive and widespread institutional failure andineffectiveness in the WSS sector. While retaining aparticular pro-poor character in its objectives,WSP should strengthen this sort of work, which willimprove WSS provision generally, not only for poorpeople. Although this has been a major part ofWSP’s aim since at least 2001, a clear strategystatement with this emphasis might help tostimulate more funding.

We have discussed our perception of confusion instrategic statements with WSP people. Theheadquarters team does not agree with ourperception; it considers that an adequately clearvision of what the Program aims for and does hasbeen discussed and internalized by all its staff, andthat this has guided them adequately through thelast five or six years. At a pragmatic level, this islargely borne out by our evaluation of the work, butwe also note that many of the Program’s regionalstaff have welcomed our evaluation framework in away that seems to us to indicate a certain unfilledneed on their part for structure and clarity. SomeWSP people at the center and the regions have toldus that they have difficulty articulating their strategy.There is also the question of the perceptions andunderstanding of outside parties and ofstakeholders, which cannot be helped by vague orconfused statements even if the insiders arethemselves clear. Strategic thinking in this sector isnot a simple matter, but its inherent complexity canbe exaggerated by opaque or vague terminology.

We conclude that there was at the start of thereview period an underlying lack of clarity aboutends and means, especially in WSP’s writtenstatements, and that subsequent formulationshave not effectively replaced those statements. Ata practical level, the consequences were not verysevere because the Program’s implied objectiveswere usually understood by its staff.

4.2.2 Focus and BalanceFocus and balance are very difficult aspects for anysmall organization working mainly by partnershipsand dependent on many funding sources with theirown varying priorities. WSP management takesgreat care with these matters. Focusing decisionsare made at several levels, from the choice of whichcountries to work in, to the balance between rural,small town, and urban work, the relative importance

of different ‘product lines’, the design and targetingof publications, and to micro-decisions within thedesign of individual interventions.

An important aspect of focus is the choice ofcountries in which to work, and how many theyshould be in a world situation where lack of adequateWSS for poor people is widespread. In this matterof geographical focus the Program is especiallydependent on the policies and priorities of itsdonors. There have been suggestions that WSPshould become active in other countries, notablelarge and important ones such as Nigeria and China.However, the Program cannot do that without secureand medium-term donor support and funding onan adequate scale; to go into such a country withjust one or two people and a small budget wouldnot serve a useful purpose. WSP has clear criteria forchoosing countries, emphasizing the long processof building up trust, which were discussed by theCouncil in 2002. The current way of doing things,especially through dialogue between WSP and donorofficers in the countries and regions themselves,seems to us the right way to proceed.

By its definition, WSP has focused on WSS for poorpeople. The effectiveness of this focus is debatable,since the poor are often served by the sameinstitutions and infrastructure as richer people, andgeneral improvements may serve the poor as wellor better than narrow ‘pro-poor’ interventions. Onearea where the Program’s focus is sometimesquestioned by stakeholders is the balance betweendifferent sorts of urban WSS. In some countries,most of its urban interventions concentrate onplaces where people live on officially recognizedplots of land, so that municipal authorities have astatutory duty towards them. In a situation wherepeople are spontaneously migrating from rural tourban areas in search of work, however, there willalways be some people, such as newly arrivedmigrants, who live in informal or illegal ‘shantytowns’. Serving these people is often hampered bypolitical or legal constraints. It is questionablewhether, in the longer run, the urban poor are betterserved by focused attention to the immediateproblems of illegal settlements, or of poor peoplemore generally, or perhaps by improving the overalleffectiveness of urban service providers even thoughthis will benefit rich people too. The answer is by nomeans obvious, and needs well-informed discussionin each country, based on a clear understanding of


interrelated objectives and cause-and-effect chains.Our view is that the Program is right to emphasizethe significance of improvements to theeffectiveness of urban service providers, intheir entirety rather than just the bits that deal withthe poor. Increasing urgency of urban work is, ofcourse, no reason to reduce efforts on rural and smalltown work.

At the region and country levels, the need to useresources efficiently means that Program activitiesoften tend to focus on one particular sector. Theregion annexes of this report give some place-specific comments about focus. WSP is veryconscious of the need to focus its limited resourceswhere they will best serve its objectives, and triesto do so partly through its internal Business Planprocess and partly through discussions withdonors, since in the end it is mainly the donorswho determine what will and will not beimplemented. Of course donors review theirpolicies from time to time, and may decide to shiftthe focus of their support. They need, however, forthe sake of their own objectives, to discuss suchshifts in good time with WSP, and generally to planfor four or five years in advance because that is thelifetime of the more useful WSP interventions. Thewithdrawal of support for continuing urban workin Bolivia in 2001, for example, appears to havetaken WSP by surprise. Many donors, in theinterests of their own focus, have restricted lists ofcountries where their funds are to be used, andwithin the framework of its own criteria theProgram has done well in accommodating this andmaintaining its own balance by working withseveral donors in each region.

Yet another aspect of focus and balance is thatbetween rural, urban, and small town work. Despiteurban migration, the vast majority of the world’spoor population continues to reside in rural areas.Serving the rural poor has traditionally been thefocus of WSP’s work, and remained the dominantportion of the workplan in the evaluation period.

At the beginning of that period, WSP’s urban workfocused on urban environmental sanitation (UES),which primarily concerned municipal solid waste.However, as it became clear that the urban WSSproblems were wider, the Program expanded itsworkplan to cover WSS for the urban poor moregenerally. This led to important work on financing

services for the urban poor, enabling consumervoice, and promoting condominial solutions.

The rural and urban work were augmented withthe hiring of a Global Rural Thematic Coordinatorand a Global Urban Thematic Coordinator (laterreplaced by a Sanitation Coordinator). The urbanprogram has generally been slightly smaller thanthe rural program.

As rural populations continue to leave thecountryside, small towns have grown rapidly. Thishas put a considerable strain on the water andsanitation sector in such towns, where thepopulation density and size of conurbation may betoo high for rural approaches but too low for astandard urban utility approach. Towards the endof the evaluation period WSP began dedicatingspecific resources to looking for solutions to theunique problems of small towns, including hostinga small towns conference in Addis Ababa in 2003.

Reported distribution of WSP’s expenditure betweenthe three spatial themes rural, urban, and smalltowns, plus other themes, is given in Appendix 4.Year-to-year comparisons must be made withcaution, since the figures depend on precisely howexpenditures were classified each year. In our viewthis classification of expenditure is not useful, sincemost of the significant aspects of the Program’s workcut across the spatial distinctions. (The MDGs donot differentiate between rural, urban or smalltowns, and some countries, such as Pakistan, havestopped using the spatial distinction.) In the pastfive years, the Program has rightly been allocatingmore resources to the non-spatial or ‘cross-cutting’themes such as institutional reform, finance,poverty reduction strategy papers, sanitation, andgender. These are discussed in Section 4.4.5.

In our view, the Program has succeeded inmaintaining a good focus on its declared coreobjective, namely helping its clients and partnersto achieve sustainable access to WSS for poorpeople. The focus has been sharpened over theevaluation period, both geographically andthematically. The continuity and henceeffectiveness of WSP’s focus would be helped if itsdonors could find ways to plan any change ofpriorities several years ahead.

4.2.3 Planning, Monitoring, Internal EvaluationAs mentioned in Section 3.1 and described in


Appendix 2, WSP’s work was defined over thefive-year evaluation period in a series of BusinessPlans, whose nature and format changed every year.Projects or activities seem often to have beenplanned for several successive years, but sincesubsequent Business Plans did not follow on fromtheir predecessors at the project level, monitoringprojects against those plans over the years wasseldom possible. This appears to have hamperedWSP’s internal monitoring, which was reported inmid-year and end-of-year reviews.11 We have notseen much evidence of useful monitoring againstplans in the regions or at project level, though theglobal reports summarize regional results, forinstance in terms of numbers of projects or productsdelivered on time (in the earlier years the work wasdefined mainly in terms of products).

Consistent multi-year work programming isneeded because most useful WSP interventionstake at least five years to bear fruit and reach a stagewhere they can be considered finished and can beclosed down, as WSP projects, without loss ofvaluable momentum. That stage can be consideredreached when other actors have taken over theprocess that WSP initiated, so that it is sustainablewithout WSP.

WSP management has been aware of theshortcomings of the Business Plan process andformats, and over the evaluation period took stepsto improve it (details are in Appendix 2; theimprovements resulted partly from the piecemealevaluations since 1999, as described in Appendix 3).Some pre-packaged and formalized procedures,such as logframes or the ‘results-based managementframework’, were brought in by outside specialistsand proved inappropriate when used in theirentirety, but over a few years WSP staff have distilledout their useful features and rejected the rest in asensible manner. The latest version of the process,embodied in BP-FY 2004 which was prepared duringour evaluation period but applies to work outsideit, is a considerable improvement on the previousfew years, and is generally welcomed by WSP staff.It would, however, benefit from some tightening ofdefinitions and objectives. We understand that thecurrent preparation of BP-FY 2005 builds on thatprevious plan, not changing its format significantly

but trying to ensure more consistent application ofoutcome-oriented planning and project design.We welcome both these developments.

The present planning procedure is somewhatinflexible, especially regarding project budgets.Development is an inherently uncertain business,and new priorities and concerns sometimes emergeunexpectedly. There has been a tendency in recentyears, not just in WSP, to attempt to specify the waymoney should be spent in increasing detail, evendown to the number of days to be spent by eachperson on each activity. This may be useful as amanagement tool, but not as a criterion formonitoring, and not when it inhibits responsible re-allocation of resources in response to changingconditions. To some extent the budget rigidity isimposed on WSP by donors, who in turn may besubject to their own government rules, so it is difficultto avoid. But one of the comparative advantages ofthe Program should be its ability to respond toemerging situations more flexibly than larger andmore bureaucratic organizations. A good example ofwhat can be done is the innovative agreementbetween AusAid and WSP-SA, which provides aflexible fund that can be called upon to support anyactivities which comply with certain agreed criteria.

Flexibility in planning and budgeting is not alwaysbeneficial, of course, since it can provideopportunities for vagueness, undisciplined use ofresources, and loss of accountability. All these can,however, be countered by simple and pragmaticprocedures for authorizing any reallocationof resources.

The mid-year review for FY 2004 (outside our reviewperiod) indicates that flexibility may be improving,and that advantage is now being taken of the muchimproved definition of projects in the BusinessPlans. That review’s overall report includes detailsof projects that have been dropped or modified forvarious specific reasons, which is a welcomeindication of some flexibility.

It is evident from our discussions, and borne out bydocuments, that the willingness of different WSPstaff to undertake coherent planning andmonitoring is very variable. At least in recent years,headquarters staff tend to try hard to achieve

11 There were also Annual Reports, but these were for general information and not intended to evaluate performance againstBusiness Plans.


systematic and coherent planning, while someregional staff, under pressure to keep up momentumon their projects, see this as an unnecessary andtime-consuming task that distracts them from moreuseful things and also inhibits their freedom to workcreatively. We consider that better application ofclear but flexible multi-year planning, with asensible degree of recording (in Business Plans) andsubsequent monitoring, would enhance efficiencyand effectiveness enough to repay handsomely thetime it takes to do. We also consider that the presentBusiness Plan format (as in BP-FY 2004) is a goodone, and when consistently used at project levelshould serve the purpose well and not waste time.

We conclude that planning and monitoring wereweak in the earlier years of our evaluation period,but that WSP has taken effective steps to improvethem and they are now much better. There remainsa need to consolidate cost-effective ways ofcombining coherent, outcome-focused, multi-year planning with creativity and adequateflexibility, and to achieve consistent applicationof such planning and monitoring procedures overall regions and all projects.

4.3 Evaluation against Our Framework4.3.1 RelevanceThe successive subsections of this Section 4.3evaluate the Program’s work against the five-levelhierarchy of Table 2.1, using the terminologyexplained in Appendix 1. This sequence ofsubsections begins with relevance and ends withthe related topic of impact.

Relevance is a measure of whether or not a set ofoutputs and outcomes (if achieved) are likely tocontribute to an appropriate higher-level goal. Theextent to which WSP’s work matches its comparativeadvantage is also important. We evaluate theProgram’s relevance in the five years against ourunderstanding of its objectives as written in itsvarious statements discussed in Chapter 3, and asimplied by its Business Plans and chosen priorities.

Many examples in the annexes illustrate a gooddegree of relevance in this sense. The rural watersupply programs in India and Bangladesh, andformerly in Pakistan, Nepal, and Sri Lanka, areaddressing the issues of low WSS coverage. Further,the focus on rural sanitation addresses the problemof low coverage and the implications for health,

hygiene, and development. The sector advisorysupport work is equally relevant, especially in Africa.Knowledge and capacity building is well targeted atsector politicians and practitioners. Work in LatinAmerica has been focused on countries withsignificant numbers of poor people, and withinthose countries has usually been focused on servicesfor those people, with imaginative and useful linesof work.

The Program has, on occasion, taken the lead inpromoting and developing new ideas, for examplewith the work on strengthening the WSS orientationof Poverty Reduction Strategy Papers in Africa. It isdifficult to guarantee at the outset that suchinitiatives will be effective in achieving usefuloutcomes and impacts, but the potential for thefuture is valuable.

WSP’s work on disseminating new knowledge andchanging attitudes is also often relevant to its higherobjectives. There have been clear attitude changesin some areas. For instance, Kenyan officials are nowprepared to consider the extension of water andsanitation services into informal areas whereas theprevailing government view, even as late as 1997,was that such areas were completely illegal andshould not be tolerated.

WSP clearly has a comparative advantage for thework it does, because of its global extent despite thelimited number of focus countries. Another reasonis the experience of many of its staff and theirknowledge of where to find the expertise of others,as well as its skill in drawing out and communicatinglessons from one place and time to another. Also,because of its practical knowledge of participatoryand collaborative methods and their integrationwith appropriate and sustainable technology.In general, the Program has positioned itself to useits comparative advantages in its regionsand countries.

The Program’s main lines of work have thusstrengthened each other and have been relevant tothe stated objectives. WSP’s work is directly relatedto Targets 10 and 11 of Goal 7 of the MDGs, and alsoserves several other of those goals more orless indirectly.

There are a few activities of doubtful relevance,for instance the work on solid waste managementin Bangladesh, but the positive examples


predominate. We conclude that the large majorityof the Program’s work is highly relevant to its statedand implied objectives.

4.3.2 EffectivenessEffectiveness is success in crossing the importantinterface between outputs (under the control ofWSP) and outcomes (outside that control). The testof effectiveness is whether a WSP project or activity,having achieved its outputs, has a lasting outcometending towards impact. It is an important measureof how well an intervention is designed and adaptedto its context, and of how well WSP relates to theother actors whose changed attitudes or behaviordetermines the outcomes.

The interaction of different lines of work (‘productlines’ in WSP terminology; see Table 3.1, even thoughthe list of lines is not ideal) means that the processesleading to outcomes are not simple and linear. Thechain from action to outcome does not lead onceand neatly up the hierarchy, but loops back andacross between the ‘product lines’ such aspolicy support, knowledge-management, andcapacity building, each enhancing the effectivenessof the others. This is an endorsement of WSP’sproduct lines, which are most effective when usedin parallel.

Among projects close to the WSS-provision end ofthe project range, the most effective ones wereprobably those relating to outcomes to whichgovernments and other local stakeholders werealready committed. This confirms the importanceof the upstream persuasion and attitude-changingwork that WSP increasingly undertakes, often underthe title of policy support. A range of WSP activitiescontribute to the development of ideas andattitudes, and the creation of ‘informed demand’for appropriate outcomes among national andregional stakeholders. These include engagementwith stakeholders through country-levelcoordination support, the dissemination of reformproposals through publications and workshops,technical support to reform processes, regionaladvocacy, financing mechanisms, and incentivestructures. WSP also rightly emphasizes theimportance for lasting outcomes of ensuring thatdifferent thematic and country productscomplement each other, for example by a countrystrategy that links regional work on WSS in PRSPswith country-specific work on the same subject.

Since effectiveness depends on the reaction of otheractors to WSP’s work, the Program’s relationship tothose actors is important here. WSP not only relatesto other actors one by one, but also influences theway they perceive and work with each other. Inmany contexts, especially in the LAC region, peopleregard WSP as an ‘honest broker’ and a valuableforger of links. In other contexts, the Program issometimes seen, justly or not, as a partisan playerin a conflict situation or a clash of attitudes todevelopment (especially where the World Bank isperceived as partisan and the Program is seen as itsclose ally). This requires sensitive handling byregional WSP staff, and it must not be assumed thatwhat works in one place will work everywhere.Perceptions and attitudes of NGOs are alsoimportant. Institutional reform and strengtheningis a politically sensitive process, and here WSP canbe most effective when it establishes a close andtrusting relationship both with donors/funders andwith government officials and other local decision-makers. Good examples are work on rural WSS inPeru, and some urban work in India. Due to thepolitical sensitivity of institutional reform, goodopportunities for WSP to achieve lasting outcomestend to be quite rare.

WSP contributions to projects and programssupported by the World Bank were often veryeffective and useful, because:❚ Links with Bank-funded projects often provideopportunities for WSP’s limited funds to be usedstrategically to leverage benefits from the muchlarger funds available through a Bank loan.❚ Due to its reputation as an ‘honest broker’ and itslinks with local stakeholders, the Program was oftenin a position to achieve results that would not havebeen possible for the Bank working alone.❚ WSP involvement in the design of World Bank-funded projects has often made them better focusedon poor people that they would otherwise have been.WSP’s work in this area is most effective when itguides or challenges, rather than follows, the usualor prior way of thinking of the other parties involvedin project design.

The Program works in an environmentcharacterized by uncertainty and resistance tochange. In such circumstances, effectiveness is likelyto be enhanced if procedures allow flexibility torevise projects in the light of the information andunderstanding that accrue as they unfold.


Effectiveness may be undermined if standardprocedures endeavor to tie down activities tootightly. WSP’s mid-year and end-of-year reviewsprovide opportunities to assess progress forindividual projects, and to consider any changes thatmight increase effectiveness. For most of the reviewperiod, however, these reviews focused, at least atproject level, on specific product deliverables ratherthan outcomes, and were not systematically usedto improve effectiveness. Recent changes to theinternal reviews have begun to improve this.

In finding that WSP’s work is generally effective inleading to outcomes, we note that this involvesmatching approaches to conditions in the verydifferent regions. In EAP, the Program works largelyby influencing projects in rural and peri-urban areasof participating countries. Here WSP advocates theuse of community-driven development and self-provisioning approaches (including a high level offacilitation and dialogue with communities in thedevelopment of, and presentation of choices topotential beneficiaries) for both water supply andsanitation. In some places, especially the LACregion which was affected by a funding crisis in 2001,effectiveness has sometimes been limited by a lackof continuity over several years, but was stillachieved in many ways. These included beneficialinfluence on the design of investment projects,WSP’s attitude-changing work, and the introductionof sustainable methods in both management andtechnical fields. Further discussion is in the‘effectiveness’ sections of this report’s four regionalannexes (available on request).

We conclude that most of the Program’s work waseffective in achieving the desired outcomes, overall product lines and regions. We do not detectsignificant time trends within the five years. Themain limits on effectiveness were inflexibility ofbudget control and lack of predictable continuity offunding, both of which are often imposed on theProgram by donors.

4.3.3 EfficiencyEfficiency, or the cost-effectiveness of convertinginputs to outputs, is difficult to assess where manyactivities have to be carried out for many years beforea desired output is achieved. In addition,partnership activities tend to use resources fromseveral sources. Even for WSP’s inputs alone,because of the lack until recently of finely

differentiated job codes on staff time sheets, it hasnot been possible to relate particular outputs to thequantitative inputs that produced them. Checkingof whole projects against their previously estimatedinputs has usually been possible for WSP, and as faras we are aware no serious problems havebeen detected.

Time and the availability of detailed records overthe five years do not permit us to check all projectsone by one for efficiency, so we base our assessmentof efficiency on looking at WSP’s work over the fiveyears and four regions, and studying a number ofprojects more closely. In the case studies that weanalyzed, we detected no significant waste of WSPresources, though inevitably some activities are moreefficient than others are. It can be argued that abilityto ‘stay the course’ over the period required toachieve an outcome is more important than anarrow view of efficiency. The sharpening of theProgram’s focus over the period of the evaluationhas helped to ensure that scarce resources are usedas cost-effectively as possible and so shouldcontribute to the efficiency of the Program.

Use of bottom-up, demand-driven approaches,instead of top-down, supply-driven approaches,may be somewhat slow and logistically challengingin scaling-up, but the former are much moresustainable and replicable when well done, whichraises their effectiveness and outweighs any loss ofcrude efficiency. In the LAC region particularly,it was noticeable that WSP’s collaborative wayof working naturally tends to produceconsiderable leverage.

In the AF and LAC regions at least, the imaginativeuse of medium-term contracts funded by individualinternational agencies, interns and short-termconsultants helps to ensure the flexibility withoutwhich efficiency would be impossible, given theuncertainties about funding. The Programendeavors to optimize the use of scarce resourcesby identifying ways in which costs can be reducedwithout compromising effectiveness. For instance,WSP staff often use cheap airfares, and emphasizethe need to ensure that procurement processesrepresent value for money. Efficiency in the use ofWSP staff time is usually good, and although thosestaff are expensive compared with some otherorganizations in the sector, their caliber and abilityto produce useful work fast compensates for this so


that the Program, in our view, provides good valuefor the donors’ money.

We conclude that WSP’s efficiency, thoughinevitably variable, is generally high. The overalleffect achieved by a small number of people isimpressive. As always, efficiency could in principlebe improved by a narrower focus, but focus is a widerissue than that. Improved multi-year planningcould also make slightly better use of resources.

4.3.4 Sustainability and ReplicabilitySustainability is the capacity of an intervention toproduce continuing impact over many years, andreplicability is its usefulness over a widergeographical area than the site of the intervention.

Questions related to sustainability include:❚ Is the change achieved through the initiativesufficiently robust to prevent it being graduallyeroded over time as those who are opposed tochange seek to return to the previous status quo?❚ Do the stakeholders who are seeking to sustainand replicate change have the tools, in terms ofattitudes, knowledge, and systems, required forthis task?

With regard to the first question, the strong WSPemphasis on supporting or ‘upstream’ processesthat lead to sector reform means that, if successful,individual initiatives are likely to be effective insupporting movement towards the higher goals.The indications are that this emphasis has becomemore pronounced over the period covered by theevaluation. The second question points tothe importance of capacity building andattitude-changing for achieving sustainabilityand replicability.

Assessment of sustainability and replicability canonly be definitively done several years after thecompletion of an intervention, and we have not seenany deliberate evaluations or lesson-learningreviews of this nature. On the basis of what we haveread, seen, and heard in the regions, our judgementis that most WSP interventions are achieving a gooddegree of sustainability and replicability in all theregions, or showing a strong likelihood of doing soin the future.

The annexes give some examples. One of the mostinteresting features of the WSP-SA program has beenthe replicability of approaches across national

boundaries; RWS development approaches havebeen evolved through an iterative learning andreplication process spanning India, Nepal, SriLanka, and Pakistan. Sustainability is more difficultto assess, but good reports are being received fromthe Swajal Project in India. In the LAC region also,deliberate replication between countries, with dueattention to varying contexts and adequate time,has shown considerable success, and some clearevidence of sustainability. That sustainability ismuch helped by WSP-LAC’s skill in promotingcompatible and mutually reinforcing packagesof technical and management-institutionalinnovations. In EAP, the bottom-up, demand-drivenapproach for community WSS service deliveryappears to be genuinely sustainable and replicablewhen well implemented. Influence on nationalpolicy, norms, and laws is also positive, for instanceinvolving PRSPs in Africa and SIFs in LAC. WSP’scollaboration with international NGOs, such as CAREor WaterAid, also helps to propagate lessons widely.

We conclude that WSP is achieving bothsustainability and replicability in nearly all of itswork; this is commendable in a sector wheresustainability is difficult to achieve.

4.3.5 Outcomes and Impact ofProgram InterventionsSince effectiveness, relevance, sustainability, andreplicability all concern outcomes and impacts,these latter topics have already been discussed to aconsiderable extent above. This section seeks tobring together a few conclusions about them. OurTerms of Reference identify three questions relatingto the impact of Program interventions:❚ What has happened as a result of an intervention?❚ What real difference has that intervention made toits intended beneficiaries?❚ How many people have been affected?

The first question is more concerned with outputsand outcomes, in our terminology, and can beanswered for most interventions. The second andthird relate more to impacts, which take time to betake effect so that, in many cases, it is still too soonto provide a definitive answer to them.

There are clear indications that Programinterventions have brought about changes,including many at the outcome level. Our annexesprovide examples. Naturally, not all Programproducts have been equally successful.


The sanitation guidelines for Uganda provide anexample of a good output that was felt, both bygovernment stakeholders and government staff, tohave had less influence on field practice than hadbeen hoped. Another example of impact failing toreach what was hoped for is the provision ofmunicipal capacity building materials for SouthAsia, though the causes were multiple. The move inthe more recent Business Plans to view groups ofproducts in terms of their contribution to overallprojects will help to ensure better impact in the future.

An example of valuable outcome, with goodprospects for impact, is the expanded rural watersupply and sanitation program in India, with itsfruitful linkage with the Rajiv Gandhi Foundation.The Government of India has embarked on anational scaling-up program. While other partieswere also involved, it was based on the policies andapproaches that WSP-SA had a major input indeveloping. WSP’s current focus in Uganda is oncapacity building for improved sanitation and thisprovides an example of an integrated process thatensures that publications are linked to other fieldactivities, thus increasing the probability that theywill have a significant impact.

In the EAP region, donors and participatinggovernments are largely adopting methodspromoted by WSP for community facilitation anduser choice. These methods appear to be workingwell in the field, in a variety of cultural andgeographic situations, to improve project outcomesand sustainability. WSP-EAP also reports positivechanges in attitudes and practices among nationalgovernments and their participating institutionsfollowing the use of MPA methods. When directlyinvolved in design of Bank WSS projects, WSP-EAPhas been largely successful in incorporating provenmethods. Extensive impact can also be observed inthe LAC region, especially in conurbations inBolivia and Peru, where ongoing full-scaleinvestments have been heavily and beneficiallyinfluenced by WSP’s earlier work on pilot projects,diagnostic studies, knowledge management, policysupport, updating of standards, and generalattitude-changing. In view of the evidentsustainability of the interventions, greater numbersof people are likely to benefit in the future.

There is evidently a need for the Program to regularlyand consciously review the relevance of its work,

ultimately meaning the capacity to produce impactswhile taking advantage of its comparative advantageand respecting that of others. As discussed in othersections of this report, relevance needs to beassessed against a coherent vision of anorganization’s objectives and how they relate to eachother and to those of other organizations, and forWSP that vision is currently unclear.

In summary, the Program’s work has evidentlyachieved relevant outcomes in diverse situations,with considerable impact already observed andmore likely to follow. Some interventions have beenless successful than others on these criteria, butthat is to be expected in a difficult field and for aninstitution that is often leading the way rather thanfollowing proven recipes.

4.4 Evaluation against Other Aspects4.4.1 Institutional and Governance ArrangementsThe Program’s formal and governance arrangementshave been described above in Section 3.4. Somestakeholders consider that it might be helpful toconsider a few fundamentally different hypotheticalmodels, each with advantages and disadvantages,to replace the current institutional set-up. Examplesmight be:❚ Independent foundation status, not linked to theWorld Bank but still controlled by a council of donorsand other stakeholders. This would require WSP tocompete on more equal terms with some othersector actors and might lead to more cooperativeperceptions of WSP by other sector actors. This maymake it easier to approach new funding sources suchas Bill Gates.❚ A private service provider to the sector, as ‘thinktank’ or otherwise, according to the comparativeadvantages.

Having considered the potential merits ofsuch arrangements, and the difficulties anddisadvantages that they would also entail, we donot consider that any fundamentally differentarrangement would be better than the current onewhereby the Program is guided by a Council whilebeing hosted and administered by the World Bank.An organization of WSP’s size needs to belogistically hosted somewhere, since it could notcost-effectively set up a completely independentsystem for fiduciary control and accountabilitythat would command the confidence of donors.The logistic, fiduciary, and administrative


arrangements of the World Bank, while somewhatrigid and bureaucratic as is inevitable in a seriousorganization of the Bank’s size and turnover, are asappropriate as any other set. Moreover, thepractical and functional advantages to WSP ofbeing located in or near World Bank offices aroundthe world are considerable. The closeness usuallyworks to the advantage of both parties. We considerthat improvements to the present arrangement,as discussed in this section, would be more usefulthan any attempt at fundamental change.

What difficulties remain, under the present set-up,concern mainly the relationships between theCouncil, the Program Director and staff, and theWorld Bank, and especially the way thoserelationships are (rightly or wrongly) perceived byother sector actors.

The 2001 Charter defines the governance with thewords ‘…directed by its participating partners andadministered by and within the World Bank onbehalf of its donors’, where the ‘participatingpartners’ are presumably intended to berepresented by the WSP Council. The meaning ofthis depends on what ‘directed’ and ‘administered’mean. This Charter sentence could be thought tomean that WSP answers to its Council and is onlyadministered by the Bank, but the clause fails tomake that clear. Moreover, the Charter goes on tosay that the WSP manager and staff ‘report to’ seniorWorld Bank officials. In many bureaucracies thewords ‘report to’ are almost synonymous with ‘arecontrolled by’. If the intention was that WSP wouldreport to a Bank official on administrative matters,while answering to the Council for other matters,that has not been stated in the Charter. Takentogether with the Charter’s stipulation that theCouncil is automatically chaired by a World Bankofficial, this can lead people to perceive WSP asunder a high degree of control by the Bank.

Thus, the Charter is not entirely clear about theWorld Bank’s role in directing the thematic andtechnical work of the Program. Many people do notrealize that WSP’s Business Plans and other strategicdocuments are approved and given authority by theCouncil. WSP sometimes says it is under the‘oversight’ of the Council, but this is not anchoredin the Charter and its meaning is not clear.

Our discussions with stakeholders have revealed thatsome people do indeed, for better or worse, see theWSP as an arm or agent of the World Bank. The Bankitself sometimes encourages the same impressionby behaving as if it were in control of the Program,rather than just administering it on behalf of theCouncil. Some stakeholders also take the view thatWSP would be more effective if it were more distinctfrom the Bank and less under the Bank’s control.Some hold that one of its most valuable functions isto challenge the Bank’s ways of thinking and itsapproach to pro-poor service provision.

In practice, the Program is indeed largely dominatedby the World Bank, but this does not necessarilyindicate an inappropriate controlling urge on theBank’s part. Indeed, the persons on each side of thedelicate divide, namely the relevant Bank Director12

(who chairs the Council) and the Program Manager(who is a former senior Bank official and is still onthe Bank’s Sector Board) take some care to separatetheir roles and distinguish between Bank andProgram interests and priorities. To the extent thatit does steer and even control and direct the Program,the Bank does so partly because, in practice,no one is willing to do so adequately. (This alsoreflects the Bank’s high opinion of the Program’susefulness and comparative advantage.) All Councilmembers have many other roles and activities, anddo not devote many days per year to WSP’s concerns.We understand that some members are alsoconstrained by the regulations and divisions ofresponsibility within their own governments orother organizations. So the Council in effect choosesto take a relatively passive role and to delegate muchof its authority and responsibility for WSP to theBank. In many instances, including significant staffappointments, WSP or World Bank managementgives Council members opportunities to influencedecisions, and they opt not to do so.

While any alternative may be difficult to achieve,we consider on balance that this degree of WorldBank domination may not be a very satisfactory stateof affairs. This is both because of its effect on thefunctioning of the Program, and because of its effecton the perceptions of other parties. The World Bankis a lending and funding organization, and isnaturally accustomed to a high degree of control overwhat happens with interventions that it finances.

12 Currently the Director, Energy and Water.


It is staffed by highly competent and motivatedpeople, and is inevitably seen in some quarters asarrogant. It is formally associated with other BrettonWoods institutions, and is perceived by some as apartisan actor in wide global debates on suchsubjects as private sector roles and globalization.The Program, on the other hand, is a partnershipand needs to have its own distinct positions on suchmatters, especially private sector roles and pro-poorpolicies where it innovates fruitfully in its veryparticular sector. We consider that the Program mightbe even more effective than it already is if its Counciltook a more active part in directing its work. Thiswould make WSP more clearly distinct from theWorld Bank, both functionally and in terms of theperceptions of others, to the advantage of both.

The perceptions of other parties in the sector, andmore widely, are significant. Negative perceptions,whether they are well founded in fact or not, canreduce people’s willingness to co-operate inpartnership ventures, or can hamper the persuasionand attitude-changing work of the Program. Weconsider that more attention should be paid to smallmatters of labelling, such as the use of the WorldBank logo on WSP products or visiting cards, or theuse of standard World Bank e-mail addresses, withthe aim of highlighting the Program’s distinctiveness.The use, even by others, of the misleading term ‘theWater and Sanitation Program of the World Bank’needs to be avoided.

A subsidiary but related issue is the fact that the WorldBank automatically chairs the Council, which weconsider fundamentally unhealthy because the Bankhas such a particular and special place among theProgram’s partners and clients. We are notconcerned here about who presides at Councilmeetings, and the fact that the Chair is given todifferent members for different sessions appearssatisfactory. Nor are we concerned with the style oractions of the current chairman. Our concern israther with the role of the Chair between the annualCouncil meetings when it is required to ‘provideleadership and resolve conflicts, represent the WSPCin interactions with other WSP stakeholders…’.We see in the present arrangement the potential fordifficulties, especially for the Chair himself or herself,when ‘interactions with other WSP stakeholders’means interactions with the Bank. Ideally we wouldprefer to see the Chair elected by the Council forsuitable periods of time. This would enable the Bank

representative to speak more freely to theCouncil, or to its Chair between meetings, as therepresentative of one of the partners, and a veryimportant one. Separating the Chair role from theBank should also reduce misunderstandings,improve the perceptions of outsiders, and enableother stakeholders, whether represented on theCouncil or not, to approach the Chair more easilyon matters concerning the World Bank. Inexpressing this opinion about the chairmanship ofthe Council, we are aware that any alternativearrangement might be difficult to achieve, given thecurrent limited engagement of most Councilmembers and the fact that all other Trust Fundorganizations hosted by the Bank have theirgoverning or steering bodies chaired by Bankofficials. But we consider that an alternativearrangement, such as an elected Chair, should atleast be seriously considered.

Within the decentralized internal governance ofWSP, the relationship between the headquarters andthe regions is both fruitful and difficult. The largelyautonomous regional managers (formally calledRegional Team Leaders or RTLs) are senior andexperienced persons with their own distinctive anddifferent styles and ways of working, and they facevery different circumstances in their respectiveregions. WSP’s headquarters staff, especially theProgram Manager, have to tread a fine line betweentrying to impose too much uniformity and centralcontrol on the one hand, or on the other handallowing healthy autonomy and diversity to mergeinto a counter-productive lack of accountability andconsistency. Planning procedures is one area whereheadquarters has struggled, over the last five or sixyears of changing methods, to maintain thenecessary minimum of control over the regions.Diversity and creativity are sometimes misused asexcuses for avoiding responsible planning anddisciplined use of resources in the service of cleargoals. We consider that the current management hasstruck a good balance in this difficult area. Ifanything, slightly more disciplined behavior by theregions would be an improvement.

The annual meetings of the Regional AdvisoryCommittees (RACs) represent a valuable elementof the Program’s decentralized governance structure,complementing the less structured contact betweenWSP regional staff and regional stakeholders, oneby one or in smaller groups, that occurs throughout


the year. Some mechanism for reporting the viewsof the RACs to the annual Council meetings mightbe an additional enhancement.

One practical consequence of the Program’s statusis that its staff are formally employed by the WorldBank, but do not have long-term contracts likenormal World Bank employees. In fact, theircontacts are typically for three years at a timebecause of the short horizon of assured funding fromdonors. This is discussed further in Section 4.4.7.

We conclude that the fundamental institutionaland governance arrangements are satisfactory,and for practical purposes the best available, butthat the details could be improved and clarified.In particular the degree to which WSP is in practicecontrolled by the World Bank should be discussedby the Council and an agreed statement shouldreplace the ambiguous provisions of the Charter.We consider that the provision for the Council to beautomatically and always chaired by a World Bankofficial could with advantage be changed.

4.4.2 Responsiveness to OpportunityThe Program’s responsiveness to opportunity hasbeen good. This is partly from necessity, whendonors offer funds for specified sorts of work, butWSP also talks to stakeholders and identifiesopportunities itself, sometimes then bringing themto the attention of donors. When a donor offers fundstied to specific work, the most appropriate WSPresponse depends on the circumstances. Is the workproposed by the donor compatible with theProgram’s other activities? Will it take place in acountry in which the Program already has apresence? The Program’s response to SIDA’sproposal for ‘ecological sanitation’ in Africa, namelynegotiation leading to agreement on a project thatincluded ecological sanitation but was not confinedto it, provides a good model for the way in whichsuch opportunities can be handled.

Examples of good responsiveness are in the regionannexes to this report. Contexts includedecentralization in Pakistan, Latin America, andelsewhere; arsenic mitigation in Bangladesh; ‘TotalSanitation’ in India; PRSPs in Africa. Sometimes,WSP identified fruitful fields of work that had notbeen identified by other organizations. WSP’sresponsiveness to opportunity is often aconsequence of its ethos, in particular itsrecognition of the need to allow space for

experimentation and innovation, not at randombut relating to key sector issues. The Program’sresponsiveness to opportunity is evidentlyappreciated by many of its sector partners,especially government people who work insomewhat inflexible institutions. They tend toturn to the Program with requests or ideas.The Program is usually able to respond, subject tothe willingness of donors which may take some timeto arrange.

The Program has, in some regions and at some timesin the past, tried formulating Memoranda ofUnderstanding (MoUs) signed with particulardonors or particular departments of the World Bank.Although these should in principle help to promoteand systematize responsiveness, they do not seemto have borne much fruit beyond a year or two. (Therecent ones with donors in South Asia and Africamay be a positive exception.)

There has sometimes been a tendency for WSP staffto follow current fashions of thinking in thedevelopment community without adequatelyexamining their relevance and practicability for theProgram’s particular work. This responsiveness tofashion rather than opportunity can divertattention, distort priorities, and the design ofprojects, and also adversely affect other people’sperception of WSP. On the other hand, WSP is ofteninnovative and sets fruitful trends which othersfollow. Examples are practical approaches to costrecovery and private sector involvement in pro-poorWSS. Our view is that the Program works best whenit retains its confidence in its own judgement andresponds to real problems and issues, only followingcurrent trends when they match that judgement.Clear strategic thinking would help WSP people todistinguish appropriate ideas and ways, for theProgram’s particular priorities and situations, frominappropriate ones.

Flexibility is needed to take advantage ofresponsiveness. Financial and budgetary rigidityis the main constraint on the Program’sresponsiveness, and some possible improvementsare discussed below.

We conclude that the Program has shown a highdegree of responsiveness to opportunity andlistens well to its partners and clients. More flexiblebudgetary arrangements might enable it to doeven better.


4.4.3 Finance and ResourcesThis evaluation does not include any financialauditing, but it appears to us that the finances aremanaged in an accountable manner. This is anadvantage of having the Program administered bythe World Bank. Transparency is not alwaysparticularly good, however. Summaries for ourevaluation period have been hard for WSP to provide,and we have heard of a case where a donor’s auditorshad problems obtaining figures for particular regionsor projects. This may be due to the rigid use of theWorld Bank’s standard accounting procedures,which are not necessarily ideal for the Program’spurposes. It should not be difficult to providenecessary information in addition to the standardreporting formats, where there is a specific need,and as far as we know any past problems havealready been solved. Efforts to control centralmanagement costs over the evaluation period havesucceeded in limiting them to about nine percentof the turnover.

The funding situation has been described inSection 3.5. Given the effectiveness of WSP and thelarge global need for sustainable and pro-poor WSS(expressed in the ambitious MDGs), there was andremains scope for WSP to expand its scale ofoperations, which was in 1999-2003 effectivelylimited by donor funding and not by the Program’sinherent capacity. We consider that the currentmanagement structure, with four regions, couldfruitfully handle an increase in turnover of up to50 percent, if the increased scale of operations wereto be justified and the strategy were clear andconvincing. This would, of course, depend ondonor funding, and on being able to attract extrastaff of the necessary caliber and experience. Anexpansion much beyond that level might requiresome restructuring.

The Program’s responsiveness to opportunity, andthe continuity of its work at some times and places,have been limited by the fact that a large proportionof its funding is tied by donors to particularcountries, types of work or projects. The trend fordonors to make allocation decisions in regionalcontexts increases this difficulty. WSP does have ameasure of untied or ‘core’ funds, some given as amatter of policy by particular donors and somederived from the percentage of all Trust Fund moneywhich is allocated to core funding. These untiedfunds, and partially-tied funds allocated to regions,

are important to the working of the Programbecause they enable it to adjust budgets to changingcircumstances, within or between years, and onoccasion to bridge unexpected funding gapswhile minimizing the loss of staff, continuity,and momentum. We consider that the Program’seffectiveness would be further enhanced if theproportion of untied or flexible funding could beraised, provided that conscious measures were inplace to avoid the increased financial flexibilitybeing misused to allow vague or ill-consideredresource use. We are aware that donors may face theirown constraints in reducing the extent of theircontrol over use of their funds.

We have noted elsewhere that most of WSP’s workrequires sustained effort over periods of the order offive years or more, which is longer than most donorsare able to commit to. The Program generally copeswith this situation by starting an undertaking with anominal three- or four-year project, then followingit with a related project or a new phase, under a newfunding commitment though in effect acontinuation of the same work. In the absence ofclear multi-year planning up to about 2003, this wasdone in a pragmatic and ad hoc manner, and withthe improved planning procedures now in place itshould become easier.

We conclude that financial arrangements aresatisfactory, that WSP could usefully increase thescale of its operation by up to about 50 percentwithout major changes, and that the proportionof untied or core funding could with advantagebe increased.

4.4.4 Relationships between ActorsWSP is a small organization working throughmultiple partnerships, so its relationships with otheractors in the sector are crucial. They are generallygood, and the Program’s role is highly appreciated bymany people in governments, the World Bankcentrally and regionally, international developmentorganizations and NGOs, national and local NGOs,and the private sector. Many sector actors takeadvantage of WSP, some even apologize for doing so,but in our view the Program is there to be takenadvantage of. If people take advantage of the Programthey are recognizing its comparative advantage.

Where there are tensions they are sometimes due toinsufficient clarity about objectives and how theyserve each other (Section 4.2.1), so that roles


become confused. Any institution working indevelopment shares its higher goals with manyothers, while lower ones are shared with fewerothers, and some goals are almost entirely its ownniche. Poverty reduction, for instance, is one of thehighest goals of WSP, sometimes articulated andsometimes not. It is shared with many institutionsin the fields of food, health, wider infrastructure,economics, and finance (such as FAO, WHO, UNDP,ILO, IMF, GWP, innumerable bilaterals, and NGOs).Improved WSS is a lower goal, just one of many waysof reducing poverty, but it is still shared with manyinstitutions, including WSSCC and some NGOs,and private sector groups, some of whom by theirnature do not put central importance on poorpeople. While looking up its objective hierarchytowards poverty reduction, WSP needs also to lookacross towards sister organizations that seek thatsame end by different means. By being clear aboutwhich particular ends and means are its specialbusiness and which ends are shared, with whom,and how, WSP could more clearly relate to andengage with the other actors, enhancing thecomplementarity and effectiveness of all of them.

Remarks about particular categories of othersector actors:❚ Governments: This is probably the most importantcategory, since the crucial step from WSP outputsto outcomes and impacts often depends mainly onthe attitudes, policies, priorities, and actions ofgovernment departments. Relationships are usuallygood, often excellent, with government peoplegoing out of their way to tell us how much they valuethe Program’s work. There appear to be few or notensions in the LAC, AF, and EAP regions. In SouthAsia also, many government officials expressedappreciation of WSP’s role and work, and evidentlyhad good working relationships with the Program;even allowing for some gap between rhetoric andreality, this is a positive sign. This applied to bothstate and federal levels in India, though there isreportedly some tension between WSP and somestate governments. This is in part an inevitablereflection of the perennial Indian federal/statetensions, which WSP cannot necessarily avoid. Butthere are anecdotal indications that WSP’s perceivedassociation with the World Bank is one of the causes.Tensions, and perhaps hostile attitudes to WSP, canarise if the Bank is seen as crudely prescriptive, forinstance on the merits of simple models of private

sector participation, and if WSP is then seen as anuncritical follower of the Bank’s line. The Programhas not always taken a considered and rational viewand has sometimes indeed been too willing to adoptwhatever attitude the Bank was displaying at anyone time. These problems do not, however, appearto be widespread in any region.

❚ NGOs: WSP works closely with many NGOs, bothinternational ones such as CARE, Catholic ReliefServices, Save the Children, and WaterAid, and alsomany local ones. We heard positive comments onWSP from many of these. They often appreciate theProgram’s work as link-maker – setting up networks,establishing communication between governmentofficials and NGOs, bringing NGOs’ concerns tobear in the design of the World Bank, and otherinternationally funded projects, initiating andguiding collaboration with the private sector. Thereremain some small NGOs, especially outside the bigcities, which are not effectively reached by WSP (thiswas emphasized to us in Kenya.)

❚ Service providers, in both public and privatesectors: WSP’s work has been highly appreciated bythe staff of some such organizations, who stillwelcome contact with WSP people and make use ofthe networks and publications even after formalWSP projects are finished. Aguas del Illimani, theurban service provider in La Paz, is an example of alarge private sector body that shows lastingappreciation of WSP’s role; it is part of theinternational group SUEZ whose WSS operationsclaim to serve nearly nine million poor people in 10Africa, Asian, and Latin American countries, onlyone of which is a WSP focus country.

❚ Other private sector bodies: WSP has goodworking relationships with some private sectororganizations outside the WSS sector for particularpurposes, for example in connection with thehandwashing initiative in Africa and Latin America.In that connection, people from a radio station anda soap manufacturer are very appreciative of WSP’ssuccess in pushing the initiative ahead.

❚ Consultants: WSP sometimes hires the services oflocal consultants, and their contributions to thesector are enhanced by the formal and informalnetworks of contacts set up by the Program.

❚ Donors: In the regions WSP generally has very goodrelationships with the local representatives of the


donors who fund most of its work. These peopleappreciate WSP’s general contribution to the sector,and also make use of the Program’s capacity infurthering their own projects and programs.In South Asia, where many donors are active, mostof them are highly appreciative of the Program; someof them told us they would like to be kept moreclosely informed and more effectively involvedin decisions.

❚ The World Bank: WSP has often helped Bankpeople in the preparation and design of loanprojects in the WSS field, and we have noted howwell this can serve the objectives of bothorganizations, and can give WSP valuable leverage.The Program’s relationship with the Bank issometimes affected by tensions surrounding itsdegree of working independence. Despite this, ourobservations during this evaluation, and what wehave heard from others, lead us to conclude that inpractice the relationship between the Bank and WSPis good, fruitful, and positive, and that suchproblems as remain are largely ones of perception.This is unfortunate, since the undoubted and widelyrecognized advantages of the collaborative but arms-length relationship between the Program and theBank are weakened when it is not correctly perceivedby relevant sector actors. Other small institutionshosted by large and well-known ones suffer similarproblems, but that fact does not diminish them.

Our region annexes contain specific commentsabout the four regions. In most regions andcountries, the WSP staff collaborates satisfactorilywith regional or country-based World Bank staff, andare appreciated. One slight indication of tensionwas reported in India, where the Bank’s CountryDirector told us he was not fully aware of the WSPwork program although he understood the natureand objectives of WSP. Other Bank staff in New Delhiexpressed a concern that the difference between theBank and WSP may not be understood by allgovernment counterparts, and that there was adanger of giving conflicting messages to thegovernment during the preparation of new projects.Our discussions, however, found that this was notthe case and that the differences were wellunderstood. This issue seems to arise from the factthat there was a discontinuity in the Bank’s contactwith some WSS stakeholders in India, while WSPhas maintained a higher degree of continuity in itsrelationship to GoI. This misunderstanding should

not be overstated and can be easily resolved throughbetter dialogue.

❚ Knowledge partners: This category overlaps withsome others but refers to organizations which arealso engaged in the generation, management, anddissemination of knowledge concerning WSS.Examples are WEDC and IRC. Occasionally WSPcompetes with these as a knowledge serviceprovider, just as they sometimes compete with eachother, but more usually they co-operate, shareknowledge, and use each other’s services. Theknowledge partners play valuable roles in the sector,complementing that of WSP, in particular inbuilding up the capacity of institutions forsustainable WSS development. Relationships withWSP are good.

❚ Other international bodies: The Program Charter’sclause 8, in describing the partnership way ofworking and naming various categories, makesspecific mention of these, naming the Water Supplyand Sanitation Collaborative Council (WSSCC) andthe Global Water Partnership (GWP) as relevantexamples. There is specific provision in clause 14for one such body to be represented on WSP’sCouncil (as one of the types of ‘strategic partnerorganization’, though none is currently on theCouncil), and WSP is formally linked to both thoseorganizations. It also has formal and working linksto others, such as UNDP and WHO. Relationshipsbetween WSP and these organizations are generallyadequately good, though there are some slighttensions and confusions over relative roles.

WSSCC has superficial similarities with WSP in itstitle and its focus on WSS, including pro-poor WSS,and it is supported by many of the same donors.But the two are functionally distinct and different.We have met with the team who is engaged on acurrent evaluation of WSSCC, to discuss thepositions of both bodies. Any overlap of rolesbetween them has reduced since about 2000 whenWSSCC moved away from knowledge developmentto concentrate more on advocacy and thepromotion of sanitation. WSSCC, though muchsmaller than WSP and thinly spread, has a wideglobal reach and a recognized advocacy activitywhich is enhanced by its complete and clearlyperceived independence of all funding bodies. It caninfluence policies from this position, while WSP caninfluence them from its quite different position as a


partnership involved not only in policy advice butalso in institutional reform and the design ofsustainable interventions. WSP cannot speak for thewider constituency that WSSCC represents,while WSSCC cannot match the technical andprofessional resources of WSP. However, sincepersuasion and attitude-changing form importantparts of what both organizations do, there is scopefor collaboration and mutual assistance withoutwasteful duplication. This has been happening,especially in Africa generally, and in connectionwith Africasan and SACoSan. We conclude that thetwo organizations do have distinct roles and arecomplementary to each other. More collaborationcould thus be fruitful in the service of the objectivesof both organizations, although it must berecognized that the staff of both have limited timeand many conflicting demands on their resources;informal case-by-case collaboration may be morecost-effective than formal ties and routine meetings.WSP could make more use of the link fordissemination and advocacy beyond its own focuscountries, and to enhance its own degree of real andperceived independence from funding sources. Weconcur with the following statement from thecurrent WSSCC evaluation: ‘There is a need for someharmonization on key policy issues whileemphasizing the different roles and interests,ensuring that there is no apparent discord, noapparent confusion of roles, and no opportunity forothers to exploit perceived differences between thetwo organizations.’

The Global Water Partnership (GWP), of which WSPhas always been an Associated Program, hasgenerally enjoyed good mutual co-operation withWSP. This was quite intensive in GWP’s early yearswhen it covered the whole water sector, includingWSS. More recently GWP has narrowed its focus,largely onto integrated water resourcemanagement, so role overlap is now less and thereis less need for detailed collaboration; therelationship remains good. WSP could perhapsengage more in the GWP debate on water resourcesmanagement, to ensure that WSS received adequateattention alongside other water uses.

With all these sector actors WSP works both in one-institution-to-another meetings and contacts, and

through national and regional networks, many ofwhich WSP has set up and supports. It oftenorganizes workshops at which sector actors from allthe above categories can learn from each other, andit was evident to us during this evaluation that thisfunction of WSP is fruitful and highly appreciated.

We conclude that WSP, as a partnershiporganization, has many important relationshipswith other sector actors, and that most of themare very satisfactory. There are some tensions, notnecessarily of WSP’s making, which could bealleviated by good communication and by clearthinking and discussion about shared andseparate objectives.

4.4.5 Cross-Cutting IssuesThere are a number of aspects of WSP’s work whichdo not fall neatly into sub-sector categories such aswater supply or sanitation, nor into spatial onessuch as rural, urban, and small towns. In fact theytend to cut across all those, being relevant forinstance in a rural water supply context or in anurban sanitation one. This section discusses a fewsuch issues and aspects, and relates them toparticular Millennium Development Goals (MDGs).

a) Gender, Poverty, and LivelihoodsWSP’s efforts in relation to gender and poverty donot usually form distinct projects alongside others,but pervade all projects and interventions, linkedto the general drive towards participatoryapproaches and community involvement. A majorfocus of WSP’s work in the review period was,however, the Participatory Learning and ActionInitiative (PLA), which ran from 1998 to 2000. Itbrought together a global core group of individualsand agencies into a partnership which worked formainstreaming gender and poverty concerns inwater supply and sanitation.13 Within its overall aimof improving the understanding of the links betweengender, participation, demand and sustainability ofcommunity-managed water supply, and sanitationservices, the initiative had two objectives:❚ to add to the knowledge that greater attention togender pays off in increased sustainability (povertywas soon added, recognizing that poverty is asimportant as gender among the elements of acommunity’s heterogeneity).

13 Linking sustainability with Demand, Gender and Poverty – A study in community-managed water supply projects in 15 countries; WSPand IRC (Gross, van Wijk and Mukherjee); January 2001.


❚ a capacity building phase, to use the knowledge tohelp change the way programs and projects aredesigned and implemented.

The initiative began with a series of rapidassessments in projects known to have someparticipatory and demand-responsive features, tosee if more gender-sensitive demand was linked tobetter outcomes. The core group reasoned thatagency officials responsible for water supply andsanitation services would pay more attention togender, if they had evidence that doing so pays off inbetter sustained services. The action research in 15countries provided empirical evidence that bettersustained and used services were indeedsignificantly and positively associated with theuse of gender- and poverty-sensitive demand-responsive approaches in project implementation,institutional practices, and policies.

This PLA initiative led to the Methodology forParticipatory Assessment (MPA), also led by WSPand IRC.14 By 2003, thanks to these WSP-IRCinterventions, MPA was a well-established and well-documented methodology that was being used inAsia, Africa, and Latin America. It has also beeninfluential in WSP’s East Asia Pacific work.

We have examined these two reports and have seensome results of the application of MPA in the regions.The documents are well-written and clear. Theypersuasively link the sustainability of WSS servicesto equity, to accountability and to the inclusionof gender and poverty aspects by involvingcommunities systematically and effectively in theservices’ design, implementation, and operation.They go on to present a well-thought-throughmethodology to help sector actors to apply theselessons all over the world. Sustainability is not justheld up as a desirable objective – practical andstep-by-step ways of achieving it are presented.

In our view, these initiatives have contributed verysignificantly to improvements in the effectivenessand sustainability of WSS interventions in manycountries, and not only the interventions in whichWSP was directly involved. In so doing, WSP hasgiven practical and useful expression to gender andpoverty concerns at the heart of its sector, thus

directly serving MDGs 1 and 3. It is notable thatmany of the individuals involved in the PLA and MPAwork between about 1998 and 2003 are still active inWSP’s regions, while others have moved on torelevant work in the World Bank. This further extendsand reinforces gender- and poverty-conscious waysof approaching WSS work.

There have also been a few specific gender-relatedactivities by WSP, notably some multi-phase projectsin the LAC region and, in 2002, a diagnostic study ofgender aspects in the WSS sector in Peru and Boliviawhich guided the design of major projects. Inearly 2004, four macro-regional gender workshopswere held in Peru, to disseminate WSP-LAC’sproposed methodology.

b) Hygiene and Public HealthThe connection between water and sanitation onthe one hand and health on the other is so obviousthat people do not often bother to emphasize it.Water supply and sanitation are major componentsof public health efforts, and there is also a directlinkage to such problems as HIV-AIDS. Caring forsomeone with AIDS-related symptoms is obviouslymore difficult in the absence of adequate watersupply and sanitation facilities. Since most of WSP’swork serves to promote sustainable WSS servicesfor poor people, it directly serves health objectives,and, in particular, MDGs 4, 5, and 6.

c) Corruption, Accountability, and GovernanceCorruption is a problem in the provision of WSSservices, and many other sorts of services andinfrastructure too, in many countries. Because of itspolitical, legal, and ethical implications it is,however, seldom mentioned, and then often usingpolite and obscure wording such as ‘goodgovernance’ or ‘rent-seeking behavior’. WSP seldommakes any claims about contributing to thereduction of corruption, but in our view it could withjustice do so. The Program often promotestransparency and community participation inprocurement processes, which must usually makecorrupt practices, for instance bribery of municipalofficials or service providers by contractors, moredifficult to initiate and to hide. During one of ourfield trips, we discussed this with local NGO peopleworking in rural water supply, and they agreed that

14 Sustainability, Planning, and Monitoring in Community Water Supply and Sanitation: WSP/IRC (eds. Mukherjee and van Wijk); 2003.(This document followed an earlier one called the Metguide [Methodology for Participatory Assessment with Communities, Institutions,and Policy Makers] which had been developed during the PLA initiative in 1998.)


this is one of the positive by-products of themethodology that they had refined and were using.

There has also been one WSP activity specificallyrelated to corruption within a broader context ofgovernance.15 The study started by noting that interestin private sector involvement was largely driven by ‘agrowing consensus on the inability of public sectoragencies to improve performance. This consensusemerges from a collective disappointment withstagnant coverage levels, as well as increasingaccounts of inefficiency, ineffectiveness, and rent-seeking in public W&S agencies.’ But it went on toobserve that, in South Asia at least, private sector actionwas not likely to take over the task of WSS provisionentirely, so the public sector needed support. WSS andits partners studied four particular South Asian publicsector institutions that had ‘exceeded the expectationsplaced on them by sector professionals’. Thesefour success stories were analyzed in terms ofresponsiveness, accountability, and service to low-income households. In one of them, the study noted adecrease in corruption (involving both meter readingsand procurement) following a wider set of governancereforms that paid special attention to accountability.Overall, the study provided valuable data and insightsinto governance in the public sector in general,including corruption as one aspect of it, andcommented on the limitations of private sectorsolutions. This is of special value in contexts whereprivate sector approaches sometimes become thefashionable thing to do, and are applied unthinkinglyor inappropriately.

Our finding is that WSP takes adequate accountof gender and poverty issues, which are inherentlysignificant in all WSS, and handles them well.Other cross-cutting aspects of its work give positiveeffects in a number of fields, reinforcing theProgram’s direct WSS objectives and also servingwider ends and helping towards several ofthe MDGs.

4.4.6 Knowledge, Communication,and PublicationsIn contrast to its early years from 1978, the Program’sknowledge generation work is now more likely toemphasize systems and processes, includingmanagerial or organizational aspects as well as

technical ones, which we consider appropriatesince the more difficult constraints on sustainableWSS provision for the poor are in those fields,especially concerning the weakness of institutions.WSP’s most useful role in the field of technologicalinnovation is now probably that of encouraging andenabling others, such as local NGOs or researchbodies, to undertake research and development.WSP has sometimes played a leadership role in thethinking of the WSS sector, for instance aboutappropriate rather than stereotyped ways ofinvolving the private sector, or ways of mediatingsubsidies through a variety of institutional models.

When it comes to knowledge management and thedistillation, filtering, and propagation of lessonslearned by or with others, WSP is very active andeffective. Our annexes contain examples, such aspractical ways of scaling-up rural water supply andsanitation in India and in Peru, arsenic mitigationin Bangladesh, condominial approaches to peri-urban WSS, and hygiene promotion as in thehandwashing initiative. Often lessons are learnedby front-line workers such as NGOs or governmentofficials who do not have the time or skills to digestand articulate the lessons and express them in formsthat can be used by others. WSP is good at thosethings, so such people are often grateful for its help.Another valuable activity is diagnostic work,especially when it contributes to the design ofprojects with significant funding. WSP is well placedfor this work, both by using its own experienced staffand by hiring and supervising consultants.

Sharing of knowledge between WSP’s regions takesplace through publications and occasional visits,through a few inter-regional joint projects, throughthe attendance of key staff at the annual Water Weekin Washington, and through the regular meetings ofregional team leaders.

WSP takes great care with its publications, which itrightly regards as an important category of output.It now has communications specialists inheadquarters and regional offices. Somepublications are generated internally, but WSP staffalso help other organizations with the preparation,editing, polishing, targeting, production, anddissemination of their written material; this isa valuable activity.

15 In Pursuit of Good Governance: Experiments from South Asia’s Water and Sanitation Sector; WSP and Massachusetts Institute ofTechnology; April 2003.


The quality of the Program’s published material thatwe have seen is usually high, with a distinctive styleand effective use of photographs. The technicalcontent of publications is generally good andappropriate, though it is, of course, not possible tosuit every possible audience. Inevitably in such a largeand diverse set of publications, there are some weakerexamples. An evaluation in 2002, described inAppendix 3, noted some shortcomings, and our reviewof some WSP publications tends to confirm this. Theannexes contain some region-specific examples anddiscussion. Closer supervision or editing ofpublications by experienced WSP staff might help, butthe benefits of this must be weighed against the highopportunity cost of their time. Some stakeholders inSouth Asia told us that fewer publications with a greaterdepth of research and analysis would be preferable,but we do not see a need for any major change onpriorities in response to that.

Our overall finding, however, based on what we haveseen in the time available and not on any systematiccollation and analysis of large numbers ofpublications, is that WSP’s efforts since 2002 toimprove the quality of publications, and make themmore consistent, have borne considerable fruit. TheProgram now systematically distinguishes differentsorts of publication, with different target audiences,and different degrees of sensitivity to presentationalstyle. It has a clear system of checklists andprocedures for the distinct sorts, and if well usedthis should steadily improve the consistency ofquality in such aspects as clear objectives, technicalcontent, readability, and visual presentation. Weconsider it right that some classes of publicationshould receive more of WSP’s scarce time andattention than others which are less sensitive or lesswidely distributed.

Continual attention to both content andpresentation remain essential. Sometimes thereduced effectiveness of a publication is not due toany failing of its own but to a lack of coherentplanning of the project or theme it is related to. Thecase of the Uganda Sanitation Guidelines illustratesa generic problem with publications, that theysometimes fail to make a significant impact, perhapsbecause a publication on its own is insufficient tobring about change in deeply entrenched attitudes

and ways of working. (In that specific case, theproblem was recognized by WSP-AF and led to a moreintegrated approach to capacity building; details arein Annex AF). More generally, WSP always needs tomake deliberate and structured choices about whatto publish, ensuring relevance to its objectives, andavoiding publishing work mainly to publicize successor promote the Program’s or an author’s reputation,which has happened in a few cases.

WSP could with advantage use a wider range ofmedia, for instance virtual field trips on video, tobring field lessons to wide audiences without theneed for extensive travel and simultaneoustranslation. The Program does sometimes distributethe publications of other organizations, but weconsider this could usefully be done more, and theycould distribute those of WSP.

We conclude that WSP’s knowledge-related workis important and fruitful as a complement to itsother activities, and is mostly well done. Qualityand appropriateness of publications is usuallygood, though inevitably variable in a difficult fieldwhere all staff are involved, including the lessexperienced ones. Standards have improved in thecourse of the evaluation period.

4.4.7 Staffing and Staff SkillsWSP has an excellent core of staff with relevant skillsand experience, but they are spread quite thinly.Staffing policy is regularly considered by WSPmanagement, and the regions have distinct policiesto match their diverse situations. Recent losses ofthematic specialists from the headquarters teamhave left a gap.16 We understand that WSP intendsto reduce these two headquarters posts to one, andthat some specialists in regional offices are to begiven a global co-ordinating role. This raises thedifficult question of how WSP should distribute itsthinking capacity between the center and theregions. Although a regional base has advantagesfor innovative thinkers, it limits their range of inputsfrom others, and makes the propagation of theirideas more dependent on publications, which maybe less useful (partly because less interactive) thanworkshops and informal contacts. If the Program isto reduce the strength or number of thematicthinkers in its headquarters it may need to counter

16 In the course of FY04 two headquarters staff, senior WSS specialists functioning as global thematic co-ordinators, left WSP to take uppermanent appointments with the World Bank. At the moment there are transitional arrangements for them to continueserving WSP part-time.


this by giving those in the regions an explicit remit,plus time, and funding, to travel to other regions orto global fora to interact with their peers. The size ofthe HQ staff has been cut to a very small core (partlyin response to calls for reduced central expenditure),and we consider it should not be cut further. Thestaff numbers in particular regions depend on thechoice of focus countries, for which there areclear criteria, and on donors’ funding decisions.Language barriers do not appear to us to be asignificant problem.

A balance is also needed between staff with well-grounded technical training and experience plus agood understanding (usually from experience, nottraining) of people, motives, governance, andinstitutions, and on the other hand staff with formaltraining in institutional aspects or social sciences.The former are not easy to find and recruit; WSP hasseveral but could use more. A third balance thatneeds to be struck in staffing policy is that betweenusing young but very talented people and usingpeople with long experience in the sector, usuallygained with other organizations in the past. Using amix of experienced and more junior staff is not abad thing, but requires that the former have enoughtime and incentive to give guidance and leadership;this has often been good, especially in Africa, butremains patchy. Often the Program’s imaginative useof consultants, and subcontracting tasks to them orNGOs, helps to alleviate problems.

Since the Program’s funding is not usually known morethan three years ahead, the staff are normally employedon relatively short contracts, leaving them withoutsecurity of employment and with an incentive toaccept more secure posts elsewhere, such as with theWorld Bank. This reduces the Program’s continuity andinstitutional memory, though it also benefits the sectormore widely in that WSP serves as a training groundfor other organizations. The rate of turnover puts extrasupervision and training burdens on seniorWSP managers.

We consider that the staff base in the evaluationperiod has been strong, and remains so today.But the Program’s work poses big challenges, andwe see the need for continuing the internal reviewof future staffing needs, grounded in a veryclear understanding of WSP’s special objectivesand its comparative advantage relative toother organizations.

4.4.8 Languages and Sub-RegionsWSP works in several languages. The LAC regionworks within its region mainly in Spanish, andoccasionally in Portuguese, but in English for mostdealings with the other regions and withheadquarters. Many of the staff are fluent in Englishas well as Spanish, while some speak French also;language does not appear to be a significant problemor constraint. Several WSP publications, originatingin LAC and in other regions, are available in bothSpanish and English versions. In South Asia, themain language is English, but some publications aretranslated into some of the region’s numerous otherworking languages; WSP-SA appears to havelanguage differences under control. The situation issimilar in East Asia Pacific.

In the Africa region, the main office in Nairobioperates mainly in English, while the Dakar officeoperates mainly in French and deals with theFrancophone countries in West Africa andMadagascar. The small office in Mozambiqueoperates in Portuguese. Communication betweenthe offices is mainly in English. The regionalmanagement has made considerable efforts in recentyears to overcome language barriers and in particularto improve the integration of the Francophone work.Meetings of the Regional Advisory Council arebilingual, in French and English, this having beenintroduced during the evaluation period, and aresometimes held in Dakar. The region has appointeda French-speaking communications specialist, basedin Dakar, in order to increase the number and qualityof French-language publications. Several French-speaking people have been appointed to the Nairobioffice, and substantial language training has beenprovided for the staff. WSP leadership remainsconscious of the need for further efforts, not only inthe Africa region, in provision for Francophonecountries and partners, and is seeking help with thehigh cost of translation and interpretation; wesupport this effort.

A related issue is that of sub-regions within some ofthe four regions. The main examples are West Africaand Central America. As described in Annex AF(especially its Appendix AF-3), since merging theformer two African regions into one the Programhas made considerable efforts to strengthen theWest Africa and Francophone elements, subject, ofcourse, to funding constraints. The use of a sub-regional office in Dakar, with four professionals,


seems to us a sensible and satisfactory way to goabout this. The relationship between the CentralAmerica sub-region and its regional office in Limais different in that there is no language barrier, butagain the arrangement seems to us to be sensibleand cost-effective (some details are in Annex LAC).

4.4.9 Response to Evaluationsand Self-EvaluationAs described in Section 3.6 above and inAppendix 3, the Program’s work was the subject ofmany piecemeal evaluations during the reviewperiod. The Program and its donors responded tothose which dealt with governance by setting upthe Council in place of the PAC, which as describedabove has been a marked improvement. WSP alsoresponded to evaluations of its own work with somebeneficial changes that have been incorporated intoits evolution. In particular, it initiated a long processof revising its planning procedures, which as notedabove is now bearing fruit.

We notice, in connection both with thoseevaluations and this one, that the Program’s stafftend to respond to any criticism, even if mixed withpraise, in a markedly sensitive and defensive way.Discussing this with them, we are told that it isnormal in World Bank circles, which in turnprobably reflects the high standards associated withthat organization. It may also reflect a desire toachieve measurable targets, and a tendency to judgepeople’s work in such terms, but we consider thattoo much emphasis on such targets tends toundermine the value of a ‘learning organization’.

In a positive sense, this sensitivity reflects the WSPstaff’s dedication, professionalism, and pride intheir work. But it does not seem to us entirelyhealthy, especially in an organization dedicated topartnership work and therefore to listening toothers. In view of their caliber and achievements,WSP people should not need to be defensive. TheProgram works at the leading edge of a field whichis professionally and intellectually difficultand challenging, and no one should expectperfection. Inevitably in development work, someinterventions fail, or succeed only partly. It appearsto us that a more self-questioning attitude on WSP’spart, together with a less defensive attitude toconstructive criticism, would strengthen ratherthan weaken its effectiveness. At the same time,WSP could take a systematically critical view of its

own work, to ensure that lessons are learned andused for the design of future projects.

4.5 WSP’s Comparative AdvantageThe above systematic evaluation of the Program’swork in the fiscal years 1999 to 2003, backed by theregional annexes to this report, lead to theconclusion that it is generally relevant and effectivein achieving outcomes and impacts in its chosenfield. It is also generally efficient in its use of limitedresources, often highly so, and this is any waysubject to the natural constraints of innovativepartnership work. The Program’s partnership styleand its influence on the design of projects fundedby others gives it great potential for leverage, whichis often used well. Its institutional arrangements aregenerally good, though require clarification andperhaps changes on some governance aspects. WSPhas improved its planning procedures considerablytowards the end of the five-year period (which is nowa year behind us), and continues to do so.

Since its early years concentrating largely onsustainable technology, WSP has rightly moved intoother fields, covering ways of doing things in a widesense. For example, the condominial approach toperi-urban WSS brings together some technologicalinnovations (such as local drains and pipes locatedand connected to main systems in a new way) withnew forms of private sector involvement and costrecovery, and also with context-specific ways ofinvolving communities and service users.

The ways in which other sector actors use WSP indicatethat they appreciate its comparative advantages,particularly in the following sorts of activities:❚ Attitude-changing advocacy, persuasion, andcapacity building activities generally, for instancethe Program’s skills in

❚ setting up and supporting networks, especiallyin areas where political, social or logistic factorsmake contact between sector actors difficult;❚ facilitating dialogue and exchangeon important issues, while not dominatingthe discussion, for instance by conductingworkshops;❚ documenting and disseminating the lessonsof experience; and❚ transferring those lessons from one countryto another.


Aspect

Attitude-changing,advocacy,persuasion,andcapacitybuilding

Advice andsupport on‘upstreamissues’

Innovativethinking

Collecting,distilling,anddisseminatinginformation

Assisting inthe design ofinvestmentprojects

Professionalandtechnicalcapacity

Continuityof contact

Table 4.1: WSP’s Comparative Advantage

WSP Strengths

Especially in recent years, WSP has beenengaged largely in persuasion and attitude-changing, with central and local governmentsand other public sector people, and also withthe private sector and communities.

WSP has developed considerable strength inthis area in recent years, and is a trusted sourceof advice in many countries.

Useful innovation often needs goodknowledge of local cultural, social, andorganizational situations. WSP is well placedfor this because of its field presence in manycountries and its continuous contact withother sector actors.

WSP has a long tradition in this field,and its improved and still improvingcommunications strategy give it furtherstrength. Its many contacts in the sector giveit a wide source base.

Largely because of its historical andcontinuing relationship with the World Bank,WSP is uniquely placed to influence projectdesign, giving it important leverage.

With about 80 staff, a high public profile in thesector, and World-Bank-level salaries, WSP canrecruit and retain a good number of highlycompetent professionals, covering a widerange of technical and organizationalknowledge and experience.

In its focus regions and countries, WSP canand does maintain continuous contact withkey actors such as senior government people.

Strengths of Other Actors,and Comments

Some other actors are also active inadvocacy, notably WSSCC. But WSP’swork in this area is much enhanced bythe combination with its othercharacteristics noted in this table.

Although some other organizationsalso do this, WSP puts particularemphasis on it and has a track record.

Other organizations, especially NGOs,may be well placed in some countries,but there is probably none with ageneral advantage over WSP.

Many others also do this, but WSP isone of the strongest and has anespecially wide range of contacts.

None other in this sector has a similarrelationship with the World Bank,which is a major funding source,though some may be strong with otherfunders or in specific regions.

Few others can match this:some NGOs have high technicalcompetence in their fields but tend tohave high staff turnover and/orlimitations in recruiting due tolower salaries. The internationalorganizations (WSSCC, GWP) do notmatch the staff numbers and technicalrange of WSP.

GWP and WSSCC are usually restrictedin their advocacy and persuasion workto occasional visits or missions to


Aspect

Globalknowledgebase

Independenceof funding

Relationshipwith fundingagencies

Track record

Ability tolearn

WSP Strengths

It can persuade and change attitudespatiently and continually, and can wait untilpeople are ready for the next message.

WSP has strong regional, sub-regional, andcountry bases on three continents, coveringa good diversity of situations. It also has astrong and improving system for extraction,storage, and dissemination of information.

WSP’s functional independence from donorsand funders can help to engender trust andconfidence in its judgement and advice(provided that WSP is perceived asindependent by the relevant actors).

The Program’s special relationship with theWorld Bank, and to a lesser extent otherfunders, gives it valuable leverage andinfluence, including indirect influence onclients who are dealing direct with the Bank.

Twenty-five years in the sector, with anevolving role.

WSP has always included learning in itsrhetoric. Most of its staff are willing andinclined to learn from experience, though theinternal monitoring of current and pastprojects is not systematic.

Strengths of Other Actors,and Comments

particular countries, which may not fitwell with the evolution of thinking intarget organizations.

Probably no other organization in thesector can match this, except perhapsbig funding agencies.

This is a key advantage in comparisonwith funders, especially the WorldBank. Many other organizations sharethis advantage, for example, WSSCCand NGOs.

No other organization can match this.But the special relationship with theWorld Bank needs to be continuallywatched to ensure that it does not giverise to perceptions of Bank control.

Most international organizations areyounger and do not have as strong atrack record (for example, WSSCC,GWP). Some NGOs do have strong trackrecords (for example, CARE, WaterAid).

While this is a strength of WSP, manyother organizations also learn, and wedo not consider this a particularlysignificant area of comparativeadvantage for WSP.

❚ Providing advice and support to clients on‘upstream issues’ such as policy and methods,especially related to institutional effectiveness.This flows from the Program’s collective experience,and more specifically the experience of its key staff.

Maintaining this comparative advantage requires thatWSP makes a conscious effort to base itsrecommendations and advice on real experience andnot on the currently fashionable prescriptions thathappen to pervade the development community atany one time.

❚ Innovative thinking, especially the Program’sability to bring together organizational and technicalaspects, and testing innovative ideas in particularcultural and geographic settings (or persuading andhelping others to test them).

❚ Collecting, distilling, and disseminatinginformation, often derived from the work of others,on technologies and on organizational methods andmodels, and more importantly on ways ofcombining those. WSP does not often have particularcomparative advantage on purely technologicalmatters, partly because they are covered by many


others, but can usefully support those others withknowledge and advice. The Program is widely trustedand respected as an information source, by both localand global sector actors.

❚ Guiding and assisting others in the design of theirinvestment projects, where WSP is especiallyeffective when it maintains its distinct pro-poorinsights, guiding rather than following the thinkingof the funders and implementers of projects.

Table 4.1 gives a brief discussion of WSP’scomparative advantages relative to other sectoractors which pursue the same or overlappingobjectives. It starts with the above five sorts ofactivities and then continues to mention a few otheraspects. This evaluation has concentrated on WSPand has not systematically examined thecomparative advantages of other organizations,though some comments about a few of them are inSection 4.4.4 above, so comments in this table aboutother organizations are tentative. A systematicreview of how WSP fits in the spectrum oforganizations would be part of the updating andclarification of its strategy mentioned inSection 4.2.1 and in our recommendations.

An important component of WSP’s comparativeadvantage is that the Program’s different types ofwork and of expertise are not separate efforts butcomplement each other very powerfully. To take thehypothetical but realistic rural water supplyexample used in Chapter 2 to illustrate ourframework, WSP’s success in persuading and

training members of the government departmentdepended on the Program’s prior knowledge of howto make rural schemes sustainable (collected andfiltered from many sources), on its understandingof the country and the department, and on the trustthat the government people already had in WSP’sunderstanding of policy issues.

We have identified in this chapter some issues wherewe consider there is room for improvement. Thesejudgements lead to our recommendations in Chapter 6,but it should be said here that the situation is not static.In many aspects we have noted improvements duringthe evaluation period and since.

There is some danger that the Program as currentlystaffed may be overstretched, especially in Africawhere so many countries have urgent WSS needs.South Asia may be in a similar position – althoughIndia is nominally one country, its vast size, itsdiversity, and its federal structure make it more likea continent for many practical purposes.

In view of our positive conclusion aboutcomparative advantage, the question arises as tohow far WSP could expand the scale of its operationswithout losing efficiency or effectiveness. Oursubjective judgement is that, given adequatefunding and the corresponding staffing, theProgram could handle 30 to 60 percent more workwith its present institutional set-up of four regions,some sub-regional hubs such as Central America andWest Africa, and a headquarters staff comprising upto about 10 percent of the total.


FUTURE CONTEXT AND WORKING ENVIRONMENT55.1 Trends in the SectorWe consider that the most relevant trends in theWSS sector in the next few years, continuing presenttrends, are likely to be increasing urbanization, andcontinuing decentralization by governments(though perhaps more subtle and refined forms ofdecentralization than in recent years). Generaleconomic growth will probably continue in mostrelevant countries. We see the main implicationsfor WSP to be:

a) Increasing pressure due to urbanization:While the importance of rural and small-townWSS will not diminish, the increase in urbanpopulations will make urban work even moreimportant than it is already. One key feature ofurban work is the fact that a wide range of urbanservices are mutually interdependent, so thatconsidering WSS in isolation from other urbanservices is not fruitful. Another feature is that anurban WSS service provider typically serves poorcommunities through the same physical andorganizational systems as richer people.There is a need to recognize the ‘horizontal’dependency between different services in thesame area, and ‘vertical’ dependency of localservices on higher order facilities and services,especially for networked systems such assewerage and piped water supply. Serviceprovision has to be organized into institutionaldomains one way or another, and any way maybring difficulties at the boundaries. Horizontallinkages mean that an integrated or coordinatedapproach to service provision will usually bebetter than a narrow sectoral one. Verticalintegration means that efforts to improve servicesfor the urban poor cannot focus only on localservices to low-income communities but ratheron those services within their overall context.This may mean that the most effective andefficient service improvements will benefit bothrich and poor. Ways may need to be found tosupport or subsidize service provision toparticular groups.

b) Institutional strengthening underdecentralization:The last few years have been marked by policiesof decentralization by many governments, and

we would expect this trend to be maintained inthe next few years, probably with better and moresuccessful decentralization models as countriesgain experience and improve on simplisticmodels. This will place heavy demands on localgovernment bodies such as municipalities,regional or provincial governments (some ofwhich are newly constituted), who often havelittle experience to help them meet thechallenges. Traditional ways of providing WSSservices, through state organizations and withlittle or no community participation or choice,have been widely found to be ineffective and/orunsustainable. WSP has already played a leadingrole in documenting and understanding theseproblems and in finding or developing solutionswhich are compatible with decentralization, andwhich make appropriate and cost-effective useof both public and private sector capabilities. Theneed for this sort of work is expected to increase.There is an equally pressing need to examinethe advantages and disadvantages of newapproaches and paradigms. Questions to beasked include what forms of communityparticipation work in what circumstances andwhich aspects of service provision should bedecentralized to what level. WSP will oftenneed to remain critical of the latest ‘conventional wisdom’ on issues of governanceand participation.

c) Economic growth and the need forbetter systems:Whether generally or in urban areas specifically,economic growth cannot be expected to outstrippopulation growth by any significant margin, sothere will continue over the next few decades tobe a need for innovative and cost-effectiveapproaches to the provision of WSS services forpoor people. But economic growth will fueltechnological development that can be used forpro-poor WSS. The main need will be not for‘appropriate technologies’, but rather forappropriate systems, encompassingtechnologies, financial, and managementsystems within an integrated whole that takesaccount of social attitudes and structures. WSPis well placed to work on this.


5.2 Needs and Demands of WSP’s Partners and ClientsWSP’s strategic and direct partners (agencies anddonors), as well as its direct clients (governmentsand service providers) will face continuing needs inthe sector, and the above trends. In that context theywill continue to place demands on the Program.The donors rightly use WSP to further their evolvingdevelopment objectives, and those in turnare influenced by the donors’ dialogues withgovernments. With both donors and WSPincreasingly using decentralized ways of working,the relevant discussion of demands often goes onin the focus countries, which is healthy in that theparticipants are exposed to local conditions and areusually in close touch with governments and othercountry-level actors such as local NGOs. The WorldBank and other funding agencies will also probablycontinue to make demands on the Program,especially for help with project design, andparticipation in the global discussion about waysand means in the sector.

After studying the Program’s work and listening to awide range of stakeholders, we consider thefollowing aspects particularly important:

a) There is evidently a widespread view in the sectorthat urban WSS, especially sanitation, is of highand increasing importance, and WSP probablyneeds to respond to this systematically ratherthan piecemeal or locally. It raises difficult issuesabout targeting efforts towards serving the poor,since (as mentioned above) the same urbanservice providers usually serve rich and poor, anda crucial determinant is often the effectivenessof those institutions and the sustainability of theirmethods and products.

b) Many people in the sector still see a needfor increased emphasis on sanitation, whilehygiene promotion and education arealso considered important as necessarycomplements to WSS provision if the desiredoutcomes and impacts are to be achieved. Workin these fields is often difficult to finance, andthe handwashing initiative, mobilizing privatesector funds, shows promise.

c) The need for WSP’s work in support ofdecentralized organizations will persist in mostregions for decades to come. The Program hasbeen of considerable assistance to central and

large provincial governments in the last fiveyears, but there is now a real and perceived needto extend support to regional and localgovernment, which is fragmented and involveshundreds or thousands of institutions instead ofa few in each country. WSP is not likely to beasked to serve these numerous institutionsdirectly, but to contribute to joint efforts in orderto build up their capacity and knowledge andgive them effective and sustainable modelsand methods.

d) Institutional weakness is still seen as a majorproblem in most parts of the sector. This is notcaused by lack of capacity alone but also byinherent weaknesses in current institutionalstructures and systems. So sector reform is likelyto remain important among the clients’ andpartners’ demands on WSP.

e) There will probably be an increasing need todemonstrate and make known the linkagesbetween WSS and other development aspectssuch as health, education, poverty. This will needclear thinking about objectives and how theyrelate to each other in effective integrateddevelopment.

f) More generally, and underlying the above, acentral need of the sector is the need to changeattitudes, particularly attitudes to communityparticipation, to collaboration between publicand private sectors, and to appropriate matchingof technical and managerial innovation.

The Millennium Development Goals are animportant expression of collective will, though theones relating to WSS (Targets 10 and 11) merely setrelative aims in terms of halving percentages ofpopulations without sustainable access to basicservices. This means that they refer to access ratherthan higher-level outcomes, and rely on changes instatistics that are of questionable meaning anyway.The necessary linkages between differentdevelopment aspects and goals are not effectivelydealt with by the MDGs themselves, so that they donot take adequate account of integrated policies andmethods. But the MDGs remain a significant way ofraising political awareness of needs. The world as awhole is evidently not going to meet the MDGseasily by 2015, if at all, so there is a large demand formore of what WSP has to offer.


6.1 Continuing with the Program’s WorkOur principal recommendation is that WSPshould continue with its valuable work, and shouldseek a significant funding increase for the next fiveto 10 years.

By ‘significant increase’, we mean something of theorder of 30 to 60 percent, rather than 10 percent or100 percent. This is not intended as a precisemeasure, nor does it rely on any quantitativeanalysis; it merely reflects our subjective views ofthe sector’s needs, as discussed in Chapter 5, of theProgram’s comparative advantages described inSection 4.5, and of the scale of work that could behandled without the major restructuring of theProgram. The next five to 10 years will be crucial forthe sector, not least in relation to the MDGs, whosetime-scale runs to 2015.

To attract a funding increase of this order, WSPwould, of course, have to persuade existing donorsto give more, and/or persuade new donors to startgiving. The remainder of this chapter presents ourmore detailed recommendations for action by WSPand its Council which we consider would help to dothis, as well as improving the Program’s effectivenessand efficiency.

The recommendations are arranged in six groups,generally following the same order as the discussionand evaluation in Chapter 4, and are not in order ofimportance or urgency. Relevant section numbersof this report, where discussion of our reasoning canbe found, are given where possible in square brackets[…]. Most of the specific recommendations areaddressed to a named party such as the WSPmanagement or the Council, identified in thischapter by bold italic type, while for convenientreference a few key words in each recommendationare highlighted by means of bold upright type.

6.2 Finance and Resources [Sections 4.4.3, 4.2.3, 4.3]a) WSP should continue to operate under World

Bank financial and accounting systems,while seeking efficient ways to make relevantinformation available and transparent toits partners.

RECOMMENDATIONS6b) The Council and WSP management should

continue to seek ways to increase the level offunding. Many of the recommendations whichfollow, and especially those concerning strategyand governance, could form the basis forimproved confidence in the Program and a freshapproach to potential new donors.

c) WSP should both plan and monitor its resourceuse with a multi-year perspective, rather thanjust within fiscal years as it has tended to do untilabout 2003. The date-specific outcome indicatorsin the Business Plan for FY 2004 provide a goodbasis for this. What is needed now is to relate eachyear’s Business Plan to previous and future ones,by keeping consistent names and referencenumbers of projects and repeating their definedobjectives and timelines in successive BusinessPlans, unless they are deliberately and explicitlychanged for stated reasons.

d) Donors should, if possible, seek ways to commitfunds several years in advance, typically fiveyears for most purposes. The necessaryconfidence should be provided by a clarifiedstrategy and enhanced multi-year planning, asrecommended below.

e) WSP should use its improved multi-yearplanning to make systematic and transparentarrangements for coping with the fact thatdonors do not usually commit funds more thanthree years in advance; this can includeplanning longer projects but breaking them upinto phases for piecemeal funding.

f) The Program and the donors together shouldseek ways of achieving a joint flexibility inresource allocation by increasing the proportionof partially tied or untied or core funding, sothat new opportunities identified by any partycan be quickly grasped. This should include theformalization of intermediate funding categoriesthat are not global core funding but are notcompletely tied either. This can counter thereduction of flexible funding that tends toaccompany the shift to regional fundingdecisions by donors. Ways of maintainingresponsible resource use and adequateaccountability should be agreed by the Council,


so as to give donors the necessary confidence inthe more flexible allocation.

g) WSP should seek ways to improve flexibilityof resource allocation in the course of eachproject’s life, since conditions change orproject design may turn out not to be ideal.The structured multi-year planning ofprojects which we advocate should includeprovision for adjustments every year. Waysshould also be sought to allow mid-yearreviews of projects to adjust work programsand budgets for the second half of eachfinancial year, with financial provision tomake this possible (such as unallocatedreserves within core budgets or projectbudgets). Where practicable, donors and localproject partners should be involved in thesereviews, to take advantage of their insights,to keep them informed, and to enhance theculture of multi-party ownership of projects(which in turn can enhance sustainability).

6.3 Institutional Arrangements and Governance [Section 4.4.1]a) The Program should remain a partnership

organization directed by its main stakeholdersrepresented on its Council, while being hostedand administered by the World Bank.

b) Within that structure, and as provided inclause 13 of the Charter, the Council shouldconsider taking on a more active role thanat present, through a delegated executivecommittee or otherwise, in the direction andcontrol of the Program. For example, it might takean active part in the choice of focus countriesand the selection of senior WSP staff such as aProgram Manager or a Regional Team Leader.To this end:

i. The Council should consider changing itsrules so that it elects its Chair from time totime instead of being chaired automaticallyby an official of the World Bank. (This refersnot to presiding at meetings but to the role ofthe Chair between the annual Councilmeetings when it is required to provideleadership and resolve conflicts, representingthe WSPC in interactions with otherstakeholders.)

ii. The Council should debate the extent towhich WSP’s thematic and technical workought to be controlled by the World Bank,and, if possible, replace the ambiguousprovisions of the Charter with a clear andagreed statement.

iii. The Council, WSP, and the World Bankshould all take conscious steps to ensure thatother parties understand the distinct natureof the Program and do not perceive it as asubservient agent or arm of the World Bank;this should include but not be limited toattention to small presentational signals suchas the use of logos.

iv. The Council should consider whetheror not any changes resulting fromthose recommendations require smallamendments to the precise terms ofthe Charter.

c) WSP central management should maintain itspresent balance between controlling the regionsand allowing them more autonomy, andthe regions should respect that balance andcomply with the necessary central disciplineregarding planning, monitoring, and resourcemanagement.

d) A way should be sought for reporting the views ofthe Regional Advisory Councils to the annualCouncil meetings.

6.5 Objectives and Strategy [Sections 4.2.1, 4.2.2]While not seeing a need for any change in theProgram’s basic identity and objectives, werecommend that WSP should seek to clarify andupdate its strategy by sharpening the definition ofits objectives, working out carefully what aims areat the ultimate level, such as poverty reduction, andwhat are intermediate levels, as means to the higherends. (The Program’s various current statements ofmission, objectives, and ways of working are notwrong, but they lack clarity and do not well reflectthe present range of activities.) The distinctionbetween outputs and outcomes could be useful forthis process, though it must always be kept clearwho is the relevant implementing agency. Thisupdated strategy formulation should reflect theemerging emphasis on addressing the massive and


widespread institutional failure and ineffectivenessin the WSS sector, and the ways in which this canbenefit poor people along with others.We recommend that WSP retains a focus on WSS forpoor people in its objectives, but takes a broad viewof the means to achieve them, not necessarilyconcentrating on specifically ‘pro-poor’interventions. One of the benefits of a clear andupdated strategy would be to help persuade donorsto increase funding levels.

Steps in this updating and clarifying process, whichshould not need to be repeated more often thanabout once in five years but do not need to refer to afixed time period, could include:1. Reformulation of WSP’s objectives in the form

of a hierarchical tree of ends and means, set incontext alongside wide development andhuman welfare concerns using povertyreduction and health; consistently using thesimplest possible terminology.

2. Identification of higher and intermediateobjectives, activities, and fields of work whichare WSP’s particular niche; noting distinctionsand overlaps relative to the niches of othersector actors.

3. Using this to define and describe WSP’scomparative advantages relative to those ofother actors such as networks and fora,advocacy specialists, research institutes, globaland local NGOs, governments, donors, and theWorld Bank.

4. Applying the results of those steps to a reviewof the Program’s thematic focus, aimedat maximizing its outcome-generatingeffectiveness in the light of WSP’s comparativeadvantage.

5. Describing the resulting objectives and fields ofwork clearly, without mixing ends and means.

6. Finally reviewing the way the clarified strategy isexpressed in brief form as a slogan or ‘missionstatement’ (which in order to describe WSP brieflydoes need to mix ends and means in onestatement).

6.5 Staffing and Staff Skills [Section 4.4.7]a) WSP should continue regularly to review the mix

of staff skills, based on the recommendedclarification of its objectives and role in thesector. Moving into a wider field than WSS maybe inadvisable, if it would be too difficult to findand retain the necessary skills. Some skills canand should be developed by providing trainingfor existing staff, in fields such as changemanagement, risk management or outcome-oriented project planning. In particular, werecommend that WSP should:❚ try to recruit more people with technical

backgrounds plus a good understandingof people, motives, governance, andinstitutions;

❚ if not placing a core of thematic thinkers inheadquarters, give those in the regions anexplicit remit, plus time and funding, to travelto other regions or to global fora to interactwith their peers;

❚ if using a significant proportion of relativelyinexperienced staff, make specific provision(with enough time and funding) for theirsupervision, guidance, and monitoring bymore experienced staff; and

❚ accepting that most staff have to be employedon relatively short contracts, develop anexplicit and transparent policy of providingin-service training and experience to staff whowill then go on to carry WSP’s insights andways of thinking into other sectororganizations in their subsequent careers.

b) Regarding WSP’s position of not being able togive its staff long or open-ended employmentcontracts (any change would require much morereliable medium-term funding), we hear withconcern that recent or impending changes inWorld Bank procedures may make continuityof employment even more difficult for WSP toachieve. If this is the case, WSP should examinethe implications of any such changes forits staffing, and take steps to mitigate anynegative consequences.

6.6 WSP’s Ways of WorkingOur recommendations for the ways WSP goes aboutits work follow on from, and would crucially dependon, the recommended clarification of objectives andof relationships with other organizations. Therecommendations in this section apply generallyto WSP management and staff.


a) The Program should strengthen its efforts toidentify and spell out intended outcomes andimpacts when designing its own projects, ratherthan concentrating on outputs (results of thoseefforts are patchy so far). It should thus work outimproved and outcome-centered ways ofdesigning projects, not just describing projectsalready designed, making the distinctionbetween outputs under WSP’s control andintended outcomes-impacts beyond thatcontrol. The intended outcomes should then beincluded in periodic monitoring and in a simplestandardized project completion report, usuallyrelying on the subjective judgement of WSP staffand key local partners to asses how far theoutcomes have been achieved. Internalguidelines and staff training may be useful, andthe improvements to project designing shouldprimarily be an internal exercise, building on theinsights and experience of WSP staff, with littleor no reliance on outside experts. We do not seea need to change the format of the annualBusiness Plans significantly, only to improve theway they are prepared. The partners and clientsof a project should preferably be involved in itsdesign. [Chapter 2, Section 4.2.3]

b) When a WSP project is being designed, this carefulformulation of the intended outcomes andimpacts should lead to the identification of theother actors whose attitudes and behavior needto change, and then to an analysis of theirconstraints, habits, motives, and priorities. Thenthe WSP project can be designed to work withthose factors, and can, if necessary, includeelements to change them. A clear framework andmethodology for project design, using thisanalysis of the non-WSP actors, should make iteasier to achieve effectiveness and relevancemore efficiently and consistently. Both this andthe previous recommendation should not leadto staff spending more time on project design,merely to using the time more efficiently.[Sections 4.2, 4.3]

c) When asked to help others to design aninvestment project, WSP should take care alwaysto offer and apply its own distinctive experienceand judgement, avoiding being drawn into theways of thinking of the other parties unless thoseare demonstrably appropriate.[Section 4.3.2]

d) In view of the political sensitivity of institutionalreform, WSP should further strengthen its effortsto build up wherever possible close and trustingrelationships both with donors/funders andwith government officials and perhaps localpoliticians; this can enhance its distinctadvantages as a partner independent of fundingsources. [Section 4.3.2]

e) WSP should collate and develop itsgrowing experience in serving decentralizedorganizations as well as central or provincialgovernments, and seek ways to make it widelyavailable within its own operations and to others.[Section 5.1]

f) WSP should consider using a wider range ofmedia, for instance virtual field trips on video tobring field lessons to wide audiences, withoutthe need for extensive travel and simultaneoustranslation. [Section 4.4.6]

g) The Program does sometimes distribute thepublications of other organizations, but weconsider this could be done more usefully, andthey could distribute those of WSP. [Section 4.4.6]

h) WSP should seek more ways to exchange ideasand information between its regions, forinstance by occasional visits by regional staffmembers to other regions. Examples ofinnovative work in particular regions that mightbe applicable elsewhere are ways to reducefixed costs, ways of working with WSSCC,upstream approaches, and private sectorinvolvement modes.

i) WSP should continue and strengthen its effortsto overcome language barriers and to foster thework of sub-regions such as West Africa; donorsshould consider helping with the costs oftranslation and interpretation.

j) WSP staff should have enough confidence intheir own caliber and achievements to adopt amore self-questioning culture, together with aless defensive attitude to constructive criticism.[Section 4.4.9]

6.7 Relationships with Other Global Organizations [Section 4.4.4]WSP should reconsider how it works with other


organizations on the global WSS and relatedfields, building on its review of objectives,comparative advantages, and strategy. Work withNGOs is important, but we are more concernedabout global bodies such as GWP and WSSCCwhose fields abut or partially overlap that of WSP.In many cases, the various bodies in the sector

could do more to help each other in such mattersas dissemination of publications and knowledge,advocacy, persuasion, and the changing ofattitudes. While care has to be taken not tooverstretch the capacity and staff time of anyorganization, we consider that WSP could useexisting institutional links more forcefully.


Terminology for this EvaluationFigure 1.1 shows how the RBM planning levels relateto various aspects of this evaluation, in particular tothe aspects identified in the ‘objectives of theevaluation’ section of our ToR.

This diagram shows the following evaluation aspects(we use the neutral term ‘intervention’ to mean aproject, program or any other defined packageof activities):

Efficiency: This describes an activity’s orintervention’s cost-effectiveness, as opposed towastefulness, in using inputs (resources) to produceoutputs; inputs include financial resources, theProgram’s human resources (its main asset), and theinputs of contributing partners.

Effectiveness: This refers to the interface betweenoutputs (under the control of WSP and participatingpartners) and outcomes (outside that control),

APPENDIX 1

which in our hierarchy is the most importantboundary. Effectiveness describes an intervention’sability to bring about or stimulate the outcomes,given the outputs. It is an important measure of howwell the intervention is designed and adapted toits context.

Sutainability and replicability: An inter-vention’s sustainability makes it last over manyyears, and its replicability enables its approachto be applied in many places, not just the place wherethe intervention was undertaken; together theseattributes enable the progression from outcomes toimpacts. These two characteristics do not necessarilycome together; it is possible to replicate a projectthat is later found to be unsustainable, and a goodsustainable project can still be place-specific. Forvaluable impact replicability is useful, whilesustainability is essential.

Figure 1.1: Planning Levels and Evaluation Aspects

Hierarchy of Levels

Impacts of WSP work

Outcomes of WSP work

Outputs of WSP activities

Activities

Inputs

Evaluation Aspects

effi

cien

cyef

fect

ive-

nes

ssu

stai

nab

ility

& r

eplic

abili

ty

rele

van

ce

stre

ngt

h o

f in

stit

uti

on

al li

nks

resp

on

sive

nes

s to

op

po

rtu

nit

ies

(nee

ds

and

reso

urc

es o

f oth

er a

cto

rs)


Relevance: This concerns the levels from outputsup to impacts; it is a measure of whether or not a setof outputs and outcomes (if achieved) are likely tocontribute to an appropriate higher level goal.Relevance is linked to sustainability andreplicability; if a project is relevant, it is more likelyto be sustainable and replicable.

Strength and nature of institutional links:These are crucial for several of the otheraspects, especially efficiency, effectiveness, andsustainability/replicability.

Responsiveness to opportunities: This aspect,closely related to the previous one, concerns theextent to which WSP matches its work to the needsand strengths of other sector actors, all the way frominput and activity levels where WSP works with other

actors, through outputs that are designed tocomplement those of other actors, to the design ofinterventions so as to maximize the chances ofachieving outcomes (effectiveness).

In accordance with the Terms of Reference and ourview of relevant topics, the above six aspects arejoined in Chapter 4 by impact, and by the followingaspects which are not specifically related to thehierarchy of levels:❚ Finance and resources❚ Focus and balance❚ Planning, monitoring, and internal evaluation❚ Institutional arrangements and governance❚ Relationships with other sector actors❚ Gender❚ Staffing and staff skills


Evolution of WSP’s Planning andMonitoring Methods, 1998-2003WSP’s work was defined over the five-year evaluationperiod in a series of Business Plans, whose natureand format changed every year. At the start, theplanning was being done by calendar years, and the1998 document contained large numbers of ‘taskplanning sheets’. Some such sheets describedbundles of several activities extending over severalyears. As well as listing activities, most of these 1998sheets explicitly set out the task’s ‘overall objective’,‘operational objective’, ‘learning objective’, ‘linkto country activities’, ‘comparative advantage’,‘assumptions’, ‘milestones’, ‘outputs’ (and‘communications/products’), ‘inputs’ (in terms ofstaff weeks), ‘impact on sector at task completion’,and ‘key partners’. This represented a systematic andquite detailed planning system, though using adifferent terminology from ours (the ‘outputs’, forinstance, often imply actions by other actors, so thatsome of them are what we would call outcomes).

This system was, however, not carried through tosubsequent years. The planning shifted to afiscal-year basis, July to June, with a transitionalarrangement for the first half of calendar 1999. Inthat calendar year, partly as a result of a number offragmented reviews of bits of WSP’s work by or forvarious donors, the Program hired a planning andmonitoring specialist to redesign the procedures.He introduced the formal terminology of LogicalFrameworks (‘logframes’) for the whole Program andfor regions and countries, but not for projects,together with a comprehensive annual process ofpreparing Business Plans, a mid-year review(normally in January), and an end-year evaluation.Planning was intended to be done on a 10-yearperspective, rolling so that in each year the periodextended 10 years ahead, and within this a moredetailed rolling three-year perspective. Thisplanning scheme was adopted in varying degrees,and with varying levels of enthusiasm, by all partsof WSP over the next year or two. The task sheetswere discontinued, and instead the Business Plansincluded, alongside some logframes for regions andcountries, tables in which tasks were representedby rows, while columns covered such matters as (inthe plan for the remainder of 1998-99) ‘objectives

APPENDIX 2

and key activities’, ‘outputs/results’, and ‘indicators/milestones’. In successive years the table formatchanged, for instance the one for fiscal year 2000(FY 2000, running from July 1999 to June 2000) had acolumn for ‘indicators of impact’, while the nextyear’s plan featured a column headed ‘outputindicators’ that was explicitly linked to the three-year rolling perspective. But the individual ‘tasks’from previous years, even if they had been scheduledto run on into FY 2000 or FY 2001, were no longer tobe found under their old names (they had apparentlynever had reference numbers), so there was noclearly discernible continuity of planning from onefiscal year to the next. WSP staff gradually modifiedthe logframe approach, dropping much of what hadbeen introduced in 1999 but keeping some aspects,including the annual cycle.

In the Business Plan for FY 2002, there were regionaland country impact statements, and also explicitlogframes for the regions. This time the three-yearindicators were more specific than for FY 2001, andrelated to named countries.

The more detailed Business Plan for FY 2003included regional strategy statements, but nologframe. They did not relate clearly to the previousyear’s impact statement and logframe, though forpractical purposes the main themes continued. Atthe global level it had descriptions of a ‘global ruralstrategy’ and a ‘global urban strategy’. In thisBusiness Plan the table was set out in terms of‘projects’ with individual reference numbers, butthey now had no individual statements of objectives,outcomes or intended impacts, and their ‘product’column usually comprised a list of activities.

Then, in FY 2003, partly as a result of a review andsuggestion by the World Bank’s OED, WSP hiredanother specialist, charged with ‘modifying theapproach from an impact assessment framework toa results-based management [RBM] one’. Oneoutcome of this was the more conscious anddeliberate packaging or clustering of work intodefined ‘projects’, fewer in number and broader inscope that the typical ‘tasks’ of FY 1999. There wasalso a shift towards defining and monitoringoutcomes rather than outputs (the second specialistused the same five planning level names as we do),


and towards coherent planning for more than oneyear at a time.

The last few months of FY 2003, which ends our reviewperiod, saw the preparation of the Business Plan forFY 2004 on this new basis, as an experiment to bedeveloped further for FY 2005. In the Business Planfor FY 2004 there are still country-specific strategystatements, and tables with one row per project, butthere are also individual ‘project sheets’ that specifyeach project’s ‘desired outcome’ (usually in the sensein which we use the word, though sometimesconfused with outputs), and some explicit outcomeindicators with target dates, and there are separatelists of activities and of products. There is again noclear continuity from previous years, the projectnumbering system being a new one, and mostprojects have start dates in FY 2004 itself, even whenthey are obviously the continuation of processesbegun in previous years. The specification of inputsin the FY 2004 plan is related to a newly introducedset of project-specific job codes for staff to enter ontheir timesheets (called IOs in World Bank parlance),which will make monitoring of inputs easier than ithas been in the past.

The detailed Business Plan for FY 2003 identified‘flagships’ for each region and for two globalstrategies. These ‘flagships’ were just some of thethemes on which WSP was working at the time. Theflagship table at the end of BP-FY 2003 notably dealsonly with activities and outputs (‘productdeliveries’), not objectives at outcome or impactlevels, which are described in the text of the regionalstrategies. Another interesting feature of the FY-2003flagship table is that it explicitly deals withcontinuity from the year before, by having a columnfor FY 2002 and sometimes showing what activitieshad been held over because of resource constraints.This is a rare and limited sign of explicit multi-year

planning within the series of Business Plans. In theBusiness Plan for FY 2004, however, these ‘flagships’were not mentioned as such, even though theactivities they had embodied continued. So the‘flagships’ seem to have been ephemeral, merelyfashionable flags briefly planted on lines of workwhich existed long before the label was introduced,and often persisted after it faded.

The fact that many of WSP’s plans and documentsshow only activities and outputs, leaving theoutcome-level and impact-level objectives to beguessed or inferred, is largely a reflection of the lackof clear statements of objectives, at what we calloutcome and impact levels, in the strategydocuments and the successive annual BusinessPlans, or at least the earlier ones. Those documentsdid have logframes and ‘impact’ or ‘strategy’statements in various forms, but they tended to bevague and general, describing problems and needsand laying claim to comparative advantage, butseldom saying precisely what was to be done andwhat it should achieve. Right at the end of our reviewperiod, the preparation of the Business Plan forFY 2004 began to remedy this, and WSP is alreadymoving forward.

It is evident from this brief history that the planningand monitoring system was in a state of change overmost our review period. The process has, however,improved considerably since about 2003,and continues to do so. For the earlier years theunsatisfactory and changing planning formats hinderany monitoring of multi-year processes (by WSP orus), and have been frustrating for the WSP staff. To alarge extent they have held to consistent long-termor medium term visions in particular sub-sectors,regions or countries, but they have often done thisdespite the planning system, not because of it.


APPENDIX 3

Previous EvaluationsThe following table lists some relevant evaluations that we have seen, and a brief discussion follows it.

Title

EVALUATIONS SPECIFICALLY OF WSP:

An Evaluation of the UNDP-World Bank Water andSanitation Program

The UNDP-World Bank Water and SanitationProgram: The Andean Region; Results from theMonitoring Mission 1998

Three -Year Review 1996-1999; Andean Region

External evaluation of the Swiss Agency forDevelopment and Cooperation contribution to theUNDP-World Bank Water and Sanitation Programand to the Pan-American Center for SanitaryEngineering and Environmental Sciences(CEPIS/PAHO)

Joint Assessment of the Regional Water andSanitation Group for Eastern and Southern Africa(RWSG-ESA)

DANIDA evaluation reports on West and CentralAfrica and on South Asia

WSP comments on that DANIDA evaluation

WSP action plan in response to all those1999 evaluations

Review of WSP’s governance

WIDER REVIEWS WHICH ALSO COVER WSP

Bridging Troubled Waters: Assessing the World BankWater Resources Strategy

The World Bank’s Approach to Global Programs:An Independent Evaluation – Phase 1 Reportand Annexes

External Consultant Evaluation of Knowledge Bank:Water Supply Sector

Author

‘an independentteam’ appointed byWB and UNDP

Boman & Peck, forSIDA

WSP

WSP andCEPIS/PAHO

SIDA

DANIDA

WSP

WSP

Jon Lane

OED of WB

OED of WB

OED of WB(Elizabeth Kleemeierand Keith Stallard)

Date

Feb 1996

Feb 1999

May 1999

June 1999

July 1999

July 1999(3 volumes)

undated

Sep 1999

Mar 2000

2002(Published document)

Aug 2002 (Grey coverBank document)

Nov 2002


Title

Efficient, Sustainable Service for All? An OED Reviewof the World Bank’s Assistance to Water Supply andSanitation

Shared Knowledge: Innovations and RemainingChallenges

Water & Sanitation Sector Board Assessment… draft

Author

OED of WB

OED of WB(Catherine Gwin)

QAG of WB

Date

Sept 2003 (anotherversion dated July andAug 2003) (World Bankdocument)

2003(Published Bankdocument)

Dec 2003

The 1996 EvaluationThis overall review was conduced for the World Bankand UNDP in 1996, well before our review period.There are some relevant points in it but many ofthem are repeated in later evaluations – which ismaybe a lesson in itself. We note the following fromthe 1996 executive summary:

1. No mission statement – that has beenattended to.

2. Need for reviews to evaluate progress againstsome work plans – some progress but stillroom for improvement.

3. Need to develop means of measuring whetherthe Program is making a difference to the livesof primary stakeholders . This relates tooutcomes and impacts and is difficult to do.

4. Need to move away from Bank rules andprocedures relating to personnel and needto develop opportunities for careerdevelopment for staff members – probablyimpossible to implement under the presentgovernance and funding arrangements.

5. Regional groups should be strengthened andperhaps made centers for specialized sectorexpertise – this has been done to aconsiderable extent.

Perhaps the main lesson to learn now from the 1996evaluation is that there are some things that can bechanged and some that are very difficult to change.

The 1999 Regional Evaluationsand Their OutcomesThe SIDA evaluation of the East and SouthernAfrica Program, carried out in 1999, noted the needfor a strategy that clearly states WSP’s role in Africa

and defines the limits to its involvement in thesector. It recommended that WSP should developa strategic plan for Africa, clarify its scope forinvolvement in different countries and itsmodes of operation, and precisely define the roleof the Nairobi office and the Country SectorAdvisers. Other recommendations related tothe development and implementation of amore effective management system and thedevelopment of marketing, communications, anddissemination strategies.

The DANIDA evaluation of WSP’s activities in Westand Central Africa and South Asia reached someconclusions that were similar to those reached bythe SIDA team. It too noted the need for a strategicframework, within which the Program could be moreselective about its operations. Noting the strongpositive experiences in countries in which theProgram had a strong country presence, it arguedthat support for sector policies, performed withinindividual countries, constituted a strongcomparative advantage for the Program.

As in East and Southern Africa, existing planningsystems were found to be inadequate whilemonitoring and reporting systems were describedas rudimentary. The evaluation found that there wasroom for improvement in financial reportingsystems. It expressed concern that the closerelationship with the World Bank might restrict theactivities of the Program and force it to follow theBank’s agenda to the detriment of its own prioritiesand concerns. Last, but not least, it found that thethen Program Advisory Committee (PAC) hadlimited influence and that there was thus a need todevelop a more effective channel of communicationbetween WSP and its donors.


WSP responded to these evaluations by:❚ Taking steps to improve project managementsystems, as described in Appendix 2 of this report.❚ Producing a Strategy Update and Progress Report(2001), which attempted to refocus the Program’sagenda, clearly identify product lines, strengthengovernance and management processes, andidentify the actions to be taken to secure asustainable financial base for the Program.

The governance arrangements were summarized ina single WSP Charter, adopted in March 2001 at thefirst meeting of the WSP Council (see this report’sSection 3.4). This had replaced the PAC in a changewhich represented a direct response to the findingsof the evaluations.

The 2000 Review of WSP GovernanceIn March 2000, a consultant was commissioned bythe then Program Manager to produce a report onWSP Governance. Much of this report wasconcerned with options for strengthening the PAC.It produced a number of possible governancemodels, thus facilitating the introduction of theWSP Council.

This report makes a number of other importantpoints which still appear to be relevant. Theseinclude the following:

1. WSP does not exist as a separate legal entity.Rather, it is legally a part of the World Bankand its finances are handled through theWorld Bank system. WSP must follow WorldBank financial and administrative proceduresat all times.

2. No written guidelines are available on the wayin which the line should be drawn between‘the Bank’s proper accountability and theWSP’s necessary degree of independence’.

3. Some stakeholders felt a need for greater effortsto be made to ensure that WSP is a truepartnership with beneficiaries, donors, andothers, feeling that they are respected asequals. (Is the fact that WSP routinely refersto government departments as clients ratherthan partners a factor here?).

This report also noted that donors weredecentralizing their budgets and staff to regionaland/or country offices and suggested that thismight have repercussions on WSP’s financingarrangements. It seems that this trend has

continued and that the need to explore new theconsequences for WSP’s finances remains.

Lessons from OEDand Other EvaluationsMore recently, the Operations EvaluationDepartment of the World Bank (OED) has carriedout a number of evaluations with implications forWSP. Each of these is now briefly reviewed.

An OED review of the World Bank’s water supplyand sanitation-related assistance was completed inSeptember 2003. This review does not refer to WSPdirectly but its findings have implications for theactivities and future direction of the Program. Thereview identified a need to modify the paradigm,predominant in the 1990s, which focused ondeveloping systems for regulating private sectorparticipation in the sector. Noting that there werefew examples of institutionalized regulation throughan independent regulator and that efforts to regulatethrough contracts could be undermined bymacro-economic and political pressures, the reviewnoted the need to move from prescribing toimplementing regulatory systems. WSP-AF isalready providing inputs to efforts to implementdesign and implement regulatory systems that arerelevant to conditions in the countries in the region.It may be that other regions could usefully increasetheir efforts in this area. This increased focus ondeveloping and regulating appropriate forms of PSPshould form one element of a drive to developimplementable sector strategies that can, in thewords of the OED, chart a route towards the MDGs.

In November 2002, the OED completed an ExternalConsultant Evaluation of water sector publications.The review included a number of WSP publications.Publications were reviewed in terms of theirrelevance, comprehensiveness, content, clarity,objectivity, and practicality. Depending on theresults of this review, they were classified in terms oftheir content (relevance, comprehensiveness, andcontent) and their potential for ‘sharing andapplication’ (clarity, objectivity, and practicality).The results of the review for the 19 WSP publicationswere as follows:Publication of ‘benchmark’ quality 5Satisfactory 5Weak content 2Weak sharing and application 3


Unsatisfactory with respectto both content and sharing/application 4

The main specific weaknesses identified by thereview were:❚ A lack of objectivity – in some cases a documentset out to prove rather than test a hypothesis andmarshalled its arguments accordingly.❚ Some documents were too general and failed toprovide practical advice on their findingsand conclusions.❚ Some documents failed to adequately addressimportant issues, particularly politics andsocial factors.

The methodology and quality of this OED reviewmay themselves be open to criticism, and itsfindings were challenged in detail by WSP, but ourrapid review of a number of the documents suggeststhat the OED review was generally fair and that therewere indeed problems with a significant percentageof publications. Given the fact that knowledgedevelopment and management is one of WSP’s‘product lines’, the finding that almost 50 percent ofpublications reviewed were unsatisfactory in someway is worrying. Since the completion of that OEDreview, WSP has taken positive steps to improve itsquality control procedures for publications. This isto be welcomed and, together with the employmentof communications specialists in Africa and LAC,should help to address the problem. However, thereis no room for complacency.

One issue for the future, relating to the relationshipbetween the Program and the World Bank, concernsthe latter’s commitment to become a ‘globalknowledge bank’. It aims to do this through threeglobal initiatives, the Development Gateway, theGlobal Development Learning Network (GDLN)and the Global Development Network (GDN). AnOED evaluation report, produced in 2003, notes thatthese initiatives have sparked considerableinnovation, knowledge sharing programs, andactivities but that they will only achieve their broadobjectives if they are better integrated into the Bank’score business processes. The report concludes thatthere is a need for Bank management to exercisemore strategic direction and oversight over theBank’s knowledge processes. The recommendation

is thus for a greater degree of central control overknowledge processes. At present, WSP has littledirect involvement with the Bank’s globalknowledge initiatives. However, given the fact thatknowledge generation and dissemination is animportant part of its remit, any move to centralizethe control of these initiatives within the Bankwould have important consequences. Not least, itwould represent a move to link the various programsmore closely to the Bank and would arguably resultin a reduction in WSP’s independence.

WSP’s Response to EvaluationsWSP has taken some important decisions on the basisof the evaluations. The most important of these relateto the Program’s planning systems and thereplacement of the PAC with the Council. However,it has not accepted all criticisms and has on occasionresponded to evaluations with detailed rebuttal ofmany of the points contained in them. The defencehas usually been on the grounds either that the pointsmade in an evaluation were not justified or that theevaluators were dealing with the past and had nottaken account of WSP’s more recent efforts to dealwith problems. While WSP’s responses doundoubtedly include valid points, they do point tothe existence of a rather defensive culture within theorganization. Our brief review of some of thedocuments covered in the 2002 OED review ofdocuments, for instance, led us to similar judgementsto those made by OED. This is an issue that needs tobe addressed. It will require action both:❚ By WSP itself, to develop a more self-questioningattitude; and❚ By donors, Bank managers and other to whom itreports, to provide suitable incentives for thedevelopment of this self-questioning attitude.

One aspect of the latter would be recognition that,given the complexity of issues relating todevelopment in general and WSS in particular, theresults of interventions will not always be asexpected. Close adherence to a system thatemphasizes targets above everything else is unlikelyto produce lasting results in such circumstances.Rather, there is a need for a greater emphasis onevaluation of the strengths and weaknesses of pastinitiatives, in order to learn from both.


APPENDIX 4

FinanceDetailed financial reporting is not a part of thisevaluation. The table below summarizes theProgram’s expenditure in the last four years of theevaluation period (FY 1999 cannot be includedbecause amounts were broken down in differentways at that time). Some details are in the regionannexes to this report.

The table gives a more detailed breakdown of mostof these expenditures for the same four fiscal years,plus FY 04 for interest. The expenditure is brokendown into the three spatial categories – rural, urban,and small towns, plus three other categories whichinevitably cut across the spatial ones. Classifyingexpenditure that pertains both to a spatial categoryand to another one, such as gender work for rural

sanitation, is obviously difficult, so the relativeamounts are subject to some uncertainty. Year-to-year comparisons, in particular, must be treated withcaution because any change in the way things arecategorized could distort the relative amounts. Forexample, one gender-related rural sanitation projectmight be classed as rural expenditure while a verysimilar one in another time or place might beclassified as cross-cutting because of the genderaspect. In particular, the apparent increase fromFY 00 to FY 01 in rural, urban, and small town work,with a simultaneous decrease in cross-cutting work,may be at least partly due to this sort of effect (theaccounting procedures were different in those yearsfrom what they are now, and these statistics havehad to be reconstructed more recently usingproduct descriptions).

WSP Expenditures, million US$

FY00 FY01 FY02 FY03 Total Percent

HQ&G* 2.5 2.7 2.3 2.1 9.6 19%

LAC 1.7 2.2 1.2 1.0 6.1 12%

EAP 2.3 2.3 2.6 3.1 10.3 20%

AF 2.9 4.4 3.4 3.9 14.6 28%

SA 3.1 3.3 2.5 2.3 11.2 22%

Total 12.5 14.9 12.0 12.4 51.8 100%

Source: Information from WSP on April 27, 2004

*This category includes expenditures of the WSP global managementand the global thematic program (approximately 50 percent each)


APPENDIX 5

People Met and Consulted

AFRICA REGION

WSP-AF (Kenya)Piers Cross, Regional Team LeaderAlain Morel, Urban SpecialistAndrew Makokha, Senior Water and

Sanitation SpecialistSapheth Mbuvi, Operations AnalystAndreas Knapp, ConsultantMeera Mehta, Senior Financial SpecialistThomas Fugelsnes, EconomistJean Doyen, ConsultantKeziah Kihara, Office AdministratorChristine Njung’e, Budget Analyst

WSP (Dakar)Annie Manou Savina, Senior Community

Development Specialist

WSP (Uganda)Maimuna Nalubega, Country Sector AdviserAnthony Waterkeyn, Consultant

WSP Communications teamSarah de Villiers Leach, Communications

SpecialistCecilia Martinsen, JPO Communications

Specialist

World BankMakhtar Diop, Country Director, Kenya,

Eritrea, and SomaliaWambui Gichuri, Economist AFTU1

Government officials in KenyaEngineer Lawrence Mwangi, ex Nairobi

City CouncilDavid Stower, Deputy Director, Ministry

of Water Resources

Government officials in UgandaPaul Luyima, Assistant Commissioner of Health

Services and Head Environmental Health(MoH)

Engineer Ephraim Kisembo, Reform Manager,DWD

Edward Jaombe, Manager of Luwero watersupply system(on behalf of Management Contractor)

Representatives of partner organizationsDennis Mwanza, WUPSamuel Wambua, NETWASRosemary Ropp, Maji na UfanisiSimon Kenny, (DFID), Telephone conversation

EAST ASIA-PACIFIC REGION

WSP-EAP (Jakarta, unless otherwise noted)Richard Pollard, Regional Team LeaderAriyanto Istandar, Water and Sanitation SpecialistRatna Josodipoera, Hygiene Education SpecialistAlfred Lambertus, Senior Water and Sanitation

SpecialistNilanjana Mukherjee, Senior Community

Development SpecialistJanelle Plummer, Senior Institutional

Development SpecialistAnn Thomas, Water Supply and Sanitation

ConsultantYosa Yuliarsa, Regional Communication SpecialistThomas Meadley, Country Team Leader

(based in Vientiane, Laos)Jemima Sy, Country Program Officer

(based in Manila, Philippines)Jan Rosenboom, Country Team Leader

(based in Phnom Penh, Cambodia)

Government officials in IndonesiaDr Umar Fahmi, DG Communicable

Disease ControlDjoka Wartono, former Director, WSLIC 2Zainal Nampira, Project Manager, WSLIC 2Basah Hernowo, Head of WASPOLA

Working GroupDr Ir. Suyono Dikun, Deputy Minister

of Infrastructure

Representatives of partner organizations(Jakarta, Indonesia)Jim Woodcock, Water and Saniation Adviser,

USAIDZabeta Moutafis, First Secretary, AusAIDHilda Winata, West Java Representative, UNICEF


Jaco Mebius, First Secretary, Water ResourcesMgt., Royal Netherlands Embassy

World Bank (Indonesia Country Office, Jakarta)Andrew Steer, Indonesia Country DirectorAniruddha Dasgupta, Infrastructure Sector

CoordinatorJanet Hohnen, Health Sector Coordinator

(WSLIC2 Project)

LATIN AMERICA & CARIBBEAN REGION

WSP staffLuis Tam, Regional Team LeaderRafael Vera, Peru Country CoordinatorOscar Castillo, Institutional and Community

Development SpecialistRocío Flórez, Handwashing CoordinatorMartin Ochoa, Coordinator for Central AmericaJorge Luis McGregor, Small Towns Management

Models SpecialistMaria Luz Perez, Communications ConsultantNorma Chávez, Language Team AssistantCecilia Ubillus, Administrative AssistantLuciana Mendoza, Communications AssistantZoraida Mantilla, Regional Team Assistant

StakeholdersJean Bernard Parenteau, CIDA/ACDI, PeruMaría Elena Bodero de Eran, CIDA/ACDI, PeruBeatrice Meyer, SDC/COSUDE, PeruGilbert Bieler, SDC/COSUDE, PeruGuillermo León Suematsu, National Director of

Sanitation, Ministry of Housing Constructionand Sanitation,

also Vice-President of SEDAPAL, PeruJavier Hernández Campanella, PRONASAR, PeruRoxana León, PRONASAR, PeruAndrew Michell, Alicorp, PeruDalmi Villar, Radio Programas del PerúJuan Manuel Calvi del Risco, Radio Programas

del PerúNora Reyez, MINSA, PeruCarmen Calle, MINSA, PeruJaime Salcedo, SEDAPAL, PeruJavier Acosta, SEDAPAL, PeruRolando Vásquez, SEDAPAL, PeruLuis Valencia, CEPISJuan Carlos Valencia, Plan InternationalCarlos Bendezú, Fondo de las AmericasRoger Agûero Pittman, Servicios

Educativos Rurales

Omar Macedo Ruiz, Centro de Estudios ySolidaridad con América Latina

Carlos Escalante Estrada, CENCA (Instituto deDesarollo Urbano)

Virginia Baffigo, CARE, PeruJuan Francisco Soto Hoyos, CARE PROPILAS,

Cajamarca, PeruWalter Cabrera, CARE PROPILAS, Cajamarca, PeruMirtha Villanueva, CARE PROPILAS, Cajamarca,

PeruAlfonso Aguirre, CARE PROPILAS, Cajamarca,

PeruPercy Suarez, CARE PROPILAS, Cajamarca, PeruJuan Salazar, CARE PROPILAS, Cajamarca, PeruEduardo Bustamante, President of JASS (users

association for water supply scheme),Monterey, Cajamarca, Peru

José Barragán, Vice Ministry of Basic Services,Bolivia

Antonio Terrazas, Vice Ministry of Basic Services,Bolivia

Enrique Torrico, Vice Ministry of Basic Services,Bolivia

Juan Carlos Sumaeta, Fundación SUMAJ HUASI &CAUDAL, Bolivia

Francisco Guachalla, Catholic Relief Services,Bolivia

Betty Soto, BoliviaEnrique Torrico, BoliviaRoberto Bianchi, Aguas del Illimani, BoliviaTania Jaldin Trigo, Aguas del Illimani, BoliviaMario Flores Chávez, Aguas del Illimani, BoliviaAlbero Chávez Vargas, Aguas del Illimani, BoliviaHumberto Puerto, RASCA (Central America

Regional Watsan Network), HondurasHugo Cobo, FIS/KfW, HondurasGivanny Ayestas, FIS Watsan Pilot Project,

HondurasThomas Walder, SDC, HondurasYadira Recinos, SDC, HondurasEdith Rivera, Save the Children, Honduras

SOUTH ASIA REGION

WSP-SA(New Delhi, India, unless otherwise noted)Junaid Ahmad, Regional Team LeaderMarianna Kullappa, State Coordinator

(Andhra Pradesh)Vandana Mehra, Regional Communications

Specialist


David Savage, Sr. Municipal DevelopmentSpecialist

Mark Ellery, Water and Sanitation SpecialistShekhar Shah, Lead EconomistSalman Zaheer, Lead Utilities Specialist

WSP-SA (Dhaka, Bangladesh)Shafful Azam Ahmed, Water and Sanitation

Specialist and Acting Team Leader,Bangladesh Country Team

Khawaja Minnatullah, Sr. Water and SanitationSpecialist

Tanveer Ahsan, Urban SpecialistKazi Adil Ahmed Shafi, Consultant (RWSS)Md. Abul Fayez Khan, Program Assistant

WSP-SA (Islamabad, Pakistan)Raja Rehan Arshad, Team Leader, Pakistan

Country Team

OthersIndiaAbhyankar, G V, World Bank Water and Sanitation

SpecialistAndersson, Owe, SIDA Head Development

CooperationBanerjee, S, Ex. Joint Secretary, Ministry Urban

DevelopmentBarrett, Alison, World Bank Regional Adviser

Cities AllianceBenmassaoud, Rachid, World Bank

Operations AdviserBriscoe, John, World Bank WRM AdviserChakravaty, Sunita, SIDA Program ManagerChary, Prof V Srinivas, Senior Faculty,

Administrative Staff College of IndiaChaudary, Nazmaul, World Bank Economist DECCurtis, Ian, DFID Senior AdviserDoolan, Sean, DFID Environmental AdviserDurgaprasad, CEO, Zila Parishad Chittoor (AP)Khatua, BC, Government Maharashtra Principal

Secretary, W&SKumar, Ajith, WSP State Coordinator, MaharashtraKutty, PV Valsala, Director, Rajiv Gandhi National

Drinking Water MissionMisra, Smita, World Bank NDO EconomistMohandas, Palat, Secretary, Ministry of

Rural DevelopmentMurty, JVR, WSP State Coordinator, MaharashtraNagireddy, Secretary, Panchayati Raj and Rural

Development, Ministry of Andhra Pradesh

Patel, Sheela, Director, SPARCPatra, Lalit Mohan, Project Officer, UNICEFRajamani, M, Ministry of Urban Development,

Jt SecretaryRay, Samarjit, Chairman, Andhra Pradesh Water

ConservationSanan, Deepak, Government of Himachal

Pradesh, Secretary Rural Development &Panchayati Raj

Shah, Ajay, Ministry of Finance, DEASrivastava, Vivek, Ex WSP India Country TeamSubramanium, Ramesh, AusAID Senior

Project OfficerTripathi, PK, CEO, Delhi Jal BoardThirupathiah, Dr, APPARD, HyderabadVan Norden, Henk, UNICEF Chief, WES

BangladeshAbdullah, Md., Deputy Chief, Local Government

Division, Ministry of Local GovernmentAhmed, Prof M Feroze, Department of Civil

Engineering, BUET(Bangladesh University of Engineering &Technology)

Ahmed, Shehlina, PLAN Health AdviserClaydon, Timothy, Country Representative,

WaterAIDDe Groot, Carel, Danida CoordinatorDyer, Rodney, DFID Engineering AdviserEdwards, Paul, Chief, Water and Environmental

Sanitation Section, UNICEFHossain, Khandker Zakir, Program Director,

WaterAIDHossain, Yakub, Deputy Director, VERC

(Village Education Resource Center)Howard, Guy, DFID Engineering Adviser

(Arsenic Mitigation)Kemper, Karin, World Bank, SASEKhan, Rifat Shahpar, Senior Program Officer,

AusAIDMotaleb, Abdul, Danida Program OfficerQasem, A H M Abul, Ministry LGRD and

CooperativesRacki, Jeffrey, World Bank, SASERashid, SMA, Executive Director,

NGO Forum for DWSSSwann, Peregrine, Senior Infrastructure and

Livelihoods Adviser, DFIDWaldvogel, Markus, Counsellor (Development),

SDC, BangladeshWallich, Christine, World Bank

Country Director


WashingtonSaghir, Jamal, Director, Energy and Water

Department, World BankStottmann, Walter, Program Manager, WSPIyer, Parameswaran, Senior WSS Specialist, WSPKolsky, Pete, Senior WSS Specialist, WSPRoa, Donna, Senior Communications Officer,

WSPPaynter, Nat, Consultant, WSPGerrard, Chris, Senior Evaluation Officer, OED,

World BankHaxaire, Marie Claude, Resource Manager, INF,

World BankJagannathan Vijay, Sector Manager - Middle East/

North Africa, World BankJanssens, Jan, Program Manager, Bank-Netherlands

Water PartnershipKoenig, Peter, Senior Water Resource Management

Specialist, LAC, World BankLocussol, Alain, WSS Specialist, South Asia, World

BankMcKecknie, Alister, QAG, World BankMejia, Abel, Sector Manager, Latin America/

Caribbean, World BankRavat, Anwar, Chief Administration Officer, INF,

World Bank

Roche, Robert, WSS Specialist, Africa, World BankSara, Jennifer, WSS Specialist, LAC, World BankVan den Berg, Caroline, Senior WSS Specialist, INF,

World BankVan Ginneken, Meike, Water and Sanitation

Specialist, World Bank

OthersArmon Hartmann, Retired, former Water Adviser,

SDCWillem Ankersmit, Retired, former Water Adviser,

DGISMartin Walshe, Water Adviser, EUDennis Mwanza, former Managing Director, WUPBrian Baxendale, Divisional Engineering Adviser,

West Africa, DFIDDarren Saywell, former Program Manager, WSSCCGourishanker Ghosh, Executive Director, WSSCCBarbara Evans, Consultant, former Global Urban

Thematic Leader, WSPBruce Gross, Retired, former Program Manager,

WSPStephen Bass, Head of Profession, Environment,

DFIDand two others


APPENDIX 6

Documents Seen and Consulted[see also Appendix 3 concerning previous evaluations, which are not repeated here]

Title

WEBSITE

RWSG-ESA, Jan-Dec 1998 (reputedly the 1999Business Plan)

WSP Workplan, Jan-Jun 1999

FY 2000 Business Plan(untitled, loose papers in file)



WSP FY 2003 Business Plan

WSP Mid-year review (apparently FY 2000)

2001 Mid-year review

WSP FY 2002 Mid-term review

WSP Fiscal Year 2003Mid-term review

98-99 WSP Report

99-00 Report

00-01 Report

01-02 Report

WSP Annual Report(July 2002-June 2003)

Author or Issuing Organization

WSP (presumed)

WSP

WSP

WSP

WSP

WSP

WSP

WSP

WSP

WSP

WSP

WSP

WSP

WSP

WSP

Date

undated

Jan 1999

undated

undated

undated

undated

undated

undated

undated

undated

undated

undated

undated

undated

undated (draft)

www.wsp.org: a well-designed and informative website, regularly updated, with news and many contacts

WSP GLOBAL ANNUAL PLANS, MID-YEAR REVIEWS AND REPORTS


Title

PUBLICATIONS

Learning What Works: A 20-Year Retrospective Viewon International Water and SanitationCooperation

Taking Sustainable Water Supply Services to Scale:A Discussion Paper

Condominial Water and Sewerage Services: Costsof Implementation of the Model (El Alto-Bolivia,Pilot Project) – Economic and Financial Evaluation

Sistemas condominiales de alcantarillado sanitario– Cambios en la disponibilidad de infraestructurasanitario y en hábitos de hygiene a partir de lasimplantación del proyecto piloto: Un enfoquecuantitativo; Evaluación de Impacto (ProyectoPiloto El Alto – Bolivia)

Sistemas condominiales de alcantarillado sanitario– Guía de procedeimientos (Proyecto Piloto El Alto– Bolivia)

Informe del Taller destinado a la ‘Presentación delos Resultados de la Implantación del SistemaCondominial en La Paz y El Alto’

Better Water and Sanitation for the Urban Poor:Good Practice from sub-Saharan Africa

Linking sustainability with Demand, Gender, andPoverty – A study in community-managed watersupply projects in 15 countries

Sustainability, Planning, and Monitoring inCommunity Water Supply and Sanitation

Meeting the Financing Challenge for Water Supplyand Sanitation

In Pursuit of Good Governance: Experiments fromSouth Asia’s Water and Sanitation Sector


Maggie Black

BNWP/WSP (Jennifer Davis,Parameswaran Iyer)

WSP-AND/Viceministerio deServicios Básicos, Bolivia/ASCID (Vivien Foster)

PAS-AND/Viceministerio deServicios Básicos, Bolivia/ASCID (Nelba Canelli)

PAS-AND/Viceministerio deServicios Básicos, Bolivia/ASCID (Mery Quitón, AdeleMartínez)

Vice Ministerio de ServiciosBasicos (Bolivia); PAS-AND;ASDI

Water Utility Partnership/WSP

WSP and IRC (Gross, van Wijkand Mukherjee)

WSP and IRC (eds Mukherjeeand van Wijk)

Meera MehtaWSP/World Bank

WSP and MassachusettsInstitute of Technology (Davis,Ghosh, Martin, Samad, Tankha,Zia and Prunier)

Date

1998

December2002

undated(ca 2002)

undated(ca 2002)

December2001

2001

July 2003

January 2001

2003

May 2003

April 2003


Title

Water Supply and Sanitation for Small Towns andMulti-Village Schemes (Proceedings of aninternational conference in Addis Ababa, June2002), Vol 1, conference summary

Willingness-to-Pay for Arsenic-Free, Safe DrinkingWater in Bangladesh (Fighting Arsenic: Listeningto Rural Communities)

Estudios de Base para la Implementación deProyectos de Agua y Saneamiento en el Area Rural

Abastecimiento de Agua por Gravedad paraPoblaciones Rurales Dispersas

La Contratacion por la Communidad –Empoderamiento y Saneamiento Rural enel Perú

Water Tariffs and Studies in South Asia (a series ofpapers): 1. Understanding the Basics;2. A Scorecard for India; 3. Tariff Structures in SixSouth Asian Cities

Achieving Sustained Sanitation for thePoor – Policy and Strategy Lessons fromParticipatory Assessments in Cambodia,Indonesia, Vietnam

Desinfectión Solar del Agua – Guia de Aplicacion

El Crédito para Agua y Saneamiento Rural: unaPropuesta

Memoria de Taller sobre Sistemas de InformaciónSectorial para el Saneamiento Básico Rural


WSP/World Bank/Ministry ofWater Resources, Ethiopia,BNWP/World Bank Insititute

WSP/World Bank/BangladeshRural Advancement Committee(Junaid Ahmad, BN Golder,Smita Misra, M Jakaria)

Vice Ministerio de Construccióny Saneamiento, with help fromDirección Nacional deSaneamiento, UGP-PRONASAR,COSUEDE, AGUASAN-COSUEDE, PAS-LAC

ProAnde (Nicolas Marinof),with editorial support from PAS-AND, also supported byUNICEF, COSUDE, CHRISTIANAID, VOLENS

PAS-AND (Oscar Castillo)

PPIAF/WSP (ed ClarissaBrocklehurst)

WSP-EAP (Nilanjana Mukherjee)

SANDEC (translation ofEAWAG/SANDEC publicaqtion(ISBN 3-906484-24-6)) with helpfrom PAS-LAC, COSUDE,UNICEF

IEP (Carolina Triveli), for PAS-AND

PAS-AND, Vice Ministerio deInfraestructura

Date

2002

August 2003

May 2003

November2001

November2001

December2002

April 2001

June 2003

February2003

November2000


Title

PROPILAS: Nota de Campo 1:la Selección de Contratistas para EjecutarProyectos de Saneamiento Rural

PROPILAS: Nota de Campo 2:la Selección de Municipios y Comunidades

PROPILAS: Nota de Campo 3:La Participación del Sector Privado en elSaneamiento Rural

PROPILAS: Nota de Campo 4:Dos Modelos en la Provisión de Servicios de Agua ySaneamiento Sostenibles

Lecciones Aprendidas del Proyecto Piloto de Aguay Saneamiento PROPILAS en Cajamarca, Perú

Peru: Saneamiento Básico Rural – AnalisisSectorial y Estrategia

Los servicios de agua y saneamiento en el Perú: undiagnostico y estadisticas

Project Appraisal Document on a Proposed Loanin the amount of US$50 million to the Governmentof Peru for a National Rural Water Supply andSanitation Project

Servicios de agua en zonas periurbanas de LimaMetropolitana – la experiencia del ProyectoAlimentación de Agua para Pueblos Jovenes (APPJ)

La Sostenibilidad de los Sistemas Autonomos deAbastecimiento de Agua del Projecto APPJ in Lima,a incios del 2003

Estudio de las Practicas de Hygiene en Indivduos yFamilias de Poblaciones Peri-urbanas de LimaMetropolitana

Planificacion del Saneamiento Ambiental Urbano:Lecciones de Bharatpur, Rajasthan, India

Health and Hygiene Education in Water andSanitation Projects (also available in Spanish)


CARE-PROPILAS, with editorialhelp from PAS-AND, andsupport from COSUDE, CARE




COSUDE, with help from PAS-LAC (1st and 2nd editions)

World Bank (confidential draft,report 19209)

ACDI and PAS-LAC

The World Bank,Report No. 24173

PAS/Unión Europea(Giovanni Bonfiglio)

SER for SEDAPAL(SER seems to be partof CASMA)

SER

WSP-SA with acknowledgementto DFID

CARE and WSP

Date

September2000

August 20012002

May 2002

July 2002

Nov 2002Aug 2003

April 1999

Aug 2001 and2nd editionMay 2003

August 2002

March 2002

none (2003)

none

August2002

April 2001


Title

Metodologias Participativas en EducaciónSaniataria – un Adaptión de PHAST paraComunidades Rurales Andinas del Perú

Evaluación Programa Aguasan – Informe

Health in your Hands – Public-Private Partnershipsfor Health: A review of Best Practicesin the Health Sector

La Salud en tus Manos – Alianza Público/Privadapara Promover el lavado de manos…Plan de Negocios

La Salud en tus Manos…Executive Report

La Salud en tus Manos…la alianza global entre lossectores publico y privado para promover el lavadode manos con jabon

La Salud en tus Manos…estudio decomportamientos en lavado de manos conjabonen zones urbano perificas y rurales del Péru

Red Nacional de Agua y Saneamiento de CentroAmerica (RRAS-CA)

Construyendo la equidad de género:metodologia e instrumentos para su inclusión enproyectos de agua y saneamiento

Norma Boliviana NB 688, Installaciones Sanitarias– Alcantarillado Sanitario, Pluvial y Tratamiento deAguas Residuales

Building Partnerships to Serve the Urban Poor:Summary Proceedings – Final Workshop of WUPProject No. 5, Abidjan, Côte D’Ivoire,November 19-21, 2001

List of Water and Sanitation Program-Africa(WSP-AF) Publications, 1998-2003

Meeting the Challenge of Water and SanitationServices for the Poor in East and Southern Africa:A Three-year Regional Business Plan FY 2000-2002


ProAnde and UNICEF,published by WSP-LAC

COSUDE

WSP(Ann Thomas, Valerie Curtis)

the alliance, Lima, Peru

WSP-LAC, mentioningCOSUDE, MINSA, USAID

Apparently by PRISMA,for the alliance

RRAS-CA, with support fromWSC (Martin Ochoa)

( Jesica Nino de Guzman,Mecedes Zevallos C) WSP of theWB (sic), with acknowledgmentto the WB Gender Division

Dirección Nacional deSaneamiento Básico, DIGESBA,Bolivia

WUP/WSP

WSP-AF

WSP-ESA

Date

May 2003,secondedition

Aug-Nov2002

July 2003

? 2003

none,probably2003

none,probably2002 or 2003

none

2nd ed.,May 2003

2001

February2004

1999


Title

New Designs for Water and SanitationTransactions: Making Private Sector ParticipationWork for the Poor

Private Sector Participation in Water Supply andSanitation Services in Sub-Saharan Africa:Decision-Maker’s Workshop: SummaryProceedings and Outline for a Roadmap: Dakar,Senegal, February 13-15, 2002

Report of the Regional Program AdvisoryCommittee (RPAC) Meeting, March 1-2, 1999,Nairobi, Kenya

Report of the Regional Advisory Committee (RAC)Meeting, November 6-8, 2002, Dakar, Senegal

Small-scale Independent Providers of Water andSanitation to the Urban Poor:A case of Nairobi, Kenya

Urban Water and Sanitation Reform Strategy

Water and Sanitation in Poverty ReductionStrategies, Workshop proceedings, RegionalWorkshop, Nairobi, June 17-19, 2002

Water Supply and Sanitation in Africa: How tomeasure progress toward the MillenniumDevelopment Goals?

Water Supply and Sanitation in Poverty ReductionStrategy Papers in Sub-Saharan Africa:Developing a Benchmarking Review andExploring a Way Forward

WSP-ESA Mid-Year Review ofWork Program FY 2000

WSP Strategic Program Planning and ManagementSystem – PPMS

NETWORK NEWSLETTERS

Agua – Boletín del Comité Sectorial de Agua ySaneamiento


WSP/PPIAF

BNWP, World Bank

WSP

WSP

Farid Mohamed

Government of Uganda,Ministry of Water, Lands,and Environment

Unpublished paper

WSP Pre-print

WSP

WSP-ESA

Date

May 2002

November1999

July 2003

October2003

February2000

Undated

Comité Sectorial de Agua y Saneamiento,Lima, Perú


Title

AguaYaku – Comite Sectorial de Agua ySaneamiento

Caudal – Revista Sectorial de Agua y Saneamiento

Cuenta Gotas — boletin de la RRAS-CA

WSP NEWSLETTER

ACCESS – an electronic newsletter available bysigning on or accessible through the WSP website

WSP FIELD NOTES

Urban Sewerage and Sanitation: Lessons learntfrom case studies in the Philippines

The Best-laid Plans – Revisiting Community-basedRural Water Supply Schemes in Uttar Pradesh

Lower Costs with Higher Benefits – Water andSewerage Schemes for Low Income Households –Lessons from the El-Alto-Bolivia Pilot Project

Fighting Arsenic: Listening to Rural Communities– Findings from a Study on Willingness to Pay forArsenic-Free, Safe Drinking Water in Bangladesh

Promocion de la salud y la hygiene a través delsistema escolar en el Péru

Nota de Campo – la micromedición en el arearural: una experiencia exitosa en El Salvador

Nota de Campo – escuela y casa saludable: unaexperiencia exitosa en Honduras

Nota de Campo – soluciones innovadura para elsuministro de agua en comunidades ruralesdispersas de Honduras

Nota de Campo – Las normas técnicas y laampliaciónde cobertura de acueducto y alcantarillado hacis lossectores pobres – El proceso de modificación de laNorma Boliviana NB 688

Nota de Campo – la Asociacion de Usarios en laGestion de Servicios de Agua en LocalidadesRurales Multiples


WSP

Joint publication with DILGPhilippines and AUSAID

WSP-SA

WSP/Vice-Ministry of BasicServices/SIDA (Vivien Foster)

WSP/World Bank/BangladeshRural Advancement Committee(Junaid Ahmad, B N Golder,Smita Misra, M Jakaria)

Field note by WASP-LAC,mentioning COSUDE and Kallpa

PAS-LAC, mentioning CARE,USAID (Ricardo Mairena)

PAS-LAC, mentioning UNICEF,SANAA, ASDI, CRS

PAS-LAC, mentioning FHIS,KfW, SANIPLAN, INBAS, SODIS

PAS-LAC

PAS (several offices), withcontributions from ACDI, SUMCanada, and the users association

Date

continuingseries

June 2003

March 2002

undated(ca 2002)

December2002

2nd ed.,April 2003

June 2003

September2003

September2003

–

October2001

Comite Sectorial de Agua y Saneamiento,Ecuador

[email protected]

RRAS-CA


APPENDIX 7

Terms of Reference for this EvaluationThis appendix reproduces the Terms of Referencethat were issued by WSP in 2003. At the time ofcontract agreement in December 2003 the timingwas changed to accommodate a start in Januaryinstead of in December, and during the course ofthe evaluation various other changes were agreedbetween WSP and the evaluation team.

These Terms of Reference describe activities requiredfor the full evaluation of the operations of the Waterand Sanitation Program (WSP) during the periodof fiscal years 1999-2003. The evaluation is beingcarried out by WSP management on behalf of theWSP Council.

BackgroundThe Water and Sanitation Program is an internationalpartnership which aims to provide increased accessto improved water and sanitation services for poorpeople living in rural and urban areas across theworld. Financing is provided by a consortium ofdonors while the Program is managed by the WorldBank, which also retains fiduciary responsibility forthe partnership. The Program has four decentralizedregional operations (Africa, South Asia, East Asia andPacific, and Latin America) and approximately 60 staffbased in field offices. Management is provided by ateam comprising regional managers based in the fieldand a small financial, technical, and managerial unitbased in Washington, DC. The Program’s governingCouncil, consisting of core donors and partners,oversees the direction, strategies, and work programof WSP. Governance issues are outlined in theProgram Charter.

Comparative AdvantagesWSP is one of a range of agencies operatinginternationally and regionally in the water supplyand sanitation sector. WSP aims to deliver a uniqueset of services which complement both the work ofits clients and of other international organizations.WSP specifically seeks to have impact on thecapacity of local and national governments toeffectively support the delivery of sustainable waterand sanitation services. To achieve this impact WSPrecognizes that some interventions must be madeconsistently over long periods of time at national

government level (to modify policies and removestructural constraints, as well as to create nationalincentives for innovation); at local government level(to maximize innovation and learning acrossmunicipalities and to build up capacity); with localand national civil society and the private sector(to increase understanding of sustainable servicedelivery and to develop new marketable skills);and with local communities (to participateeffectively in the reform debate and to becomeresponsible consumers).

WSP’s field-level interventions thus complementand combine with the work of other groups workingmore at the level of financing (such as internationaldevelopment banks, UNICEF, bilateral programs),international advocacy, monitoring, and evaluation(such as WSSCC, WHO, and UNICEF through theJMP), international networking and capacitybuilding (IRC, WEDC, Sanitation Connection) andpure research (MIT, LSHTM).

The Water and Sanitation Program aims tomaintain the following combination of relevantcharacteristics which together give it a comparativeadvantage to deliver impact in multiple countriesdown to the local level:

1. Global Presence. The network of WSP operationsand skills deployed around the world enables itto quickly draw in relevant global experience withlow transactions costs and create links betweencountries grappling with the same problems.Additionally, the network provides an excellentvehicle to maximize impact of lessons learnedwithin regions.

2. Regional Presence. The ability to flexiblydeploy staff (from WSP and through partnerorganizations) facilitates regional learning andthe sharing of capacity and experiencebetween countries.

3. Positioning Within Countries. The Program’slong-term engagement with policy developmentand capacity building provides an opportunityfor partner-projects to influence radical changesat a national scale and to transfer their influencebeyond project or national boundaries.


Program Strategy, 1999-2003The current Program Strategy Document (Waterand Sanitation for the Poor: Innovating from FieldExperience, 1999-2003) provides the overall directionfor WSP operations. The strategy had four areas ofsubstantive focus; Rural Water Supply andSanitation, Services for the Urban Poor, SupplyChains, and Small Towns. To define WSP activities,four ‘product lines’ were identified: InvestmentSupport, Pilots, Networks, and KnowledgeManagement (which included Studies, Field Notes,and Learning Events).

The 1999 strategy document was developed inresponse to a multi-donor evaluation of WSPoperations carried out in the same year. Theevaluation called for tighter management controlsand greater accountability for delivery of productsand assessment of impact. In response, a numberof management innovations were introduced,including more rigorous business planning andmonitoring. The formation of the WSP Council(replacing the earlier, less formal Program AdvisoryCommittee) was also a direct outcome of this earlierevaluation. Since 2000, formalized consultationwith Regional Advisory Committees (RACs),formalized business planning (documented inAnnual Work Plans, or AWPs), regular progressreporting, and the development of detailed strategypapers for the Council’s consideration and approval,have all led to greater clarity in WSP operations.

The current strategy and activities of the WSP cantherefore best be understood by reference to:❚ the strategy document and subsequent updatespresented to the Council;❚ the proceedings of Regional Advisory Committee(RAC) meetings;❚ AWPs, six-monthly Progress Reports;❚ the Annual Report;❚ background documents prepared for WSP Councilmeetings; and❚ minutes of WSP Council meetings.

Work PlanningNomenclature, 1999-2003The evaluation will focus on, and be structured withreference to, the annual work plans prepared by WSPcountries, regions, and thematic groups. Workplanning has been refined over this period, withprogressive improvements being made to thesystem of planning and monitoring. Annual work

plans are produced by each region, by each globalthematic group (Rural and Urban) and by WSPmanagement. There are thus at least seven workplanning areas in each annual WSP Business Plan.These annual work plans are reviewed in the contextof overall program resources and finalized by theWSP program management team (PMT). Whilesome changes in nomenclature have occurred inthis period, annual work plans broadly containreference to:❚ Flagship Activities: Significant areas of work, witha clearly defined objective of measurable scaleand significance;❚ Outputs: The impacts expected of each flagshipactivity; and❚ Products: Discrete deliverables which are expectedto contribute to each of the outputs.

Work Planning, 2004In FY03, WSP undertook a comprehensive study ofits business planning and monitoring structure,specifically examining how impact could bemeasured more accurately. The findings showed thatwhile WSP was planning well, a few changes couldgreatly improve its ability to evaluate the effect of itsactivities. The changes recommended included:❚ Project Sheets: Generating page-long summariesof the initial goals of every project.❚ Outcome Indicators: Identifying the tools to helpmeasure the impact of projects at a level beyondsimple outputs; and❚ Flagships: Limiting the number of flagship activities.

Under the new planning structure, the Business Plannow contains an annex with all the project sheetsfor the Program. Included in the sheets are theoutcome indicators, along with the desiredoutcomes and background information of theprojects. The indicators, as developed, should bereferenced in developing the consultants’ strategyfor 2004-2008.

Evaluation of Impact,1999-2003The current strategy paper was intended to guideWSP operations up to 2003. Recognizing that thesector is continually evolving, a periodic review andevaluation was part of the strategy identified in 1999.Given the existence of a Council body which bringstogether key funding partners and technicaladvisers, it is proposed that such a periodic reviewnow be carried out under the auspices of the Council,


with full buy-in from individual financing partners,to minimize costs and maximize the resourcesavailable to support a detailed evaluation andplanning exercise. The evaluation will cover theperiod 1999-2003.

ObjectivesOverall objectiveThe overall objective of the current exercise is toevaluate the impact of WSP activities over the pastfive years (1999-2003).

Objectives of the Evaluation, 1999-2003The evaluation will assess WSP operations in theperiod 1999-2003. This assessment will be madein terms of relevance, impact, effectiveness,efficiency, responsiveness, and the sustainablebenefits of the Program’s activities, and overallinstitutional arrangements. Each of these aspectsof WSP operations will be assessed for the periodunder consideration as follows;

A. Relevance (The extent to which the activity issuited to the priorities and policies of the targetgroup, recipient, and donor)a) To what extent are the objectives of the

Program still valid?b) Did WSP have a comparative advantage in

the sector, and if so, was it correctly definedand articulated?

c) Were the stated objectives of WSP operationscommensurate with this comparativeadvantage, clearly defined in strategydocuments and measurable?

d) Are the activities and outputs of the Programconsistent with the intended impactsand effects?

e) Are the activities and outputs of the Programconsistent with the overall goal and theattainment of its objectives?

B. Effectiveness (A measure of the extent to which anaid activity attains its objectives)a) To what extent were the objectives achieved/

are likely to be achieved? (This aspect to beassessed across all annual work plans.)

b) Which activities were most effective incontributing to the achievement/non-achievement of stated objectives, what werethe characteristics of these activities and towhat extent could they have been replicated

in other annual work plans (that is, in otherregions or thematic areas)?

C. Efficiency (A measure of the outputs – qualitativeand quantitative – in relation to the inputs)a) To what extent were the impact and benefits

arising from activities commensurate withthe level of effort and resources expended?

b) What were the most efficient areas ofoperation for WSP activities (by country,region or thematic area of work)?

c) Were activities cost-efficient?d) Were objectives achieved on time?e) Was the project implemented in the most

efficient way compared to alternatives?

D. Impact (The positive and negative changesproduced by a development intervention, directlyor indirectly, intended or unintended)a) What has happened as a result of the Program

or project?b) What real difference has the activity made to

the beneficiaries?c) How many people have been affected?

E. Institutional Arrangements (An assessment ofthe appropriateness and effectiveness of thepartnerships and other institutionalrelationships in which WSP engages)a) How appropriate, effective, and efficient

were the institutional and managementarrangements of WSP during this period?

b) What were the most effective relationships(and specifically partnerships) under whichWSP operated?

c) What were the advantages and disadvantagesof WSP’s institutional relationship with theWorld Bank?

d) How appropriate and effective were WSP’sgovernance, organizational structure, andstaffing profile in realizing a relevant,effective, and efficient business plan?

F. Sustainability (A measure of whether the benefitsof an activity are likely to continue after donorfunding has been withdrawn)a) To what extent did the benefits of a project

continue after the project ended?b) What were the major factors that influenced

the achievement/non-achievement ofsustainability of the project? For example,what were the lessons learned?


G. Responsiveness to Opportunitya) How responsive is the Program to

opportunities and demands that arise fromclients, regional or international sector bodies?

b) Is the management, planning, and financialstructure flexible enough to allow the Programto respond to demands as they arise?

c) Do the benefits of such responsivenessoutweigh the inevitable delay or disruptionto the planned program of work?

d) How highly valued is such responsiveness bythe Program’s clients?

H. Financesa) Financial Sustainability: Has the Program

effectively raised funds in this period? Has itpursued the right type of funding (for example,global vs. regional, large, untied vs. small,tied)? What can be done to improve thesustainability/efficiency of fundraising?

b) Financial Management: Has the Programmanaged its finances effectively andefficiently in this period? Do the task managersmanage their budgets appropriately? Whatcan be done to improve the situation?

c) Impact of Financial Management andFunding Uncertainties upon ProgramEffectiveness: How has the fundingavailability affected the Program’sperformance (for example, have delays inavailable funds hurt performance? Areregional or global funds more effective)?

I. Recommendations for the Futurea) What is the new reality of the sector and how

does it impact on WSP operations?b) What are the new needs and demands of WSP

clients and partners?c) What are the most effective interventions

(products, thematic areas, modes ofoperation, etc.) for WSP with its given trackrecord and comparative advantages?

d) What are the potential financingopportunities for WSP services in the comingperiod; and

e) Are WSP’s mission, goals, and strategiesstill appropriate?

This ConsultancyPurposeWSP management requires detailed inputs to thisprocess from a team of independent experts inthe sector. The consultancy team will carry outthe evaluation exercise, whose objectives aredescribed above, and will oversee a consultativeprocess which will enable WSP Management tomake recommendations to the WSP Council.

OutputsThe consultancy would result in thedevelopment of:

a) Criteria and evaluation indicators forassessing the relevance, effectiveness,efficiency, and responsiveness of WSPoperations;

b) Specific criteria or an approach for assessingthe impact of WSP activities relating to sectorreform and strengthening;17

c) An assessment of performance against theevaluation indicators in the period 1999-2003;

d) An assessment of one project from eachregion that has been completed in FY04. Theassessment will be compared against theterms and indicators developed in the projectsheets as found in the FY04 business plan;

e) An assessment of the effectiveness, efficiency,and responsiveness of the WSP institutionalarrangements in force in this period; and

f) Recommendations for future strategic focusfor WSP.

DeliverablesThe consultancy will have the followingdeliverables:

Inception ReportThe inception report will contain:a) List of key documents and resource people for

the evaluation development exercise;b) Work program for the evaluation development

exercise;c) Draft of detailed program for regional visits and

consultation meetings;

17 This is a focus area in the 1999-2003 strategy and annual workplans, but the impact of discrete WSP activities is difficult to measurebecause: (i) change may take place in a long timeframe; (ii) changes are unlikely to occur without input from a range of partners, makingthe impact of discrete WSP activities hard to assess; (iii) to be really effective, WSP has to create a sense of real ownership on the part ofgovernments, rendering it even more difficult to assess the impact of WSP specifically.


d) First cut of criteria and indicators for assessingthe relevance, effectiveness, and responsivenessof WSP operations; and

e) List of specific questions and concerns relatingto the evaluation to which WSP managementand/or Council will respond.

Draft Global Evaluation ReportThe draft global evaluation will form the basis formore detailed consultation at the regional level andwill contain:a) Draft assessment of general performance against

agreed indicators in the period 1999-2003;b) Draft assessment of the general effectiveness and

efficiency of the institutional arrangements inforce in this period;

c) Preliminary proposal for important activities(two-three per region and one-two global) whichwill form the basis of case studies to be completedduring the rest of the assignment;

d) Preliminary assessment of the key global sectorissues relevant to WSP strategy, as well as likelylevels and sources of funding in the coming five-year period; and

e) Detailed work program for finalization ofevaluation through regional meetings, andpreparation of regional strategies.

Four Draft Regional EvaluationsBased on four regional consultations and furtherinvestigations at the regional/country level, theconsultancy will produce for each region:a) A finalized assessment of performance against

agreed indicators in the period 1999-2003;b) A finalized assessment of the effectiveness and

efficiency of the institutional arrangements inforce in this period; and

c) A finalized assessment of the key sector issuesrelevant to WSP strategy as well as levels andsources of funding in the coming five-year period.

Global EvaluationThe consultancy will finalize the four regionalreports, incorporating feedback received, andalso prepare a brief global overview documentwhich provides:a) Synthesized overall evaluation of performance

and institutional arrangements in the period1999-2003, with reference to regional documentsfor further detail;

b) Synthesized assessment of the key sector issuesrelevant to WSP strategy and assessment of likelylevels and sources of funding in the coming five-year period;

c) Global recommendations for key activities,outputs, and product types which should formthe basis of WSP work programming in thecoming five-year period; and

d) Global recommendations for institutionalarrangements for both the coming five-yearperiod, and, if relevant, in the longer term.

ActivitiesPreliminary Global Evaluationa) Assemble and review current strategy document,

annual Business Plans, six-monthly progressreports, proceedings of regional retreats andRACs, documentation prepared for the WSPCouncil, minutes of Council meetings, minutesof PMT meetings, and any other relevantmanagement documentation which is available.All available documentation will be madeavailable by WSP management;

b) Make a preliminary assessment of the internalconsistency between the strategy, annual workplans, and activities/products delivered anddevelop draft indicators for evaluation of therelevance, effectiveness, efficiency, andresponsiveness of WSP operations, theinstitutional structure, and the impact ofinstitutional reform activities;

c)Hold preliminary meetings/phone conver-sations with WSP management, key staff(past and present if appropriate), and WSPCouncil members;

d) Develop a list of key contacts (individuals andorganizations) who will participate in theevaluation exercise; and

e) Prepare and present to WSP management aninception report (see above).

Regional Consultationsa) Prepare a draft workplan, in consultation with

WSP regional management staff, for thecompletion of a regional consultation in each ofthe four regions;

b) Prepare a draft global evaluation (see above)which will form the basis of regionalconsultations. This should benefit fromdetailed input from WSP management, andwill not be released to partners and clients


without the prior approval of relevant regionaland global WSP staff;

c) Hold detailed meetings (by phone ifappropriate) with regional WSP managementto review the draft global evaluationand plan for regional consultation meetings.Agree on a final workplan in each regionincluding timetable, and a list of keyinvitees and people/organizations whowill be consulted at the regional andnational levels;

d) Incorporate feedback into draft globalevaluation;

e) In each region – prepare a program andbackground material for a regionalconsultation meeting which will be organizedby WSP regional staff. This meeting will beattended by at least one member of theconsultant’s team, at least one member of WSPmanagement staff from outside the region, andWSP staff from the region;

f) Participate in regional consultation meetingsin each region. Contact and consult on anindividual basis with key members of the PAC,RACs, country advisory groups, WSP staff, andWSP Council to assess quality and impact ofinterventions at local, regional, and global level(scope likely to include consultations with atleast five key actors in each region);

g) Analyze within each region the activities andimpacts of the Program over the pastfive years, taking into account relevance,effectiveness, impact, efficiency, responsive-ness, institutional arrangements, andsustainability;

h) Analyze the current sector situation and keyrelevant issues in each region which shouldshape WSP future operations (issues wouldinclude, for example, the MDGs, currentthinking on sector reform, the role of the privatesector, the call for greater action on hygienepromotion and sanitation, etc); and

i) Prepare four draft regional evaluations.

Completiona) Assemble and review feedback on regional

evaluation papers and write draft interimreport for submission to WSP for comment –March 1, 2004;

b) Review feedback on draft interim report.Incorporate Program’s comments into interim

report for submission to WSP Donor Sub-Committee for comment – March 22, 2004;

iii) Incorporate Sub-Committee’s comments intodraft report for donor meeting – May 1, 2004; and

iv) Produce final version of the global evaluation –June 15, 2004.

Staffing and ResourcesStaffingThe firm will provide a review team consisting ofthree to four professionals exhibiting a range of skills.Specific skills of the team will constitute:

Team Leader/ Institutional SpecialistThe team leader will be a specialist with at least 15years of experience in the area of infrastructure andservice delivery, with a focus on water supply andsanitation, of which at least five years should beexperience of working for, or closely partnering with,international organizations working in developingcountry contexts.

Specialist expertise should relate to institutionaltransformation and reform in social service deliveryareas at the national level in developing ordeveloped countries.

The team leader/institutional specialist will:❚ Lead the team, providing direction and oversightto the evaluation exercise;❚ Design and oversee the Program for evaluation,including taking responsibility for planningregional travel, coordinating meetings with keyinformants, and ensuring there is adequate liaisonand coordination with regional WSP offices;❚ Liaise directly with the WSP Program Manager andkeep him informed of the activities of the team;❚ Take the lead in developing the indicators forassessing WSP performance, particularly in the areaof institutional reform;❚ Take the lead in evaluating the institutionalarrangements within which WSP operates,including its institutional positioning, govern-ance, management, partnerships, financing,and staffing;❚ Provide oversight and continuity for the fourregional evaluation exercises; and❚ Take lead responsibility for ensuring all reportsare produced in time, and be available for reviewand discussion with WSP Council members in thelater stages of the consultancy.


Urban Poverty/Sanitation SpecialistThe urban poverty/sanitation18 specialist will haveat least 10 years of developing-country expertise inthe area of improving service delivery to poorcommunities and households in urban areas. Atleast five years of this experience should be directlylinked to the delivery of water supply and sanitationservices. This expertise may have been gained in theprivate sector, NGO or public sector.

The urban poverty/sanitation specialist will:❚ Participate as a full member of the team in theevaluation development;❚ Lead the evaluation of the urban portfolio of WSPwork, including evaluation of regional/nationalproducts and activities, the development ofappropriate indicators of success, and thedevelopment of a future strategy; and❚ Take the lead in two of the regional evaluationexercises.

Rural Water Supply and Sanitation SpecialistThe rural specialist will have at least 10 years ofdeveloping-country expertise in the area ofimproving water supply and sanitation servicedelivery to poor rural communities. At least fiveyears of this experience should be directly linked tothe delivery of water supply and sanitation serviceswithin a wider social development framework. Thisexpertise may have been gained in the private sector,NGO or public sector.

The rural water supply and sanitation specialist will:❚ Participate as a full member of the team in boththe evaluation development;❚ Lead the evaluation of the rural portfolio of WSPwork, including evaluation of regional/nationalproducts and activities, the development ofappropriate indicators of success; and❚ Take the lead in two of the regional evaluationexercises.

WSP ContributionsWSP will provide significant inputs to the processof the evaluation exercise. These contributionsinclude:❚ Staff time: Up to two staff weeks for each regionalteam. Up to four weeks for Program Manager andthe management team in Washington, DC.

❚ Costs of four regional consultation meetings;❚ Costs associated with the participation of WSPCouncil members, RAC members, and otherpartners in all consultations;❚ Rapid and timely input to draft reports producedby the team; and❚ Costs of production of final evaluation documentfor formal distribution to WSP partnersand clients.

Cost EstimateThe cost estimate should include the costs ofconsultant time, travel and associated expenses,communications, production of all reports (eightcopies of each), and other expenses.

Payment will be made on a lumpsum basis on theproduction of the following reports:Mobilization 30%Interim Report 40%Final Report 30%

ManagementThis consultancy will be managed by WalterStottmann, Program Manager, WSP. All writtensubmissions will be made to Mr Stottmann. At therequest of the Council, the consultancy team maybe requested to make presentations directly to theWSP Council either as a whole, or through theChairman, Jamal Saghir, Director, Energy and Water,World Bank.

Proposed Methodology and TimeframeFor this evaluation, WSP suggests that it will take ateam of four approximately 40 person-weeksaltogether to complete. Specific requirements foreach of the team members are defined above.WSP proposes the following timeline for theevaluation exercise:

1. Early December. The four-member evaluationteam comes to Washington, DC to meet withWSP management. The purpose of this visit is forthe evaluation team to become familiar withWSP and its processes. This will also allow the teamtime to examine WSP’s files and documents.WSP staff members (five in total) will plan ondesignating two work-days each to assist the team.

18 Expertise in sanitation would be an advantage in all three members of the team, but at least one member must have specific expertisein this area.


Additional administrative support will be availableas necessary.(One-two weeks)2. Early January. The team evaluates the regions. Oneteam member travels to each of the regions to meetwith the WSP regional offices, client representatives,and other officials.(Two weeks)3. End-January to February. The evaluation teamwrites report, producing it at the end of February.(Four weeks)4. Early March. Team leader returns to Washingtonto discuss the draft interim report with WSP andincorporate suggested changes.(Two-three weeks in DC)

5. End-March. Team incorporates changes anddistributes interim report to donor sub-committee,allowing three weeks for comments.(Three days)6. Mid-April. Sub-committee responds withcomments. Evaluation team incorporatescomments, producing the draft report in time forthe Council meeting in early May.(Three days)7. End-May. Report is presented to Council, andcomments are noted for the final version.(—)8. Mid-June. Council’s comments are incorporated,and final version produced.(Three days)

REPORT

The ITAD team comprised:

Michael Snell

Robert Boydell

Kevin Tayler

Charles Chandler

Water and Sanitation Program1818 H Street, N.W. Washington, D.C. 20433, USA


Africa (WSP-AF)World Bank, P.O. Box 30577, Nairobi, Kenya


Latin America and the Caribbean (WSP-LAC)Water and Sanitation Program, Banco Mundial, Mision Residente del Perú

Álvarez Calderón 185, San Isidro, Lima 27, PerúPhone: (51-1) 615-0685; Fax : (51-1) 615-0689; E-mail: [email protected]

East Asia and the Pacific (WSP-EAP)World Bank, P.O. Box 1324/JKT, Jakarta 12940, Indonesia


South Asia (WSP-SA)World Bank, 55 Lodi Estate, New Delhi 110 003, India

Phone: (91-11) 2469-0488, 2469-0489; Fax: (91-11) 2462-8250, 2461-9393; E-mail: [email protected]

Website: www.wsp.org

REPORTExternal Evaluationof the Water andSanitation Programfor the period1999 to 2003

External Evaluationof the Water andSanitation Programfor the period1999 to 2003