INTERNATIONAL 2 o 2 r

DRINKING WATER SUPPLYAND SANITATION DECADE

Publication No. 6

MINIMUM EVALUATIONPROCEDURE (MEP)for water supply and sanitation projects

1981-1990 May 1985

FOREWORD

The Minimum Evaluation Procedure (Ml-I}) was prepared by WHO in cooperation with theIjOtulon School of Hygiene and Tropical Medicine. The full document describes a relatively low-cost, simple and quick method of evaluating water supply and sanitation projects. The guidelinesare directed primarily towards managers of water supply and sanitation programmes in Membercountries.

The purpose of this booklet is to introduce you to MEP. It was prepared by WHO with theassistance of Mr Brian Appleton and based on material prepared by Mr S Cairncross of the LondonSchool of I lygiene and Tropical Medicine for the International Reference Centre for CommunityWater Supply and Sanitation, The Hague. The drawings were prepared by Ms Iois Carter.

The generous cooperation of IRC and the Royal Tropical Institute, Amsterdam, in providing theoriginal material is gratefully acknowleged.

OF WATER AND SANITATION PROJECTS

^ J . / • • \ . . . . . . . - ' - •

r...

WHY EVALUATE?

1. To get more for our money

With enormous needs and scarce resources, it isvital that money spent on water supply andsanitation schemes should be used in the mosteffective ways. We need to repeat successfultechniques and correct less successful onesquickly, if expected health and social benefits are tobe maximized

Evaluation means testing our assumptions againstactual experiences from completed facilities, sothat faults can be rectified, strengths identified, andfuture plans improved. Only by studying anoperating system can we find out how futuresystems can best be designed.

And it works

Evaluation of a low cost sanitation project in oneAfrican city led to a new. and cheaper design oflatrine. Whereas the project had been almostparalyzed for lack of finance, hundreds of latrinesare now being built each month and the project isexpanding rapidly, paid for by the users.

I"he idea of village-level maintenance workersarose independently in several countries fromstudying the functioning of rural water supplies. Insome areas where village-level handpumpmaintenance has been introduced, breakdownrates have been reduced from a typical 40% to 10%or less.

2. Donors like it

Water supply and sanitation facilities are commonly built with subsidies or loans fromgovernments and external support agencies. They too want value for money. Evaluation is not ahead hunting exercise, to find out who is to blame for mistakes. The ultimate aim is todemonstrate the benefits of water and sanitation projects, to justify expenditure on them, andhence to obtain funds for the sector. At the same time, the backers want to learn from experience,to improve existing and future projects, and to back winners. Ixxal agencies which havecooperated with donors to conduct evaluations have usually found the exercise extremely usefulto both parties.

3. To detect invalid concepts

Breakdowns have technical causes which areusually easy enough to diagnose, but there is oftena social or organizational reason when they occurfrequently and are not dealt with.

Evaluation may show that other technology, whileappearing less efficient, is in fact more appropriate.

WHAT TO EXPECT

Improvements and better understandingThe aim of evaluation is to learn and to improve. Bystudying what we have done to see how far we areachieving our g«ils, we want to gain fromexperience and produce better work

New alternativesMany shanty towns, such us this one in latinAmerica, are subject to fkxjding. An evaluation of alatrine programme in a town like this showed thatraising the ground above flood level cost little moreper plot that a new latrine. With the landfill givingsecurity from flocxling, householders then builtnew houses and new latrines with their ownmonev.

Better functioningIt is easy to see when facilities are not functioningproperly, lftheydonot function, they can hardly beexpected to be used To diagnose the reason andprescribe a remedy may not be just a technicalproblem. A field visit of a day or two will usuallythrow up plenty of ideas for improvements.

An evaluation of functioning will show how manytaps are dry, or only run for a few hours a day, likethis one. Ways may then be found of improving thesituation and of planning future projects better.

Better utilization

If the facilities are not being used properly, theycannot produce their full potential impact. Amixture of observation and questioning of userswill bring out difficulties and ways of promotingand improving utilization.

More impact

Measuring the impact of water supply andsanitation projects on things like healthimprovements and economic status is far from easy.However, the process of evaluation, followed bycorrective action, can be one important way ofobtaining the most benefit from projects.

FUNCTIONING

A r c t h e f a c i l i t i e sfunctioning properly?

Step-by-step

There are three parts to evaluation:

1. Are the facilities functioningproperly?If the answer is NO, ways ofimproving the functioning shouldbe sought before a full evaluationcan be made of the next part;

2. Are the facilities being utilizedproperly?Some information on utilization willhave been collected during the firststage, more will come whenfunctioning defects have beenremedied. Again, proper use offacilities should be assured beforepnx'eeding to —

3. Are the optimum health, socialand economic impacts beingachieved?This last step is not always needed.Basically a system which functions andis used properly can be expected to produceimpact, The idea behind the procedure is thatexperience from the real world helps in two ways: itshows how to get the most out of an operating systemby making improvements where necessary, and it providesknowledge for application elsewhere. The focus is not just onengineering; functioning and utilization depend equally on many

factors, which are reflected in the MF.F.

NO

How can the functioningbe improved?

What to measure

Collecting too little information during evaluation may put the wholeexercise in jeopardy, while collecting tcx> much is both time-consuming and expensive. In MEH, WHO identifies the keyparameters which should be measured when evaluating, for example,the quantity of water provided from a water supply facility.Comparison of theoretical and actual values for demand andproduction is one of the ways to judge the functioning of the facility(along with water quality, reliability, and convenience).

In the second step, the evaluator will want to know the proportion ofhouseholds using the facilities and the way that different amounts ofwater are being used.

MINIMUM EVALUATION PROCEDURE

What is MEP?

The letters stand for Minimum Evaluation Prcxedure — a relatively inexpensiveand simple method of evaluating water supply and sanitation projects, developedand tested by the World Health Organization.

The technique is quick, allowing judgements to be made in a matter of weeksrather than months, with few resources in terms of money or manpower.

MEP follows the three step system, but to keep the method simple, description ofdata collection and analysis is limited to the first two steps: evaluation offunctioning and of utilization. Evaluation of impacts will be dealt with in a laterdexument.

jj

YESUTILIZATION

: YESAre the facilities beingused properly?

NO

How can the utilizationbe improved?

IMPACT

Are the optimum health,'s<x~ial and economicimpacts being obtained?

YF.S

NO

What complementaryinputs are needed toimprove impact?

'.••'?•• :--r".'.-".-'iV- "'••• - V. • '•'->'-••,'.'j! i;'•.'.•"• ' • ^ ' . y - ' j . ' " ; . ' '

^^••'fS^^'^r-

Going by the book

MEP is described step by-step andwith progressively increasing detailin a 52 page txx>klet* issued byWHO. Starting with the initialdecision to evaluate, the b<x>klettakes its reader through the planningstages, collection and assessment ofdata, preparation of recommendations, and follow up actions.

It suggests indicators to be lookedfor under the separate headings ofC o m m u n i t y Water Supply,Sanitation, and Hygiene Education,and suggests data-gather ingtechniques, including samplequestionnaires.

* The booklet - Reference ETS/H11 isavailable in English, French, orSpanish, free of charge, from WHO,1211 Geneva 27, Suitzerland

WHICH EVALUATOR?

Practical lessons

Useful studies have been conducted by donoragencies, by consultants, and by academicresearchers. If the results are to be put into practicethough it is very important to involve staff of thewater or sanitation agency who will be involved infuture projects, and representatives of theconsumers, who will see ways that operations canbe improved in future. Staff chosen must not be soclosely linked with the project that results may beseen as biased.

Field experience

Ixxal staff, like this health worker in Iran, have abetter knowledge of the programme and thepeople it serves than outsiders can ever have.Programme staff make excellent evaluators. At thesame time, an evaluation can be a training exercisefor them, and they will then be more likely toput itsfindings into practice.

The best information of all comes from consumers.Villagers are using (he services every day; whobetter to tell you what you want to know?Handpump "doctors" seen here on a trainingcourse, have a wealth of experience which anevaluation seeks to collect.

10

You the expert

If you organize an evaluation, you rapidly becomesomething of a multi disciplinary expert yourself.So don't rush to call in the specialists until you aresure that you need them. Remembertixuhat bcxiksand manuals carry lots of information you willneed.

Level achice

Evaluation will usually yield lessons at all levels.I>ata collected at the field level may identifyproblems, but the .solutions can .sometimes meanchanges in organization or procedures higher up.Once the information is in, a study at regional ornational level is easy — and often revealing!

Really, an evaluation can be made at any levelwhere there is a will to earn' it out. Ideally, ofcourse, it is conducted by a team of national,regional and local staff.

It doesn't always need an expert to spot whensomething is wrong. Anyone can see that thisAfrican handpump has not been used for a longtime Important facts that an evaluation can bring tolight are often as obvious as this — once one getsout into the field.

MEP AT WORK

Building on success

Botswana's village water supply programme issuccessful. In the first six years, supplies reachedexactly the number of people planned, utilizationwas very high, and the beneficiaries were happy.

So, when Botswana's Department of Water Affairs(DWA) and the Swedish InternationalDevelopment Authority (SIDA) decided lo evaluatethe programme, the idea was to l<x>k ahead.

What factors had contributed to the success of theprogramme? 1 low could those lessons be spread toother programmes? Could these positiveexperiences influence programmes supported bySIDA in other countries?

Collaborative efforts

Results of the evaluation were important to SIDA, to the DWA, and to the Ministry of Finance and Planning.I-essons identified at field level might (and indeed did) have implications higher in the command structure. Soeveryone participated.

From a randomly selected sample of 10 projects, field data were collected by a locally recruited consultant.MEP was used to make sure that the data collected would be both relevant and consistent.

Positive results

Tables gave a gixxd indication of how things wereworking, and helped to identify problems. Fromthe results, it was clear that Botswana's programmeowes its success to: good participation; small scaleactivity; standardization; and good administrationThat is good news for the future.

Not surprisingly, there were some shortcomingsidentified. Future water sector agreementsbetween the Botswana Government and SIDA willpick up these lessons, like: better provision forrecurrent funds (particularly for transport); betterDWA.'MIJG1. cooperation for rehabilitation needs,including tools and training.

y#

AYds/yf

TS0C/

/

I

6

&

310

*//

onnno(5)

onn

®o®

A/S

( • )

O

o

(•)f)o

o®

®noooo

Ifo»

o*>n»

C)»C)'

o»

r/ou

j /

, / ?

/3

Xi

/ «

<»^

i < »

/5»«

^ >̂ «

i 3 /

?/6

1??/

*SDO

t4SZ

N*t

4/1

$79/

3f3

if7

/or

t/y

HI

/ * V

M>

If

t i

6

/ X

f

3

9

/

t

n

«°COM-

mi

i

2

i

o-<® '

12

For the futureOf course evaluation is not the end of the story.Above all, MEP is a planning tcxil. In Botswana, ».iwell as encouraging SIDA and the DWA in futureactivities the evaluation has prompted research toanticipate future problems.

Handpumps and animal powered pumps will bestudied for small settlements, where diesel powermay be unsuitable, and research has beenrecommended into solar technology. 10 anticipatechanging economics.

Satisfied customer

SIDA was pleased with the Botswana study. The Authority's Policy Development and Evaluation Divisioncommented:

"Our experiences of using MEP concepts are encouraging . . . . MEPwas able both to supply relevant information and to pinpoint

important issues in the programme."

Some other reactions" . . . invaluable for all field staff who have to design a formal

programme for evaluation . . . . users can tailor the guidelines to suittheir needs . . . . most impressed with the simple and clear way in

which the guidelines are given."

UNICEF - Bangladesh

" . . . it is simple, to the point, comprehensive and informativerecommend teaching this report to undergraduate and postgraduate

Sanitary Engineering candidates in developing countries"

Head, Sanitary EngineeringSection, University of Khartoum,

Sudan

"All in all the MEP serves a real need to combine indicators,assessment and.action. Congratulations."

Mary Elmendorf, ConsultingAnthropologist

13

DECISION TIME

How long does it take?

It depends what you do. A functioning study mayproduce useful results in only a few days; a detailedevaluation of health impacts may be a majorresearch undertaking and last a year or more. So ingeneral, the MEP will be applied to functioning andutilization only.

How long have you got?

Any evaluation is better than none. Functioning is the easiest of all to studyand, as it usually will result in someactions to improve performance of facilities, it is best to plan each element on an individual basis anyway.Clearly, the time needed depends on the scale of the project under review.

A village scheme may be evaluated for functioning in just a few days. On the other hand, evaluating thefunctioning of a country-wide programme of thousands of handpumps may require substantial time andresources.

Utilization too can sometimes be evaluated in a relatively short time -• particularly with the benefit ofinformation collected during the functioning study. The hardest element to programme is impact evaluation.Again, some aspects are simple and quick. The saving of women's time spent in collecting water, for example,can be assessed easily in a study lasting a week or two.

The rule should be: if you do not have the time and money for a big evaluation: do a smaller one. And do onlywhat you really need to do.

When to start

Information from evaluation studies is especiallyuseful when planning major new investments.Having said that, one can almost always find somenew investments in the planning stage in the watersupply and sanitation sector. Similarly, there willusually be existing systems to be evaluated.

In practice, evaluation is commonly carried out atthe end of a major investment phase, but mid terminformation is just as valuable, so there is no fixedrule.

14

GONE TO

r-nr

W/f

Community Participation (ETS/83.8)A guideline for planning water supply and sanitation projects

which identifies opportunities for popular participationthroughout the project cycle.

Operation and Maintenance (ETS/839)A checklist of items to be considered in rural water supply

development to ensure at the planning stage that operation andmaintenance will work

IRC Publications (available from IRC, PC) Box 93190,2509 AD TheI lague, Netherlands)

Evaluation for Village Water Supply Planning (Technical PaperSeries No 15)

Evaluation for Better PlanningA set of modules for use \i workshops on evaluation.

^ ^ ^

For further information write to:Environmental Health Technology

and SupportDivision of Environmental Health

World Health Organization1211 Geneva 27, Switzerland