Service Delivery Indicators for Strengthening Local Monitoring of Rural Water Service Delivery in Uganda

PoIntS foR actIon

• MinistryofWaterandEnvironment(MWE)toadoptalimitednumberoftheproposedServiceDeliveryIndicators(SDIs)thatfocusontheactuallevelofservicedeliveredtosupplementtheGoldenIndicators.

• MWEshouldtrackadherenceofwaterservicestonationalnormsandstandards

• MWEandrelevantpartnerscarryoutanational-levelsurveyandusethereportsasinputfortheJointTechnicalReviewprocesstoprovideanopportunityforallsectorstakeholderstoassesstheusefulnessoftheSDIs.

• Organiseawarenessraisingandtrainingactivitiesforsectorprofessionalsrepresentingthewholerangeofpotentialusers:MWE,TSUs,DWOs,DonorsandNGOs

This paper presents a framework of service delivery indicators (SDIs) for monitoring rural water services in Uganda. This has been developed for use by the sector to broaden the scope of the existing national monitoring system, the golden indicators, beyond tracking performance of systems to actual services delivered. The SDIs were developed based on sector norms, standards and guidelines set by the Ministry of Water and Environment. This paper describes the process of developing the indicators, and shows how they complement the golden indicators, tracking critical service parameters that were not captured so far, such as reliability, users’ satisfaction or performance of districts. The additional information from SDIs for instance highlights that a district may have a functionality rate of 82% and yet users are only assured of getting water at any time from 70% of the systems.

BRIEfInG PaPER November 2014

Background and rationale for SDIsTheperformanceofUganda’sWater,SanitationandHygiene(WASH)sectorismonitoredthroughtrackingof11“golden”indicators.DataontheseindicatorsiscollectedbyDistrictLocalGovernments(DLGs)staff,undertheguidanceoftheMinistryofWaterandEnvironment(MWE).TheMWEcompilesandanalysesthisinformationintheannualWaterandEnvironmentSectorPerformanceReportwhichispresentedanddiscussedamongsectorstakeholdersduringyearlyJointSectorReviews(SsoziandKerstinDanert,2012).Althoughveryhelpfulfornationalstakeholderstotrackgeneralprogressinthesector,thegoldenindicatorshaveanumberoflimitations:

lBydefinition,theyonlycaptureasmallpartofallissuesrelatedtoruralwater.Outofthe11goldenindicators,7arerelatedtoruralwater,focusingonaccesstowater,percapitainvestments,waterquality,andfunctioningoftheserviceprovideri.Theydon’tincludeinformationonaspectsthattheliteratureonruralwatersupplyidentifiesasbeingimportant,suchasthelikelihoodofsustainabilityofwaterfacilities,servicelevels,users’satisfaction,ortechnicalbackstoppingprovidedtowaterserviceproviders.

lBecauseofthelimitednumberofparameters,theydonotallowmakingcorrelationsthatcouldinformthereasonsbehindtheprogressandtrendsthatthegoldenindicatorstrack.

lTheyhavebeendesignedforusebynationalpolicy-makersanddecision-takers,anddonotprovidesufficientinformationtoallowlocalactorssuchasstaffsofDistrictWaterOffices(DWOs)ofTechnicalSupportUnits(TSUs)iitotakeinformeddecisionsanddirectremedialactionsatlocallevel.

InviewoftheshortcomingsoftheGoldenIndicators,IRC/Triple-SinUgandatogetherwiththeMWEdevelopedServiceDeliveryIndicators(SDIs),withtheaimofsupplementingthealreadywell-establishednationalmonitoringframework.TheSDIsshouldprovideadeeperunderstandingoftheresultsonthegoldenindicators,andthereforeallowenhancedmonitoringofruralwaterservicesdelivery,sothatactioncanbetakenatlocalandnationallevel.

Forthepurposeofthedevelopmentofthisframework,aSteeringCommitteewasestablishedwithmembershipfromgovernmentalandnon-governmentalbodies.ThetechnicaltoolsandplatformaredesignedwithsupportfromMakerereUniversityinKampala.Thispaperpresentsthe

proposedframeworkformonitoringusingtheSDIs,whichencompassesmorethanthemeredefinitionofmonitoringindicators.Itentailsalsothedefinitionoftheprinciplesforthemonitoring,thestructureandcontentoftheSDIsthemselvesandadetailedmethodologyfordatacollection,guidelinesandformatsforanalysisandpresentationofresultsfromthecollecteddata.

Key principles and approach for the development of SDIs in UgandaProcess,scopeandkeyconsiderationsforthedevelopmentofSDIsTheSDIsweredevelopedbycomparingtheGoldenIndicatorstothevariousruralwatersubsectorpoliciesandguidelinesincludingservicedeliverynorms,standardsandprescribedpractices,androlesandresponsibilitiesofstakeholders.Inthisway,weidentifiedgapsnotcoveredbytheGoldenIndicatorsthattheSDIscouldfill.Forinstance,withrespecttoservicelevels,thegoldenindicatorscaptureinformationonwaterquality,accesstoanimprovedfacilityanditsfunctionality.SDIscouldsupplementthat,withdataondistanceofthefacility,quantityofwateraccessed,reliabilityofthefacilityandqualityofthewatersource.

Unit of analysis and scope of applicationThepurposeoftheSDIsistoprovideaquickunderstandingofthesituationoftheruralwatersupplysector,whichaffects–positivelyornegatively–thedeliveryofsustainablewaterservices.Forthis,SDIscanprovidethestatusofservicesatacertainmomentintimeand,whendatacollectionisrepeated,measurechangesovertime.However,theSDIsare

Example of complementarity of SDIs and golden indicators on service delivered

TheMWEsectorperformancereport2013showsthatfunctionalityforruralsystemsinKabaroleandLiradistrictsreach82%and74%respectively.However,preliminaryanalysisoftheSDIsshowsthat,ifwetakeintoaccountreli-abilityofthefacilities,only70%ofruralwatersystemsinKabaroleandLiraarefunctioningandreliableiii.Sincefunctionalityisderivedfromcalculatingthenumberofwaterpointsdeliver-ingwateratthetimeofspotcheck,anindica-toronreliabilityhelpsaddingatimedimen-sion,showingthatactuallyfortwothirdofthefacilitiesusersareassuredtofindwaterattheirwaterpoint.

notmeanttobeusedfortrackingtheperformanceofeverysinglewaterfacility,servicemanagerorserviceauthorityiv.Theyratherprovideinformationofthemaintrendsinabroadergeographicalarea,likeadistrictorsub-county.Theunitofanalysisisthereforethesub-countyordistrict.ThisalsoimpliesthatdataontheSDIsiscollectedonasampleofwatersystems.TheSDIsarealsodesignedinsuchawaythattheyareapplicabletothetwomainruralservicedeliverymodelsfoundinUganda,soastofacilitateaggregation.

GiventhatthemainpersonswhowouldusetheseSDIarestaffofwaterserviceauthorities,datacollectionandanalysistoolshavebeenkeptsimpleandquicktoapply.Forinstance,thedatacollectionprotocolisdesignedinsuchawaythatenumeratorsdonothavetogotohouseholdstoconductinterviews,butratheraskquestionstouserswhocomeandcollectwater.Datacollectionquestionsarekeptassimpleaspossible,oftensimplyrequiringayes/noansweroraresponsethatcanbeselectedfromalistofpre-setoptions.

Table 1. Parameters monitored by Golden Indicators and SDIs for rural water services in UgandaLevel Golden Indicators (GIs) Service Delivery Indicators (incl. sub-indicators feeding into these indicators; (sub)

indicators specific to WSSBs)

Service deliv-ered

Water quality (E. Coli) Water quality (E. Coli; TDS; Turbidity)

Not included in GIs Water quantity (Water quantity delivered; Water quantity accessed)

Access (distance to improved water facility)

Accessibility (Number of users; Distance to water facility; Walking time)

Functionality water facility Reliability (Uptime water facility)

Not included in GIs Quality of water source (Non exposure to pollution; No seasonal variations water qual-ity; No seasonal variations of yield)

Users’ level Not included in GIs Users’ satisfaction with the service delivered (Users’ satisfaction with water quality; Users’ satisfaction with water quantity; Users’ satisfaction with accessibility; Users’ sat-isfaction with reliability)

Not included in GIs Users’ sense of ownership of water facility (Users’ financial contribution to O&M; Cleanliness water facility surroundings)

Service manage-ment level

Management (actively functioning WSC/WSSB)Gender (women in key positions in the WSC/WSSB)

Service manager and operator composition and activeness (Activeness service man-ager; Gender; Viability of scheme operator)

Not included in GIs(although some aspects may be cap-tured under the WSC/WSSB being active)

Service manager’s and operator’s performance of tasks (Financial records; Collec-tion user fees; Cost recovery; Meetings with users; Preventive maintenance; Scheme operator’s reports; Meetings with tap committees; Relationships with service authority and support)

Not included in GIs Service manager and operator internal governance (Records on decision points; Transparency on O&M fund; Transfer water fees to WSSB)

Service author-ity and support mechanisms

Not included in GIs District Water Office (DWO) staffing (No staff DWO)

Equity (Deviation from the District average no. of persons per water point)

District planning (Planned investment based on equity)

Per Capita Investment Cost

Not included in GIs Community mobilisation pre-construction (Signed Memorandum of Understanding; Community capital cash contribution; Land agreement)

Not included in GIs Support and supervision to service managers by service authority and Handpump Mechanics / HPMs (Reactivation of service managers; Responsiveness DWO to major breakdowns; Responsiveness HPMs to breakdowns)

Not included in GIs Support and supervision to service managers by Umbrella Organisation (Support visits; Responsiveness to major breakdowns)

Not included in GIs Construction supervision (Quality construction supervision)

Not included in GIs Monitoring (Use of water facilities’ functionality data; Monitoring of service managers’ activities)

Not included in GIs District coordination (Functionality of District Water and Sanitation Coordinating Com-mittee)

Not included in GIs Responsiveness of TSU (Responsiveness of TSU to request for support from DWO)

Structure and content of the Service Delivery Indicators in UgandaSDIsdescribethewayruralwatersupplyservicesaredeliveredandsupportedacrossthefollowinglevels:

1.Userslevel:Thisreferstotheirsatisfactionwiththeserviceandtheirsenseofownershipofthefacility,measuredthroughproxyindicatorssuchasfinancialcontributiontoO&Mandcleanfacilitysurroundings;

2.Servicedelivered:Coveringwidelyrecognisedserviceslevelparameters–waterquality,quantity,distanceandreliabilityofthefacility(KayseretAl.,2013)–aswellasqualityofthewatersource(i.e.nonexposuretopollutionandseasonalvariations);

3.Servicemanagementlevel:Referringtothecompositionandactivenessofservicemanagers,theirperformanceintheirtasksandresponsibilities,aswellastheirinternalgovernance;

4.Serviceauthorityandsupportmechanismslevel:Focusingonperformanceofserviceauthorityfunctionsbeforeconstruction(planningphase,communitymobilisation),duringconstruction(supervision)andafterconstruction(supportandsupervisiontoservicemanagers,monitoring),aswellasgeneralconditionsfortheperformanceofthesefunctions(DWOstaffing,districtcoordination).

ThedetailedcontentoftheSDIsispresentedinTable1indicatingalsohowtheyexpandontheGoldenIndicators.

Table1highlightshowSDIscanprovideadditionalinformationtothegoldenindicators,differingmainlyinthefollowing:lTheSDIscontainthecompletesetofgenerallyacceptedservicelevelparameters,ascomparedtothethreecoveredbytheGIs

lTheSDIscontaininformationaboutusers’satisfaction,anareacompletelynotcoveredbyGIs

lArguablythebiggestdifferenceliesintheservicemanagementandsupportlevel,wheretheSDIscontainmoredetailedinformation,whereastheseareweaklycoveredbytheGIs.

framework for SDIs data collection and analysis

Sampling StrategyKeepingthecostsoftheapplicationoftheSDIslow

isakeyconsiderationwhendesigningthemonitoringframework.Thisimportantaspectwascentralinthedesignofthesamplingstrategythatworksasfollows:

lServiceauthoritylevel:allserviceauthoritiesaresampled

lServicemanagementandservicelevel:thesearecollectedforwaterpointsbutnotforallinageographicarea.Thedesignedsamplingstrategyallowsasmallsamplesizewhilestillensuringgoodaccuracyoftheresults.Basedonanumberofassumptions,i.e.thatthesub-countywouldbethelowestgeographicalunitforanalysisoftheresults,andthatalowerprecisionisacceptableaslongasaccuracyvishigh,aminimumsamplesizeissetat11waterpoints.Thismeansthatforeachdistrict,datahastobecollectedfromallsub-counties,with11pointsourcesand(whenavailable)at11tapsvisitedineachsub-county.

lUsers.Foreachwaterpoint,10userswhocometofetchwaterareinterviewed;inordertoallowrepresentativenessoftheusers,interviewsareconductedatdifferentmomentsoftheday,andevery5thvisitorofthefacilityisinterviewed.

Thisapproachhastheadvantagesofbeingastandardstatisticallybasedsampling,thatreducesthecostsoffullsurveys,withveryreliableestimatesatthedistrictlevel(within10%pointsconfidenceinterval)andwithin5%pointsconfidenceintervalatthenationallevel.Themaindisadvantagesofsuchasmallsamplesizeisthatthesamplingstrategyneedstobestrictlyadheredto,toensureaccuracyoftheresults,andthatserviceauthoritiescannottakeactionsonspecificwaterpointsthatfalloutsidethesample.

Data collection methodology and toolsTheinformationrequiredfortrackingthesub-indicatorsandindicatorscomesfromvarioussources,sothatthebestinformedindividual/groupisinterviewedforeachspecificissue:

lHouseholds(representedbythepersoncollectingwateratthefacility)forsomeoftheindicatorsrelatedtotheservicedelivered,users’satisfaction,andperformanceoftheservicemanager

lWSC/WSSB(representedbyatleasttwomembersofthecommittee/board)fortechnicalinformationonthewaterpoint,someoftheindicatorsrelatedtotheservicedelivered,users’paymentfortheservice,performanceofsomeservicemanager’stasks,supportprovidedbythe

serviceauthority

lStaffsofsub-countyandDWOforperformanceofsomeoftheserviceauthoritytasks

lUmbrellaOrganisationStaffformattersrelatedtopost-constructionsupporttoWSSBs

Standardmobilephones(non-smartphones)areusedtorecordthedatacollected.Apartforthequestionsdirectedtotheserviceauthorityandsupportinstitutions(DWO,sub-countyandumbrellaorganisation),theentrypointfordatacollectionisthewaterpoint.Mostofthedata,particularlyatthewaterfacilitiesaswellasinterviewswithWSCs/WSSBsandwaterusersiscollectedbylocalHandpumpmechanics(HPMs)whoarefamiliarwiththeareasandlocationofwaterpoints.

Inaddition,mostofthemhavealreadybeeninvolvedincollectingdatafortheDWOtofeedintothegoldenindicatorsandalreadyareconversantwithdatacollectionusingmobilephones.TheHPMsparticipatedinatwodaytrainingonthedatacollectionprotocolandonhowtousemobilephonestocollectdata.Forthefirstroundofdatacollection,whichtookplaceinDecember2013,theinformationfromstaffsoftheDWOs,sub-countiesandumbrellaorganisationswasgatheredbyTriple-Sstaff.

Thedatacollectedisthendirectlystoredonanonlineplatformthatisaccessibletofieldsupervisorstomonitorinrealtimeincomingdata.Theplatformisnowaccessibletoallrelevantstakeholders.Dataprocessing:calculationandpossibleaggregationoftheSDIsvalues

Eachsub-indicatoristrackedthroughoneorseveralmeasurableparameters(alsoseeAdanketAl.,2013).Foreachwaterpoint,avalueisallocatedtoeachsub-indicator:l“0”whentheresponseisnegative,orunderaset

benchmark,or

l“1”whentheresponseispositive,orreachingthebenchmark.

Thetablebelowshowsexamplesoflinksbetweensub-indicatorsandparametersandhowscoringisdone.

Valuesforeachsub-indicatorarecalculatedatthelowestgeographicalunitforanalysis,i.e.thesub-countyfordataonusers,servicedeliveredandservicemanagementandthedistrictfortheonesonsupportauthority.Itisobtainedbycalculatingtheaverageofthescoresforthe11waterpoints(pointsourcesortaps),convertedintoapercentage.Forinstance,ifforthesub-indicator“Walkingtime”,7waterpointsscoreda“1”and4a“0”,thevalueofthesub-indicatoris64%.Eachsub-countyhenceobtainsavalueforeachsub-indicatorandcanbecomparedtoothersub-counties.

Thevalueofagivenindicatorissimplyobtainedbycalculatingtheaverageofthevaluesofthecorrespondingsubindicators.Furtheraggregationcanbedonetoobtainavalueforeachindividuallevelofservicedelivery(servicedelivered,users’satisfactionandsenseofownership,servicemanagers,andserviceauthorityandsupportmechanisms).Forinstance,thevalueforservicedeliveredistheaverageofthevaluesforthe5individualindicators(Waterquality,Waterquantity,Accessibility,ReliabilityandQualityofwatersource).Again,thevalueobtainedinasub-countyoradistrictcanbecomparedwiththeonesobtainedforotherareas.Resultscanbepresentedforpointsources,forpipedschemes,orforallsourcestogether.Aggregationscanalsobedoneforvariousgeographicallevels:sub-county,district,ornational.

Data analysisAnalysisofdatacollectedinDecember2013shallbefinalisedinApril2014.Resultsforeachsub-indicator,eachindicatorandeachlevelofservicedeliverywillbeobtainedthroughdownloadingofshort(1-2pages)automaticallygeneratedPDFreportsthatpresentkey

Table 2. Sample of the water Service Delivered Indicators & scoring system at facility level

Indicator Sub Indicator Measurable parameter

Score

Accessibility No of users of water facility

No of regular users registered by the committee

No. users ≤ norm = 1No. users > norm = 0(norm is 300 users for a borehole, 200 for a shallow well or a protected spring, and 150 for a tap)

Distance to water facility

Distance between household and water facility for 10 users

At least 80% of users are within 1 km = 1Less than 80% of users are within 1 km = 0

Walking time Time spent (minutes) by 10 users for a round trip to the facility

At least 80% of users do a round trip within 30 min = 1Less than 80% of users do a round trip within 30 min = 0

graphsandtablesandareaccessibleonline.Oncethesereportsareaccessible,staffsofDWOsandsub-countieswillinterpretthemwithsupportfromTriple-SandtheMWE.ThejointinterpretationofwhattheSDIsmeanshallleadserviceauthoritiestothedesignofremedialactionsbasedontheidentificationofgapsandissuesinservicedelivery.

Way forward and conclusionThispaperhaspresentedanadditionalframeworkformonitoringruralwatersuppliesthatcomplementsthewell-knownGoldenIndicators.ThetestingoftheSDIshasshownthattheyhavepotentialtogenerateinformationonactualservicelevels,reliabilityoftheservice,usersatisfactionandperformanceofserviceauthoritiesthatisnotcapturedbythegoldenindicators.AsaresulttheSDIsprovideopportunityfor

identifyingissuesandgapsacrossthe‘entireservicedeliverychain’thathavebeenoverlooked.

Thepapershowsevidenceonhowtrackingreliabilityofruralwatersystemstellsthewaterusers’storyhiddenbehindfunctionalityfigures.Despitetheadvancementintheprocessofdevelopingtheindicators,thereisstillcontentionamongsectoractorsthatparameterstrackedbySDIsareverymanyandthatthesystemmaynotbeeasytoreplicatebygovernmentgivenitsavailableresources.Theongoingprocessofanalysisofdatafromthefirstroundofdatacollectionwillbeusedtofurtherprioritisetheindicatorstoselectthosethatproducethemostrelevantandactionabledata.

References

1. Adank,M.,Smits,S.,Bey,V.,Verhoeven,J.andPezon,C.(2013)Developmentanduseofservicedeliveryindicatorsformonitoringruralwaterservices.IRCSymposium2013:MonitoringSustainableWASHServiceDelivery(http://www.irc.nl/page/79345).IRC:TheHague.

2. Kayser,G.L.,Moriarty,P.,Fonseca,C.andBartram,J(2013),DomesticWaterServiceDeliveryIndicatorsandFrameworksforMonitor-ing,Evaluation,PolicyandPlanning:AReview.InternationalJournalofEnvironmentalResearchandPublicHealth,2013,Vol.10,pp.4812-4835.

3. Ssozi,DisanandKerstinDanert(2012),NationalMonitoringofRuralWaterSupplies.HowtheGovernmentofUgandadiditandlessonsforothercountries.FieldNoteNo2012-4,RuralWaterSupplyNetwork.

About IRC

IRC is an international think-and-do tank that works with govern-ments, NGOs, entrepreneurs and people around the world to find long-term solutions to the global crisis in water, sanitation and hygiene services. At the heart of its mission is the aim to move from short-term interventions to sustainable water, sanitation and hygiene services.

With over 45 years of experience, IRC runs projects in more than 25 countries and large-scale programmes in seven focus coun-tries in Africa, Asia and Latin America. It is supported by a team of over 80 staff across the world.

For more information please visit www.ircwash.org

Visiting addressAqua Complex, 1st FloorPlot 2220 Port Bell Road, Kitintale

Postal addressP.O Box 40398Kampala, Uganda

tEL: +256 758 200782uganda@ircwash.orgwww.ircwash.org/uganda

End Notes

i. There are two main service delivery models for rural water in Uganda: point sources (boreholes, shallow wells and protected springs) and piped schemes. The service manager (also called service provider) is the body that manages and delivers the water service to a defined population in a defined service area, taking care of operation, maintenance and administration (book keeping, tariff collection, etc.) of the system. For rural water services in Uganda, the service manag-ers usually are community-based Water Source Committees (WSCs) for point sources and Water Supply and Sanitation Boards (WSSB) for piped schemes.

ii. TSUs are regional units established by the MWE to build capacity and offer back-up support to district local governments in their fulfil-ment of roles and responsibilities as regards to WASH services.

iii. A water point is considered as reliable when it provides water 95%

of the time or only breaks down for a maximum period of 2 weeks in the whole year.

iv. The service authority is the body with legal responsibility for guaranteeing a water service in a defined area, fulfilling functions such as planning, coordination, oversight of services. A service author-ity usually is responsible for technical assistance to service managers although this can be contracted out. In Uganda, service authority functions are split between two administrative levels: districts and sub-counties.

v. Accuracy is the trueness of the result, the proximity of measure-ment results to the true value; this should not be confused with preci-sion, which consist of the repeatability or reproducibility of the meas-urement. Accuracy is determined by the sampling strategy which sets how representative the sample is, while precision is largely determined by the sample size and to some degree to the sampling strategy.