M&E System

1

M&E System

Workshop on Indicator Setfor

Rural Water Supply and Sanitation

22nd November 2007

2

Principles 1. Simplicity. The M&E system should be as simple and robust as

possible in the first phase. Once the first phase is successful the system can be made more sophisticated.

2. Improvement rather than replacement. The M&E system should be based as far as possible on improving existing practices rather introducing new ones

3. Flexible and province specific. The M&E system should be flexible, province-centred and adaptable to province specific characteristics. Each province has a different starting point and the transition plan to move from the existing to the proposed system will be different.

4. Cost effectiveness. The M&E system should cost effective and adequately resourced. Provinces can be adopt a sampling approach where appropriate.

3

Principles

5. Accountability. Responsibilities for M&E should be clearly allocated by each province and correspond to mandates of the various decentralised institutions and make use of the best qualified available personnel at local level.

6. Nationally coherent. The centre will ensure coherence by defining the indicators, specifying the minimum provincial reporting outputs, providing provincial support tools for launching and maintaining the M&E system and ensuring timely and coherent national reporting.

7. Independent. Although certain tasks can be outsourced, the database should be under state management and the reporting system should not depend on availability and knowledge of proprietary IT tools

8. Focussed on the end use. Results from the M&E should be made available for planning and decision-making at all relevant levels.

4

Implementation Strategy10 point Province support package

1. Mandatory minimum specifications – indicator report2. Mandatory reporting tools3. Mandatory QA procedures 4. Default data collection system 5. Default templates for data collection6. Default circulars for officialization7. Default Training package8. Budget options and cost norms9. Guidance on sampling strategy10.Default Action plan

5

The A-B-C of theViet Nam RWSS M&E system

B. Program levelData related to the particular programs e.g. NPT IIBased on the MPI Aligned Monitoring Tool.

C. Research level Special studies on sector issues e.g. Value for Money studies; water consumption patterns, gender

A. Sector level Sector Indicators

6

A - Sector monitoring

A – B – C

B –Program Monitoring

C – Research /studies

•NPT II•134 / 135•Donor projects e.g. WB/ ADB

•E.g. •Value for Money Studies•Benchmarking/ best practice•Human Rights/ Gender studies•Functionality•Hand-washing

Sector coverage indicators 1-8

Program outputs & performance indicators 9-14

7

Sector indicators - AArea Sector level Indicators (national, provincial, district, commune)

Water supply 1) Proportion of rural population using safe water (%)

2) Proportion of rural population using clean water satisfying standard in decision No.9 issued by MoH

Environmental sanitation

3) Proportion of schools having safe water and hygienic latrine (%)

4) Proportion of commune health centers having safe water and hygienic latrine(%)

5) Proportion of public works (Market, cultural house, CPC office) having safe water and hygienic latrine (%)

6) Proportion of HH having hygienic latrine(%)

7) Proportion of livestock farming HH having hygienic farms (%)

8) Proportion of trade village having wastewater and waste treatment system (%)

8

Program indicators - B

Area Program level Indicators

Financial progress 9) Proportion of budget implemented compared to budget planned (%)

Physical progress 10) Number of people served by newly constructed or upgraded facilities per year (#)

11) Number of newly constructed latrines per year (#)

Investment efficiency 12) Average investment per capital on piped schemes (Vnd/p)

Sustainability and service efficiency

13) Proportion of water supply schemes operating sustainably (%)

14) Proportion of piped water supply systems operating under recognized management models (%)

9

Examples of C level surveysArea Potential indicators Comment on survey

Access Water & sanitation use Water quality Gender

Observation and interview to understand e.g. in ethnic minority areas. Detailed water quality

Sustainability and Service efficiency

Cost recovery Institutional models

Tariffs,Billing and collection efficiencyExpenditure on maintenance.

Investment efficiency

Equity Poverty targeting Value for money Private sector Improvement in Health

MOLISSA poverty listsSpecialist surveys

Behavior Change

Handwashing Knowledge, Attitude,

Practices (KAP)

Surveys to test handwashing practice and other hygienic behavior.

10

Indicator: #

Variables to be measured: Determination of indicator value:

Notes on survey instrument, sampling and frequency of measurement

Supportive Definitions:

Notes on outstanding issues, options, uncertainties:

Indicator specification

11

Indicator 1 water coverage

Indicator: 1 Proportion of rural population using hygienic water (%)


1.Population using hygienic water from dug well (#)2.Population using hygienic water from tube well (#)3.Population using hygienic water from tap (#)4.Population using hygienic water from other water sources, like spring water, rain water, treated surface water, (#)5.Total population (in commune or sample) (#)

Indicator value = variable( 1+2+3+4)/ variable 5 i.e. The Population using water from one of the safe sources/ total population expressed as a percentage.

12

Indicator 1 continued


Survey instrument is a question on a HH questionnaire which is filled out by a commune appointed official Sampling: there is a choice of a 100% sample or if not a random sample (> 5% unless data indicates a smaller sample) Frequency of measurement: one a year


Safe water has two definitions: Definition i) is that it is water water that satisfies the following quality requirements: No color; No smell; no strange odor; No harmful components that can negatively affect the human health; No harmful bacteria – all by observation. Definition ii) relates to Decision No-09 /2005/QD-BYT issued by MOH .

Safe dug well & Safe tube well & Other safe water sources are defined

13

Indicator 1water coverage


The issues to be decided are: The survey instrument will capture private sector investment as well as public sector

investment The survey instrument is dependent on a correct answer from the householder – it should be

considered to introduce a random verification check to improve the assurance of data quality. Use of HH or individuals as the unit of measurement (better to use HH - more inline with

VLSS methodology) Sample size: A 100% sample would be very expensive. A 5% sample would normally be good

enough but the statistical significance can only be known once the degree of variation is known – thereafter the sample size needed can be calculated – GSO might be able to advise given VLSS experience. VLSS sample size is approx 0.07%

Sample selection : a random sample would be best but difficult to guarantee. Possible sources of bias (e.g. near commune center etc) will need to be identified and avoided.

Safe water quality definition i) is used for this indicator. Definition ii) is used for indicator #2 Should the indicator be “access to” rather than “using” – see definition of the sanitation

indictor (#6)

14

Indicator 2 water quality

Indicator: 2 Proportion of rural population using clean water satisfying standard in decision No.9 issued by MoH

Variables to be measured:

Determination of indicator value:

The 22 variables to be measured are defined in the Decision No.9.

The water either passes or fails. It passes if all 22 variables are below the maximum limit. It fails if any one variable is above the limit


The survey instruments are defined under testing methods in the Decision No.9. Sampling should be 100% but until the budget is available, a random 5% sample size is

suggested.Frequency of measurement (6 months for the variables (parameters) 1 to 12 and

parameter 21 and 22) the rest depending on the circumstances.

15

Indicator 2 water qualitySupportive Definitions:

“Safe Water Standard” under decision No-09 /2005/QD-BYT issued by MOH on 11 Mar. ’05 (applied for water supply construction with less than 500 users), and “Drinking Water Supply standard” No-1329 issued by MOH on 18/4/2002 (applied for water supply construction with more than 500 users) should be applied for water quality control in the rural areas.


Should the indicator by extended to water supplies over 500 users? at present the indicator only covers less than 500 users.

Sample size: A 100% sample would be very expensive – although legally required. For sector information purposes a 5% sample would normally be good enough but the statistical significance can only be known once the degree of variation is known – thereafter the sample size needed can be calculated


A precise water quality testing of non-piped systems is not measured under this indicator set. Should this indicator be just for piped systems?

16

Indicator 3 schools

Indicator 3 Proportion of schools having safe water and hygienic latrine (%)


1. Presence of safe water & hygienic latrines at primary school point (#)

2. Presence of safe water & hygienic latrines at kindergarten point (#)

3. Total number of primary school points (#)4. Total number of kindergarten points (#)

Indicator value = variable( 1+2)/ variable (3+4)

i.e. The number of schools (primary and kindergarten) using water from one of the safe sources and having sufficient latrines/ total number of schools expressed as a percentage

17

Indicator 3 schools Notes on survey instrument, sampling and frequency of measurement

Survey instrument is a question on a school questionnaire which is filled out by a commune appointed official and checked by observation. Sampling: there is a choice of a 100% sample or if not a random sample (> 5% unless data indicates a smaller sample) . Frequency of measurement: one a year


Definitions for safe water are given under indicator 1. Definitions for hygienic latrines are given under indicator 6

Notes on outstanding issues, options, uncertainties: Note: there is a multiple variable involved (both water and sanitation have to be measured at the

same school/ kindergarten point) Should secondary schools be included? What about satellite school points (especially in mountainous areas) Sample size: A 100% sample would be very expensive – although legally required. For sector

information purposes a 5% sample would normally be good enough but the statistical significance can only be known once the degree of variation is known – thereafter the sample size needed can be calculated


Not clear what the acceptable rate of water points / hygienic latrines per pupil. A suggestion is sufficient water for at least 5l/pupil/day and not less more 40 pupils per hygienic latrine

18

Indicator 4 health centers Indicator 4 Proportion of commune health centers having safe water and

hygienic latrine (%)


1. Presence of safe water & hygienic latrines at commune health centers (#)

2. Total number of commune health centers (#)

Indicator value = variable 1/ variable 2 i.e. The number of commune health centers using water from one of the safe sources and having sufficient latrines/ total number of health communes expressed as a percentage


Survey instrument is a question on a health center questionnaire which is filled out by a commune appointed official and checked by observation. An alternative could be the commune level health authorities to directly provide the information. Sampling: 100% i.e. all commune health centers Frequency of measurement: one a year


Definitions for safe water/ hygiene latrines (as indicator 1/6).

Notes on outstanding issues, options, uncertainties: multiple variable involved (both water and sanitation) Not clear what the acceptable rate of water points / hygienic latrines per commune health

center. A suggestion is sufficient water for at least 5l/visitor/day and not less 2 hygienic latrines

19

Indicator 5 – public worksIndicator 5 Proportion of public works (Market,

cultural house, CPC office) having safe water and hygienic latrine (%)


1. Presence of safe water & hygienic latrines at commune markets (#)

2. Presence of safe water & hygienic latrines at commune culture house (#)

3. Presence of safe water & hygienic latrines at commune CPC office (#)

4. Total number of commune markets (#)

5. Total number of commune cultural houses (#)

6. Total number of commune CPC offices (#)

Indicator value = variable (1+2+3)/ variable (4+5+6)

i.e. The number of public facilities using water from one of the safe sources and having sufficient latrines/ total number of public facilities expressed as a percentage

20

Indicator 5 – public works


Survey instrument is a question on a commune questionnaire which is filled out by a commune appointed official and checked by observation. An alternative could be the commune level health authorities to directly provide the information.

Sampling: 100% i.e. all commune public buildings of the 3 types specified Frequency of measurement: one a year


Definitions for safe water are given under indicator 1. Definitions for hygienic latrines are given under indicator 6


Note: there is a multiple variable involved (both water and sanitation have to be measured at the same health point)

Not clear what the acceptable rate of water points / hygienic latrines per commune public facility. A suggestion is sufficient water for at least 5l/visitor/day and not less 2 hygienic latrines

21

Indicator 6 - latrinesIndicator 6 Proportion of HH having hygienic latrine(%)


1. HHs having a hygienic double vault compositing latrine (#)

2. HHs having a hygienic VIP latrine (#)3. HHs having a hygienic pour flush water seal

latrine(#)4. HHs having a hygienic septic tank latrine (#)5. Total number of HHs (#)

Indicator value = variable( 1+2+3+4)/ variable 5

i.e. The number of HHs having a hygienic latrine / total number of HHs expressed as a percentage


Survey instrument is a question on a HH questionnaire which is filled out by a commune appointed official – because of the technical nature of definition, a confirmatory visit should be made.

Sampling: there is a choice of a 100% sample or if not a random sample (> 5% unless data indicates a smaller sample)

Frequency of measurement: once a year

22

Indicator 6 - latrinesNotes on survey instrument, sampling and frequency of measurement

Survey instrument is a question on a HH questionnaire which is filled out by a commune appointed official – because of the technical nature of definition, a confirmatory visit should be made.Sampling: there is a choice of a 100% sample or if not a random sample (> 5% unless data indicates a smaller sample) Frequency of measurement: once a year


The definition of hygiene latrines is given in Decision No.08/2005/QD-BYT of 11/03 2005


Captures private sector investment as well as public sector investment Needs correct answer from HH / interviewer– training and random verification check Sample size: A 100% sample would be very expensive. A 5% sample would normally be

good enough but the statistical significance can only be known once the degree of variation is known – thereafter the sample size needed can be calculated – GSO might be able to advise given VLSS experience. VLSS sample size is approx 0.07%


23

Indicator 7 - livestockIndicator

7Proportion of livestock farming HH having hygienic farms (%)


1. Number of hygienic pig livestock pens2. Number of hygienic cattle livestock pens3. Total number of pig & cattle livestock pens

Indicator value = variable( 1+2)/ variable 3i.e. The number of hygienic livestock pens / total number of livestock pens, expressed as %


Survey instrument is a question on a HH questionnaire which is filled out by a commune officialSampling: there is a choice of a 100% sample or if not a random sample Frequency of measurement: one a year


“Hygienic livestock pens” – temporarily apply 2 criteria to classify whether the pen is hygienic or not:- At least 10m away from house- The animal excreta volume requires good management to avoid environmental pollution (pens are daily cleaned, excreta volume are covered in storage holes)


Survey instrument largely depend on the proper response from households – need to consider random checking to ensure the data quality.

24

Indicator 8 trade villagesIndicator 8 Proportion of trade village having waste water and waste treatment

system (%)


1. Number of villages with solid waste & waste water treatment system

2. Total number of investigated villages

Indicator value = variable( 1)/ variable 2expressed as a percentage


Survey instrument is a question on a questionnaire on trading village which is filled out by a commune appointed official. Sampling: 100% sampling. Frequency of measurement: once a year


“Trading village” has standard definition. The hard definition if adequate waste water waste treatment will be that the discharge standards satisfy the environmental regulations – this requires a sample test or verification that an acceptable technology is in place and adequately maintained and operated. As a softer definition the technical opinion of the commune official will be accepted.


- Either solid waste or waste water is discharged at some certain villages- Available waste water or solid waste system but treatment effectiveness is beyond the requirement.??- commune officials will need training on what is meant by adequate waste treatment

25

Indicator 9 - budgetIndicator 9 Proportion of budget implemented compared to budget planned (%)


1. Annual budget planned2. Annual expenditure

Indicator value = variable( 2)/ variable 1 expressed as a percentage


Survey instrument is simply a recoding of the values in the annual RWSS plan and expenditure records. The AMT should be able to provide this data.Sampling: 100% i.e. all communes and districts and provincesFrequency of measurement: once a year, note expenditure will normally take at least 3 to 6 months after the end of the financial year before it is fully ready.


The annual plan is the first PPC approved NTP-RWSS budget and work plan. The expenditure is the audited statements of NTP-RWSS expenditure against plan.

Notes on outstanding issues, options, uncertainties: Does this indicator refer to only capital or also recurrent? if recurrent then it presumably does

not include wages which would be very difficult to allocate. Expenditure of DOH, DARD and the various agencies involved will be needed to be captured

– this will not be easy especially for the recurrent budget.

26

Indicator 10 – people servedIndicator 10 Number of people served by newly constructed or upgraded facilities per year (#)


1. Design capacity in terms of people served for each new scheme

2. Design capacity in terms of people served for major extensions

3. Calculation of ration of actual operational / capacity level /%) for each scheme (phase 2)

Indicator value = variable 1 + variable 2


Survey instrument design capacity from the final design documents Phase 2:actual use . Sampling: 100% i.e. all communes and districts and provinces Frequency of measurement: once a year


Design capacity in terms of people served : from daily design capacity divided by the assumed daily usage.(need to check the normal design presentations to determine the best way of defining this figure)Calculation of operational level : those who actually receive their main water supply from the newly constructed works

Notes on outstanding issues, options, uncertainties: Distinguish between design capacity and actual service Nos achieved Each year the actual capacity

increase will change as more people connect to a network. Extensions of existing schemes will often need to be estimated and there is a danger of double

counting if the extension is done over several years.

27

Indicator 11 latrines constructedIndicator 11 Number of newly constructed latrines per year (#)


No of latrines constructed through subsidiesNo. of latrines constructed without subsidies

(phase 2)

Indicator value = variable 1


Survey instrument expenditure records of the number of latrines that were subsidized. A cross check of the controls (visits) undertaken should be made on a spot samplePhase 2: No. of latrines constructed without subsidies should be obtained by calculation / estimate using the difference between variable 1 and changes in year to year in sector indicator # 6 . Sampling: 100% i.e. all communes and districts and provinces. Frequency of measurement: once a year


See sector indicator # 6


It is important in phase 2 to also capture the latrines built not using subsidies.

28

Indicator 12 unit costs Indicator 12 Average investment per capital on piped schemes (Vnd/p)


1. Total annual investment costs for piped schemes2. Number of served people from piped schemes

(invested in the year)

Indicator value = variable 1/variable 2Expressed in VND/p


Survey instrument : recoding of of investment costs for constructed piped schemes and number of served people from those schemes. The AMT should be able to provide this data. Sampling: 100% i.e. all communes and districts and provinces. Frequency of measurement: once a year.


Piped scheme is a water supply system for over 50 households?


- Construction cost price (project) varies due to the market price of materials, equipments, salaries.

- Besides water schemes under NTP, there are also schemes funded by other programs/projects.- How to calculate for rehabilitated, expanded schemes (almost like 2nd investment)?

29

Indicator 13 sustainabilityIndicator 13 Proportion of water supply schemes operating sustainably (%)


1. Number of schemes operating sustainably.

2. Total number of current schemes

Indicator value = variable 1/variable 2Expressed in percentage


Survey instrument is a statistic table to be done by a commune staff based on technical visits. Sampling: 100% i.e. all communes and districts and provinces. Frequency of measurement: once a year.


Water supply systems operating sustainably must meet the following criteria: proper model (this criteria is already measured in the next indicator) , properly utilized capacity (≥ 70%), sufficient collection of water tariff, healthy financial mechanism.


- Should the water quality be on criterion in the definition or not?- What if only 3 out of 4 criteria (on criterion is little poor) are satisfied?- Commune staff will need very careful training by pCERWASS

30

Indicator 14 Indicator 14 Proportion of piped water supply systems operating under

recognized management models (%)


1. Number of piped schemes with proper management model

2. Total number of current schemes

Indicator value = variable 1/variable 2Expressed in percentage


Survey instrument is a statistic table to be done by a commune staff. Sampling: 100% i.e. all communes and districts and provinces Frequency of measurement: once a year.


Piped water schemes under recognized management models include 4 types, namely:- Community management model: - pCERWASS management - - Private management

model: - Water supply service business model:


In case new model(s) with effective operation but not recognized, then what ?

Documents

M&E System