1

Lao Poverty Reduction Fund II Impact Evaluation Pre-Analysis Plan Sept 2015

I. Introduction

1. This pre-analysis plan outlines the methods to be used and specifications tobe employed to test a set of hypotheses concerning the impact of the Lao PRF Phase II program over the period 2012-2015 on access to health and education services, clean water, all-weather roads, and perception of and satisfaction with the above services and local development planning. The stated Project Development Objective for PRFII (Additional Financing) is to improve the access to and the utilization of basic infrastructure and services for the Project's targeted poor communities. This PDO would be achieved through inclusive community and local development processes with an emphasis on ensuring sustainability.

The set of hypotheses considered in this evaluation are directly relevant to the indicators established in the Results Framework of the PRF II Project Document and reflect what the program will be evaluated on. Additional hypotheses pertaining to impacts on other indicators (chiefly household welfare and social capital/governance), which are not included in the Results Framework will be considered and enumerated at a later date in an annex to this plan. However, this additional set of hypotheses and accompanying indicators should not be viewed as impacts PRF II will be held accountable for.

II. PRF II Implementation During the Period of Evaluation (2012-2015)

PRF II Treatment Activities

The primary intervention component to be evaluated by the IE is the use of kumbanh-based block grants for community development planning and subsequent construction of infrastructure sub-projects. The kumbanh planning process is undertaken on a four year rolling basis. Villagers prepare development plans that are integrated at the kumbanh level through an inclusive process led by elected village representatives. Kumbanh facilitators assist communities to develop plans and also monitor progress. Sub-projects are then implemented at the village level. Village meetings with facilitators were held during the first cycle of implementation, in cycles when subprojects were implemented (generally only 1 per village over the period of evaluation) and occasionally in non-subproject implementation cycles.

Each targeted kumbanh receives upfront a four-year budget to inform its planning and prioritization. An average annual budget allocation of US$35,000 is provided to PRF II target kumbanhs, for a total average investment amount per kumbanh of US$140,000 over four years with sub-projects financed and implemented on an annual basis. Sub-projects are selected for financing at the kumbanh level by the PRF kumbanh committee (consisting of elected villagers including women and,

103247P

ublic

Dis

clos

ure

Aut

horiz

edP

ublic

Dis

clos

ure

Aut

horiz

edP

ublic

Dis

clos

ure

Aut

horiz

edP

ublic

Dis

clos

ure

Aut

horiz

ed

2

where feasible, members of all ethnic groups present in the village). PRF district staff, district local government and sector officials provide technical validation of proposals. The final decision for sub-project financing is made at the kumbanh level by the PRF kumbanh committee based on transparent criteria and process as specified in the Project Operations Manual including:

75% of sub-projects must benefit directly the poorest communities within the kumbanh.

Sub-projects must meet appropriate technical standards for infrastructure agreed upon with relevant sector ministries.

It should be noted that not all villages in a given kumbanh necessarily receive subprojects. Second, these subprojects may vary in type, e.g. water, roads, schools etc. This variation presents challenges to the impact measurements. PRF II was implemented during Cycles IX-XII of the overall PRF program and corresponding to the period November, 2012 to August 2015. The table below details the extent of sub-project construction during this period: Table 1: Number of sub-projects constructed by type

Sub-projects Total Kumbanh

Roads and Bridge 18 12

Schools 16 12

Water 15 12

Health 2 2

Kindergarten 3 3

Table 2 below details when meeting were held with PRF facilitators over the period of evaluation: Table 2: PRF facilitated meetings held by Cycle

Planning Stage (Cycle IX Attapeu, Cycle X others)

Subproject Construction

No Subproject Construction during the cycle

Attapeu Yes Yes 1 out of 3 cycles Luang Prabang Yes Yes 1 out of 2 cycles Oudomxai Yes Yes 1 out of 2 cycles Phongsaly Yes Yes 1 out of 2 cycles

3

III. Hypotheses Overview The stated Project Development Objective for PRFII (Additional Financing) is to improve the access to and the utilization of basic infrastructure and services for the Project's targeted poor communities. This PDO would be achieved through inclusive community and local development processes with an emphasis on ensuring sustainability The IE is designed to address the following set of Project Development Objective level indicators from the Results Framework:

1. Greater than 75% satisfaction levels reported by beneficiaries in targeted villages regarding improved services and local development planning

2. Improved access to and utilization of basic economic and social services in kumbanhs supported by PRF

• % increase in access to and utilization of health services • % HHs with improved access to and utilization of safe water resources • % increase of HHs with access to all weather roads

3. Lowest two quintiles benefit from above services The analysis will be structured to test a set of hypotheses to provide estimates of impact for indicators directly relevant to the Results Framework. In addition, the pre-analysis plan proposes a hypothesis for school sub-projects (to be discussed below) due to the face that approximately a third of investments are for schools or kindergartens and roads/bridges. Table 5: Hypotheses and indicators (indicators requiring new questions for the endline survey instrument are in red)

Hypothesis Indicators dir so secq lvl fo I. PRF II increases access to protected water sources

i. access to protected water source in the dry season ii. access to protected water source in the rainy season iii. distance to reach water source iv. time to reach water source

+ + _ -

HH HH HH HH

WS-8 WS-10 WS-9 WS-N

HH HH HH HH

B B R R

II. PRF II increases accessibility

i. months of the year access to road

ii. villages with access to roads in the rainy season iii. villages with access to roads in the dry season iv. time to district center and nearest two villages by car or

truck v. cost to district center

+ + + - -

V V V HH HH

IV-N IV-3 IV-4 MA-3, N MA-2, N

V V V HH HH

R B B R R

III. PRF increases

i. times per months goods are sold outside the village ii. quantity of goods sold outside the village

+ + -

HH HH HH

MA-18 MA-11 MA-15

HH HH HH

R R R

4

access to markets

iii. price differential for goods sold in the village and market outside the village

MA-16

IV. PRF II increases access to health care services

i. seeking care at medical facility if sick

ii. time to nearest point of health care service

iii. cost of transport to nearest point of health care service

+ - -

I I I

H-9 H-13 H-14

I I I

B R R

V. PRF II increases perception of school quality and quality of school infrastructure and facilities

i. perception of school quality (categorical scale) ii. satisfaction with school quality (primary school)

iii. perception of quality of school building and facilities

+ + +

HH HH HH

E-N E-N E-N

HH HH HH

C C C

VI. PRF II increases satisfaction with loc al development planning processes

i. satisfaction with planning process to select village projects ii. satisfaction with capacity of subprojects constructed to meet

village needs iii. perception of influence of community over village affairs iv. perception of community participation in decision-making

+ + + +

SC SC SC SC

SC-N SC-23 SC-25 SC-24

HH HH HH HH

C C C C

VII. PRF II increases satisfaction with and perception of services

i. satisfaction with and access to education services ii. satisfaction with and access to health services

iii. satisfaction with and access to safe water

+ + +

SC SC SC

SC-N SC-N SC-N

HH HH HH

C C C

VII. Households in the lowest two quintiles across all households benefit from access to roads, health care and safe water

Consumption quintiles created via consumption aggregate from HH module and Market Price survey.

HH CF, CNF HH R

dir Direction of Hypothesized Effect: +: positive, - : negative so Source of Indicator: HH: Household module; I: Individual module; SC: Social dynamic and governance module; V: Village module lvl Level of Indicator: HH: household; I: individual (sub-household); V: village for Format of Indicator: B: binary; R: real number; C: categorical secq Section and Question Number: WS: Sanitation-Water; IV: Part IV of village questionnaire; MA: Roads, Transport and Market Access; H: Health, E: HH Roster and Education; SC: Social Capital and Government; CF: Consumption-Food; CNF: Consumption Non-Food; N: new question.

5

Discussion of Hypotheses Selection Hypotheses and indicators were selected based on the needs identified in the Results Framework. With respect to Hypotheses I, II and IV, this is straightforward, utilizing indicators directly related to indicator 2 from the Results Framework. For Hypothesis II, additional questions were asked on access, mode of transport and time to the nearest and second nearest villages to capture the fact that many projects are only able to improve certain sections of roads rather than the entire link to the district center. With respect to Hypothesis III, access to markets, the concept of the most immediate impact of new roads aside from accessing services was developed for the baseline survey to look at the extent which communities were able to sell goods in markets and the price they receive external to the village. While not specified in the Results Framework, impacts are potentially detectable via the IE at the household level. For Hypothesis V, education, although not explicitly stated in the Results Framework, stakeholders (including the World Bank and the PRF Management Team) felt that since a third of investments throughout the program were in schools and kindergartens, some approach to measuring their impact should be developed. It is important to note that at baseline 95% of villages had a primary school and enrollment rates across grades was at 90% or above. The majority of school projects involved expansion of an existing building or replacing an existing building with a new building. Thus, school projects were not providing a new school where none had existed before for the majority of cases. A number of indicators were then considered:

Enrollment: rejected due to already high enrollment rates and the perception that remaining lack of enrollment is not due to infrastructure conditions.

Completion rate and Attendance: while a new building might provide a greater perception of school quality and of benefit in attending from the parents’ perspective, it is not clear that this effect could be strong enough to be detected. For attendance, there are significant measurement difficulties for households to recall absences and administrative data may be hard to access and unreliable. For completion rate, given the longer term nature of indicator, there are more factors outside the control of the project, such as teaching quality and curriculum which are likely to be more important than a building.

Perception of quality: the indicator proposed is perception of overall quality and perception of quality of facilities. It should be possible to capture impacts from the new school building via perception questions. These questions are currently placed in the new Services Satisfaction module to be answered by

6

the designated randomly assigned male or female respondent so that the results can be disaggregated by gender.

For Hypotheses VI and VII, local development planning and services, new questions will be added (see below) to capture satisfaction with service access and quality, and the planning process, with new questions to address the former placed in a new Service Satisfaction module. Indicators derived from existing questions from the baseline survey around the community’s role in and influence over planning will also be evaluated. Additional Questions to Be Added to the Endline Survey

Household Instrument Water and Sanitation Module: o How long does it take to fetch water? o How many times per week do you fetch water?

Village Instrument Part IV o How many months of the year does this village have access to a road

by car or truck? o How long does it take to reach the nearest and 2nd nearest villages in

the dry and rainy seasons by car or truck? (intended to capture changes in location where roads to do not eventually lead to the district center)

Household Instrument Road, Transport and Market Access Module Household Instrument Social Capital and Governance Module

o How would you rate your satisfaction with the development planning process in the village? (5 point scale, to be added to the end of the Social Capital module)

Household Instrument NEW Service Satisfaction Module o Are you satisfied with the quality of the school building and facilities?

(5 point scale) o Are you satisfied with the overall quality of the school? (5 point scale) o Are you satisfied with your ability to access health care services? (5

point scale) o Are you satisfied with the quality of health care services? (5 point

scale) o Over the past three years, has your access to health care services

improved, declined or stayed the same? (5 point scale) o Are you satisfied with your ability to access education services

(primary school)? (5 point scale) o Are you satisfied with the quality of education services? (5 point

scale) (primary school) o Over the past three years, has your access to education services

improved, declined or stayed the same? (5 point scale) o Are you satisfied with you ability to access safe water? (5 point scale) o Are you satisfied with the quality and availability of safe water? (5

point scale)

7

o Over the past three years, has your access to safe water improved, declined or stayed the same? (5 point scale)?

o What is your perception of the quality of the school building and its facilities ? (Six point step using show card)

o What is your perception of the overall quality of the school? (Six point step using show card)

III. Experimental Design The impact evaluation employs a randomized controlled experimental design to estimate the impacts of the Lao PRF II program on access to health and education services, clean water, all-weather roads, and perception of and satisfaction with the above services and local development planning. The evaluation compares changes in outcomes between 22 treatment kumbanh (sub-districts), receiving funding and activities through PRF II and 22 control kumbanh, which have not yet participated in the program. For indicators where no baseline data is available, the evaluation will compare differences between treatment and control kumbanh at baseline.

Treatment Assignment For the purpose of the impact evaluation, participation in Lao PRF II for the eleven districts in the four provinces was assigned randomly by kumbanh, the unit of project implementation. The sampling frame of kumbanh was limited by the number of districts and kumbanh planned for the four new provinces joining PRF for Phase II. The plan for implementation was for 14 districts and 114 kumbanh in Phongsaly, Oudomxai, Luang Prabang and Attapeu provinces. Selection of the initial 114 kumbanh was made by taking all kumbanh in the 14 districts with poverty rates greater than 40%. Of these 114, forty-four kumbanh were selected to participate in the evaluation - 22 of these being assigned to the treatment group and 22 to the control group. The sampling frame consisted of all 114 kumbanh with an estimated poverty rate greater than 40% . All districts were newly beginning participation in Lao PRF II and were not a part of the original PRF I project. Sampling Design Sample selection was then conducted via the following steps:

District Selection: a total of 11 out of the 14 districts were selected for the sample, apportioned to each province based on population. Phongsaly and Attapeu each contained only two PRF II districts, each of which was assigned automatically to the sample; for Luang Prabang and Oudomxai, 4 and 3 districts respectively were selected randomly from the planned five in each province, giving a total of 11 districts.

8

Kumbanh Selection: two treatment and two control kumbanh were selected randomly from within each of the 11 selected districts for a total of 44 kumbanh. Control kumbanh will not receive project support during the period of evaluation.

Household Selection: within each Kumbanh, 100 households were selected randomly, for a total of 4400 households. The 100 households are distributed evenly across all villages in the kumbanh by dividing the 100 households by the number of villages.1 Households were then selected randomly using lists constructed in each village with the assistance of the village government. The same households will be reinterviewed during the endline survey, with tracking of households remaining in the district.

For the qualitative component, 16 villages were chosen to be surveyed under the qualitative component. Within each province, 2 districts were selected with one treatment and one control village in each district. Districts and villages were chosen purposively to reflect geographical, ethnic and socio-economic variation.2

IV. Data

Data required to estimate impacts for PRF is provided by two quantitative surveys and two qualitative research studies:

Baseline Survey and Study: conducted between September 12th and October 31st 2012 prior to all activities in the sample kumbanh with the exception of some initial planning meetings held in Attapeu province at district and kumbanh levels by PRF staff (but not in villages). Endline Survey and Study: to be collected between September 12th and October 31st 2015.

The surveys and studies utilized the following instruments:

i. Household Questionnaire: containing individual modules (to be answered by individuals or an adult best able to answer), household modules (to be answered by the head of household or adult best able to answer), and a social dynamics and governance module consisting of 50% female and 50% male adults best able to answer, with gender randomly assigned. Summarized in Table 3 below.

1 Post selection weights using population data were created to ensure estimates reflect equal probability of selection for each observation. See Section 3.3 below. 2 Discussions were held between the research team and PRF II district teams given the limited information on village characteristics at the national level. Villages which demonstrated higher poverty rates, remoteness from district and village centers and ethnic group variation were selected by the field teams in consultation with the World Bank evaluation team. Every effort was made to ensure that the treatment and control villages in each district had similar profiles with respect to the characteristics discussed above.

9

ii. Village Questionnaire: village head is the respondent. iii. Market Price Survey: prices of goods from consumption module at

nearest market. iv. Qualitative Instrument: administered to focus groups and key informants

as outlined in Table 4 below Table 3: Survey Instrument Modules and Corresponding Respondents

Module Instrument Section Respondent Household Household Roster, Housing

Conditions. Access to Markets, Consumption

Head of Household (or adult best able to answer)

Individual Access to Education, Health and Employment

Individuals (or adult best able to answer in the case of children)

Social Dynamics and Governance

Social Dynamics and Governance

One respondent per household: 50% male, 50% female

Village Village Survey Village Head

Table 4: List of Key Informant Interviews and Focus Group Discussions

Key informant interviews (at district, kumbanh and village levels)

District Government Head Village head Other village official

PRF District Head and Staff (other development project staff in control areas)

PRF Kumbanh Facilitator (other development staff in control areas) Other community leaders/elders (including 1 woman and 1 member of a minority group at village level)

Total Interviews:

1 1 1 3

1

3

10

Focus group discussions (in each village)

Poorer villagers – male

Poorer villagers - female

Ethnic Minority villagers – male

Ethnic Minority villagers – female

Ethnic majority villagers – male

Ethnic majority villagers - female

Formal groups (village associations, women’s groups, savings groups etc.)

Total FGD’s:

1 1

1

1

1

1

4

10

10

V. Methodology

The impact evaluation seeks to test the seven hypotheses listed in Table 5 above to determine whether PRF has had a statistically significant impact on access to health services, water and roads, and satisfaction with local development planning processes and quality of services. The results will be examined for villages, households and, separately, for households in the lowest two quintiles of per capita consumption. Indicators to test these hypotheses are created using data collected in the baseline and endline surveys.

Two different specifications will be used to test the hypotheses:

1. Impact of treatment effect across the entire sample: the IE will test the overall impact on indicators for all hypotheses using a standard difference-in-differences estimator. Baseline data will be used when available, and an alternate ANCOVA specification will be tested for robustness.

2. Impact of specific sub-project types: due to the smaller sample size associated with a given type, hypotheses seeking impacts deriving from infrastructure sub-projects (I-V) will utilize an approach designed to identify the impact of that particular type for the relevant treated locations: in some cases it is likely that construction was not widespread enough to generate detectable impacts across the entire sample.

Estimation of Treatment Effects for the Entire Sample (Hypotheses I-VII) All hypotheses with indicators for which baseline data is available will be tested using a standard difference-in-differences estimator of the following form: (1) Ytvi = a + tr1Tv + tr2Tvtt +tt + gv + uvi

Where Ytvi is the outcome of interest at time t (0= baseline, 1 = endline), in village v and household i, Tv is a dummy representing PRF treatment, tt is a dummy representing t, g are village fixed effects and u is the error term. The coefficient of interest representing the impact of PRF is represented by tr2. All specifications will include village level fixed effects with standard errors clustered at village level.

11

In cases where baseline data is unavailable, the following specification will be used, where tr1 represents the impact of PRF on the indicator of interest: (2) Y1vi = a + tr1Tv + gv + uvi The standard difference-in-differences method was chosen given the relatively immediate and consistent impact that new infrastructure is expected to have on the indicators of interest. In particular the assumption of parallel time trends between treatment and control groups in stronger for impacts of this type. For robustness, an ANCOVA approach which allows for random variation in the time trend and which sometimes provides greater statistical power in comparison with difference-in-differences will be employed.3

Estimation of Effect of Specific Types of Projects on Individual Indicators4 (Hypotheses I-V)5 As noted above in Section II, Lao PRF II funds the implementation of subprojects selected by the village community. Given that for all but two villages in the sample, only one type of sub-project was built over the period of evaluation, detection of overall treatment effects across the entire sample which are directly related to specific infrastructure subproject types is less feasible. Thus, for hypotheses pertaining to specific infrastructure types (I-V), estimation of the overall treatment effects of the program is supplemented by estimating the treatment-on-treated effect of particular infrastructure types in villages where each type has been located or villages which stand to benefit from such infrastructure constructed in other villages in the kumbanh. To undertake this approach, the choice of project type needs to be accounted for as it is endogenous to the treatment assignment: the choice is only revealed for villages which receive the project. Once treatment status is determined the choice of project type is driven by a set of unobserved variables. These unobservables may not be distributed equally between the entire control sample and the subset of villages choosing a particular infrastructure type, potentially biasing results in a comparison of a subset of households in treatment villages with the entire control sample. In order to control for such unobservables, we need to compare the villages from the treatment group that have chosen a project of a particular type to villages in the control group that would have chosen the project of the same type if they received

3 McKenzie, D. (2012). Beyond Baseline and Follow-Up: The Case for More T in Experiments. Journal of Development Economics. 4 This section closely follows the methods used in the NSP Impact Evaluation and can be found at the following web address: 5 For a detailed discussion, see Annex 1 below

12

support from PRF II. Formally, denote by Cvj a dummy that indicates that a village v will choose the project of type j, if given a chance and the corresponding outcome of interest by Yvj. Denote by Y0vj and Y1vj potential outcome Y in village v without treatment and with treatment respectively. Cvj is only observed in treatment villages in the case where Tv =1; the realized value of Yvj is thus: (3) Y1vj if Tv = 1 and Cvj =1 Y0vj otherwise The estimate of interest is the treatment on the treated effect: (4) ttot = E[Y1vj - Y0vj | Cvj =1] It can be demonstrated that the treatment on the treated effect can be estimated using an instrumental variable regression with Cvj as the endogenous variable and Tv as the instrument.6 The treatment assignment is uncorrelated with unobserved variables in the error term driving project choice because treatment was randomized and not conditioned on such variables. However, it is correlated with Cvj as project selection only occurs in villages where treatment is received. To then estimate the effect of particular infrastructure subproject type, the following IV regression is used: (5) Y1vi = a + tivj Cvj + gp + uvi

Where Cvj is instrumented by Tv. Standard errors are clustered at the village level.

6 See, http://www.nsp-ie.org/paps.html Appendix A for formal proof.

13

Annex 1: Minutes of Lao PRF II Endline Survey Revision Workshop Minutes

World Bank Office, Vientiane June 10th, 9am-12pm

Minutes

The workshop was held on Wednesday June 10th. Participants included key PRF staff, DFAT and WB IE team staff. The following topics were discussed:

1. Review of IE Process: a brief review of the methods (including randomized, mixed methods approach), content of the survey instruments, research questions and key findings from the baseline survey.

2. Review of interventions by PRF and subproject construction in control

kumbanh: number and type of subprojects built (see table below in Meeting with PRF Staff) to reach a common understanding of what PRF II has implemented in treatment kumbanh. In addition, the number of subprojects built in control and treatment areas (non-PRF) were presented to confirm lack of threat of resource diversion, confirming the viability of the randomized design.

3. Review of PRF Results Frameworks and relevant indicators to be

collected by the IE: Greater than 75% satisfaction levels reported by beneficiaries in targeted

villages regarding improved services and local development planning Improved access to and utilization of basic economic and social services

in kumbanhs supported by PRF o % increase in access to and utilization of health services o % HHs with improved access to and utilization of safe water

resources o % increase of HHs with access to all weather roads

Lowest two quintiles benefit from above services

4. Module by module walk-through of the survey instrument by sector (Education, Health, Road and Market Access, Water, Social Capital/Governance). The following five questions were discussed for each sector.

o What has PRF built? o What types of impacts can we expect from this type of subproject? o Do we have indicators which capture these impacts in the IE

survey instrument? o Do these indicators fulfill RF reporting requirements?

14

o What additional indicators should be added to the IE survey instrument (including the qualitative component)?

The following tables review the discussion by sector:

5. Education

What was built 21 primary schools and 3 kindergartens. The PRF team indicated that over 90% of primary school construction projects are reconstructions where old facilities are replaced by new facilities or expansion to existing facilities.

Types of Impacts

Discussion centered around whether this kind of project could produce greater enrollment or attendance rates 1) via parents perceiving that the school was of better quality and therefore providing better learning and 2) via increase in capacity or grades offered. It was noted that the new infrastructure likely creates a positive perception among parents, as the projects were selected by villagers. Other potential impacts mentioned were attendance, teacher attendance and incidence of sickness among students (due to cleaner facilities). It is not clear that the former two can be measured accurately via HH responses. For the health indicators, given the addition factors influencing illness it is unlikely that impacts will be detected.

RF indicator Currently, none. Listed at output level Relevant indicators from the baseline

% of children enrolled in appropriate age cohort (7-11) years of schooling completed by age cohort (7-11)

Feasibility of new indicators

It was noted the already high rates of enrollment in primary schools (over 90%) may make the detection of impacts difficult as the constraints to the remaining 10% not enrolled are likely not due to infrastructure quality. With respect to attendance, there are difficulties in accuracy of measurement in the field due to recall. For something like completion rate which measures attendance and enrollment over time, other factors such as teaching quality and curriculum will play a larger role, reducing the potential for detectable impacts from the new facilities. There was agreement that a short module on motivations for and perceptions of infrastructure that was built should be included. For primary school construction, this would involve asking:

satisfaction with the outcome of the project

15

perception with the quality of facilities and overall quality of the school

whether the project has addressed identified needs

It was noted that given that 30% of subproject funds went to school projects, having an outcome indicator beyond perception would be preferable. However, the WB IE team does not recommend pursuing non-perception impacts as they are likely not detectable given the capabilities of the IE design.

Additional indicators needed in the survey instrument

Perception and satisfaction questions (to be developed by the World Bank IE team) as described above.

6. Health

What was built 4 dispensaries in kumbanh centers. Two of these are new and two are replacing old facilities.

Types of Impacts

Increased seeking of care, reducing travel times and cost to reach facilities.

RF indicator % increase in access to and utilization of health services

Relevant RF Indicators from the baseline

% of those sick seeking care time to point of health care service cost of transport to health care service

Feasibility in addressing RF

No issues.

Additional indicators needed in the survey instrument

None needed.

7. Water Systems

What was built 16 clean system renovations, spring gravity constructions and wells. Water sources are protected.

Types of Impacts

Access to protected water source. Reduction in travel time to access water source.

16

RF indicator % HHs with improved access to and utilization of safe water resources

Relevant indicators from the baseline

% of households with access to protected water source in dry season % of households with access to protected water source in rainy season Distance to water source

Feasibility in addressing RF

No issues.

Additional indicators needed in the survey instrument

Time needed to fetch water and number of trips per week to allow for calculation of time savings.

8. Roads

What was built 19 road projects including repair, improvement and spot

improvement. One was a bridge project. Types of Impacts Reduced travel time to markets and services. RF indicator % increase of HHs with access to all weather roads

Relevant indicators from the baseline

% of villages with access to roads in the rainy season % of villages with access to roads in the dry season access to district and kumbanh markets

Feasibility in addressing RF

Sufficient. However, it was noted that not all roads are intended to connect to the network leading to the district center and may have other intended purposes including access to fields and nearby villages. The 19 projects were placed in four categories:

o Road to village closer to Kumbanh Center (KBC) or District

o Road to village away from KBC or District o Road to village closer to other KBC or District o Road within village

Measuring impact via access to all-weather roads should be based on the intended destination rather than only looking at access to district centers. The survey will construct a map to identify which communities benefit from the PRF road and how they benefit in terms of destination. The evaluation will consider impacts for the villages which can benefit in terms of access to roads and markets.

17

Additional indicators needed in the survey instrument

Modifying existing road and market access questions in the HH module with the intended destination rather than only district center. Months of the year with access to roads

9. Social Capital, Participation and Satisfaction Levels

What meetings were held

Planning meetings were held in the first Cycle of implementation (Cycle IX for Attapeu, Cycle X for other provinces). Implementation meetings are held frequently in villages targeted to receiving funding during the subproject construction process. Meetings are held occasionally in other years for non-funding receiving villages.

Types of Impacts Participation in planning and subproject construction processes.

RF indicator Greater than 75% satisfaction levels reported by beneficiaries in targeted villages regarding improved services and local development planning

Relevant indicators from the baseline

Receipt of information on PRF and other development planning processes Participation and influence of the community local development planning Satisfaction with local government response to identified problems

Feasibility in addressing RF

The survey instrument does not contain explicit questions asking about perception of quality of services or local development planning. There is some concern that only two meetings period over 3 Cycle is not sufficient to generate impacts.

Additional indicators needed in the survey instrument

Perception of and satisfaction with access and quality of services, and perception of change in quality and access over time.

10. Summary of additional research topics The following additional research topics were suggested by participants:

18

Topic Discussion O&M Determining how successful the project is at maintaining

infrastructure in comparison with control areas due to the incorporation of O&M into project process. This is likely be addressed via a planned dedicated study which will not only look at perception and participation but quality and frequency of maintenance.

Use of PRF planning process in local development decision-making

Understanding to what extent local development actors (local government, NGOs, donor supported projects) are utilizing PRF planning mechanisms for resource-targeting and planning. This should not be addressed via a HH-based IE as survey teams have limited time in the field to collect the data from government officials. A separate dedicated study on PRF influence on local development planning should be undertaken which will have an explicit focus for data collection on planning and resourcing at the district and kumbanh levels.

Capacity to attract resources

To what extent are PRF kumbanh/villages able to attract additional resources, both in the form of infrastructure projects, and staffing/materials for existing facilities? It is not clear that a limited number of planning and implementation meetings are sufficient to create sufficient capacity and motivation to draw in additional resources for impacts to be detected via the IE. A separate study (in coordination with local development planning process above) should be conducted to look at this issue.

Livelihoods What kind of impact are PRF subprojects having on livelihoods opportunities? As the project is not designed to address these issues directly and the typical impact on consumption/household welfare in CDD projects is 4-6 years, the potential to detect impacts is unlikely. However, data on market access, movement of agricultural outputs and sale of outputs at different locations outside the village is included in the survey and can be used in other research activities to look at the issues.

11. Next Steps

Pre-analysis plan and draft of new questions circulated to stakeholders (July 1st)

Plan and instrument draft are revised (July 8-10th) Field test of new questions (July 13-14th) Survey preparation (August) Data collection (September-October)

19

Data cleaning, analysis and report writing (November 2015- January 2016)

Final report (February 2016)

12. PMT support for the IE process The PMT will review all pre-analysis plans and survey instruments to

ensure that questions are properly formatted and translated. The PMT will provide support during the field test, survey preparation

and data collection to properly supervise the contracted survey firm. The PMT will provide a permit letter to facilitate data collection by the

survey firm (WB will provide an initial letter explaining the IE process to assist the PMT with obtaining the permit letter)