Impact Evaluation Sri Lanka Dry Zone Livelihood Support and

Partnership Programme

Impact Evaluation WorkshopColombo, Cinnamon Lakeside Hotel – 3 December 2013

1

Background

• IFAD-wide: 30 project impact evaluations by 2015

• This has been IFAD’s first impact evaluation with primary data collection on a large scale

• Testing internationally recognised methodologies in an IFAD-project context. Identify opportunities, constraints and limitations

2

FF

Criteria and data

• Whole set of IFAD project evaluation criteria:Relevance, effectiveness, efficiency, impact, sustainability,

innovation, up-scaling, gender equality

• Primary data collection: mixed-method approach:

Survey of 2560 households (with / without project) Qualitative survey: focus groups: 41 with beneficiaries and

project and 30 with line agencies’ staff Synthesis mission • Collaboration with: GreenTech Consultants Pvt. (SL);

Social Impact, Washington State University (USA)

3

FF

Methodology

1. Baseline survey conducted in 2006 but dataset was lost

2. Project targeted poorer communities and households

→ sampling bias

3. Farmer household’s characteristics can affect participation in project: → sampling bias

4. Economic growth and poverty reduction in Sri Lanka during past 10 years: attribution issues

5. Multi-component: different impact pathways, difficult to capture through standardized questionnaire

4

FF

Approach followed

1. Data recall (reconstructing selected baseline data)

2. Econometric techniques: (i) Propensity-score matching; (ii) Heckman treatment effect

1st step: determine community and hh characteristics associated with project participation

2nd step: compare hh assisted by project vs. hh without project

This is expected to reduce sampling bias

3. Cross validation between mixed methods (qualitative & quantitative)

5

FF

General project information• Total cost US$ 27.2m, IFAD loan US$ 22.0m + 0.3m grant

• Approved Sep 2004, effective Dec 2005, compl. Mar 2013

• Components:(i) rain-fed upland agricultural & livestock developm. (10%); (ii) small-scale irrigation rehabilitation (15%);(iii) marketing and enterprise development (13%);(iv) microfinance and income-generation (12%);(v) community infrastructure development (27%); (vi) programme management (22%)

• Slow implementation 2005-2008. Catch-up 2009-2012

6

ES

The programme

area

7

ES

Findings – Project Performance

• Relevant project design and component mix. Exceptions: (i) initial marketing component design and (ii) subsidised interest rates for credit

• Adaptation to evolving country context (emphasis on higher-value crops, dairy farming, linkages with existing value chains)

• Concomitant focus on seed production schemes increased output quality and added value for the seed producers, especially in the case of seed potatoes

• Immediate objectives achieved in majority of components. High coverage of agricultural extension packages, although quality not always at par with quantity

8

ES

Findings – Project Performance (2)• Efficiency – a sluggish first half of project implementation, with

23% disbursements after 43 months, with a vigorous turn-around after the MTR in 2009

• Common pattern in IFAD-funded projects, depressing efficiency

• The catch-up effort has emphasized quantitative targets and less consolidation of results

• The PCR infers high internal return rates while the impact evaluation is more cautious on methodological grounds

• Subsidized interest loan rates are inefficient → credit rationing. Alternative: matching grants coupled with market interest rates

9

ES

Targeting – statistical analysis

Pro-poor geographical targeting (communities)

•Communities with project are more distant from District Secretary Division’s capital

•Less equipped with basic infrastructure (e.g. school, police, health post)

Household with project had:

•Poorer asset base before project start-up (recall data)

•More diversified cropping patterns

•Slightly better educational status of household head

10

FF

Findings: impact1) Project M&E data show generalised and consolidated impacts. Evaluation findings are more nuanced. Why ?

Evaluation survey sample more representative (random sampling) and takes into account attribution issues (included households without project)

2) Impact survey finds that farmers are better aware of improved techniques, higher value crops and products

3) Knowledge and awareness ≠ adoption. Initial investment costs can be a barrier, and farmers require extended technical support

11

FF

Findings – impact (cont.)

4) Project contributed to small farmers’ investments in dairy farming in dry zone area

5) Households invested in dairy farming (and on higher value crops) out of their own savings (limited credit access)

crowding out investment in other hh assets

Mixed effects on hh expenditures (as an income proxy)

6) Project activities peaked 2009-2012, impacts still emerging

12

FF

Other findings• Moderately sustainable: creation of farmers’ federations,

linkages with public extension programmes and private businesses. But farmers’ federations still weak

• Multiple innovations (e.g., potato and onion multiplication technology, quality seeds for cowpeas, groundnuts, cooling machines for milk, FFS approach).

Partnerships with public extension and private sector facilitate up-scaling. Progress will be slow without follow-up support

• Gender equality: men and women both benefitting from technical packages. Women well represented in farmers’ society governing bodies

13

ES

Recommendations for the future

• Need for a follow-up phase to consolidate results

• Promote: (i) linkages with existing value chains through public-private sector partnerships; (ii) support farmers’ societies and federations; (iii) avoid subsidised credit

• Advocacy on policy issues needs to continue. Not only land tenure, also formal registration of farmers’ societies

• Commitments to provide a financial contribution to revolving funds to be honoured

• Better accuracy and quality control in M&E is required. Retain good practice of conducting thematic studies. Conduct and preserve baseline survey

14

ES

Take home lessons

• The DZLISPP evaluation shows that supporting smallholder agriculture, modernizing agriculture and fostering market integration can be compatible objectives

• The presence of large retailing and processing companies in rural areas provides important opportunities for market linkages

• DZLISPP provided a seminal contribution but without additional support, progress may be slow in the future

• Mixed findings on impact reflect the burden of initial investments on smallholder farmers’ households.

15

ES