BRACE Impact Evaluation Steering Committee Meeting17 January 2013ACTED, Juba, South Sudan

Agenda

TIME TOPIC14:00-14:15

Participant introductions14:15-14:45

HEA baseline – findings for Northern Bahr el Ghazal, Warrap and Western Bahr el Ghazal

14:45 -15:15

Quasi-experimental baseline – sampling methodology

15:15-16:00

Quasi-experimental baseline – review of indicators and tools

16:00-16:15

Quasi-experimental baseline – review of fieldwork16:15-16:30

Beyond the baseline – next steps and discussion

Participant Introductions

HEA Baseline - Findings

Elisabeth VikmanImpact Initiatives

Quasi-experimental baseline – review of sampling methodology Byron PakulaImpact Initiatives

Sampling OverviewObjective of the sampling is:• Get a fine-grain analysis of the food security in the

implementation areas• Compare the food security situations across:

• Areas where no FFA or GFD has been done in the past 12 months• Areas where GFD are being implemented (planned, ongoing, or

completed)• Areas where FFA are being implemented (planned, ongoing, or

completed)• Differentiate based on food security assessments (low, medium,

high)• Differentiate based on livelihood zone (Ironstone Platea, Western

Floodplains)

Sampling Theory of Analysis• Food Security Indicator is a function of:

• Control Group A: No Interventions• Control Group B: GFD Interventions• Treatment Group: FFA Interventions

FSI = CG(A)i + ∂1CG(B)i + ∂2TGi+ ∂3CG(B).TGiWhere • i=livelihood zone, food security assessment, and wealth

groups• ∂1 = impact on food security due to GFD• ∂2 = impact on food security due to FFA• ∂3 = impact on food security due to FFA and GFD in the one

location

Statistical Significance• The aim of the analysis is to be statistically significant

across the variables of:• Wealth Groups (4 categories) OR• Food Security Classification (3 categories) OR• Livelihood Zone (2 categories) AND• Control and Treatment Groups (3 categories)

• The aim for the data analysis is to be 95% confident with no more than a +/- 5% margin of error

• Due to the scale of the population sizes, to achieve this it is necessary for a sample of at least 380 households in the control and treatment groups for each of the three dependent variables • Wealth groups will not be specifically targeted, but will be

identified after the data collection process. This may affect the significance.

Clustering of Households• Clustering will ensure that the data received in

each community is sufficiently ‘random’ and ‘representative’

• Clustering in practice means the sampling will collecting at least 20 household surveys for each community or village

• To support the quantiative household surveys, each community or village will also have a focus group discussion to provide qualitative information and to support the analysis

Community Selection• Communities are selected randomly based on a

weighted average of where FFA activities are being implemented• E.g. If 25% of activities are implemented in the Ironstone

Plateau, then approximately 25% of the surveys will be collected in Ironstone Plateau

• Note that a minimum of 380 surveys for each sample group is required

• The locations are randomly selected based on WFP information for FFA interventions

• Control groups are selected from the same Payam with support from Boma Chiefs to identify similar villages meeting criteria of no FFA or GFD for past 12 months, or only GFD..

Sample SizesTotal Control A (None) Control B (GFD) Treatment (FFA)

HHs Villages HHs Villages HHs Villages HHs Villages

 Livelihood Zone            

Ironstone 1520 76 380 19 380 19 760 38

Western  4560 228 1140 57 1140 57 2280 114

       Food Security Classification

         

High 1520 76 380 19 380 19 760 38

Medium 3040 152 760 38 760 38 1520 76

Low 1520 76 380 19 380 19 760 38

Quasi-experimental baseline – review of indicators and toolsElisabeth VikmanImpact Initiatives

Quasi-experimental – Phase 1 baseline indicatorsIndicator Sub Measurement Level Type

Agegroup of household members Age dependency ratio 0-4; 5-14; 15-24; 25-49; 50+ Household Demographics

Gender of household members Female, male Household Demographics

Household member status Relationship, disability status, employment status HH members Household Status

Immigration status Returnee, IDP, refugee, host HHH and entire household Household Status

Social networks Boma kinship ties, contribution by kin Current /most recent 7 days Household Assets

Coping strategies (index) Negative' strategies Most recent 7 days Household Assets

Consumption Food and non-food items  Most recent 30 days Household Assets

Expenditure Food and non-food item, livelihood inputs (informed by HEA) Most recent 30 days Household Assets

Level of formal education current/completed primary/secondary/higher education Completed/current grade Household Assets

Income sources (time distributed)Livestock, Agriculture, Fishing,  In Kind, Labor, Trade, Remittances - primary/secondary/tertiary - % by season - link with HEA Most recent year Household Assets

Access to market Time, Cost, Challenges Current, generally Household Assets

Access to food  Type, Source Most recent 7 days Household Food Security

Availability of food (by type) Type Current Household Food Security

Nutritional status Type, source  Most recent 24 hours most recent 7 days Household Food Security

Morbidity Malaria, ARI, Diarrhoea symptoms/treatment - by demographics Most recent 14 days Household Health

Health seeking behavior Ante-natal care - female by agregroup Most recent pregnancy Household Health

Access to health care Time, Cost (transport, care, medicine/materials), Challenges Most recent 14 days, generally Household Health

Access to adequate water Source, distance Current Household Assets

Access to adequate sanitation Type - by gender Current Household Assets

Livelihood sufficiency cross-reference with income generation Lickert scale - most recent 12 months Household Perceptions

Well-being Quality of life Lickert scale - most recent 12 months Household Perceptions

Food security and changes cross-reference with access to food Lickert scale - most recent 12 months Household Perceptions

Mobility Freedom, choice Lickert scale - most recent 12 months Household Perceptions

Ability to withstand shocks vulnerability and ability to respond Future 12 months, Lickert scale Household Perceptions

Gender roles Income generating activities, division of labour Most recent 30 days/12 months, Lickert scale Household Perceptions

Gender attitudes Education, division of labour, access to care Current, completed, most recent 2 weeks, Lickert scale Household Perceptions

Changes in population Size, Composition - Hosts, IDPs, Returnees, Refugees - by gender Current Community Demographics

Quality of healthcare Primary Health Care 1) Unit and 2) Centre, 3) Hospital Current Community Health

Quantity of healthcare Primary Health Care 1) Unit and 2) Centre, 3) Hospital Current Community Health

Community assets Infrastructure, Social services, BRACE assets Current Community Assets

Community priorities Current Community Perceptions

Quasi-experimental baseline – review of Phase 1 fieldwork Elisabeth VikmanImpact Initiatives

4.1 OrganigramIMPACT Geneva

Luca PupulinSupervision of the

outputs

PM /Evaluation manager

Elisabeth VikmanSupervision of the

overall impact evaluation

Assessment managerMalika Baymatova

Supervision of the field teams

1 Assessment Team Leader

9 Assessment Assitants / 1 Driver

4 Data Entry Assistants1 Assessment Team Leader 9 Assessment Assitants / 1

Driver

1 Assessment Team Leader 9 Assessment Assitants / 1

Driver

ACTED South Sudan

Emilie PoissonSupervision of the

operations

DB/GIS/Research Assistant

In charge of the data base

Technical expertise from IMPACT Geneva

Assessment expertByron Pakula

Impact evaluation technical expertise

MIS/GIS expertRenaud Zambeaux

DB/MIS/GIS technical expertise

REACH in country support team

Punctual support

REACH Country Manager

REACH GIS/DB officer

Michael HOpfesnberger

4.2 Staffing & Output STAFFING – Dinka/Arabic/English speakers• 3 Team Leaders• 3 Community Focal Points• 24 Assessment Assistants• 4 Data Entry OfficersOUTPUT• 6080 HH level interviews• 304 Community level Key Informants interviews

4.3 Data collection plan – Wau, Western Bahr el Ghazal

21 January 2013• Day 1: Training Team Leaders & Community Focal

Points – ACTED Wau base• Day 2: Field testing – Wau communities• Day 3: Training Assessment Assistants – ACTED

Wau base• Day 4: Field testing – Wau communities• Day 5: FULL-SCALE SURVEY

28 January – 29 March 2013 • Phase 1 baseline survey – Northern bahr el Ghazal

and Warrap states

Beyond the baseline – next stepsElisabeth VikmanImpact Initiatives

5. Beyond the baseline 

 

 

Consolidation of key project monitoring (output/ outcome) indicators Monitoring of exogenous environmental factors affecting outcomes not linked to the project

January 2013 April 2013 August 2013 April 2013

Start of Phase II End Phase II

 

Baseline Survey – Phase I

HEA Baseline Process Household

Questionnaires Focus Group

Discussions Key Informant Surveys XXXX HHs, 9 counties,

2 states

Endline Survey– Phase I & II

HEA Final Analysis Process

Household Questionnaires

Focus Group Discussions Key Informant Surveys XXXX HHs, 20 counties,

4 states

Mid-Term Review – Phase I

Household Questionnaires Review of Indicator

Progress Focus Group Discussions XXXX HHs, 9 counties, 2

states

Baseline Survey – Phase II

HEA Baseline Process Household

Questionnaires Focus Group

Discussions Key Informant Surveys XXXX HHs, 11 counties,

2 states