Upload
others
View
3
Download
0
Embed Size (px)
Citation preview
U.S. DEPARTMENT OF STATE THIRD ANNUAL CONFERENCE ON PROGRAM EVALUATION:
NEW PARADIGMS FOR EVALUATING DIPLOMACY IN THE 21ST CENTURY
June 8-9, 2010
Health Track
PowerPoint presentation from workshop on Validation of a Measure of Household Hunger for Cross-
Cultural Use
Session transcript: http://www.state.gov/s/d/rm/rls/rm/2010/148053.htm
Food and Nutrition Technical Assistance II Project (FANTA-2)Academy for Educational Development 1825 Connecticut Ave., NW Washington, DC 20009Tel: 202-884-8000 Fax: 202-884-8432 E-mail: [email protected] Website: www.fanta-2.org
Validation of a Measure of
Household Hunger for Cross-
Cultural Use
Findings of work by:
Megan Deitchler, Terri Ballard, Anne Swindale and
Jennifer Coates
What is Food Security?
“When all people at all times have physical and economic
access to sufficient food to meet their dietary needs for a
productive and healthy life.” (USAID 1992)
This definition is founded on 3 elements:
1. Food availability
2. Food access
3. Food utilization/consumption
3
Food
Access
Food
Consumption
& Utilization
Food
Availability
Food Security
National level
Individual level
Household level
Food balance
sheets (FAO)
Various, no
standard
method
Anthropometric
indicators
Le
ve
l of
Me
asu
rem
en
tS
tan
da
rd M
eth
od
of
Mea
sure
me
nt
Defining and Measuring Food Security
Prevalence of “undernourishment” by country,
FAOSTAT 2009
Why do we need information on household
food access?
• To understand who is affected by food insecurity
• To understand to what extent households are affected by
food insecurity
• To understand how households are affected by food
insecurity
Useful measurement criteria for an indicator of
household food access
• Simple, internally, externally valid measure
• Provides timely information
• Possibility to obtain disaggregated data by region,
urban/rural communities
• Comparable, cross-culturally valid measure
Motivation for a comparable measure of
household food access for cross-cultural use
• Decision making: Facilitates prioritization of geographic
regions for intervention
• Monitoring and evaluation: Enables multi-regional and
cross-country evaluation of programs
• Policy advocacy: Increased attention to food
deprivation with availability of comparable data for
resource-poor areas
7
Process to identify a comparable measure of
household food access for cross-cultural use
Phase 1. Development of Initial Scale
• Background concept paper
• Multi-year field validation studies
• Literature review
Phase 2. Validation and Revision of Scale
• Identify partners and datasets for validation study
• Assess internal, external and cross-cultural validity
• Revision of scale according to validation results
8
Phase 1: Development of initial scale
9 scale items (I) defined:
1. Worry that the household would not have enough food
2. Not able to eat the kinds of food preferred
3. Eat a limited variety of foods
4. Eat some foods that you really did not want to eat
5. Eat a smaller meal than you felt you needed
6. Eat fewer meals in a day
7. No food to eat of any kind in your household
8. Go to sleep at night hungry
9. Go a whole day and night without eating
4 frequencies (F) defined:
Never, Rarely, Sometimes, Often
9
Phase 2: Validation and revision of scale
Data sets for validation study:
Mozambique Round 1 (Moz R1) 2006 n=591 rural; 4 districts
Mozambique Round 2 (Moz R2) 2007 n=299 rural; 2 districts
Malawi (Mal) 2007 n=1161; 3 districts
Kenya 2006 n=152 urban; HIV/AIDS affected hhs; (purposive sampling)
Zimbabwe (Zim) 2007 n=176; HIV/AIDS affected hhs in 3 districts
South Africa (S Africa) 2006 n=491; 1 district municipality
West Bank Gaza Strip (WBGS) 2006 n=1973; nationally representative
Primary validation analysis method:
Rasch modeling using CML for polytomous and dichotomous data
10
Example of Rasch analysis: Scale items and
households on a logit continuum
Less severe
Item 1
-5.1
Item 3
-3.4
Item 4
-2.5
Item 2
-4.8
Item 6
0.7
Item 5
-0.9
Item 7
2.2
Item 8
3.9
Item 9
5.8
HH 1
-1.4
HH2
-2.9
HH 3
2.5
More severe
Scale measurement criteria and method for
evaluation
• Internal validity: Infit and outfit statistics
• External validity: Direction and strength of association
with related indicators
• Cross-cultural validity: Cross-cultural comparison
plots of standardized item and household calibrations
Example of Rasch analysis: Perfect cross-
cultural equivalence
item1step1
item2step1item3step1
item4step1item5step1
item6step1
item7step1item8step1
item9step1item1step2item2step2
item3step2item4step2
item5step2item6step2
item7step2
item8step2item9step2
item1step3item2step3
item3step3item4step3
item5step3
item6step3item7step3
item8step3
item9step3
pop
ula
tio
n2
population1
Two of the alternative [shorter] scales tested
5 Item 3 Frequency (5I 3F) Scale:
5. Eat a smaller meal than you felt you needed
6. Eat fewer meals in a day
7. No food to eat of any kind in your household
8. Go to sleep at night hungry
9. Go a whole day and night without eating
3 Item 3 Frequency (3I 3F) Scale:
7. No food to eat of any kind in your household
8. Go to sleep at night hungry
9. Go a whole day and night without eating
14
Internal validation results: Assessing equal
discrimination of items and item steps
15
Infit
Range % of Items and Items Steps Outside of Infit Range
5I 3F Moz R1 Moz R2 Mal WBGS Kenya Zim S Africa
0.7-1.3 33.3% 26.7% 0.0% 0.0% 0.0% 40.0% 6.7%
0.6-1.4 6.7% 20.0% 0.0% 0.0% 0.0% 6.7% 6.7%
0.5-1.5 0.0% 6.7% 0.0% 0.0% 0.0% 0.0% 0.0%
3I 3F Moz R1 Moz R2 Mal WBGS Kenya Zim S Africa
0.7-1.3 11.1% 22.2% 0.0% 33.3% 0.0% 33.3% 55.6%
0.6-1.4 0.0% 0.0% 0.0% 0.0% 0.0% 22.2% 22.2%
0.5-1.5 0.0% 0.0% 0.0% 0.0% 0.0% 22.2% 0.0%
Cross-cultural validation results for item steps:
5I 3F, Moz R2 and Moz R1
5step1
5step2
6step1
6step2
7step1
7step2
8step1
8step2
9step1
9step2
-6-5
-4-3
-2-1
01
23
45
6
Mo
zam
biq
ue
R2
-6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6Mozambique R1
16
17
Cross-cultural validation results for item steps:
5I 3F, Zim and Moz R1
5step1
5step2
6step1
6step27step1
7step2
8step1
8step2
9step1
9step2
-6-5
-4-3
-2-1
01
23
45
6
Zim
bab
we
-6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6Mozambique R1
Cross-cultural validation results for item steps:
3I 3F, Moz R2 and Moz R1
q7step1
q7step2
q8step1
q8step2
q9step1
q9step2
-6-5
-4-3
-2-1
01
23
45
6
Mo
zam
biq
ue
R2
-6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6Mozambique R1
Cross cultural validation results for item steps:
3I 3F, Zim and Moz R1
q7step1
q7step2
q8step1
q8step2
q9step1
q9step2
-6-5
-4-3
-2-1
01
23
45
6
Zim
bab
we
-6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6Mozambique R1
Cross cultural validation results for HH
measure: 3I 3F, Moz R2 and Moz R1
1
2
3
4
5
-4-3
-2-1
01
23
4
Mo
zam
biq
ue
R2
-4 -3 -2 -1 0 1 2 3 4Mozambique R1
Cross-cultural validation results for HH
measure: 3I 3F, Zim and Moz R1
1
2
3
4
5
-4-3
-2-1
01
23
4
Zim
bab
we
-4 -3 -2 -1 0 1 2 3 4Mozambique R1
22
Results for Household Hunger Scale (HHS) by
data set
Little to no
Household
Hunger
Moderate
Household
Hunger
Severe
Household
Hunger
Mozambique R1 42.8 46.4 10.8
Mozambique R2 43.1 48.8 8.0
Malawi 51.9 37.2 10.9
West Bank
Gaza Strip74.9 18.7 6.5
Zimbabwe 51.4 33.7 14.9
South Africa 31.2 46.4 22.4
External validation results: Proportion of households
classified in each HHS category by household wealth
score, Moz R1
0
10
20
30
40
50
60
70
80
90
100
0 1 2 3 4 5 6 7 8 9 10Pro
po
rtio
n C
lass
ifie
d i
n H
HS
C
ate
go
ry
HH Wealth Score
Little to no hunger
Moderate hunger
Severe hunger
23
External validation results: Median monthly HH income
by consumption unit for each HHS category, WBGS
and S Africa
0
50
100
150
200
250
300
350
400
450
500
Little to no hunger
Moderate hunger
Severe hunger
Me
dia
nH
H I
nc b
y C
U
WBGS
SA
24
0
10
20
30
40
50
60
70
80
90
100
Pre-harvest, Dec '06 Post-harvest, July '07
Little to no hunger Moderate to severe hunger
0
10
20
30
40
50
60
70
80
90
100
Pre-harvest, Dec '06 Post-harvest, July '07
Little to no hunger Moderate to severe hunger
External validation results: Is the HHS sensitive to detecting
change? Trends from 2 districts in Mozambique
Household hunger remains
relatively constant over time.
Series of climactic shocks:
Droughts, Floods in early 2007, and
Cyclone Favio just before harvest.
Household hunger improves
significantly over time.
Gondala harvest less affected by
climactic shocks.
25
External validation results: Is the HHS sensitive to
detecting change? Trends from 2 districts in Mozambique
Household hunger remains
relatively constant over time.
Series of climactic shocks:
Droughts, Floods in early 2007,
and Cyclone Favio just before
harvest.
Household hunger improves
significantly over time.
Gondala harvest less affected by
climactic shocks.
Chibabava, Sofala Province,
Mozambique
Gondala, Manica
Province, Mozambique
Limitations of Study
• Data sets used in validation study were mostly limited to
Southern and Eastern Africa
• Sample size limitations for the Kenya and Zim data sets
• Adaptation work not undertaken for the WBGS and SA
data sets
• Rasch assumption of conditional item independence was
not assessed
• External validation was limited by the variables available
in data sets
• The data used for validation of the 3I 3F scale (and other
scales tested) were collected as part of a longer, 9I 4F
scale26
Next steps
• Publish a HHS operational guide, to provide
instruction for data collection and tabulation
• Continue internal and cross-cultural validation
of HHS, as data are collected in more contexts
• More external validation analyses, with more
robust measures of food deprivation
• Potential study to evaluate if the HHS collected
with a 3I3F questionnaire performs differently
than a HHS collected with a 9I4F questionnaire
Conclusions
• There are trade-offs between identifying a measure appropriate for use within one culture and identifying a measure appropriate for use across cultures
• 3I 3F scale shows reasonable equal item discrimination for most data sets and expected associations with „external‟ variables. Appears to provide a measure for cross-cultural comparison.
• Use of the 3I 3F scale should not preclude the concurrent use of a culturally specific measure of food insecurity, when available (e.g. ELCSA in LAC region).
• Other longer scales should be explored if a measure of food insecurity (rather than food deprivation) is desired for any one context.
• In the absence of cross-cultural validation, the resultant measure should not be assumed to be appropriate for cross-cultural comparative purposes.
28
Acknowledgments
The contribution of data by the following organizations is gratefully acknowledged:
Mozambique: FAO Representation
Malawi: Dept. of HIV/AIDS and Nutrition, UNICEF
West Bank Gaza Strip: FAO Jerusalem office
Kenya: Samwel Mbugua and Human Nutrition, Egerton University
Zimbabwe: Center for Applied Social Science, University of Zimbabwe
South Africa: South African Human Sciences Research Council
29
This presentation is made possible by the generous support of the American people through the support of the Office of Health, Infectious Disease, and Nutrition, Bureau for Global Health, United States Agency for International Development (USAID), under terms of Cooperative Agreement No. GHN-A-00-08-00001-00, through the Food and Nutrition Technical Assistance II Project (FANTA-2), managed by AED. The contents are the responsibility of AED and do not necessarily reflect the views of USAID or the United States Government.
Food and Nutrition Technical Assistance II Project (FANTA-2)Academy for Educational Development 1825 Connecticut Ave., NW Washington, DC 20009Tel: 202-884-8000 Fax: 202-884-8432 E-mail: [email protected] Website: www.fanta-2.org