www.brac.net

Impact

BRAC Research Impact AssessmentSummarized by BRAC USA

www.brac.net

Why this exercise?

Q. Why do donors contribute to an organization?A. Awards, word of mouth, testimonials,

charity ratings, preferred habitat (programs, geography, culture, religion, ethnicity), vanity, emotive photos/stories, scale, quantifiable impact.

www.brac.net

Quantifiable impact is (becoming) the gold standard for NGO evaluation

• More and more funders are asking for proof of effectiveness by demonstrating a quantifiable impact

• BRAC’s independent research department is unrivaled in the magnitude of research produced

• Development organizations ignore quantitative impact at their peril

• Easier said than done

www.brac.net

What is Impact?Impact

“The positive and negative, primary and secondary long-term effects produced by a development intervention, directly or

indirectly, intended or unintended.”-- OECD Development Assistance Council

Impact Evaluation“A study of the attribution of changes in the outcome to the

intervention.“-- 3ie

www.brac.net

Impact research should be independent

Real life example of research bias

www.brac.net

Reports UsedProgram Title Time Period

Agriculture Impact of BRAC’s Agriculture & Livestock Programme in Uganda 2009-2010

Education Economic and Social Analysis of Primary Education in Bangladesh: A Study of BRAC Interventions 2009-2010

Education An Assessment of BRAC Pre-Primary Schools Environment 2012

Health Manoshi: Community Health Solutions in Bangladesh. Impact Evaluation Surveys in Dhaka Urban Slums, 2007, 2009, & 2011 2007-2011

Health Maternal, Neonatal, and Child Health Programmes in Bangladesh: Review of Good Practices and Lessons Learned 2006

Health The Pilot Maternal, Neonatal and Child Health Project (MNCH) at Nilphamari: Profiling the Changes During 2006-07 2006-2007

Health Sustaining Health: The Role of BRAC’s Community Health Volunteers in Bangladesh, Afghanistan, and Uganda 2008-2009

Health Impact of Community-based Tuberculosis Programme: Ex N/A

Microfinance The Wider Impacts of BRAC Poverty Alleviation Programme in Bangladesh 1992-2004

Microfinance An Impact Evaluation of BRAC’s Microfinance Program in Uganda 2008-2009

Ultra-Poor Addressing Extreme Poverty in a Sustainable Manner: Evidence from Ultra-Poor Programme In Bangladesh 2002-2008

Ultra-Poor Can Basic Entrepreneurship Transform the Economic Lives of the Poor? 2007-2011

Ultra-Poor Asset Transfer Programme for the Ultra Poor: A Randomized Control Trial Evaluation 2007-2011

Water & Sanitation Achievements of BRAC Water, Sanitation, & Hygiene Programme 2006-2011

Youth Social Network and Financial Literacy among Rural Adolescent Girls: Qualitative Assessment of BRAC’s SoFEA Programme 2011-2012

Youth Empowering Adolescent Girls: Evidence from a Randomized Control Trial in Uganda 2008-2010

www.brac.net

Positive impact on usage of modern inputs, cultivation methods, and vaccination services (Uganda)

Program participants are 27 percent more likely to use improved seeds than the comparison group (Uganda)

Impact Synopsis

Agriculture & Livestock

BRAC Pre-primary schools: Scores 7.7% higher than non-BRAC schools

Reduced Dropout and Repetition, Rate, Higher Pass Rate: BRAC schools are twice as effective as government schools in terms of lower dropout, higher pass rates and repetition rate

Education

www.brac.net

Impact Synopsis

Increases in MNCH knowledge, including danger signs for pregnant women, new mothers, and newborns

Increase in uptake of services, practicing essential newborn care, and increased access and follow-through of referral

Improved health metrics including: ANC and PNC visits, increase in use of Post-partum hemorrhage or PPH (the leading cause of maternal mortality) medication, increase in percentage of institutional delivery and an increase in percentage of women practicing exclusive breast feeding.

Reduction of MMR (Maternal Mortality Rate) from 236 per 100,000 live births in 2008 to 141 live births in 2010.

Health

www.brac.net

Positive long-term impact on the lives of the ultra poor, even four years after the end of the program’s two-year support phase, in terms of per capita income, employment, food security and asset holding

Among participants that entered the two-year program in 2002, 92% satisfied at least 7 out of 10 indicators of extreme poverty in 2008.

By 2008, six years after the baseline survey, the program’s impact on participants’ per capita income had almost doubled compared to the impact from 2002 to 2005.

Impact SynopsisMicrofinance

Better child survival, nutritional status, family planning, and education in households served by the program.

In Uganda, a Study published in 2013 revealed positive impacts including increase in savings, improved food consumption and increased financial and social resources leading to improved business startup and investment

Ultra Poor

www.brac.net

Water & Sanitation

Sanitary latrine use increase: Households using sanitary latrines increased significantly from the baseline (32%) to midline (41%) to end line (53%)

Reduction in water related diseases: The prevalence of water related diseases significantly reduced from 9.4% in the baseline to 7.1% in midline and 2.3% at the end line.

Impact Synopsis

www.brac.net

An RCT shows Uganda's ELA program having positive impacts on life and vocational skills, health-related knowledge, risky behavior, early childbearing, and engagement in income generating activities.

A study of the SoFEA program in Bangladesh found that the program was successful in increasing self-confidence and motivation, and in strengthening the relationship between girls and their parents and within the community.

Initiative to encourage adolescent girl’s socialization and improve financial literacy has been found to be “quite effective”.

After receiving Financial Literacy Training (FLT), SoFEA members were more financially aware and many were involved in Income Generating Activities, such as livestock and poultry rearing.

Youth

Impact Synopsis

www.brac.net

Agriculture and Livestock

www.brac.net

Timeframe of study: 2009-2010

Positive impact on usage of modern inputs, cultivation methods and vaccination services

Participants 27 percent more likely to use improved seeds than the comparison group

Improved cultivation methods: Participants are six percent more likely to follow line sowing cultivation and 10 percent less likely to follow inferior mixed-cropping

Participants are 24 percent more likely to utilize vaccination services

Participants are 16 percent more likely to be ‘market-oriented’ resulting in higher revenues from sales

Impact of BRAC’s Agriculture & Livestock Programme in Uganda

Agriculture & Livestock

www.brac.net

Education

www.brac.net

BRAC Education Program Model

www.brac.net

Economic and Social Analysis of Primary Education in Bangladesh: A Study of BRAC Interventions

Education

Timeframe of study: December 2009 – February 2010Value for Money: Cost per student completed is 2x in formal schools.

BRAC schools boost test scores: BRAC pre-primary students enjoyed ↑ in final exam scores in primary school by 7.7% compared to comparison group higher as well as ↑ secondary school enrollment (89% v. 82%).

BRAC schools have reduced dropout rates & higher completion rates, making them twice as effective as government schools.

Cost-Benefit: Private returns to primary education are higher for BRAC graduates, due to the 4 year cycle & lower cost of BRAC schools, compared to the 5 year cycle of mainstream schools

Internal Efficiency: BRAC schools advantage includes intensive monitoring of teachers & teacher input.

www.brac.net

Economic and Social Analysis of Primary Education in Bangladesh: A Study of BRAC Interventions

Did not attend pre-primary All Pre-Primary BRAC Pre-Primary Mainstream Pre-Primary0

10

20

30

40

50

60

70

80

90

100

72.6

83.888.6

82.2

Positive Effect of BRAC Pre-Primary Education on the Transition to Secondary Schools

%enrolled in secondary

www.brac.net

An Assessment of BRAC Pre-Primary Schools EnvironmentTimeframe of study: July – September 2012

Poor Overall Environment: In a 7-point scale, BRAC schools received an average score of 1.47

On the other hand, POs considered that the majority of schools were rated as good or very good.

Education

Do BRAC Pre-Primary Schools Have a Positive Impact?• Shahjamal and Nath (2008) compared performance of former pre-primary students

enrolled in formal primary schools and a comparable group who did not have a pre-primary background, and found that former BRAC students did significantly better in Grade I, but no difference was found in other grades

• Ahmad and Haque (2011) also find that scores achieved in primary school examinations are higher for students with BRAC pre-primary education

• Are these results contradictory?

Personal care, lang. learning, interaction, parents & staff, parts of activities can be improved with little additional monetary investment.

www.brac.net

Reaching Primary Education at the Doorstep of the Poor: The BRAC Experience

Education - Summary from Andrew Jenkins

Timeframe of study: 2005 Over 2.4 million children completed the primary education curriculum, due to BRAC’s educational

interventions 6.5% of total primary school students were enrolled in BRAC schools and over 90% of BRAC students went

on to enroll in formal secondary schools BRAC students had a better knowledge of health & development issues than formal school students

Socioeconomic Impact of BRAC’s Non-Formal Primary SchoolsTimeframe of study: 2005 BRAC primary students enjoyed better performance compared to formal school students. NFPE school attendees were significantly more knowledgeable on child immunization/prevention of disease

than those who attended government primary schools or never attended school at all A smaller proportion of adults who attended government primary schools had immunization cards (57.9%)

compared to NFPE school enrolled households (64.3%) NFPE school enrolled couples collected birth control at a higher rate (37.4%) than those who attended

government primary schools (32.1%) and who never attended a school (26.9%)

www.brac.net

Health

www.brac.net

Manoshi: Community Health Solutions in Bangladesh. Impact Evaluation Surveys in Dhaka Urban Slums, 2007, 2009, & 2011

Timeframe of study: 2007-2011 Increases in MNCH knowledge. Though improvement varied depending on the particular

aspect of knowledge. For example, knowledge about recommended 4+ ANC visits increased from 45% in 2007 to 59% in 2011 in the project area v. 50 to 52% in comparison area.

Increase in uptake of services, practicing essential newborn care and increased access and follow-through of referral. For example, colostrum feeding increased 32 absolute percentage points (PP) in the project area v. 18 PP for the comparison area.

Improvement in multiple health practices: ↑ ANC & PNC visits, ↑ use of Misoprostol (drug used for post-partum hemorrhage -leading cause of maternal mortality), ↑ in percent of institutional delivery & % of women breast practicing exclusive breast feeding.

For example: Women receiving PNC after delivery improved 28PP in project area v. 7 PP in comparison area.

↓Maternal Mortality Rate from 236 per 100,000 live births in 2008 to 141 in 2010, as a result of household services, timely referral, & coordinated support at referral facilities. (Source: Manoshi: A Community-based Solution to Avert Maternal Death, 13th ASCON 2011).

Health

www.brac.net

MMR declined by 50% in Manoshi areas

www.brac.net

Maternal, Neonatal, and Child Health Programmes in Bangladesh: Review of Good Practices and Lessons Learned

Timeframe of study: 2006 Saving Newborn Lives Program: A

collaborative program Funded by BMG implemented by BRAC, et al demonstrated changes in many areas of maternal care: Post-Natal Checkup increased by 37.5

to 61% from the baseline survey Newborn checkup within 24 hours

increased from 14.4% in 2002 to 27.3% in 2004

ANC increased from 22% in 2002 to 92% .

Health

*National averages sourced from UNICEFhttp://www.unicef.org/infobycountry/bangladesh_bangladesh_statistics.html) and the World Bank (http://data.worldbank.org/indicator/SP.DYN.CONU.ZS?page=1)

SNL Program has Positive Impact on Post-Natal Checkup

www.brac.net

The Pilot Maternal, Neonatal and ChildHealth Project (MNCH) at Nilphamari:Profiling the Changes During 2006-07

Timeframe of study: 2006-2007 ↓ in induced abortion. However, self-treatment and treatment-seeking from informal

providers for complication of abortion remains high. Mean number of ANC visits ↑ with increased proportion attended by skilled providers

relative to baseline. Intake of iron tabs and TT vaccination increased. Majority of deliveries continued to take place at home. Though deliveries by trained

TBAs increased since baseline, the proportion delivered by the TBAs remained substantial.

For treatment of neonatal complications, mothers increasingly preferred UHCs/district hospitals.

Management of illnesses of the under-fives is fragmentary and the thrust is on prevention only.

www.brac.net

Sustaining Health: The Role of BRAC’s Community Health Volunteers in Bangladesh, Afghanistan, and Uganda

Timeframe of study: 2008-2009 Dropout rate of 12% low compared to similar programs internationally. Self-reported motivations for CHPs primarily related to financial incentives in all

settings except Afghanistan - women see this role as income-generating opportunity:• 75% of SSs reported that their income makes a big difference. In Uganda and

Afghanistan these numbers were 80% and 15%, respectively. 97% of SS in Bangladesh reported that being an SS gave them financial independence

BRAC is meeting the demand for medicines, but CHWs in all three countries felt competition from pharmacies, private clinics, and other providers.

Importance of increased Social recognition: 18% in Bangladesh, 11% in Uganda and 16% in Afghanistan

Health

www.brac.net

Sustaining Health: The Role of BRAC’s Community Health Volunteers in Bangladesh, Afghanistan, and Uganda

Health

www.brac.net

Health

Timeframe of Study: 2008-2009Cost effectiveness: In BRAC areas, cost was $64 compared to $96 in other areas. BRAC jointly expanded DOTS services to cover a population of 89.5 million,

with services mainly provided by CHWs In 2009, 96,427 patients were diagnosed, of them, 71,946 were new

sputum-positive Strong case-detection rate of ~80% and treatment success rate of 93%. CHPs fill critical gaps in human resources at community level; reduce delays

in diagnosis, bring doorstop DOTS service, and ↑ case-detection and cure rates

Impact of Community-based Tuberculosis Programme: Experience of BRAC

www.brac.net

The Bangladesh Miracle – Significant drop in mortality rates and increase in life expectancy

www.brac.net

The Bangladesh Paradox – Source: the Lancet

Success may be attributed to a multi-stakeholder pluralistic health system. Women-focused, equity-oriented, nationally targeted programs in family planning,

immunization, oral rehydration therapy, maternal and child health, tuberculosis, and vitamin a supplementation were critical success factors

Bangladesh’s approach to equity and its widespread deployment of female community health workers to bring essential health services to local households is noteworthy and a “testament to how many services can be provided when there is a scarcity of credentialled health professionals”

BRAC and The Bangladesh Paradox After the War of Liberation, the government created the space and donors provided the

financial support for NGOs like BRAC, which, as a group “have innovated to address issues or poverty, unemployment health education, and the environment, and in many cases, the government and NGOs have worked together to achieve a common goal.”

The national implementation of an oral rehydration therapy program by BRAC enables mothers to prepare homemade oral rehydration saline

Scaling up of innovations has been strengthened by investment in research, which has enhanced program design, monitoring & evaluation, and implementation

The Bangladesh Paradox: Exceptional health achievement despite economic poverty

www.brac.net

Microfinance

www.brac.net

The Wider Impacts of BRAC Poverty Alleviation Programme inBangladesh

Timeframe of Study: 1992-1995

↑ child survival and nutritional status in households. Protein-Energy Malnutrition (PEM) prevalence declined from 23% to

12%. Also improvements in expenditure patterns, per capita calorie

consumption, family planning practices and children’s education. BRAC households % using family planning was 57% v. 49.6% for

poor-non BRAC members. Violence against women initially increased when credit was

introduced but later tapered after other services like training were introduced.

Microfinance

www.brac.net

Child survival rate better for BRAC members

www.brac.net

An Impact Evaluation of BRAC’s Microfinance Program in UgandaTimeframe of Study: January 2008- May 2009

The study finds that BRAC’s program “seems to confer significant positive benefits to borrowers” including: • Increase in total savings (1.63 log change in savings) and assets

(0.76 log change)• Greater consumption in the form of more expensive and nutritious

food (increase of 0.3 meals per week consumed with fish)• The resources and incentives to start a household business

(Household business ownership increased by 27%) Results vary considerably depending on the statistical technique used.

This coupled with data issues makes it difficult to make causal claims. One year between baseline and follow up, may not be adequate.

Microfinance

www.brac.net

Ultra Poor

www.brac.net

Addressing Extreme Poverty in a Sustainable Manner: Evidence from Ultra-Poor Programme In Bangladesh

Timeframe of study: 2002-2008Key Findings: Positive long-term impact, four years after the end of the two-year support phase, in terms of per

capita income, employment, food security and asset holding 92% of the ultra-poor households satisfied at least 7 out of 10 indicators of extreme poverty in 2008

v. 64%.

Ultra Poor

Other Findings

Income/Employment: By 2008, participants’ per capita income had almost

doubled. The study calls this “remarkable,” because it “indicates that impact on per capita income was not only sustainable but also increased over time”

Among working-aged female participants, farm self-employment rose from 0.2% in 2002 to 37.0% in 2008, versus 0.3% to 11% in the control sample.

Positive Impact on Farm Self-Employment

www.brac.net

Addressing Extreme Poverty in a Sustainable Manner: Evidence from Ultra Poor Programme In Bangladesh (Cont.)Assets:

Financial assets increased sharply compared to the control sample, for years after the end of the 2-year program, attributable to a “dramatic” change in savings behavior & participation in the microcredit market (11, 24)

The percentage of women with cash savings increased from 8% at the start of the program in 2002 to 98% in 2008, versus 21% to 34% in the control group

Ultra Poor

Food security: By 2005, self-perceived food insecurity “had fallen tremendously” for participants, or 47%

compared to the 2002 baseline versus only 11% drop in the controlHousing & sanitation: By 2008, 94% of participants had roofs made of tin and 57% of participants had a sanitary

latrine, compared to 44% and 2% in 2002Education: The program “did not have significant impact on education in the short term” but “in the

long-term a modest positive impact on boys’ net primary enrollment was observed.”

www.brac.net

Ultra PoorAddressing Extreme Poverty in a Sustainable Manner: Evidence from Ultra-Poor Programme In Bangladesh

(Cont.)

www.brac.net

Can Basic Entrepreneurship Transform the Economic Lives of the Poor?

Timeframe of study: 2007-2011Cost of evaluation: ~$2 million

Annual income increased 38% over 4 years. Food security improved 18pp after two years, and 8pp after four years,

corresponding to a 39% and 18% increase from the baseline. After 2 years, self-employed women in the treated communities increased

20pp from the baseline, compared to only 6.2pp increase in control group. This difference further improved at 4 year point.

After 2 years: ↑ ownership of livestock despite being free to dispose of these assets. Chicken ownership ↑ 2.3pp, v. a .2pp increase in the control group. Most chickens were retained at the four-year point.

Businesses are successfully retaining and expanding livestock assets even as BRAC has withdrawn direct assistance.

Ultra Poor

www.brac.net

Change in Food and Nutrient Consumption Among the Ultra Poor: Is the CFPR/TUP Programme Making a Difference

Timeline of Study: 2001-2004Key Findings: A significantly higher % of TUP households consumed meat and fish regularly (>= 4 times/week), compared to control households. A relatively large number of TUP households never consumed eggs, milk or fruit, however, which has an impact on the overall nutrient intake.

Ultra-Poor - Summary from Andrew Jenkins

Impact Assessment of CFPR/TUP: A Descriptive Analysis based on 2002-2005 Panel Data

Timeline of Study: 2002-2005Key Findings: Participants are more likely to have savings, taken a loan, & be informed of laws. Selected

households more likely to possess assets such as cows & furniture. Savings and asset accumulation are precursors for ownership of land and significant

improvements in housing “The program has contributed significantly to improving the lives of the ultra poor and helping them

help themselves”

www.brac.net

Water and Sanitation

www.brac.net

Achievements of BRAC Water, Sanitation, & Hygiene ProgrammeTimeframe of study: 2006-2011

Key Findings: Sanitary latrine use increase: from the baseline (32%)

to midline (41%) to end line (53%) Hand washing ↑: Hand washing practice with soap at

‘critical times’ significantly increased from baseline to end line. At end line 97% of respondents had knowledge about hand washing with soap before eating and after defecation, and 22% and 88% reportedly did so at that particular time.

Reduction in water related diseases: The prevalence of water related diseases ↓ from 9.4% in the baseline to 7.1% in midline and 2.3% at the end line (see detail next slide).

Water & Sanitation

www.brac.net

WASH program significant reduction in water related disease prevalence

Water & Sanitation

www.brac.net

Achievements of BRAC Water, Sanitation, and Hygiene Programme (Cont.”)

Other Findings:

↑ in hygienic practice in terms of covering water vessels during transportation and storage. The improvement in knowledge and practice in drinking pure water could be a result of the health education provided by the WASH program.

Although the prevalence of water-related diseases significantly reduced amongst children over and under five-years-old, the prevalence of water-related diseases continued to be highly pronounced among children under five.

Evidence suggests it is likely that a substantial number of mother/caregivers do not wash their hands with soap/ash after fecal contact and before feeding a child.

Water & Sanitation

www.brac.net

Youth

www.brac.net

Social Network and Financial Literacy among Rural Adolescent Girls: Qualitative Assessment of BRAC’s SoFEA Programme

Timeframe of study: April 2011- September 2012Key Findings:

• ↑ in self-confidence and motivation and in strengthening the relationship between girls and their parents and within the community and improved socialization.

• After receiving Financial Literacy Training (FLT), SoFEA members were more financially aware and many were involved in Income Generating Activities, such as livestock and poultry rearing

• The fact that many of the girls are now supporting themselves financially has empowered them immensely, not only within their own families but also within the community as a whole.

• Concerns:– Time constraints for active participation—girls stopped attending regularly during exam

periods and harvest season. – Early Marriage The fact that rural Bangladeshi women tend to marry quite young poses a

socio-cultural challenge to the SoFEA program; after they marry they tend to drop out of school to focus on post wedding responsibilities and no longer maintain a wider network of friends.

– Father opposition: biggest opponents to club participation were the fathers. Girls mentioned during sessions that even some educated fathers forbid daughters from going.

Youth

www.brac.net

Empowering Adolescent Girls: Evidence from a Randomized Control Trial in Uganda

Timeframe of Study: 2008-2010

Cost Effectiveness of program: Cost of $17.9 v income gains of $26.7

Life skills: Condom usage ↑ 50% points among sexually active. Fertility rates were 29% lower compared to a control sample. Participants’ reports of having sex unwillingly decreased by 83% from the baseline – a

clear indicator of success in empowering youth girls. Girls in treated communities who agree that “females should earn money for the family”

increased by 18 pps from the baseline level of 37%.

Vocational Skills: The likelihood of being engaged in income generating activity ↑ 35% over baseline levels. Personal consumption expenditures ↑ 33% from the baseline. No adverse effect on schooling: The ELA program does not appear to have an adverse

effect on schooling investments made by participants.

Youth

www.brac.net