Using Mixed Methods to Measure and Monitor Empowerment in Projects and

Programs:Bangladesh, Ghana and Jamaica

Nora Dudwick (PRMPR)Jeremy Holland (OPM)

Gil Yaron (GYA)

PRMPRJune 23, 2008

2

Outline of presentation• Empowerment: what is it and why measure and

monitor it?

• Bangladesh: Measuring empowerment impacts of social safety net programs on women

• Ghana: Measuring empowerment outcomes on water, education and infrastructure beneficiaries of a rural CDD project

• Jamaica: Community-based monitoring of social policy outcomes, with a focus on youth-police relations

• Conclusions: Lessons learned and implications for monitoring empowerment

3

Why measure empowerment?

• The WDR 2000/1 identified “empowerment” as an important development objective

• “Empowerment” is closely linked to other corporate agendas of “social accountability,” and the “demand for good governance”

• Empowerment enhances people’s choices and opportunities

• While a legitimate goal in and of itself, evidence suggests that empowerment improves poverty reduction outcomes

4

Operationalizing empowerment

• Empowerment: the interaction of agency and opportunity structure

• Agency can be measured through proxies -- the material, financial, social, human, informational, psychological and other assets that people deploy to achieve their goals

• Opportunity structure: the formal and informal institutions (“rules of the game”) that constrain or facilitate people’s ability to exercise agency

5

Why use mixed methods?

• Access to most assets can be measured by indicators (but qualitative methods better at evaluating psychological, social assets)

• Institutional context can be only partially measured by indicators, and calls for qualitative methods

• Empowerment (as transformative choice) is difficult to measure quantitatively and benefits from a mixed method approach

• ME pilots combined questionnaires, focus groups, Community Score Cards, individual interviews, and rapid ethnography

6

Aim of Measuring Empowerment project

• Operationalize empowerment and identify indicators for specific project, program and policy contexts

• Go beyond looking at access and satisfaction (beneficiary model) to look at people’s ability to make choices and demand better services

• Pilot a cost-effective tool for producing timely feedback to projects, programs & policies and increasing the evidence base for policy makers

• Each pilot was undertaken in conjunction with ongoing in-country monitoring of specific project, program, or policy outcomes

• TFESSD activity, supported by Country Team and Government expertise in each country

7

Measuring empowerment: The challenge

• Meaning: Can we capture the essence of empowerment? Can we observe changes that are meaningful in both direction and magnitude (i.e. identifying whether a person or group is “more empowered” and “how much more empowered”)?

• Causality: Can we attribute cause and effect to a dynamic, relational and cross-sectoral phenomenon? How can we measure implicit choice which may not be observable (i.e. people may choose not to choose)?

• Comparability: Can we aggregate data so that conclusions about empowerment impacts and changes can be inferred for larger population groups?

8

BANGLADESH: Empowerment impacts of Social Safety Net Programs on women

9

Bangladesh - motivation

The Bangladesh PRSP emphasises the: – Role of Social Safety Net Programs (SSNP) in

reducing poverty – Need to focus on the empowerment of women

The substantial literature on women’s empowerment in Bangladesh does not treat SSNP in detail

It was possible to explore this issue by adding TFESSD funds to an existing JSDF-funded survey of SSNP being implemented by the Bangladesh Bureau of Statistics

10

Bangladesh - context

The aim of the survey was to measure the empowerment impacts on women of the following SSNP:

1. Food-for-work (FFW)2. Vulnerable Group Development (VGD)3. Primary Education Stipend program (PESP)4. Vulnerable Group Feeding (VGF)5. Old Age Allowance Scheme6. Allowance for Widowed, Deserted, and

Destitute Women7. Allowance for Distressed Disabled Persons

11

Bangladesh – methodology overview

FGD (72)

Household Survey (2741)

Women’s Survey (2741)

CommunitySurvey (69)

All: Propensity score matching

Beneficiaries: ordered logit

SSI analysis

12

Bangladesh – Empowerment indicators 1

Women’s questionnaire:• Control over assets (husband, self, joint, others)• Participation in village meetings and elections (&

if not, why not)• Participation in household decision making (h, s,

j, o) including joining organisations, economics & child related

• Autonomy (visiting & purchases) & domestic violence

13

Bangladesh – Empowerment indicators 2Household questionnaire empowerment module e.g.

Programs make: no difference | a small difference | a big difference | a negative difference

Being able to resolve disputes Membership of any local groups e.g. clubs or samittiesBeing able to choose who you vote for in electionsBeing able to complain to government officialsParticipating in development projectsBeing able to get clean waterAccess to news and informationBeing able to choose what work you doKeeping children in school

14

Bangladesh – PSM methodology

Unobserved determination & social capital SSNP Empowerment

Standard PSM approach

Biased results

Observable proxies on information & attending beneficiary meetings + factors affecting “lobbying”

SSNP Empowerment Unbiased results

Our PSM approach

15

Iterative relationship

Agency

Opportunity structure

Degree of empowerment

Development outcomes

Iterative relationship

Agency

Opportunity structure

Degree of empowerment

Development outcomes

Bangladesh – methodology, cont’d

We distinguish asset-based agency (such as improved self-esteem) from actual empowerment outcomes (such as greater autonomy or more decision making in the household)

We also distinguish between female-headed households in which the female heads are formally widowed, separated, divorced or abandoned, and those where women run the household without describing themselves in these terms

Source: Alsop, Bertelsen and Holland (2006)

16

Bangladesh – findings

1. SSNP modestly contribute to women’s economic empowerment

2. FFW appears to have less economic impact than do other programs

3. SSNP have a bigger impact on keeping children in school than on access to credit, land, water or electricity

4. Old Age/other allowances pay for education of children within the extended family & PESP is used in other areas -- e.g. allowances are fungible

5. PESP is popular but could be better targeted

17

Bangladesh – findings, cont’d1. FGDs in particular show that SSNP do enhance self-

worth and self-esteem, and increase women’s access to information.

2. But -- increased self-esteem and access to information did not translate into empowerment as measured by observable changes in behaviour. Behavior change depends on changes in the norms governing acceptable female behaviour. Should husbands be part of VGD capacity building?

3. Expansion of the Food-for-Work and Money-for-Work schemes was the most frequently voiced request from FGDs, although these programs have the least economic impact. How to design better workfare programs?

18

Bangladesh – findings, cont’dSSNP had little effect on women’s social or civic

empowerment (i.e. on autonomy, involvement in household decision-making and incidence of domestic violence).

Old Age Allowances and VGD may actually result in negative impacts, perhaps because increasing women’s economic assets triggers a conservative backlash from poor and poorly educated husbands. Further in-depth qualitative analysis is needed on this issue.

19

Bangladesh – findings, cont’d

1. The combination of FGD and quantitative survey techniques worked well. Ideally, the questionnaire design would have built on earlier FGDs

2. The Empowerment Framework conception of empowerment as the outcome of asset-based agency interacting with institution-based opportunity structure is useful.

3. It is important to focus on de facto female headed households, not only those which identify themselves as “widowed, separated, divorced and abandoned”.

20

GHANA: Assessing empowerment effects of the Community Based Rural Development Project

21

Ghana - motivation

The Community Based Rural Development Project (CBRDP) aims to empower rural populations by supporting the decentralization process, and more specifically, by improving delivery of rural water, health and education infrastructure

We wanted to pilot measures to test:1. CBRDP impact on people’s knowledge about local

government and ways of demanding better service – social accountability agenda

2. Quantitative indicators of empowerment that could contribute a “demand-side” component to the new Functional Organizational Assessment Tool (FOAT) designed to measure local government capacity

22

Ghana - context

CBRDP components include capacity building for local government officials to deliver better infrastructure, & training for beneficiary communities to effectively manage it

In some CBRDP communities, work was carried out with communities on participatory planning under the Rapid Results Initiative (RRI).

A Beneficiary Assessment planned for the mid-term review presented an opportunity to integrate measures of empowerment

23

Ghana – methodology overview

CSC (6)

KI interviewsfrom tracking

study (110+153)Ordered logit & otherquantitative analysis

CRC (927)

PAM consult qualitative analysis

CSC – Community Score CardCRC – Citizen’s Report CardKI – Key Informant

24

Ghana – Citizens’ Report Card

CRC covered 927 households from 49 communities:1. CBRDP communities without Rapid Results Initiative (RRI)2. CBRDP communities from the same district with the RRI 3. A control group of non-CBRDP (but otherwise comparable)

communities

All respondents were asked questions on governance with sub-samples asked about education, health, water or roads.

Five point Likert scale questions were used to capture perceptions of service quality (e.g. very dissatisfied through to very satisfied)

25

Examples of empowerment questions in CRC

Did your wards experience any problem with this school in the last academic year? – 8 choices

If so, what did you do about your major problem)1. Nothing, I do not know who to approach2. Nothing, no one could solve this problem

3. We tried to overcome this ourselves as best as we could

4. Went to see:List of possible representatives

Were you satisfied with the solution?

1. Yes 2. No, nobody tried to help

3. No, they tried but I was not satisfied

26

Ghana – Community Score Card and Key informant interviews

CSC undertaken in 6 representative districts:1. Community members in an open meeting2. District Assembly officials as service providers3. Interface meetings with community, DA officials

and elected members

Information, voice & negotiation issues emerged in community and interface meetings

Tracking study of officials trained by CBRDP. Empowerment issues emerged in key informant interviews undertaken as part of the tracking study

27

Ghana – education findingsCBRDP school construction projects (e.g. latrines) significantly raised parent satisfaction with education

Both qualitative & quantitative analysis tell a similar story:• Parent satisfaction with services was enhanced by more direct involvement in educational matters.

• If the PTA functions, parents were much likelier to be satisfied with their children’s education

Looking at empowerment issues produced practical feedback for CBRDP. Extra value can be added byinforming and involving parents (RRI, parental involvement via PTA to improve discipline)

28

Ghana – empowerment findings

There was only limited empowerment of consumers to resolve problems of service delivery. For both CBRDP and control groups:

•Health facility users didn’t know where to go to address problems. In education, even parents who did know where to address their concerns failed to do so.

•The qualitative data suggested that parents failed to pursue complaints because previous complaints were either ignored or not addressed to their satisfaction.

29

Ghana – decentralization findingsOnly one-fifth had a good understanding of District Assemblies (the highest level of local government)

There was similar or even greater levels of ignorance about the right to attend local government meetings, see meeting minutes, and know the content of the budget.

Men from wealthier households and with greater levels of schooling were generally more knowledgeable than poorer groups, women, and people living in the Eastern Region. Will the poor lose out from decentralization?

Local people need more information to communicate effectively with their DAs to secure better services

30

Ghana – RRI findingsThe Rapid Results Initiative (RRI) procedures had interesting empowerment impacts:

They promoted greater accountability, transparency, ownership and effective management of associated sub-projects and will be adopted more widely.

Their success suggests that community-driven development can result in greater empowerment and accountability in connection with the project that is being implemented.

At the same time, evidence suggests that this success can cause local people to engage less with their existing Unit Committees.

31

Ghana – policy monitoring findings

The indicators tested by the pilot could be used to monitor national processes as well. Specifically, they can provide a demand-side to the FOAT – to monitor decentralization

The score card process itself was empowering. There is a need to establish structured consultation between consumers and service providers as part of a regular process of interaction, not as an ad-hoc event.

32

JAMAICA: Community-BasedPolicy Monitoring

33

Community-Based Policy Monitoring in Jamaica

Objective

To evaluate social policy execution, with a focus on the relationship between youth and service providers, in particular the police

34

Jamaica: Methodology

A mixed-method approach applied in three communities: Community Score Card:

– An interactive monitoring tool that generates quantitative and qualitative data

– 5 empowerment indicators (perception scores of availability and exercise of choice) added to existing score card instrument

• Peer ethnography– In-depth analysis of underlying power relations by

“peer researchers” looking at: Identity, support and authority; and police-youth relations.

35

Community Score (Scale: 1= Very poor; 2= Poor; 3= Fair; 4= Good; 5= Excellent).

Indicator Harasson Gardens

Poyuton Terrace

Coolblue Gap

Violent, poor urban

Stable, poor urban

Poor rural

Original indicators

Level of trust youth have in the police 1 4 4

Level of respect and courtesy displayed by the police

2 5 5

Level of fairness displayed by police 1 4 4

Level of responsiveness of police 3 3 2

Level of effort made by police to interact with the youth

2 5 3

Additional empowerment indicators

Level of youth access to information about police activities and services

3 5 1

Level of youth willingness to use police services (e.g. reporting incidents)

4 5 4

Ability of youth to officially complain about inappropriate police behaviour / action

5 5 2

Level of youth willingness to officially complain about inappropriate police behaviour / action

1 4 4

Level of youth hope that police-youth relations can improve

2 5 5

36

Jamaica: Findings and Recommendations

•Social policy needs to address the agency of undervalued youth: Amongst poor urban and rural communities, youth have very little power in the presence of adults

•Social policy needs to rebuild relevant institutions: Within inner city communities, in particular, young people have lost faith in traditional institutions, turning to self-reliance and alternative street/ gang institutions

•Social policy needs to be geared towards employment generation to break the cycle of gun and alternative authority structures

•While schooling is still perceived as fundamentally important institution in framing the life choices of young people, gender is becoming an increasingly significant factor in influencing choices and outcomes

•Police-youth relations are at the centre of outsider-insider contact, and are problematic but not intractable

37

Measuring empowerment in Jamaica: Reflections

•The application of a mixed method diagnostic tool allowed the evaluation to look at the problems affecting everyday service provider-user interaction, but also to delve deeper to examine the social structural issues of class, gender and social hierarchy.

•Rather than simply tinkering with policy implementation at the interface of police-youth relations (a kind of “empowerment lite” approach), the tool can generate more “upstream” and cross-sectoral social policy analysis.

38

Conclusions and recommendations

39

Cross-cutting findings

• Measuring empowerment is important where objective is pro-poor policy or social inclusion

• Empowerment measurement methodologies can themselves be empowering locally while energizing macro-level policy discussions

• Empowerment indicators can be integrated into existing surveys but care is needed over choice of indicators when aggregating

• Mixing methods results in better, more nuanced information that yields policy-relevant insights on power relations underpinning everyday interactions

• Mixed methods result in different types of evidence that appeal to diverse stakeholders

40

Cross-cutting recommendations

• Work with local stakeholders to identify empowerment indicators

• Include empowerment indicators in ongoing M & E exercises

• Actively support use of appropriately sequenced mixed methods

• Link empowerment measurement with policy interventions that focus on assets and institutions

• Track impact of reforms that change relations between citizens and officials (service provisions, decentralization, etc.) where disempowerment of poor can be unintended consequence