Participation, values, and implementation: three research challenges in developing gender-sensitive indicators

This article was downloaded by: [Florida State University]On: 12 November 2014, At: 23:07Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954 Registered office: MortimerHouse, 37-41 Mortimer Street, London W1T 3JH, UK

Gender & DevelopmentPublication details, including instructions for authors and subscription information:http://www.tandfonline.com/loi/cgde20

Participation, values, and implementation: threeresearch challenges in developing gender-sensitiveindicatorsTessa Hochfeld & Shahana Rasool BassadienPublished online: 29 Aug 2007.

To cite this article: Tessa Hochfeld & Shahana Rasool Bassadien (2007) Participation, values, and implementation:three research challenges in developing gender-sensitive indicators, Gender & Development, 15:2, 217-230, DOI:10.1080/13552070701391516

To link to this article: http://dx.doi.org/10.1080/13552070701391516

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) containedin the publications on our platform. However, Taylor & Francis, our agents, and our licensors make norepresentations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose ofthe Content. Any opinions and views expressed in this publication are the opinions and views of the authors,and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be reliedupon and should be independently verified with primary sources of information. Taylor and Francis shallnot be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and otherliabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to orarising out of the use of the Content.

This article may be used for research, teaching, and private study purposes. Any substantial or systematicreproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in anyform to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions

http://www.tandfonline.com/loi/cgde20

http://www.tandfonline.com/action/showCitFormats?doi=10.1080/13552070701391516

http://dx.doi.org/10.1080/13552070701391516

http://www.tandfonline.com/page/terms-and-conditions

http://www.tandfonline.com/page/terms-and-conditions

Participation, values, and implementation:

three research challenges in developing

gender-sensitive indicators

Tessa Hochfeld and Shahana Rasool Bassadien1

This article critically reviews three key research challenges that emerged as a result of

developing gender-sensitive indicators for an NGO in South Africa. The first

challenge faced was finding an appropriate balance between an ‘expert-led’ and an

inclusive participatory research process. The second challenge was identifying a set of

criteria, based on explicit research values, to guide indicator development. Third, we

were confronted with the task of ensuring that the indicators developed could be

implemented in the context of a resource-poor, service-driven NGO setting.

A discussion of the challenges faced exposes the complex and sometimes murky

process of developing indicators that are relevant, appropriate, and usable.

Introduction

In recent years, it has been recognised that introducing policy and programmes that

address gender equality without concomitant monitoring and evaluation is inade-

quate. Gender-sensitive indicators are tools commonly used in monitoring and

evaluation to assess gender-related changes in society over time, and provide direct

evidence of disparities between women and men in a particular society (CIDA 1997;

Derbyshire 2002; SDC undated). They are a vital means of monitoring the impact of

programmes on gender relations and roles, and in this way contribute to filling the

frequent gap between abstract policy commitments to gender equality and a coherent

and comprehensive process of gender mainstreaming (Moser and Moser 2005).

In South Africa, non-governmental and community-based organisations are facing

increasing pressure from funding bodies to develop gender-sensitive indicators to

demonstrate programme value, as well as to evaluate programme effectiveness. While

the international literature on gender-sensitive indicators can help organisations

understand the value of indicators and can guide them in a general way, the actual

process of indicator development can be overwhelming. It is a challenge to develop

gender-sensitive indicators that are specific, relevant to the particular services offered

by an NGO, and responsive to unique local conditions.

Gender & Development Vol. 15, No. 2, July 2007ISSN 1355-2074 print/1364-9221 online/07/020217� 14 – Oxfam GB 2007

DOI: 10.1080/13552070701391516

217

Dow

nloa

ded

by [

Flor

ida

Stat

e U

nive

rsity

] at

23:

07 1

2 N

ovem

ber

2014

The process of developing gender-sensitive indicators is a process of applied

research, as data need to be collected and analysed. In this paper we discuss and

critically reflect on three issues that presented themselves as challenges during

monitoring and evaluation research that we were contracted to carry out, as part of the

development of gender-sensitive indicators for a social service NGO, the Pietermartiz-

burg branch of the Family and Marriage Society of South Africa (FAMSA)2. The first

challenge was resolving the tension between an ‘expert-led’ and a participatory

research approach. The second challenge was deciding on the criteria used to develop

the indicators. We argue that as the indicators are a direct result of the particular

research decisions and choices that are made, and as this process is a complex one,

these decisions should be influenced by a conscious and acknowledged set of criteria

at each stage of the research process, rather than being allowed to arise in an arbitrary

ad hoc manner. The third challenge we identify is that the successful use of gender-

sensitive indicators to measure gender equality can be complicated by issues of

funding and resource allocation at the implementation stage.

Background

Context

FAMSA provides social services via a small core group of professional social work staff

and a large contingent of volunteers. The organisation’s services range from individual

and family therapeutic counselling to community development and community social

education. FAMSA is situated in a small city, Pietermaritzburg, in the province of Kwa-

Zulu Natal, South Africa. The city is an important urban centre for the surrounding

rural and peri-urban areas, and thus FAMSA (and its rural satellite offices) serve a

diverse set of communities across a wide geographical region, made up of individuals

from a variety of racial and ethnic groups, and of different socio-economic status.

Unemployment, poverty, violence against women, and HIV and AIDS deeply

permeate the social conditions of both urban and rural communities in this province.

While mechanisms promoting formal gender equality in South Africa are considerable,

the good protective legislation supporting women’s rights now in place is often

inaccessible to those who are poor, uneducated, or lack the social or cultural capital to

mobilise resources on their own behalf. Substantive gender equality is far from

achieved, particularly in relation to conservative gender norms and roles, and

excessive levels of violence against women.

Data collection/productionThe data used to develop indicators was collected (or rather, as feminists Ramazanoglu

and Holland (2002:154) suggest, ‘produced’) via a two-stage process. The first stage

involved speaking to FAMSA service providers (paid staff and volunteers). We

Gender & Development Vol. 15, No. 2, July 2007218

Tessa Hochfeld and Shahana Rasool Bassadien

Dow

nloa

ded

by [

Flor

ida

Stat

e U

nive

rsity

] at

23:

07 1

2 N

ovem

ber

2014

identified four social workers, whom we interviewed, and 20 volunteers, with whom

we ran two separate focus group discussions.

The questions we asked in this phase attempted to elicit participants’ under-

standing of gender, their perception of how service needs and service delivery are

gendered, and their perception of the different gendered experiences of beneficiaries.

We tried to avoid the use of difficult concepts such as ‘gender inequality’ in our

interviews by asking specifically if women and men had different access to resources

and opportunities, and whether gender influenced decision-making power. We also

asked about participants’ perceptions of how services could better meet the gendered

needs of clients.

The second stage involved talking to service users. Twenty-five clients and six

community leaders (three school teachers, one ward councillor, one Induna3, and one

pastor) were interviewed, using structured questionnaires. Community leaders were

interviewed because they play a key role in maintaining cultural norms and practices

with regard to gender roles and stereotypes within communities, and they all currently

or in the past had some form of contact with the organisation’s programmes. They were

asked about the impact of FAMSA programmes on gender relations in their community.

Clients were requested to evaluate FAMSA’s services particularly in relation to

whether they were identifiably gendered. Other questions were designed to elicit

information on gender-based violence and gendered roles in relation to childcare,

domestic duties, decision making, and family and intimate relationships.

We will now turn to the challenges we met during this research.

Research challenges

Using a participatory approachIt is entirely possible that gender-sensitive indicators can be developed based on

theoretical knowledge and a literature review. However, as gender-sensitive indicators

are programme-level, implementation-related tools, a technocratic, ‘one-size-fits-all’

approach is inappropriate for their development. Further, developing indicators

without engaging with those who will later on be asked to use them in their work,

or whose access to services may be affected by them, would be ethically questionable

for a social service organisation genuinely accountable to its constituency. There is a

growing body of literature (Cornwall 2003; Malhotra et al. 2002; Levy 1998; Mayoux

1998; Guijt and Shah 1998) that shows that taking an ‘expert’ approach to gender work

at the local level often means the practice is ‘top-down, superimposing particular

(culturally specific, some might suggest) frames of reference and barely allowing for

broader participation in agenda setting or implementation. A simplifying worldview is

thus projected onto diverse development situations’ (Cornwall 2003, 1326). We thus

chose a research process driven by stakeholder4 participation.

Gender & Development Vol. 15, No. 2, July 2007 219

Research challenges in developing gender-sensitive indicators

Dow

nloa

ded

by [

Flor

ida

Stat

e U

nive

rsity

] at

23:

07 1

2 N

ovem

ber

2014

‘Pure’ or ‘ideal’ participatory research is understood as a process in which there is

collective decision making by a team consisting of all stakeholders at every stage of the

research process (Collins 1998; Chambers 1997). This conception of participation, while

ideologically seductive to champions of participatory research, is not always realistic,

particularly when external consultants are asked to manage the process, as we were in

this case. For us, the tension was between ensuring participation, and not wanting to

demand too much of an overstretched organisation’s time and resources. However,

Estrella and Gaventa (1998) argue that for monitoring and evaluation research to be

considered participatory, the principles used are more important than the actual

research practices chosen. They suggest that if research is based on principles that

emphasise stakeholder participation, learning and reflection, negotiation and discus-

sion, and design flexibility, then it can be considered a participatory process. Our

research process was guided by the principles of participation, rather than being

entirely participatory.

In our approach, we ensured maximum participation in the data collection/

production phase of this process by choosing to use qualitative research methods with

all stakeholder groups. While this clearly links to the principle of participation, it also

resonates with the principles of negotiation, discussion, learning, and reflection.

Individual semi-structured interviews and focus group discussions provided oppor-

tunities for stakeholders to reflect on the complexity of gender dynamics in the

community, elicited the actual gendered experiences and perceptions of service

providers and service users, and allowed an engagement in constructive discussion

on necessary changes. This meant we did not, as ‘experts’, just assume what needed to

be measured, but based the indicators on areas emphasised by stakeholders.

While the balance of power in the data collection/production phase leaned towards

stakeholders, the opposite was true for the next phase. Geographical distance, time,

and resource constraints for both the researchers and the stakeholders meant there was

minimal stakeholder participation in the indicator development phase. We tried to

mitigate a privileging of ‘expert’ views both by relying on the information produced

during participatory interviews and discussions, and through the continued use of

participatory principles, particularly the reflective and discursive way in which we

approached the analysis of data. For example, it was only after the generation of the list

of 68 indicators that we coded the indicators thematically, and then grouped them into

12 thematic clusters. In essence, it was a bottom-up, reflective way of identifying core

nodes around which changes in gender relations occur for FAMSA stakeholders in the

community and in the organisation. This also prevented us from focusing only on

women, which is a common but very limiting mistake in work around gender

(Cornwall 2003). While some indicator clusters deal exclusively with women, such as

‘empowerment of women’, and some deal exclusively with men, such as ‘men’s power

and control’, the majority deal with community processes, which, when measured in a

gender disaggregated way, will indicate changes in gender inequality. For example, the



Dow

nloa

ded

by [

Flor

ida

Stat

e U

nive

rsity

] at

23:

07 1

2 N

ovem

ber

2014

indicator cluster ‘leadership and participation’ includes indicators such as ‘men

increasingly allowing opportunities for women to speak in public/community contexts’.

Nonetheless, although we were very committed to reflecting the community’s

actual experiences and concerns, it is disingenuous to suggest that as researchers we

had no authority over the list of indicators chosen. We are not claiming that the voices

of clients and staff can be directly ‘heard’ through these indicators, as no research is

unaffected by the choices of the researchers themselves (Jackson 2002; Cornwall 2003;

Guijt and Shah 1998; Estrella and Gaventa 1998). What we are sure of, however, is that

a discursive, reflective approach based on participatory principles helped enormously

to ensure significant stakeholder influence on the final product.

Strategic criteria influencing indicator development

Gender sensitive indicators need to be able to measure shifts in gender relations within

the specific peculiarities of the local context. One of the core challenges in developing

indicators is identifying the actual criteria for monitoring and evaluation in a context

of infinite possibilities (Moser & Moser 2005); in other words, deciding on the

framework that will guide choices made in indicator development.

Three strategic and theoretical criteria influenced the development of indicators in

this research process, namely: (1) a feminist and social justice approach; (2) the need to

measure outcome, not just input; (3) balancing qualitative and quantitative indicators.

Feminist and social justice approachIn this research, a feminist ideology and social justice approach shaped our knowledge

base, and was openly declared to FAMSA when the contract was negotiated.

Feminist epistemology is a branch of feminist theory which argues that ‘truth’ is

strongly shaped by our social conditions, and that the idea of value-free knowledge is

a false construct, biased towards the male experience of the world. While approaches

to knowledge and its production continue to generate many internal disagreements

within feminisms, in general, feminist epistemology attempts to raise the volume of

excluded voices, specifically women’s, and ensure that knowledge distortions that

erase women’s experiences as legitimate are unpicked and investigated (Longino 1993;

Gouws 1996; Nielsen 1990). This lens influenced indicator development in this project

in that we strove to promote gender equality and the emancipation of women by

ensuring that women’s needs and experiences were legitimised and clearly heard, and

that each decision that was made was weighed carefully to ensure no assumptions

about gender ‘neutrality’ were being made. At the same time, we concentrated on

gender relations, rather than focusing solely on women, which could have had the

effect of disregarding men’s gendered identities.

We specifically chose to combine participatory and feminist research methods as

these approaches share ‘epistemological, ethical and political principles’ (Cornwall



Dow

nloa

ded

by [

Flor

ida

Stat

e U

nive

rsity

] at

23:

07 1

2 N

ovem

ber

2014

2003, 1326). Used in an informed and ethical way, these approaches can complement

and support each other in ways that can lead to more inclusive and transformative

practice (Cornwall 2003).

In addition, we were guided by a social justice approach which contends that class,

gender, and race are forms of structural disadvantage, and ‘any social or political [or

research] programme which does not specifically question or challenge them is likely

to reinforce these forms of oppression by accepting the dominant order which

supports them’ (Ife 1999, 55). This meant that we analysed comments within a

framework that attempted to ensure that social injustices were not inadvertently

reproduced by the indicators developed. We did this primarily by privileging

community needs rather than donor requirements, as sometimes the latter are

disconnected from important local issues, and can reinforce the existing status quo

by ignoring the specificities of contemporary and indigenous gender relations.

In an example of how this approach functioned in practice, when six of the nine

male clients interviewed claimed that their partner beat or hit them, we did not

literally and uncritically mirror the comments made by these research participants. On

closer investigation we learned that a number of these same men were attending

counselling because they had been ordered to do so by the court, after they had

physically abused their partners. It is possible that these men felt persecuted

themselves because their wives obtained court orders following serious abuse.

Experience from across the world shows that advances in rights and services for

women, particularly those relating to abuse, often attract a reactionary response.

Claims may be made that men are suffering persecution and disadvantage, or

criticisms may be voiced regarding the perceived lack of focus on men (Chant and

Gutman 2000). This response can be partly explained by the lack of acknowledgement

of the substantive disadvantages women continue to face in society, especially with

respect to being the primary victims of violence in the home (Chinkin 1995). Hence

using our ideological base to guide data interpretation led to indicators that measure

the development of problem-solving skills in relationships, rather than simply

measuring a decrease in the number of men reporting experiences of abuse. For

example, one indicator states: ‘men feel increased comfort and skill in using problem

solving skills and verbal, honest and open communication instead of violence or

control in their relationships’.

In our report to FAMSA we clearly articulated the reasons behind these decisions.

In addition, we recommended some research into the incidence of abuse of men in the

communities in which FAMSA works, as ignoring the possible existence of this group,

with serious needs and who are commonly marginalised in service delivery, would be

undesirable and unethical. Hence, in this case it was not just a matter of us

‘suppressing’ data that we ‘didn’t like’, but rather finding ways to ensure that

indicators developed do not serve to reproduce inequalities, while still serving

community needs.



Dow

nloa

ded

by [

Flor

ida

Stat

e U

nive

rsity

] at

23:

07 1

2 N

ovem

ber

2014

Measuring outcome not input

It is very common in monitoring and evaluation processes for assessment to focus on

input indicators rather than focusing on outcomes (Moser & Moser 2005). But if

indicator development is aimed at the substantive improvement of gender equality

between women and men, then measuring programme outcome (rather than input or

output) is both a political and ethical imperative. Instead of only measuring the

resources or effort that have gone into programmes (an input indicator), or the number

and proportion of women beneficiaries (an output indicator), one needs to measure the

actual improvement in parity between women and men; that is, the actual impact on

real people’s lives.

There are, however, some inherent difficulties in measuring outcome in the context

of gender analysis. The most significant is that changes in gender equality usually only

occur via a long, complex, iterative process, influenced by innumerable social,

political, cultural, and economic forces. As such, changes can seldom be attributed

to the impact of one isolated programme or service (Kabeer 1994). For example, it is

quite difficult to attribute ‘increased safety of women in the context of their homes’, or

a ‘decrease in men’s discomfort and reprisal when women transgress gender norms’

(which are both outcome indicators) solely to the work of one organisation, rather than

to more generic social and cultural changes. However, the work of the organisation

may contribute to these changes in a particular community through some targeted

interventions. Thus, in this context, we needed to balance the outcome measurement

imperative with the pragmatism of choosing indicators that are actually useful

measurements of an organisation’s performance.

This tension was minimised in the development of gender-sensitive indicators for

FAMSA by ensuring that we included outcome indicators that are within the

organisation’s direct sphere of responsibility, as well as those that are only indirectly

influenced by the organisation’s activities (SDC, undated). For example, measuring

‘women’s improved ability to raise issues in couples counselling’ is an outcome

indicator that is within the organisation’s direct sphere of influence.

On the other hand, indicators such as ‘women’s increased financial independence’

and ‘increased gender role fluidity between women and men in communities’ measure

outcome in a sphere only indirectly influenced by the organisation. While the

organisation can develop programmes that help women obtain jobs and interventions

that encourage changes in gender roles, these are more strongly influenced by a variety

of economic, social, and cultural factors that would be difficult for one small

organisation to influence.

Snyder et al. (1996, 1482) recommend that employing long-term planning as well as

‘adopting an external perspective that emphasizes anticipating and shaping environ-

mental change to achieve the organisation’s strategic plans’ may improve an

organisation’s ability to impact on gender outcomes. Hence, our implementation

plan for FAMSA suggested a long-term strategic approach to indicator implementa-



Dow

nloa

ded

by [

Flor

ida

Stat

e U

nive

rsity

] at

23:

07 1

2 N

ovem

ber

2014

tion. Moreover, FAMSA was encouraged to be aware of the current local socio-cultural

context in the implementation of indicators.

Balancing quantitative and qualitative indicators

Significantly less attention has been paid to the development of qualitative as opposed

to quantitative gender-sensitive indicators in the literature. This is partly because the

large volume of literature on the development of gender-sensitive indicators is

dominated by discussions of macro-level quantitative indicators, that measure changes

at the national level such as maternal morbidity, political representation in parliament,

or gender-sensitive policy development (see Alsop & Heinsohn 2005; Saith & Harriss-

White 1998; Beck 1999; Knowles et al 1997; UNESCO 1997). Quantitative indicators also

dominate because the ‘gender impact’ of a concrete service is relatively easy to

quantify. It is relatively easy, for example, to establish the number of women as

opposed to men granted housing, or the improved gender balance in the numbers of

those who use a new rural transportation system, or the number of houses connected

to mains electricity services in a particular area (which will impact positively on

women’s domestic responsibilities).

Quantitative indicators are generally unambiguous and, with the correct data,

measurement is straightforward and easy. An example of this type of indicator as

developed for FAMSA is ‘increased numbers of women taking cases to community or

religious authorities’. This quantitative indicator requires the institution of a simple

system of counting that can be done quite easily if there is cooperation from

community authorities and FAMSA volunteers.

However, quantitative indicators have inherent weaknesses. For example, ‘in-

creased numbers of women in leadership positions, particularly at the local level’ is a

quantitative indicator developed for FAMSA which measures levels of women’s public

participation. But this type of indicator is not able to measure the quality of

participation or level of influence women have when participating, which is of crucial

importance to substantive gender equality (Moser and Moser 2005; Cornwall 2003;

Botchway 2001). Instead, measuring the quality of participation is better captured in a

qualitative indicator such as ‘women’s improved self-confidence and capacity to talk

in appropriate public and private settings’.

Social work services in particular do not easily lend themselves to useful

quantitative measurements: measuring the success of services that aim to strengthen

ailing marriages, that contribute to emotional healing after trauma, that comfort those

grieving, and that promote women’s empowerment, is a complex and sensitive

process, and is not merely a matter of counting the number of women or men who

receive the service. Although the usefulness of quantitative indicators is undeniable,

Jackson (2002, 504) argues that,



Dow

nloa

ded

by [

Flor

ida

Stat

e U

nive

rsity

] at

23:

07 1

2 N

ovem

ber

2014

a great many research questions posed by gender analysis do demand in-depth methods of

study . . . One obvious example is the area glossed as ‘decision making’. Heroic efforts may be

made to produce simple indicators on this . . . but they produce little of value because of the

complexity of the interactions called ‘decision making’ which can rarely be reduced to anything

cut and dried.

Hence, to be useful in social service organisations, gender-sensitive indicators need to

lean more towards qualitative indicators and need to measure micro- and meso-level

processes. It is for this reason that we consciously developed qualitative indicators to

assist in measuring micro changes in gender interactions, such as ‘women’s decision-

making in the family is increasingly open and unaccompanied by fear of resistance

and/or retaliation’.

Qualitative indicators are particularly suited to measuring women’s empower-

ment, especially when focused on micro-level processes that impact on individuals,

couples, and families. The intention is to measure improved gender equality in the

everyday lives of women, such as ‘increased gender balance in sharing of unpaid

housework and child-care’, rather than focus on important, but less immediate macro-

level issues for FAMSA, such as improved work opportunities for women in non-

traditional fields.

The principal disadvantage of qualitative indicators, however, is that they are

notoriously difficult to measure. It is often only via self-reporting from an individual

woman herself that an empowerment indicator such as the following can be measured:

‘women report feeling more competent, capable and empowered’. However, relying so

heavily on service-user’s self-reports heralds profound complexities for implementa-

tion and demands time, patience, and a bottom-up, systematic, and methodical

approach to measuring impact (CIDA 1997; Derbyshire 2002; SDC undated). While this

seems overwhelming, the organisation can account for these subtle changes through

careful and consistent monitoring and evaluation in the counselling, training, and

development process, if it is clear about what to look for.

We thus tried to maintain the right balance between quantitative and qualitative

indicators not only to ensure that a range of processes and issues are captured, but also

to be realistic about what it is possible to measure in a small, resource-poor service

NGO. However we also recognised that some indicators that we felt were vital to

measuring gender equality would nevertheless be difficult to implement for a number

of reasons, expanded on below.

Implementation: measurement and resource concerns

It is not just the development of gender sensitive indicators that involves a process of

research, but also the actual implementation of these monitoring and evaluation tools,

as measurement of change requires data collection/production and analysis. In the

‘messy world of project implementation’ (Gow and Vansant in Botchway 2001, 139),



Dow

nloa

ded

by [

Flor

ida

Stat

e U

nive

rsity

] at

23:

07 1

2 N

ovem

ber

2014

the way gender mainstreaming processes and tools are introduced, negotiated, and

executed will have an enormous impact on their success or lack thereof in any given

NGO. The existence of measurement tools, however good they are, will pay nothing

more than lip service to gender equality unless genuine commitment to the goals of

gender mainstreaming and indicator implementation is as universal as possible within

that organisation. If the process of indicator development and implementation is

perceived as beneficial only to donors, ‘foreign’, inappropriate, or unimportant, then

resistance is inevitable (Wendoh and Wallace 2005). Even if commitment to gender

equality is genuine and widespread in an organisation, two key issues can stymie

successful implementation, namely measurement difficulties and resource limitations.

Measurement difficultiesMeasurement challenges abound with the use of gender-sensitive indicators, but the

most basic is the lack of baseline data. That is, you cannot identify a decrease or

increase in a type of behaviour or phenomenon unless you can identify the level at

which this type of behaviour or phenomenon currently occurs (Derbyshire 2002; Saith

and Harriss-White 1998). This is true for both traditionally measured quantitative

indicators and more complex qualitative indicators. Thus, in theory, for each indicator

chosen, the organisation needs to first measure the starting or current level. In practice,

however, baseline measurement can be lengthy and complicated (Estrella and Gaventa

1998), and thus baseline estimation which is done with the participation of the

beneficiary group is a more realistic goal. Change attributed to the programmes or

services offered can then be measured against this baseline estimate, using the

indicator.

A more profound challenge lies with the difficulties in measuring changes in power

and status, particularly important indicators in work around gender relations. Despite

extensive debate in this area, no consensus exists on how to accomplish this. A

sophisticated conceptualisation focusing on ‘agency’, multiple indicators, triangula-

tion, and combining qualitative and quantitative indicators have all been suggested as

ways to deal with this challenge (Moser and Moser 2005; Kabeer 1999; Jackson 2002;

Malhotra et al. 2002). But the more complex and difficult measurement is, the less likely

it is that a small NGO will go to the trouble of undertaking this task. Thus NGOs in

consultation with experts need to reach consensus on a method of measuring elusive

and imprecise, but exceptionally significant, social processes to provide baseline

estimates, prior to indicator development and use.

Resource limitations

Financial support, time, and gender expertise are the three pivotal resources key to the

success or failure of gender-sensitive indicator use. First, it is crucial that organisations



Dow

nloa

ded

by [

Flor

ida

Stat

e U

nive

rsity

] at

23:

07 1

2 N

ovem

ber

2014

themselves realise that some of their own financial resources must be allocated to this

process. Or, if funders create the imperative for organisations to develop gender-

sensitive indicators, they need to provide the financial resources for implementation.

Second, while organisations may be able to implement many of the indicators

themselves, implementation cannot be ad hoc or done as an extra task in the daily

process of the organisation. Indicator use takes time, and in a busy social service

organisation this can often mean taking time and resources away from service delivery.

This tension needs to be resolved if success is to be achieved.

Third, working with a gender expert could provide strategic direction, ensure

efficiency, and contribute to greater success in implementation, since personnel may be

overwhelmed by the task of implementing a wide range of indicators in the context of

limited time and specialised knowledge. However, expertise on its own is not a

guarantee of effective implementation without significant resource adjustments.

Crucially, an organisation must be realistic about what can be achieved with the

available resources. While the 68 indicators developed for FAMSA are a comprehen-

sive reflection of gender issues in the community, it is entirely unrealistic to expect the

organisation to use all indicators at once. Instead, we recommended a system of

prioritisation and staggered implementation.

Conclusions

Three distinct research challenges emerged as a result of developing gender sensitive

indicators to monitor and evaluate FAMSA programmes. First, we decided to embark

on a participatory process to ensure that the indicators were relevant, appropriate,

context driven, and genuinely reflected the gender issues identified by the community

and organisation. But the level of participation was challenged by time and resource

constraints, encouraging a less than ideal ‘expert’-led approach which we problema-

tise. Resolving this tension was assisted by the use of participatory principles even

when engaging in less participatory research practices.

Second, developing indicators is a complex process that is best guided by a set of

criteria based on the political orientation of the researchers. Feminist research theory

has argued that, rather than decreasing so-called scientific rigour, the ‘explicit, self-

conscious application of values within scientific practice’ (Gergen 1988, 92) can

substantially decrease inadvertent male bias. In this case, a political orientation

informed by a feminist and social justice approach was also one of the criteria used in

developing indicators. The other two criteria, namely a focus on outcome, and

balancing quantitative and qualitative indicators, were chosen for their capacity to

assist in measuring substantive changes in gender equality. Hence, these three criteria

for selecting gender-sensitive indicators were value-driven and designed to ensure a

gender sensitive monitoring and evaluation process that has the potential to be

transformative.



Dow

nloa

ded

by [

Flor

ida

Stat

e U

nive

rsity

] at

23:

07 1

2 N

ovem

ber

2014

The third challenge discussed in this paper is the extent to which indicators can be

implemented in resource-poor, service-focused organisations. Indicator implementa-

tion is a cornerstone of monitoring and evaluation, since an inability to implement

indicators would mean that organisations are unable to measure shifts in gender

relations in the communities where they are working. Gender-sensitive indicators that

cannot be implemented are therefore worthless. Not only do organisations and funders

need to collect/produce baseline data against which to measure indicators, but if they

are serious about promoting gender equity, they must move away from piecemeal,

inconsistent, and patchy approaches to resourcing gender mainstreaming, by

accounting for implementation complexities in their planning and funding allocations.

The seemingly technical task of developing gender-sensitive indicators is in reality

fraught with challenges and contradictions. We contend that developing indicators

that will contribute to transformative practice and an increase in gender equality is

only possible using an approach that is honest, flexible, strategic, and driven by

feminist social justice values.

Tessa Hochfeld has been an activist, researcher, academic, and social work practitioner in the gender

field for many years, with a particular focus on gender-based violence. She is currently a researcher at

the Centre for Social Development in Africa, University of Johannesburg, where her work is focused

on gender, poverty, social development, and social policy. Email: [email protected]

Shahana Rasool Bassadien is a social worker by profession. She has been an activist in the gender

field through her work as a consultant, trainer, and researcher. Shahana has a Masters in

Comparative Social Research (specialising in social policy) from the University of Oxford. She is

currently coordinating the Honours Social Work Programme at the University of Johannesburg.

Email: [email protected]

Notes

1 Thank you to the director, staff, volunteers, and clients of FAMSA Pietermaritzburg for

permission to use the data for this article and for working with us so co-operatively and

openly. Thanks also to Terry Sacco for very helpful comments on an earlier draft of this

paper.

2 When using the acronym FAMSA we refer specifically to this branch, and not any other

of the numerous FAMSA offices across South Africa.3 Indunas are important tribal leaders amongst Zulu people. They wield significant local

power and report directly to, and can strongly influence, their local area Nkosi (chief).

4 By ‘stakeholders’ we are referring to the staff and volunteers of the organisation,

recipients of services, and representatives from the wider community, all of whom have

a stake in the effective delivery of FAMSA services.



Dow

nloa

ded

by [

Flor

ida

Stat

e U

nive

rsity

] at

23:

07 1

2 N

ovem

ber

2014

References

Alsop, R. and N. Heinsohn (2005) ‘Measuring Empowerment in Practice: Structuring

Analysis and Framing Indicators’, World Bank Policy Research Working Paper 3510,

Washington: World Bank

Beck, T. (1999) ‘Using Gender-Sensitive Indicators: A Reference Manual for Governments

and Other Stakeholders’, London: Commonwealth Secretariat

Botchway, K. (2001) ‘Paradox of empowerment: reflections on a case study from Northern

Ghana’, World Development 29(1): 135�/53

Chambers, R. (1997) Whose Reality Counts? Putting the First Last, London: Intermediate

Technology Publications

Chant, S. and M. Gutman (2000) Mainstreaming Men into Gender and Development: Debates,

Reflections and Experiences, Oxford: Oxfam GBChinkin, C. (1995) ‘Violence against women: the international legal response’, Gender and

Development 3(2): 23�/8

CIDA (1997) Guide to Gender-Sensitive Indicators, Hull, Canada: Canadian International

Development Agency (CIDA)

Collins, K. (1998) Participatory Research: A Primer, Cape Town: Prentice Hall

Cornwall, A. (2003) ‘Whose voices? Whose choices? Reflections on gender and participa-

tory development’, World Development 31(8): 1325�/42

Derbyshire, H. (2002) Gender Manual: A Practical Guide for Development Policy Makers and

Practitioners, London: Department for International Development (DFID)

Estrella, M. and J. Gaventa (1998) ‘Who Counts Reality? Participatory Monitoring and

Evaluation: A Literature Review’, IDS Working Paper 70, Brighton: Institute of

Development Studies

Gergen, M.M. (1988) ‘Toward a feminist metatheory and methodology in the social

sciences’, in M. M. Gergen (ed.) Feminist Thought and the Structure of Knowledge, New

York: New York University Press

Gouws, A. (1996) ‘Feminist epistemology and representation: the impact of post-modern-

ism and post-colonialism’, Transformation: Critical Perspectives on Southern Africa 30: 65�/82

Guijt, I. and M.K. Shah (1998) ‘Waking up to power, conflict and process’, in I. Guijt and

M. K. Shah (eds.) The Myth of Community: Gender Issues in Participatory Development,

London: Intermediate Technology Publications

Ife, J. (1999) Community Development: Creating Community Alternatives �/ Vision, Analysis and

Practice, Sydney: Addison Wesley Longman

Jackson, C. (2002) ‘Disciplining gender?’’, World Development 30(3): 497�/509

Kabeer, N. (1994) Reversed Realities: Gender Hierarchies in Development Thought, London:

Verso

Kabeer, N. (1999) ‘The Conditions and Consequences of Choice: Reflections on the

Measurement of Women’s Empowerment’, UNRISD Discussion Paper No. 108, Geneva:

UNRISD

Knowles, J.C., C. Leighton and W. Stinson (1997) ‘Measuring Results of Health Sector

Reform for System Performance: A Handbook of Indicators’, Bethesda, USA: Partner-

ships for Health Reform



Dow

nloa

ded

by [

Flor

ida

Stat

e U

nive

rsity

] at

23:

07 1

2 N

ovem

ber

2014

Levy, C. (1998) ‘Institutionalisation of gender through participatory practice’, in I. Guijt and

M. K. Shah (eds.) The Myth of Community: Gender Issues in Participatory Development,London: Intermediate Technology Publications

Longino, H. (1993) ‘Subjects, power and knowledge: description and prescription in

feminist philosophies of science’, in L. Alcoff and E. Potter (eds.) Feminist Epistemologies,

New York: Routledge

Malhotra, A., S.R. Schuler and C. Boender (2002) ‘Measuring Women’s Empowerment as a

Variable in International Development’, Paper of the World Bank Gender and

Development Group, Washington: World Bank

Mayoux, L. (1998) ‘Gender accountability and NGOs: avoiding the black hole’, in C. Millerand S. Razavi (eds.) Missionaries and Mandarins: Feminist Engagement with Development

Institutions, London: Intermediate Technology Publications

Moghadam, V.M. and L. Senftova (2005) ‘Measuring women’s empowerment: participa-

tion and rights in civil, political, social economic and cultural domains’, International

Social Service Journal 57(2): 389�/412

Moser, C. and A. Moser (2005) ‘Gender mainstreaming since Beijing: a review of success

and limitations in international institutions’, Gender and Development 13(2): 11�/22

Nielsen, J.M. (ed.) (1990) Feminist Research Methods: Exemplary Readings in the Social Sciences,Boulder: Westview Press

Ramazanoglu, C. and J. Holland (2002) Feminist Methodology: Challenges and Choices,

London: Sage

Saith, R. and B. Harriss-White, (1998) ‘Gender Sensitivity of Well-Being Indicators’,

UNRISD Discussion Paper No. 95, Geneva: UNRISD

SDC (undated) ‘Gender Toolkit: Instruments for Gender Mainstreaming’, Bern: Swiss

Agency for Development and Cooperation (SDC)

Snyder, M., F. Berry and P. Mavima (1996) ‘Gender policy in development assistance:improving implementation results’, World Development 24(9): 1481�/96

UNESCO (1997) ‘Gender-Sensitive Education Statistics and Indicators: A Practical Guide’,

Paris: United National Educational, Scientific and Cultural Organization (UNESCO).

Wendoh, S. and T. Wallace (2005) ‘Re-thinking gender mainstreaming in African NGOs

and communities’, Gender and Development 13(2): 70�/9



Dow

nloa

ded

by [

Flor

ida

Stat

e U

nive

rsity

] at

23:

07 1

2 N

ovem

ber

2014

Documents

Participation, values, and implementation: three research challenges in developing gender-sensitive indicators