Measuring the Impact of NGOs

8/2/2019 Measuring the Impact of NGOs

1/8

Oxfam GB

Impact Measurement for NGOs: Experiences from India and Sri LankaAuthor(s): Linda Kelly, Patrick Kilby, Nalini KasynathanReviewed work(s):Source: Development in Practice, Vol. 14, No. 5 (Aug., 2004), pp. 696-702Published by: Taylor & Francis, Ltd. on behalf of Oxfam GBStable URL: http://www.jstor.org/stable/4029898 .

Accessed: 09/01/2012 18:16

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at .http://www.jstor.org/page/info/about/policies/terms.jsp

JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of

content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms

of scholarship. For more information about JSTOR, please contact [email protected].

Taylor & Francis, Ltd. and Oxfam GB are collaborating with JSTOR to digitize, preserve and extend access to

Development in Practice.

http://www.jstor.org
http://www.jstor.org/action/showPublisher?publisherCode=taylorfrancishttp://www.jstor.org/action/showPublisher?publisherCode=ogbhttp://www.jstor.org/stable/4029898?origin=JSTOR-pdfhttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/stable/4029898?origin=JSTOR-pdfhttp://www.jstor.org/action/showPublisher?publisherCode=ogbhttp://www.jstor.org/action/showPublisher?publisherCode=taylorfrancis


2/8

Developmentn Practice,Volume 4, Number , August2004 CTaylof&rPubishingImpactmeasurementfor NGOs: experiences from Indiaand SriLankaLinda Kelly, PatrickKilby,and Nalini Kasynathan

IntroductionIn modem developmentdiscourse,one area hathas gained considerablecurrencyhas been theimpactdeliveredby NGOs, mainlybecausetheirwork is understood o have direct andobviouseffects on the lives of poor and marginalisedpeople. Despite thesehigh expectations,a numberof studies of NGO work point to a perceived lack of 'evidence' from which to establishmeasurable mpact.These studies nclude,among others,ODI (1996), the Danish NGO ImpactStudy (Oakley 1999),and the AusAID NGOEffectivenessReview (1996). There s also a viewthat heNGO sectoras a whole tends to exaggerate he impactof its work to support he case forincreaseddonor support Roche 1999; Riddell 1999; Kenall andKnapp 1999). But perhaps hemost relevantchallengeto NGOs comes frompoor people themselves:

... poor people give NGOs mixed ratings. Given the scale of poverty, NGOs touchrelatively ew lives .. . some NGOs are largely irrelevant,self-serving, limited in theiroutreach and corrupt, although to a much lesser extent than the State ... (Narayan1999)

It has been difficult for NGOs to demonstrate he worth and relevance of their work in amanneracceptable o sceptical outsiders.Most of the studies citedabove tend to report hat theevidence and frameworks or demonstratingmpactarelackingin the examplesof NGO work,rather hansuggest that theirwork has no impactas such.Furthermore,n developmentworkthere are neitheragreeddefinitionsof what impact is, nor agreedmethods for measuring t.

It was withinthis contextthatOxfamCommunityAid Abroad(CAA) decided to undertakean 'impact project' in 2000 aimed at developing a frameworkand process for the ongoingmeasurementof the impact of the organisation'swork. The impact project, which ran into2001, was not only in response to the widercritiquesabout the failure to establish the impactof developmentwork but was also a recognitionby OxfamCAA itself that ts own externalandinternalaudienceswere interested n understandingmore clearly what the impactof its workwas and how that impact may be measured.The projectran over an 18-monthperiod. It startedwith several debates and discussionsabouthow the agency might define and measureimpact, and then two specific activities inmeasuring mpact in differentcountrieswhere Oxfam CAA worked-India and Sri Lanka.The final stage broughtthe findings of this researchtogetherwith the agency's needs andissues, to look athow it might develop an ongoing model of impactmeasurement.This reportis a summaryof the researchundertakenand some of the conclusions about Oxfam CAA'simpactthat were reachedthroughthis project.

Defining ndmeasuring mpactThe impact projectsought to understand irst whatchanges had been experienced by people,and, second, how the work of OxfamCAA had or had not contributed o those changes.The696 ISSN 0961-4524 print/ISSN 1364-9213 online 050696-14 C)2004 Oxfam GBDOI: 10.1080/0961452042000239841 Carfax Publishing


3/8

Impact measurementor NGOsorganisationrequired an understandingof impact that allowed it to focus on experiencesbeyond the more immediateoutcomes of any particularnterventionand make linkagesto thebroadercontexts in which developmentactivities were occurring.The key questionsthat werebeing investigated were 'had peoples' lives improved?' and 'had we contributedto thisimprovement?'

Moving from definition to measurementcreatedfurther difficulties, the main one beingwhose perspective should be taken into account when measuring mpact. Roche (1999) hasidentified wo broadapproacheso deal with the issue of perspective n impactassessment.Thefirst is the 'projectout' approach. t startswith the aim of the projector programmeand thenfinds ways of measuring he outcomes of that aim or objective from a variety of perspectives.The second is the 'contextin' approach. t looks at the changes that are happening n people'slives, whatis significantaboutthese changes, and thenassesses the usefulness or effect of anyintervention n relation to these changes. Like the 'project out' approach,the 'context in'approach s resource ntensiveand ultimatelyvalue ladenand perspective dependent.But thedifference in this approach,which presents a significant step forward in addressing therelevance of development interventions, is that the starting point for measurement s thechange in the conditions of a particular ontext, rather han the intervention tself.

These two approaches an be appliedat a very basiclevel or at a broader ne. Forexample, inanempowerment roject,a 'projectout'approachwould look at the intervention ndthe changesthatoccurredn domestic or local powerrelationsas a consequenceof the intervention,whetherit be microfinance,education,or health. A 'context in' approachwould look at the causes ofdisempowermentthat an individual or community is facing (the context) and relate theintervention o that context. At the broaderor macro level, these two approaches o measuringimpactcould be applied o programmes uchas nationwideadvocacy campaigns.

At apractical evel, Oxfam CAA adopteda blend of bothapproachesn its studyof impact.Itwas clear that nvestigatingonly the changesthathad occurred orpeople, in the absenceof anappreciation f thechangestherespectiveprogrammes ad beentrying o achieve,would be tooartificial.Partof the tension with the 'context in' approach s thatstaff and constituentsbothvaluewhattheyhave beendoingandhavevalid reasons orconsideringan interventionuseful. Itis difficult o assumethatwe startwith a blankslateandthatall that mpactassessmentwouldbeabout s lookingafreshat whatpeoplehaveexperiencedandhow their ives have changed.Partof the value of the impactenquiryis to assess what organisationswere tryingto achieve incomparisonto the changes experienced by people, and understandwhy the two might bedifferent.

Oxfam CAA was also looking to understand hangefromboth theperspectiveof the peopleconcerned,and from otherperspectives,such as broader ocal andregional changes that hadmore indirectand onger-termmpactson thepeople.This neededto be donein a freshway thatopened upthesignificanceof suchchangefromtheperspectiveof thebeneficiaries hemselves.A further tepwas then to test the relevanceof specific interventions gainstthesechanges.Within hisframework,OxfamCAA undertook wo research tudies,one in Indiaand one inSriLanka,eachdesignedto trydifferentapproaches o definingandmeasuring mpactforpoorand marginalisedpeople. The overall aim was to further he organisationalunderstanding fwhathadbeenachieved,to consider heimplicationsof this learning orongoing programming,and also to assess how to measuresuchchangemoreeffectively in the future.The Indian studyA studywas undertakenn Indiathat looked at programmesof interventionover long periods(up to ten years) with long-standing Oxfam CAA partners,which had as their goal theDevelopment n Practice, Volume14, Number5, August 2004 697


4/8

Linda Kelly, Patrick Kilby, and Nalini Kasynathanempowerment of women. For the purposes of the study, empowermentwas defined as anexpansion of choice and a person's enhanced capacity or opportunity o act on those choices(Kabeer 1999; Hindess 1996). A total of 15 NGOs were studied, of which half were OxfamCAA partners.From each of these NGOs a sample of self-help groups(the constituentgroupof around20 women that NGOs generallyfoster and work with) was chosen and a total of 77women's groups were interviewed.A process for measuring mpowermentwas adoptedbased upon the experiencesof poor andmarginalisedwomen and how they answereda series of open-endedquestionsrelating o (any)changes in their lives (Hines 1993). The approach hen set out to test the degree to which thework of the partnerNGOs had facilitated this change, and whether the change was in factempowering, i.e. corresponding o an increased range of choices and action in the lives ofwomen.Care was taken to ask open-ended questions that did not suggest certain answers, and onsensitive issues questionswere asked in the 'thirdperson' so thatrespondentsdid not feel theywere being personally 'put on the spot'. Similarly, the researcherwas not identified with adonoragency, but ratherwith a research nstitution.These approachesenabled the women tomake an assessment of the perceived changes that had occurred n theirlives and gave themthe opportunity o attribute hese changes to the work of the NGO and explain how thatoccurred. Otherquestions were asked about village life and women's participation n it toprovidesome idea of the broadercontext and the disempowering nfluences women generallyfaced.

The results of the answersto the questions on the change that women had experiencedwereranked.The broadrankings hat emergedfromthe women themselvesrangedfrom being ableto go out of the house and engage with people in authoritysuch as bank managersand thepolice, through o engagingwith the local political processes.Theempowerment hangeswerethen statisticallytested in relation to several factors such as caste, education, land, villagesocial capital, etc., and the range of informal, semi-formal, and formal accountabilitymechanisms the NGO had established with respect to its partnerorganisations.The resultsshowed thatthree factorswere statisticallysignificant n determining mpowermentoutcomes:'downward'NGO accountability o local groups;how long the grouphad been meeting;andthe leadershipof the group.The finding that formalaccountabilityof the NGO to its constituencyis related to strongempowerment utcomes hasimplications or how Oxfam CAA field partners ngagewith theirconstituencyin terms of the level of controlthey give theirconstituencyin determining hedirection of their work. It also has implicationsfor how Oxfam CAA develops and fosterspartnerships,and how it manages its development programmes. t points to a much greaterfocus on local control at all levels.

The second point is thatempowermentof the most disadvantaged akes time andrequiresa long-term ocus. This findinghas implications or the 'project' approach o developmentandtends to favour more strategic longer-term interventions. It also has implications fordeveloping and supportingpartners or the long term,andrecognises thatseeking short-termoutcomes may not lead to empowerment.

The issue of the natureof the leadershipof groups appeared o be very important an areaoften poorly covered in NGO practice). Those groups that had a rotated or collaborativeleadershipapproachshowed strongerempowermentresults than those groupswho dependedon a single leaderor even on the NGO to provide leadership.Again, these findings can givesome insight into how Oxfam CAA develops and supports t partners.Related to these specific findings was the broad observation hat even the most successfulcommunity development programmesshould be aware of the context in which they are698 Development n Practice, Volume14, Number5, August2004


5/8

Impactmeasurementor NGOssituated. This may require some integration with advocacy to ensure that there is asupportive ocal economic, political, and social environmentat the broader evel to ensurethe sustainabilityof empowermentoutcomes. For example, a groupof waste-pickerwomenwho had enormoussuccess in empowerment hat led to significantchanges to theirworkingand living conditionswere having theirlivelihoods threatenedby a proposedprivatisationofwaste managementin their city. This change in urbanwaste managementpolicy had thepotentialof undermining he strongempowermentresultsthat had been achieved.A broaderadvocacy campaignon waste managementmay be requiredto preserve the rights of thesewomen.

The Sri LankanstudyThe Oxfam CAA programmein Sri Lanka works with poor and marginalisedpeople,especially women, living in the areassubjectto civil warbetweenthe governmentandTamilseparatistgroups. The methodology developed for the study had two parts that roughlycorrespondo thenotions of 'projectout' and 'contextin' approaches o impact assessment.Asin the Indian study, people were asked 'What is going on in your lives? What have beensignificant changes in your lives? What are the significant problems?' Second, the studysought informationagainst a set of impact indicators or 'change dimensions', developedexternallyby OxfamCAA staff, thatfocused uponthe long-termchangesthat the programmewas tryingto achieve.

Data were sought from various sources and involved a large numberof meetings withstakeholders,ncludingcommunitygroups,staff,the centralcommittee,andsubcommitteesofeach of the partnerorganisations.OtherNGOs, governmentofficials, externalexperts,andthelike were also interviewed. Finally, a study on the national and regional changes thatinfluenceddevelopment n Sri Lankawas undertaken, o thatchange could be understoodatseveral different evels beyond that of the communitiesthemselves.

For the changes sought by the programme,the 'project out' assessment, each 'changedimension' was consideredfor the seven partnerNGOs studied,and either an assessmentofchange over time or an overall evaluationof progresswas made. The dimensions thatwereexaminedwere the levels of:* communityparticipation;* social awareness;* empowerment/capacity uilding;* genderjustice;. provisionof services and benefits;. inclusion of the most marginalised;* institutionalstrengthening.The issue-based assessment also identified other key factors that were outside the directquestioning but clearly related to the possible impact of the programme.These includedthe effects of the war and issues related to incompatibility and competition amongdonors.

While the findingswere generallypositive andthe changesexperiencedwere relatedto thework of Oxfam CAA, the relevance of the programmein terms of the overall impactexperiencedby the poorest and most marginalisedpeople was not as great as expected. Thestudy suggested that the CAA programme n Sri Lankahad been highly relevantto some ofDevelopment in Practice, Volume 14, Number 5, August 2004 699


6/8

Linda Kelly, PatrickKilby,and Nalini Kasynathanthe particular roblems acedby people-human rightsviolations,genderdiscrimination, oorservices, andlack of civil society structures.The interventions ended,however,to be limitedto respondingto the local expression of these problems and were unable to address thefundamental roblemsof war,humanrightsviolations,caste,genderandethnicdiscrimination,economic disadvantage,and the failureof governance.

People were more able to take advantageof situationsof change (for example, they knowmore about theirrights andhow to organise),but the changes were not happeningat a levelthat ensured sustained difference.An example of this limited level of change came fromlooking at the degree to which women were empoweredby the programme.

The staff knew what constitutedwomen's empowermentand what the outcomes were,and they felt that the programmeswere moving in the rightdirection.The strategiesadoptedto improve effective participationof women such as policies of positive discrimination,constitutionalchanges, regulartraining,and awarenessraising were all tested and proven.However, the findings of the studyclearly indicatedthatOxfam CAA had a long way to go.While women reported a decrease in domestic violence and an increase in householdincome, many of the importantdecisions were still being made by men and, in someinstances, women were willingly handing over decision-makingpower to the men. Whatbecame evident is that the empowerment of women in a complex war-zone situationrequiresmore sophisticated nterventions han those used in other settings. This finding ledto changes in programming.

New activities and approacheswere introduced by programmestaff and communities,following discussion and analysis of the researchfindings. The directionof the programmewas improvedso that the objectivesandindicatorsmoreclosely matchedthe areasof changerequiredfor real empowerment.Second was a revision of the use of trainingas a tool forcommunitydevelopment.The content andapproaches o the trainingofferedby Oxfam CAAwere revised with a greater ocus on groupand issues-basedtraining.

The relationship between Oxfam CAA and the community-based organisations theysupported was also revised, with Oxfam CAA establishing more participatory andtransparent rocesses to bringthe relationship nto more of a partnership rrangement-i.e.promoting'downward'accountabilitymechanisms,as in the case of Oxfam CAA's work inIndia detailed above. Finally, the findings indicatedthe need for stronger inkages betweendevelopment programmesat the local level and advocacy efforts at a broader nationallevel.

OrganisationalearningThe two case studies identified ssues thathave relevancefor OxfamCAA as an organisation.First is that a process of 'downward' accountability to partner organisations and theirconstituencies nvolves mutuallearningand the ability to listen to them. In this respect it isimportanto have formalor semi-formalaccountabilitymechanismsbuiltintothedesignof theproject,includinghow local people can asserttheirrightsvis-'a-visthe NGO.

Second, actionneeds to take place at more thanone 'level' of intervention or any impactto be sustainable. That is, advocacy and policy work should be integrated with fieldprogrammesandcapacitybuilding.Action for change occurs at local, national, regional, andin some cases intemational evels. Third, t appears hatinterventionsaredynamicandshouldchange with a changingcontext and over time, andbe monitoredto accountfor the changesandensurethesechangesarerelated o the overallgoals of theprogramme.While it is possibleto measureshort-termgains in field programmes uch as OxfamCAA'sprogrammes n Indiaand SriLanka, t is clear thatanyimpacttakestime and the variouscomponentsof changethat700 Development n Practice, Volume14, Number5, August2004


7/8

Impact measurementor NGOsneed to occur will take place in different ways and at differenttimes. Managing this processrequiresregular attentionto measurementof change and difference.

Finally, both studies indicatedthat people know what impact they want. Poor people knowwhat is helpful to them and what is not. Unless the poor participate n defining the impactbeing sought and their perspective is recognised as central to assessing impact, NGOs (orindeed any aid-delivery mechanism) run the real risk of being considered 'largelyirrelevant'.

ConclusionsThe final lesson from the impact project was that the process of trying to measure impacttaught Oxfam CAA more about the meaning andprocess of impact than many of the earlierdiscussions and debateshad. While it seems that there is no one approach o measuringordefining impact,it would be a mistaketo allow this to stop an organisation rom attemptingto learn more about the contribution t is makingto sustainedandpositive changesin people'slives.

One problemwith these types of field studies is the difficulty of drawingfirm conclusionsaboutattribution.However,the attempt o learn about the impactof interventions, ather hanjust test the attainmentof objectives,was a powerfulchallenge to Oxfam CAA and its staff.In many ways the struggle to develop andtest an appropriatemethodologywas as importantas the specific conclusions.As one staffmembernoted, 'Perhaps f we keep tryingto measurewhat's really important,we might end up doing it.'

ReferencesAusAID (1996) A Review of the Effectiveness of AusAID Support to NGO Programs,Canberra:AusAID.Hindess, B. (1996) Discourses of Power: FromHobbes to Foucault, Oxford:Blackwell.Hines, A. M. (1993) 'Linkingqualitativeandquantitativemethodsin cross-cultural esearch:techniques from cognitive science', American Journal of Community Psychology21 (3):729-746.Kabeer, N. (1999) 'Resources, agency, achievements:reflections on measuresof women'sempowerment',Developmentand Change 30(3):435-464.Kenall, J. and M. Knapp (1999) 'Evaluationandthe voluntarysector:emerging issues', inD. Lewis (ed.) InternationalPerspectives on VoluntaryAction: Reshapingthe ThirdSector,London:Earthscan.Narayan, D. (1999) 'Cananyonehear us? Voices from47 countries',in D. Narayanwith RajPatel, Kai Schafft, Anne Rademacher,and Sarah Koch-Schulte, Voices of the Poor: CanAnyoneHear us?, New York,NY: OUP (for the World Bank).Oakley, P. (1999) Danish NGO Impact Study: A Review of Danish NGO Activities inDeveloping Countries-Synthesis Report, Oxford and Copenhagen: INTRAC/BECHDistribution.ODI (1996) 'The Impact of NGO Development Projects', Briefing Paper No. 2, London:ODI.Riddell, R. C. (1999) 'Evaluating NGO development interventions', in D. Lewis (ed.)International Perspectives on VoluntaryAction: Reshaping the Third Sector, London:Earthscan.Roche, C. (1999) ImpactAssessment or DevelopmentAgencies, Oxford: Oxfam GB.Development n Practice, Volume14, Number 5, August2004 701


8/8

James L. GarrettThe authorsLinda Kelly is an independent practitionerwho specialises in community development,monitoringandevaluation, and gender and aid management.Contactdetails: 37 NorthValleyRoad, Park Orchards,3114, Australia. .Patrick Kilbylectures in programmemanagementand empowermentand rights-baseddevelopmentat theAustralian National University. Contact details: Asia Pacific School of Economics andGovernment, Crawford Building, Australian National University, ACT 0200, Australia..Nalini Kasynathanhas taught at University Peredeniya n SriLankaandis currently esponsiblefor managingOxfam CAA programmeswithin South andEastAsia. Contactdetails: OxfamCommunityAid Abroad, 156 George St, Fitzroy,Vic 3065,Australia. .

Bridginggaps: collaborationbetween research and operationalorganisationsJames L. Garrett

The potential of collaborationInformation s essential to improvingorganisational ffectiveness. The potentialfor benefitsfromcollaborationbetweena researchorganisationandan operationalNGO seems large.TheNGO cantapinto the latestknowledgeand learnhow to improve ts own surveyandanalyticalmethods.This can in turnstrengthents show of impactand innovation o donors.By workingwith NGOs,researchers an get a bettersense of criticalpolicy and programmequestionsandshapetheirwork to demand,thereby ncreasing he probability hatothers will actively makeuse of theirfindings.

This PracticalNote builds on the insights of LauraRoper's 2002 article on 'Achievingsuccessful academic-practitioner esearchcollaborations'by reviewing one example of suchcollaboration, a partnershipbetween CARE and the InternationalFood Policy ResearchInstitute(IFPRI).Since 1997, CARE and IFPRIhave collaboratedon increasingknowledgeabouturban ivelihoods thatwill be of use to programmedevelopment.The two have workedin a numberof countries, ncludinglivelihood assessmentsin Tanzania, ssue-basedresearchin Bangladesh,technicalassistancein Mozambique,andprogrammeassessmentin PeruandEthiopia.Examiningtheir collaborativeefforts, this paperprovides a concreteillustrationofhow to build bridges and profit from synergies between two such organisations whilehighlightingpotentialbumpsto expect along the way and what to do aboutthem.

ThegapDespite the apparentbenefits, explicit collaborations between research and operationalorganisationsare not common.Institutionalperceptions hrowup barriers o working together.The rootsof theproblemmay be primarilydifferences n organisational ultureandintellectual702 Developmentin Practice, Volume14, Number5, August2004

Documents

Measuring the Impact of NGOs