A Basic Guide to Evaluation for Development Workers

8/7/2019 A Basic Guide to Evaluation for Development Workers

1/96


2/96

A Basic Guideto EvaluationforDevelopmentWorkers

FrancesRubin

Oxfam


3/96

First published by Oxfam UK and Ireland in 1995Reprinted by Oxfam GB in 199 6 ,19 98 ,1 999 ,200 2 ,20 03 Oxfam UK and Ireland 1995ISBN 08 55 98 275 6A catalogue record for this publication is available from the British Library.All rights reserved. Reprodu ction, copy, tran sm issio n, or translation of any part of this p ublicationmay be made only under the following conditions: with the prior written permis sion of the publish er; or with a licence from the Copyright Licensing Agency Ltd., 90 Totte nha m Co urt Road,

London WiP 9HE, UK, or from another national licensing agency; orfor quotation in a review of the work; orund er the term s set out below.

Th is publication is copyright, but may be reprodu ced by any meth od w ithout fee for teachingpurp oses, but not for resale. Form al perm ission is required for all such use s, but normally will begranted imm ediately. For copying in any other circu ms tanc es, or for re-use in other pu blications,or for translation or adaptation, prior written permission must be obtained from the publisher,and a fee may be payable.Available from:Bo urne mo uth English Book Centre, PO Box 1496, Parkston e, Dorset, BH12 3YD, UKtel: +44 (0)1202 712933; fax: +44 (0)1202 712930; email: [email protected]: Stylus Publishin g LLC, PO Box 60 5, He rnd on, VA 20172-06 05, USAtel: +1 (0)703 6611581; fax: +1 (0)703 6611547; email: [email protected] details of local agents and representatives in other coun tries, consult ou r website:www.oxfam.org.uk/publicationsor contact Oxfam Publish ing, 274 Banbury Road, Oxford OX2 7DZ , UKtel: +44 (0)1865 311311; fax: +44 (0)1865 312 6 0 0 ; email: publish@ oxfam .org.ukOur website contains a fully searchable database of all our titles, and facilities forsecure on-line ordering.Published by Oxfam G B, 274 Banbury Road, Oxford OX2 7D Z, UK.Printed by Information P ress, Eynsham.Oxfam GB is a registered charity, no. 202 918, and is a m em be r of Oxfam Internatio nal.

This book converted to digital file in 2010


4/96

Acknowledgements

In 19921 drafted the evaluation sec tion of a proposed Oxfamprocedures m anua l on project managem ent. An adapted form h asprovided mater ial for the Oxfam Handbo ok for Development andRelief, edited by Deborah Eade and Suzanne W illiams. Somesimilarities may be found be tween this guide and parts of that text,as drafts have been swapped between us .Particular thanks are due to Deborah Eade and Bridget Dillon,for comments, editing an d friendship. Thank s, too, to RobertNicholls for being a critical 'guinea-pig' at a particularly busy time.

This short guide to evaluation was originally intended for usewithin Oxfam b ut w e hope that it will also be found helpful to awider readership.


5/96


6/96

Contents

1 Introduction 91.1 Situations which might involve evaluation 101.3 The aim of the guide and wha t it covers 112 Putting evaluation in context 132.1 Evaluation in the project cycle 132.2 Appraisal, monitoring and evaluation 142.3 The relationship between appraisal, monitoring andevaluation 152.3.1 Appraisal 152.3.2 M onito ring 16

2.3.3 Evaluation 162.3.4 Appraisal, monitoring and evaluation in the projectcycle 172.4 The stakeholders in development action 173 Traditional and alternative models of evaluation 193.1 Trad itional evaluation 193.2 Reassessm ent of evaluation: the challenge for NG Os 213.3 Ideas about develop ment and the role of evaluation 213.4 Alternative approaches to evaluation 223.5 Pow er and control 234 The purpose and use of evaluation 254.1 Reasons for evaluation 254.2 Judging success and failure 264.3 M easuring quantity and quality 264.4 W hat evaluation can do 27

4.5 W hat evaluation cannot, and sh ould no t, be used for 285 Putting evaluation into practice 295.1 Developing evaluable projects and prog ram m es 295.2 The key ques tions 29


7/96

6 A basic guide to evaluation for development workers5.3 W hy and for whom is an evaluation to be done? 315.3.1 Why is the evaluation being done? 31

5.3.2 W ho wan ts the evaluation? 325.4 W hen shou ld evaluation take place? 335.4.1 W hen, in the life of a project, should evaluation takeplace? 335.4.2 Other times when evaluations take place 345.4.3 Other important factors relating to timing 345.5 W hat is the scope and focus of the evaluation? 365.5.1 The focus of an evaluation 365.5.2 W hat is the role of the funder in an evaluation? 375.5.3 W hat criteria should be used for evaluation? 385.5.4 W ho chooses the criteria? 395.5.5 W hat indicato rs shou ld be used? 405.5.6 When shou ld indicators be chosen and w ho shouldchoose them? 405.6 Who will manage the evaluation and w ho will carry it out? 425.6.1 W hat skills are needed w ithin the evaluation team? 435.6.2 W ho can do the evaluation? 445.6.3 Participatory evaluation 455.6.4 User evaluation 465.7 How will the evaluation be done? 475.7.1 W hat information w ill be needed ? 485.8 How much w ill the evaluation cost, and what resources areneeded? 515.8.1 Budgeting for evaluation 515.8.2 M atching scope to resources 515.9 Reporting back and follow up : how to make sure evaluationinformation is used 525.9.1 The uses of evaluation results 525.9.2 How will the findings be presen ted? 535.9.3 Follow up 535.9.4 W ho ow ns evaluation and evaluation results? 54

6 Some practical considerations wh en planning an evaluation 566.1 Stages in evaluation 566.2 How to make sure an evaluation will be successful 576.3 Planning and man aging an evaluation 57


8/96

Contents 76.3.1 Draw ing up the Terms of Reference (TOR) 596.3.2 Conten ts of the TOR 606.3.3 Confidentiality 616.3.4 Recruitment and selection of eva luato rs 616.3.5 Essential docum enta tion 626.3.6 Orientation an d briefing 626.3.7 Resources and logistical arrangements 636.4 Field work 646.5 M anagem ent and supervision of the exercise 646.6 Arrang ing meetings 65

6.7 Analysis 666.8 Conclusions and recom mendations 666.9 W riting the final report 676.9.1 The con tents of the report 677 Using evaluation: feedback and follow-up 697.1 Initial Feedback 697.2 Follow up 70

7.3 Dissem ination 707.4 Institutional learning 717.5 Barriers to learning and change 71Appendix 1 Checklists of ques tions 73Appendix 2 Developing an evaluation plan 77Appendix 3 Tasks for the team leader 79Ap pendix 4 Suggestions for the content of an evaluation report 81Appen dix 5 Advan tages and disadvantages of internal andexternal evaluators 83References and further reading 85Index 91


9/96


10/96

1 Introduction

There are m any books on evaluation, some of them very interestingand helpful, others full of jargon and rather confusing. Oxfam staffwh o advise on plann ing and evaluation receive many requests foradvice about different aspects of evaluation. The queries indicate aneed for a simple guide that maps a route through the confusionthat has built up around evaluation and its uses, that focuses onkey underlying principles, and promotes clearer thinking.A numbe r of books and texts on evaluation have been dra w n onin compiling this manual, in particular:A UNICEF Guide for M onitoring and Evaluation: Making a Difference?UNICEF 1991.Feuerstein, M T, Partners in Evaluation, Macmillan 1992Beaudoux, E et al, Supporting Development Action, Macmillan 1992Guide to Planning and Evaluating NGO Projects, Vols 1-3, NORA DA Workbook for Implementing Partner Organisations and SupportAgencies, AGKED/Misereor, 1991

Readers are advised to consult these books if they wish to s tudythe subject in more detail. Other useful books are listed in theReferences and Further Reading section.The manual is also based on the cumulative experience of staffin the advisory unit in doing evaluations, runn ing work shops w ithfield staff to discuss their evaluation experience, and advising staffabout specific exercises; and on insights from colleagues in Oxfam,both in the UK and other countries.


11/96

10 A basic guide to evaluation for development workers1.1 Situations w hich m ight involve evaluationHere are some examples of the situations that agency staff mayexperience in the course of their work which involve evaluation insome form: A grassroots or interme diary gro up may seek help on how tobuild evaluation into their work. A No rthern funding agency may wish to encourage the buildingin of evaluation into the work of an organisation that is puttingforward a proposal for funding. A Northern funding agency is not happ y with the progress of anorganisation that it is funding, or part-funding, and they wish tocarry out an external evaluation. A major official donor wishes to do an evaluation of a gro up ofprojects that it is co-funding with a non-governmental agency.They have already appointed the evaluators and have only alimited am oun t of time in which to complete the exercise. Questions are asked about the effect of funding decisions takenover the years. Staff w an t to un de rstan d the effect of a particular policy theyhave been implem enting, eg gender policy, disaster prepared ness.

1.2 Questions about evaluationHere are some examples of the questions most often asked aboutevaluation: How do we draw u p terms of reference (TOR)? We wan t the evaluation to be as participative as possible, do youhave any tips? H ow can we reconcile par tner agency needs and the fundingagency needs? W hat indicators should we use?


12/96

Introduction 11 W hat is the appropriate m ethod? Should it be qualitative or quan titative? Who should do the evaluation? W hat is an acceptable cost? W hen is it best to evaluate, and how long does it take? W e've had awful problem s w ith external evalua tors so w e'vedecided just to prom ote intern al evalua tions how do we goabout it?

1.3 The aim of the guide, and w hat it coversThis guide is intended to help people to understand the unde rlyingprinciples of evaluation, in order to be clearer about its uses andlimitations. It will deal with questions such as what evaluationactually is and why it is useful; when it is inappropriate; how todevelop pro gram mes and projects that can be effectively evaluated;the steps necessary for planning an evaluation; and how to useevaluation to improve development action. It will emphasise thedangers in not being clear about why an evaluation is beingrequested; the need to involve people; the polit ical nature ofevaluation and the importance of negotiation.

The second chapter of the guide puts evaluation in context.Evaluation is defined, and then its place in projects is discussed.There is also a discussion on its relationship to planning, appraisa land monitoring.There are different 'actors' involved in development, and thesedifferent actors may have very different view s on things. For thisreason negotiation is important at all the various stages in adevelopm ent initiative, not least when considering evaluation.A brief history of 'traditional' evaluation and its characteristicsis given in Chapter 3. There is a discussion on w hy NGO s need toreassess evaluation, and the alternative options available. This isfollowed by a section on the political nature of evaluation, andissues of power and control.


13/96

12 A basic guide to evaluation for development workersChapter 4 looks at why people do evaluation, and whateva luation can and cannot do . Chapter 5 outlines the main

questions that have to be addressed when evaluation isincorporated at an early stage in project planning.The next chapter considers practical considerations whenplanning a specific evaluation exercise. There is advice on drawingup terms of reference, and ensuring better feedback and follow up,including suggestions on presenting evaluation reports. The finalpart of the chapter covers particular points that evaluators need toconsider.The last chapter looks at the important questions of feedbackand use of the findings of an evaluation.There are appendices with checklists of questions, and anexplanation of the terms used in the manual are given in aglossary. There is also an annotated bibliography of the mosthelpful and accessible books on evaluation.


14/96

13

2 Putting evaluation in context

2.1 Evaluation in the project cycleEvaluation means 'to assess the value or worth of. For ourpurposes we broaden this out to mean 'to understand the value ofsomething in order to do things better in the future'.

This guide will focus on evaluation activity related todevelopment projects. The points covered can also apply toevaluations of emergency work, particularly if emergencies areunderstood from a development perspective. Because 'project' isstill the most widely-used term to describe development activities,it will be used in this guide, although it is rather a limited way ofdescribing development.

Ideafeeding into new , / \ ^project ideas / , - x ^ \and policies / / \ \ assessment ^/ / \ \ o f t h e 3/ \ situation ^

X\ ,' planning we v a l u a t i o n \ / and design \' i

m o n i t o r i n gThe project cycle


15/96

14 A basic guide to evaluation for development workersThe project cycle is a useful w ay of describing how developmentidea s tu rn in to a p r op os a l for ac t ion . As the pr op os a l i s

implemented, it should be dynamic and flexible enough to changein the light of lessons learnt; evaluation is one of the most valuabletools for learning.The diagram on the previo us pag e does not take full account ofthe complex nature of development work, and the interactionsbetween the different people involved. Each stage should be theoccasion of learning and change, and p erha ps a spiral rather than acircle would be a better representation of the way the developm entprocess builds on past experience. The diagram also shows thatevaluation may sometimes not be used, and evaluation reportsmay lie unread on dusty shelves.

This book is intended to help readers to make full use ofevaluations.

2.2 Appraisal, monitoring and evaluationA distinction is usually made between appraisal, monitoring andevaluation. Appra isal is the critical exam ination of a proposal, on the basisof agreed selection criteria, before implementation or approval forfunding.

The appraisal phase will involve asking questions such as: How has the problem to be addressed been identified? Does the proposed action add ress the problem? Do the people proposing to carry out the work have thecapacity to do it? W hich different in tere s t gro up s hav e bee n inv olv ed indefining the problem and choosing the course of action? How are different groups of men and women likely to beincorporated or affected by the project (young/old, landed/landless, single-headed households)?


16/96

Putting evaluation in context 15 Monitoring is a continuous, methodical process of datacollection and information-gathering throughout the life of aproject. The information collected can be used for regularevaluation of progress, so that adjustments can be made while thework is going on.Monitoring is also used to mean the systematic 'tracking' of aparticular condition, or set of conditions (for example politicalevents, environmental change, the situation of women), to identifytrends. Evaluation is a learning and management tool: an assessment ofwhat has taken place in order to improve future work. Measuring,analysing and interpreting change helps people to determine howfar objectives have been achieved and whether the initialassumptions about what would happen were right; and to makejudgements about the effectiveness, efficiency, impact andsustainability of the work.

2.3 The relationship between appraisal, monitoringand evaluation2.3.2 AppraisalA proposal for a new project should state clearly the objectives andactivities, and the resources required. The assumptions on whichthe project is based should also be explained, and there should besome indication of how progress will be measured. Time andthought given, at the initial planning stage, to the way in whichmonitoring and evaluation will be built into the work, will createthe conditions for the success of the project, and make it more'evaluable'. Projects should start on a sound basis of information,especially a knowledge of the roles, responsibilities andrequirements of women and men in a community, and within theproject.

Everyone who has an interest in a particular project, the'stakeholders' (funder, local NGO, and the women and meninvolved directly in the work), should identify and agree on the


17/96

16 A basic guide to evaluation for development workerscriteria for assessing the progress of the project. They shouldjointly select the indicators which will show that change has takenplace. Dur ing the life of a project, it is quite possible, and may evenbe desirable, that criteria and indicators should evolve in responseto experience gained and changing objectives. Indicators shouldprovide relevant information, of value to those who will use thefindings, and there should be a balance between quantitative andqualitative indicators (see p. 40). It is a poor use of resources tocollect too much data, of poor quality or little relevance, andgenerally frustrating for those involved in collecting it.

The ap pr ais a l s tag e of a project sh ou ld a lso inv olv e theconsideration of different options for addressing the problem. Thisis the point at which m onitoring an d evaluation expectations of thedifferent parties involved will be negotiated.2.3.2 MonitoringIn m on i to r in g , in fo rm a t ion for 't r a ck ing ' p ro g re s s aga ins tpreviously ag reed plans and 'milestones' is routinely gathered. Thechanges that are resulting from project activities can be identified:both the effects and the impact (see p. 38). If there are discrep-ancies between actual and planned progress, corrective action canbe taken. This can include changing the overall purpose and planof the activity. Monitoring can also mean keep ing a check on theuse of resources. Questions for later evaluation can be identifiedduring monitoring.

2.3.3 EvaluationEvaluation uses information gathered during regular monitoring,but may need other information as well. It often uses 'baselineinformation': information collected at the very beginning of aproject, against which progress can be measured. Evaluationhapp ens at set times in the life of a project.Evalua tion looks at the relevance, effectiveness and impact of apro jec t , wi th the a im of improving an ex is t ing pro jec t o rinfluencing future policies, programmes and projects.


18/96


19/96

18 A basic guide to evaluation for development workersThe 'target group', or the men and women who are to be theprime beneficiaries of a project, should be the most important

stakeholders. It is still rare for them to participate fully in thedefinition of development action and the planning of projects.There is increasing evidence that the more people are enabled toparticipate in decisions about development activities which affectthem , the more successful and sustainable the activities will be.The most im portant questions for the target group are:

Does the project tackle their real needs? Are the results significant? Are the results worth the effort?

The answ ers are likely to be different for men and wom en.Apart from the 'partners ' in development action, describedabove, there are other groups to relate to at the regional andna t iona l leve l : fo r example , na t iona l and loca l governmentauthorities, labour unions, development assistance agencies, otherorganised groups. Even if they are not directly involved in theproject, they may have opinions about it. It is important to find outtheir views and work out ways of relating to them.Organisations which provide funding do so on condition thatthe assistance they give is in line with their overall policies. Theyare accountable to boards of management, public opinion, andsupporters, and control is often a major concern for them. Theyhave to spend their funds according to the agreements made, andthey need to be sure that projects achieve the intended objectives.The different stakeholders will try to influence the choice ofobjectives, priorities, and methods for projects according to theirown values and principles. At times this can cause conflict andpower struggles, particularly when a donor organisation's influencepredom inates because of their control of financial resources.Planning and evaluation of projects should be based on an

acknowledgement of possible conflicting motives and interests, orthese ac t iv i t ies may become noth ing more than theore t ica lexercises, dominated by the stronger (funding) partner. Theinterests of the 'target' groups'should have the highest priority, inpractice as well as intent.


20/96


21/96


22/96

Traditional and alternative models of evaluation 22members of organisations subjected to such evaluations should feelinsecure and unco-operative.

3.2 Reassessment of evaluation: the challenge forNGOsToday, NGOs, both funded and funders, face a challenge thatdemands re-assessment of evaluat ion and i ts potent ial uses.Increasingly NGOs have been praised as effective agents for theencouragement of grassroots development, and for their ability toreach poor women and men, at lower cost. There has been anincrease in funds from multilateral and bilateral agencies to NG Os,and it seems likely that these agencies will wish to channel evenmore through Northern NGOs and directly to Southern NGOs infuture.However, NGOs have been unable to provide firm evidenceabout the results of development efforts, since few, if any, haveproduced accessible information, based on systematic evaluation ofp ro jec t s and p rog rammes . In add i t ion , the re i s a g rowingscepticism on the part of the tax-paying and donating public aboutthe results of developm ent aid.

NGOs need to be able to prove that they are effective. They mustdevelop the i r ab i l i ty to evaluate the i r work , in o rder todemonstrate the positive change their efforts are helping to bringabout; and to use the results of evaluation to learn how to imp rovetheir work.

3.3 Ideas about development and the role ofevaluationAs discussed above, the t radi t ional model of evaluat ion wasdeveloped in response to a technocratic view of development,where developm ent inputs could be measured in num bers.Today, many NGOs think of development more in terms ofpeople than of things, of enabling poor and disadvantaged womenand men to have more say in ensuring their l ivelihoods, and


23/96

22 A basic guide to evaluation for development workersshap ing their societies: deve lopm ent is seen to be more about socialchange than technocratic solutions; and partnership, participationand the dignity of people are valued. Evaluation should be integralto the development process, and it is important to find ways ofevaluating that match this perception of development. Also, if thepeople car ry ing ou t p ro jec t ac t iv i t ies are more invo lved inevaluating them, they are much more likely to use the results ofthat evaluation in present or future w ork.

3.4 Alternative approaches to evaluationOver the last two decades various alternatives to the traditionalforms of evaluation have developed. These approaches can makeuse of some of the tools used in traditional evaluation, but the aimis to m ake them more app ro p r ia t e to m ea su r ing soc ia ldevelopment.Traditional evaluation tends to see the people involved in aproject as objects of an evaluation; alternative approaches take theview that participants in development projects should be the'subjects' of evaluation and take a more active role. The methodschosen should enhance their capaci ty to col lect and analyseinformation relevant to them a nd their situation.Alternative approaches owe much to developments in LatinAmerica and Asia in the 1970s and 1980s. They are characterised by: A view of evaluation as an integral pa rt of the developm ent orchange process: 'reflection-action'. An und ersta ndin g of evaluation as an 'emp ow ering proce ss' ,rather than control by an external body. A recognition of subjectivity in evaluation . A recognition that different grou ps of men and wom en will havedifferent perceptions, which are equally valid. There is a need fornegotiation during the process of evaluation, to reach consensusabout conclusions and recomm endations. A tendency to use less formal techniques such as unstructuredinterviews, and participant observation.


24/96


25/96

24 A basic guide to evaluation for development workers A misuse of evaluation: to fudge decision-taking, for exam ple. Cu ltural differences in ways of wo rking and expressing things.

Evaluation will work best where it is understood as an integralpart of development work, and where there is an environmentwhich encourages critical reflection. Otherwise people can feelinsecure about their role and jobs. Morale can be affected byadverse f indings, that are insensi t ively communicated . Muchdepends on where the decision to initiate evaluation comes from,and how the results are used.Staff of funding NGOs should be encouraged to take an activeinterest in upgrading their knowledge and skills, to enable them toengage confidently in evaluation w ork, and to advise counterp arts.


26/96

25

4 The purpose and use of evaluation

4.1 Reasons for evaluationThere are many different reasons why evaluations are carried out.Some good reasons are to measure progress and effectiveness; tolook at costs and efficient use of resources; to find out if it isnecessary to change the way things are being done; and to learnfrom what has happened in order to make plans for the future.These are all constructive reasons, within the control of thoseorganising their work.There are other, less good, reasons why evaluations are carriedout. An evaluation might be dem anded by a funding agency, whoare wondering whether to go on supporting a project; or therem igh t be a s t a tu to ry re qu i rem en t fo r eva lua t io n f rom agovernment department. Evaluations are sometimes done as partof a research project as a way of testing out new techniques forgathering information; or sometimes because people who raisefunds for an organisation need something to put in their publicitymaterial. Evaluations may be done routinely, as a matter of policyat some high level of the organisation, without anyone being veryclear about why they are done. Evaluations done for some of thesereasons are likely to be seen by those mo st directly involved in thework under scrutiny as being imposed from outside. The peoplewho are taking part in the activities being evaluated are probablynot fully informed in advance, and have little say in the matter ofwhat is evaluated and how. They will seldom be told about theresults of the evaluation.

Sometimes, expensive and disruptive evaluations have beencarried out by funders when decisions have already been madeabout whether or not to continue financial support. Evaluationscan be used as excuses for getting rid of certain staff, when whatwas needed was a managem ent review. Where there is a suspicion


27/96

26 A basic guide to evaluation for development w orkersof serious misconduct within an organisation, it is better to do anaudit than an evaluation. Evaluation can be misused, to cover upweaknesses in a programme by only focusing on what is 'good'.Conflicting groups within a project or organisation may demandan evaluation, when what is really needed is a facilitator to dealwith the conflict, after which an evaluation should be a moreconstructive p rocess.

If used well evaluation can contribute to development action,but i t cannot solve every problem, or answer every question.Above all, evaluation cannot be a substitute for good managementand firm decision making; it can only provide information to helpin these processes.

4.2 Judging success and failureSometimes evaluations are expected to show clearly whethe r or nota project has been 'successful'. It is often very difficult to showclear evidence of success, because this may depend on so manyfactors, some of which are beyond the control of the project. It isoften easier to show failure, and unfortunately, evaluations oftenconcentrate on this. Another problem is that different groups ofpeople will have different perceptions about what constitutesfailure and success. The time-scale can also be a factor: a projectm ay be deem ed a failure a t one stage, bu t several years later it maybecome clear that there were some positive effects. A failure maywell spark off other activities that lead to positive change.

4.3 Measuring quantity and qualityEvaluat ion is concerned with measurements . Evaluators wil lalmost certainly want to know details of costs, and the quantities ofresources that the project uses. Many of the inputs which a projectuses (salaries, materials, tools, petrol) are measurable. Evaluatorswill also want to assess the activities that have taken place, and thechanges that have resulted from project efforts, and want to knowhow many people are involved, and the products or services which


28/96

The purpose and use of evaluation 27the project generates. These may sometimes be very obviouslymeasurable in terms of numbers: acreage cultivated, vaccinationscarried out, springs protected. Whatever can be expressed in termsof numbers, amounts, and quantities are termed the quantitativeaspects of evaluation.Of course, not all the inputs or changes are measurable in thisway . How do you m easure advice given, or level of self-confidenceachieved? Evaluators must be very careful that they do not justconcentrate on what can be easily counted, but must be sure theytake account of the uncountable factors too: such things as beliefsand a t t i t ud es , level of kn ow led ge o r sk i l l , be ha v io ur , andmotivation. These qualitative factors may be extremely importantin determining whether projects are successful or not.It is often what people think about a programme which is thereal ly s ignif icant factor in their level of involvement andcommitment to it, and therefore whether it achieves its objectivesor not. In order to find out why a programme has followed acertain course, it is vital to look at the pro gram me as a whole. It isnot enou gh to know that a program me has succeeded or failed: it iseven more important to know why. That means taking account ofall the unmeasurable factors as well as the things that can becounted.

4.4 What evaluation can do1 Evaluation can improve the m anagem ent of program m es andprojects and related activities, and point to the better usage offunds and other resources.2 Evaluation can help people to learn from experience so as toimprove the relevance, methods, and resul ts of projects andprogrammes, for current and future work.3 Evaluation can increase accountab ility: to donors: to meet their dem ands that resources are being usedeffectively, efficiently and for agreed objectives; of dono rs to the organisations they fund and work with;


29/96


30/96

29

5 Putting evaluation into practice

5.1 Developing evaluable projects and programmesThe aim of this guide is not only to explain wh at evaluation can andcannot do, but to help people to make sure that when evaluation iscarried out, it is useful, and lessons can be learned from it.Evaluation reports often say that 'it was difficult or practicallyimpossible to do a "proper" evaluation'. The usual complaint isthat there is no statement of how things were at the start of aproject, no idea given as to what kind of changes were hoped for,or wha t indicators could be used to measure change. If informationhas not been collected during the project, it is difficult to tracechanges in d irect ion, or measure achievements . Occasional lyeva luato rs com plain of an overload of unfocused information.Remember that evaluation is a tool for learning and bettermanagement. The aim is to assess what has taken place in order toimp rove fu ture work . Evaluat ion nee ds to be thou ght about ,discussed and negotiated from the very early stages of a projectbecause: It can be a highly sensitive event, often bringing toge ther peoplewith different agendas. It de pe nd s on information gen erate d at several stages of aproject.

5.2 The key questionsAsking a few key questions during the appraisal and planningstage of a project will help to make evaluation an integral part ofthe design and methodology of the project, and to plan for futureinformation needs.


31/96

30 A basic guide to evaluation for development workers1 Why and for whom: the purpose of the evaluation; why is itbeing d one, and who is asking for it?2 When: the timing of the evaluation; at what point in the life ofthe project or program m e will it take place?3 What: what is the scope and focus of the evaluation; what will itlook at, wha t are the key questions to answ er?4 Who: who is responsible for managing the exercise, who isgoing to carry it out (do the evaluating)?5 How: how is the exercise to be carried out, how will informationbe gathered ; how long will it take; w ha t criteria and indicators willbe used?

6 How much: what resources will be needed, financial an d other?7 What then: how will findings be reported and to whom; howwill findings be shared and presented to different audiences; whatis the procedure for follow up?

These questions should be asked when considering a projectp roposal fo r fund ing , o r decid ing how bes t to o rgan ise anoperational project. In the case of funded wo rk, it will be im portan tto distinguish between monitoring and evaluation that are internalto the organisat ion of the project , and that which relates toexpectat ions or agreements with the funders or sponsors . Afunding NGO must be explicit about how it is going to judge apar tner o rgan isa t ion ' s per fo rmance over t ime, and what i t sexpectations are in terms of evaluation. It is equally importan t thatan NG O, when receiving funds, is clear abou t wha t commitments iti s m ak ing to a pa r t i cu la r o rga n i sa t ion , an d how i ts ownperformance will be judged. These expectations should be thesubject of negotiat ions when f inancial and other support isdiscussed.

Each question will now be looked at in more de tail.


32/96

Putting evaluation into practice 315.3 Why and for whom is an evaluation to be done?5.3.2 Why is the evaluation being done?The reasons for doing eva luations can be group ed as follows:i To improve performanceThis is ' formative' evaluation, helping to 'form' or shape workwh ile it is still going on. This kind of eva luation is useful for thosedirectly involved in, or in charge of work. It can identify problems,and things that are working well and can be built on. It can alsocheck whether or not objectives have evolved, and be a way ofkeeping the different people involved in the project informedabout progress or the need for change.ii To make choices and decisionsThis is 'summative' evaluation, a summing-up of a project to makea judgement of how effective it has been in achieving results.Information from such evaluat ions can be used to comparedifferent ways of doing things, to help people make choicesbetween types of development action. This kind of evaluation canbe used by funders to decide on whether to continue support ornot.iii To learn lesson sThe prime purpose of an evaluation may be learning, so that theresults can be shared within a project, between projects, betweenorganisat ions and so on. For example, a funding NGO mayev alu ate a va r ie t y of w ork w i th a pa r t ic u l ar se c to r of thepopulation, to see which approaches have the best results, andwhy.iv To increase accountabilityEvaluations may be a condition of receiving funds, as donors wantto be sure that their funds are being used effectively. They mayalso want to find out if there are alternative ways of doing.thework.

Ev a lua t io ns to p ro m ote acco un tab i l i ty of do no rs andimplementing agencies to the wom en and m en they are su pporting


33/96


34/96


35/96

34 A basic guide to evaluation for development workersterm evaluation. An annual report, written by a project manager,can be a useful evaluative docum ent.iv. Final evaluationThis will happen at the end of a project, in order to learn lessonsabout how the project has been implemented and the results. Afinal report of a project written by a project manager can beevaluative, comparing objectives with what w as achieved.v. Ex-post evaluationsThese happen some time (often two years or more) after a projecthas finished. They look at impact and sustainability. They alsoconsider broader 'policy' issues. This kind of evaluation is rare inNG Os, and m ore effort should be put into prom oting such studies .5.4.2 Other times when evaluations take placeAn eva luation may be called for when:i A problem is identified by monitoring or on-going evaluation,and it is decided to investigate further.ii. A don or, for exa m ple a major co-funder , m ay requ est anevaluation to fit in with their own institutional requirem ents.iii. An organisation decides it wants to look at a particular aspect ofwork, maybe in relation to areas identified in its prog ram me plan.5.4.3 Other important factors relating to timingThere are other time-related factors to consider at an early stage,particularly when planning a large and expensive evaluation, thatinvolves a lot of different people:i. Climate, seasons and key events in the calendar W hen are the key religious festivals? W hen do peop le hav eother commitments? When might it be inappropriate to do fieldwork, or schedule m eetings? W hat polit ica l ev en ts can be foresee n, eg elec tions , pa rtycongresses?


36/96

Putting evaluation into practice 35 W hat are the bus y times in the year for orga nisa tions andprojects and governm ent d epa rtm ents, eg end of the financial year. Are there times in the year wh en access to par ticula r projectareas is impossible? Or when it is so hot that people will havedifficulty concentrating in discussion sessions? Are there significant patterns of work over the year? There maybe periods when there are few men around, because they havemigrated for work. The women left behind may have little time tospare for evaluation sessions. W hen are the harvest and 'le an' periods? H ow do they affectwomen and men an d their capacity to engage in an evaluation?ii . Use of people's time, or opportunity cost Do project staff and beneficiaries feel the re will be a benefit tothem in participating in an evaluation exercise, even though theyhave p lenty of other things to do? Are there some days when people are especially busy? Are theresignificant differences between men and women's timetables? Are particular interest grou ps unavailable at certain times?iii. The availability of external evaluatorsIf people from outside the project/organisation are going to beinvolved, they may well be in demand, and only available duringspecific periods.iv. The length of time the evaluation will takeRegular on-going evaluation may only involve project staff in acouple o f days every month o r every few months . Formalevaluations will need several months of preparation (but shouldnot last beyond one year, because there is a risk of losing continuityif people change their jobs) . The t ime wil l depend on whatinformation is available or needs to be prepared, and how manyweeks of field work are necessary. When the evaluation has beendone, several weeks or mon ths m ay be needed to study the results,prepare the report, and discuss it with those involved. The amountof time needed for more formal exercises should not be under-estimated. It is importan t to plan this into work .


37/96

36 A basic guide to evaluation for development workersField visits need to be long enough to enable the collection ofnecessary information and should avoid the 'fly in-fly out' style of

development. Discussion time should not be forfeited to endlessjourneys to make a fleeting visit to the most far-flung group. Themore actively involved the women and men who participate in aproject a re in the evaluation, the longer the exercise is likely to take.

5.5 What is the scope and focus of the evaluation?Three sets of questions should be built into any evaluation: W hat changes have taken place, and are these changes the oneswhich the original plan hoped for? Evaluations should give anaccount of what the project has achieved, or not achieved, andcompare this with expectations. W hat were the reasons for the success or failure? It is im portan tto know why things happened as they did, and analyse the factorswhich influenced the way the project progressed. W hat actions should now be taken? Evaluators should suggestcourses of action, in the light of answers to the first tw o questions .5.5.2 The focus of an evaluationAt the planning stage of a project decisions can be made aboutwhat to focus on in particular evaluations. The focus m ight be: a geographic area groups of beneficiaries types of activities time period a par ticular kind of activity over a gro up of projects to compareeffectiveness (a thematic evaluation) a group of projects all wo rking in the same area to see how theyinteract with each other.


38/96

Putting evaluation into practice 37Some pro jec ts may be so smal l tha t such d iv is ions areinappropriate.Nearer the time of preparing for the actual evaluation, it will benecessary to review, and possibly to change, the focus.If monitoring and evaluation are taking place within projects,th is in format ion can be gathered together fo r a themat icevaluat ion . Th is i s one way of making resources spen t onevaluation go further. It also helps to build up a picture to enableevaluation of overall programmes.There are a number of 'levels of interaction' which could be the

focus for an evaluation: the funding NGO's relationship to theproject, or the implementing agency; or agency relationships withproject participants.Sometimes it may be appropriate to have a much narrowerfocus. For example, those in charge of a project may be aware of aproblem bu t not sure exactly what it is. A 'process ' evaluation maybe carried out which will look at how a project is functioning, andits management.In some instances an evaluation is not appropriate. If there areserious doubts about a particular project, then a project audit m ightbe in order. An aud it will examine w hether a project conforms withrequired policies and procedures, and whether resources are beingused as planned.

5.5.2 What is the role of the funder in an evaluation?The interests of the funder may dominate the evaluation process,given its relative power. If an NGO is a sole funder, it may wish toevaluate the whole of the project. If it is only funding part of aproject, it may only feel able to evaluate that particular aspect ofthe work. In this case it may be worthwhile to involving all thefunders in an evaluation, so that donors' demands are coordinated,resources are pooled, and they gain a shared understanding of thework.However important the role of the funders, the implementingorganisation m ust also be able to state its views.


39/96

38 A basic guide to evaluation for development workers5.5.3 What criteria should be used for evaluation?The criteria that are commonly used as a focus for shapingevaluation questions are:i. Effectiveness: how far is the project or programme achievingobjectives? For example, the project might be concerned with thetraining of primary health care workers to improve their technicalskills, or disseminating information about sustainable agriculturalpractices. Achievements at this level are project outputs, or whatwas done . The i n p u t s are the human, f inancia l and mater ia lresources that were provided, to achieve the objectives.ii . Efficiency: what is the cost of achieving the objectives?A project may be very 'effective' in working towards its objectives,b u t it may be d o i n g so at very high cost (both social ly andeconomically), which is neither replicable or sustainable. Theremay be ways of achieving the sam e things mo re cheaply. This m ayinvolve look ing at how th ings are o r g an i s ed , w h a t t y p e oftechnology is being employed , as well as financial management.iii. Relevance: is the project relevant?A project may or may not prove to be appropriate to the needs ofthe women and men it is designed to help. There might be otherproblems that should take priority. The overal l approach andstrategy of the project should be consistent with the problem andintended effects.iv.Impact: what are the effects of the project?T h e i m p ac t of a pro jec t is the soc ia l , economic , t echn ica l ,environmental and other effects on individual men and women,and communities directly or indirectly involved in the project. Theimpact on women and on men may be different. Part of the impactmay have been changes in the institutions involved. The number ofpeople affected shou ld be estimated .Impacts can be intended and unintended, positive and negative,immediate or long-term. They can operate at the micro level (forexample, at household level) or macro level (they may affect awhole sector).Sometimes a distinction is made between shorter-term results(ou tcomes) and longer- term resu l t s ( impacts) . For ex am p l e ,


40/96

Putting evaluation into practice 39outcomes could be a change in the way people do things as a resultof the project, e.g. domestic servants may no longer be prepared towo rk for less than m inim um wag es. How ever, if the overall impac thoped for is an improvement in standard of l iving, and basichealth, through improved income and conditions, it could be verydifficult to prove the causality between the intervention of a smallp ro jec t and wh at ac tual ly ha pp en s . So m any o ther ex te rnalinfluences can have an effect.v. Sustainability: will project activities and benefits continueafter external support i s withdrawn?M any types of w ork canno t be financially self-sustaining. The issueis how far the men and women directly involved can take chargethemselves of finding the resources necessary. There are two keyaspects of sustainabi l i ty for social development programmes:social/institutional an d economic.Another aspect of sustainability is the effect the project has onthe environment and natural resources.vi. Progress: is the project achieving the original objectives, orhave these changed?An evaluation can also question the objectives and design of theproject itself. It ma y be concluded that a project is progressing verywell, even though it is far from meeting the original objectives.These may have been too ambitious, or irrelevant. The evaluationmay also look at who was involved in setting objectives: was itonly project leaders or were staff and beneficiaries also involved?5.5.4 Who chooses the criteria?Dif feren t pe op le invo lv ed in a p ro jec t ma y ha ve d i f fe ren tperceptions of what is 'success' . Funders and funded should beopen about the criteria that each will use for determining changeand progress , and how these cr i te r ia wi l l be t rans la ted in toindicators relevant to a particular situation. Project organisersshould also be allowed to state how they will judge their ownwork, given what they say they are setting out to do.Much more work needs to be done to enable the women andmen at the grassroots level to be able to state wh at 'me asur es' theywill use to judge the prog ress of projects design ed to benefit them.


41/96


42/96

Putting evaluation into practice 41that change is taking place. Sometimes it may be helpful to identifythese crucial indicators by using ranking exercises.

There is some debate about the extent to which indicatorsshould be objectively verifiable; that is, that different people wouldtake the same view about how change can be measured. At theinitial planning stage, the different participants in a project shouldhelp to choose the indicators. Different groups of people are likelyto choose different indicators according to what they think theproject is trying to do, and the terms in which they see positivechange. It is especially important to elicit the views of both womenand men. It may be difficult to reach agreement on how changeshould be measured, but unless everyone involved is clear aboutwh at the project is trying to achieve, and the changes which shouldresult, it is unlikely that the project will operate without conflictswhich could damage its effectiveness. Discussion about the choiceof ind ica to rs can reveal the d i f feren t expecta t ions amongparticipants, and negotiations should continue until a satisfactoryconclusion is reached.

As well as choosing indicators, decisions need to be made at theplanning stage about how the information about them will be col-lected. This could be through regular reports and the keeping of reg-isters, as part of the routine administration procedures; by researchstudies; or by special workshops. Indicators should be 'tracked' dur-ing implem entation, as part of systematic m onitoring, and it is likelythat some may prove to be inappropriate and can be dropped. Insome situations, tracking indicators can identify the distributionaleffects of a project, for example, the different effects of a forest pro-tection project on people who are totally dependent on forest prod-ucts compared to those who have other sources of livelihood.In the case of project proposals whose objectives are initiallyvery vague, such as 'to improve the quality of life' or 'to helppeople understand the forces that affect their lives', the process ofdeciding on indicators can be very helpful. By defining the actual

changes which would need to take p lace in order for theseobjectives to be achieved, the objectives themselves can be mademore realistic. 'Improving the quality of life' could be made moreprecise by ident ify ing the indicators which would show thedetailed improvements that are expected.


43/96

42 A basic guide to evaluation for development w orkersIf programme objectives and indicators were not defined at theoutset of a project then it will be necessary to define them when

carrying out an evaluation. This can be done by asking the variousparties involved to identify what they are actually doing, and whatresults they are expecting.

5.6 Who will manage the evaluation and who willcarry it out?Deciding who should be involved in any evaluation and on whatterms, can be a cause of conflict. The decision should be negotiatedin principle during the appraisal and planning stages of theproject. The people w ho m ight be involved are: the men and wom en beneficiaries/users of the project project staff project managem ent representatives of funding agencies: eg foreign N GO , UN , EEC,bilateral and mu ltilateral agencies, local corporate dono rs local or national government officials consultants.

The different responsibilities, depending on the type and size ofthe project and the proposed evaluation, are: deciding that evaluation will happ en funding the evaluation m anaging the evaluation carrying out the evaluation prov iding information.

Responsibility for ensuring that an evaluation is carried out willlargely be determined by who is controlling it. This may be theimplementing agency, project staff, the funders, the men andwo men at the grassroots, or a combination of these.


44/96

Putting evaluation into practice 435.6.1 What skills are needed within the evaluation team?Skills needed may be som e or all of the following: ability to write a clear, concise report in lang uag e X ability to lead and manage an evaluation facilitation skills and know ledge of gro up dynam ics capacity to analyse insight into the local situation, practices, beliefs knowledge of the coun try and the locality know ledge and experience of a particular type of work experience of participatory meth ods ease of access to wom en an d men in the project area wide know ledge of the NG O scene in the country sound un derstan ding of how interventions can affect men andwomen differently.

In an evaluat ion team, i t is bet ter to have members withdifferent skills rather than expect to find one evaluator with everyskill need ed. Teams should be in propo rtion to the size of the w orkto be evaluated . Two to four may be a reasonab le number ,allowing different eva luato rs to look at different aspec ts of work. Ifthere are evaluators who need translators this will also affect theteam size. Obviously if the project or evaluation is very small thenit is likely that one evaluator will be sufficient.In some areas, projects and organisations have engaged in 'inter-p ro jec t ' evaluat ion , when s taf f o f d i f feren t o rgan isa t ions ,undertaking fairly similar work, have evaluated each other. This isonly effective where staff can trust each other.


45/96

44 A basic guide to evaluation for development workers5.6.2 Who can do the evaluation?

The choice of who should evaluate, will depend to a large extenton the purpose of the evaluation and the stage of the project atwhich the evaluation happens. (See Appendix 5 for the pros andcons of external and internal evaluation.)Many women and men involved in a variety of developmentprojects are actively involved in monitoring and evaluating theirwork (self-evaluation). But sometimes community members andproject staff collect information to be analysed by other people, and

never see the results. If they do eventually receive a report it maybe in a language that they have difficulty understanding. If thishappens evaluation can be seen negatively as something that isdone by and for 'somebody else'.The people who actual ly carry out the evaluat ion may beexternal evaluators, who are not part of the project, or staffmembers who have training or experience in evaluation, or acombination of these.

i. External evaluationTraditional evaluation relies on formal short-term exercises doneby outside 'experts', often reporting directly to the funding agency.The assum ption is that people w ith no relationship to the project ororganisation will make a more 'objective' assessment.Formal external evaluation is appropriate when looking atissues of accountability, or if there is a need for particu lar expertiseo r exper ience wh ich an ex te rna l pe r son migh t have . Suchevaluat ions can be expensive, part icular ly when contract inginternational consultants who may cost many times more thanlocally contracted consultants in some places.ii . Internal evaluationProject staff with suitable training can carry out an evaluation, or aspecial evaluation function can be given to a group of staff. (Aperso n from another p art of the same organisation, bu t not directlyinvolved in the project, may still be considered internal.)Internal evaluation can contribute to increased understandingand better planning within a project, and help to strengthen theorganisation, because reflection and learning is done by thosepeople responsible for carrying out the work.


46/96

Putting evaluation into practice 45iii. Self-evaluationA group o f peop le can eva lua te and make judgemen ts andsuggestions about work they are directly responsible for. Internalself-evaluation is valuable if done regularly as part of the normalproject work, perhap s six-monthly or an nually.An outsider can be asked to help with the process. The outsiderin this case is not an evaluator passing judgement, but someonewho can help the group itself in its questioning.iv. Joint evaluationAn evaluation can be unde rtake n jointly by different stakeholders.For example, the directors of a funding agency and the head of animplementing organisation may decide to carry out an evaluation.They would agree the terms of reference (see below) together andthe evaluation could be carried out by one or more people chosenby them, e i the r members o f the o rgan i sa t ions o r ex te rna lconsultants. However, there is a danger that the evaluation teammay become too large to function efficiently, because all thedifferent stakeholders involved want to appoint someone to theteam wh o will represent their views.Many funders have tried to do this kind of evaluation in thespirit of partnership. But if the evaluation is insensitively imposed,and is closely related to questions of renewed funding, it is nottruly a pa rtnership.5.6.3 Participatory evaluationParticipatory (or sometimes participative) evaluation involves thewomen and men staff and beneficiaries of a project. It can be aninternal or external evaluation. True participation is when thoseinvolved are deciding and have control of the evaluation. Theydecide when it should happen, how it should be carried out, andby whom.

In its ideal form participatory evaluation is a way of doingevaluation which makes it an integral part of the planning andimplementat ion of development act ion. The women and mendirectly involved in the project are at the centre of the process ofevaluation. As a group, they come together to think about theproject and what it has achieved.


47/96

46 A basic guide to evaluation for development workersParticipatory evaluation is based on a people-centred approachto development and the belief that ordinary men and women have

the capacity to change things. It is a form of learning, where theexperience of the process of evaluation is as important as thefindings. It is pa rt of a whole app roach to developm ent, rather thana tool or technique that can be applied (as an afterthought) in anycontext. Project staff in a conventional top-down organisationcannot suddenly be expected to be confident participants in anevaluation; nor can they necessarily be expected to enable theconf iden t par t ic ipat ion o f benef ic iar ies , when th is has no tprevious ly been part of their way of working.As with all evaluation, it is important to define clearly who isparticipating and on what terms. People often ask how they canmake an evaluation 'as participative as possible', and if there are'participatory techniques' they can use. But such questions show alack of understanding about what participation really means, andits full implications.Sometimes what is called participatory evaluation should morecorrectly be termed consultation. Different groups of men andwomen are asked their views about a project, or a communityhosts the team of evaluators as they go about their field work.There i s cu r ren t ly a g rowing in te res t in u s ing so -ca l l ed'Participatory Rural Appraisal' techniques in an evaluation; buts imply us ing the techn iques wi l l no t in i t se l f ach ieve rea lparticipation. Without real input and control in the design andimp lementation of the evaluation, people cannot be said to be trulyparticipating.However, there are occasions when a participatory approach isprobably not appropriate; for example, when an audit is beingcarried out, or if there are serious management problems to beinvestigated.5.6.4 User evaluationThis is a process in which the male and female beneficiaries ortheir representatives assess an organisation and/or the work it ispromoting. These evaluations can be organised by the projectparticipants themselves or by project staff, or by an externalconsultant.


48/96

Putting evaluation into practice 47I t is important to ensure that the quest ions d iscussed arerelevant to the participants, and the organisers should be honest

about how much change is possib le. Many apparent ly 'open 'evaluations of this type have ended in demoralisation for thepeople taking part because of the limits to change.Feeding back f indings is important in al l evaluat ion, butespecially in this kind of exercise, where participants are beingasked their views. It is important for project participants to knowthe reaction of project organisers to the findings. Such evaluationsare a way of increasing accountability of funders and implementers

to project beneficiaries, and require confidence on the part ofproject managers and staff.

5.7 How will the evaluation be done?Answering questions about why and for whom an evaluation isbeing done, and who should do it, should make it easier to decidethe approach that is appropriate and what methods and toolsshould be used . You will need to make choices about:i . Formal or informal methods?Formal m ethods: are carefully plan ned eva luation exercises, wh ich have formalterms of reference(TOR) are special even ts at a particular stage of a project (eg, mid-term ,final, ex-post) use external eva luators for increased accountability (this is oftendemanded by funders or government departments) cost more than informal exercises because of the need to contractconsultants, the detailed preparation and the scope need a final comprehensive report can be disruptive to the rhy thm of the project.


49/96

48 A basic guide to evaluation for development workersInformal methods: are simpler and do not require as mu ch preparation as formalmethods tend to be undertaken by people from the project itself or thoseassociated with it may occur as regular events throu gho ut the project, or as one-off even ts as and w hen need ed are most app rop riate for prov iding information for m akingdecisions about the way a project is being imp lemented may need a short repo rt documenting discussions, conclusionsand recommendations, often only useful to those who participated.(Final project reports prepared by project staff, or half yearlyinternal evaluation meetings would fall into this category.)ii. External or internal evaluators?In the discussion on 'who' should be doing the evaluation (section5.6), we looked at the relative pros and cons of internal andexternal exercises, as well as issues around participation. (See alsoAp pend ix 5.)5.7.2 What information will be needed?At the planning stage of a project, and certainly when discussing aproposal for support, it is important to build in the gathering ofinformation, for on-going monitoring and later evaluation.Information needs are shaped by the purpose, scope and focusof an evaluation. They should relate to available resources, timeframe, research and evaluation capacity at different levels.i. Baseline informationThe initial project document should provide information on aparticular situation, location, the men and women who are thefocus of a project, implementing agency and so on. Baselineinformation provides a point from which to measure change, toenable 'before' and 'after' comparisons to be made. Participatoryappraisal studies are being increasingly used to provide this typeof information.


50/96

Putting evalua tion into practice 49If this is not possible an evaluation may compare the targetgroup with similar groups of men and women who have not been

involved in the project; but because of all the other factors affectingpeople's lives, it is difficult to identify causal links between theproject and any differences which may be found.ii . On-going information gatheringWorkplans which state as clearly and simply as possible theobjectives and what is expected of different project activities helpto focus information gathering. At the planning stage of a projectan evaluation plan can be worked out, which defines points in thelife of projects (say quarterly, six-monthly or yearly) when resultsof activities and progress towards goals will be assessed. The planshould state how information on progress is to be gathered and bywhom, what indicators will be used, and how the results will befed back into on-going work.It is helpful if yearly work plans fit into a system of longer-termgoals. In this way progress can be reviewed annually, and overalldirection modified as needed. Doing this can help projects bringwork dow n to a more realistic level.

It is quite acceptable for project objectives and strategies tochange as a result of on-going monitoring and evaluation. Anychange s need to be ag reed be twe en the peo p le conce rned ,recorded, and the reasons g iven. In terms of learning fromdevelopment, it is as important to understand the processes atwork, as to reflect on resu lts.iii. Existing informationSome of the evaluation questions may be answered by existinginformation. This could be: docum ents such as the diaries and notes of project workers , fieldvisit reports, team meetings, and reports of evaluations that havebeen conducted at other times adm inistrative docum ents of project staff information col lected by other bodies that could providebackground and comparative data eg official reports and researchfrom government, UN and independent research institutes


51/96

50 A basic guide to evaluation for development workers re po r ts f rom o ther s im i lar p ro jec ts , espe cia l ly da tadisaggrega ted by gende r, area, age, or ethnic group .iv. New informationIt is im portan t for issues of cost, time, practicality an d usefulness totry and define how new information will be used, when it is neededand the level of accuracy required . The value of the information m ustma tch the resources needed for gathering it.

There are two m ain wa ys of collecting new information: ques t ion ing peop le th rough in terv iews, focus g roups andconducting surveys observing people and things on site visits.

These techniques are often combined, for example the growingpractice of rapid appraisal/assessment (RRA/PRA); knowledge,attitude and practice surveys (KAP); and case studies.v. Quantitative or qualitative measures?A good evaluat ion wil l use both quanti tat ive and qual i tat iveinformation. (See also the ear l ier d iscussions on measuringquantity and quality in section 4.3, and section 5.5.5 on the choiceof indicators.) Quantitative information is often described as'objective', yet someone has made a subjective judgement aboutwhat measure shou ld be used as the s tandard! The re la t ionbetween the two types of information is that quantitative data canbe used for measuring 'what happened' (the changes which tookplace), and qualitative information can be used for analysing 'howand w hy' things happened.Qualitative methods include observing activity, interviewing,and trying to gain a more complete understanding of complexch an ge s . A cr i t ic i sm of qual i ta t iv e d ata i s tha t i t i s no t'representative', or scientific, and therefore generalisations cannotbe drawn from it.S tudies are being done to compare information generatedthrough these two methods, to see what the difference is infindings, versus relative costs.


52/96

Putting evaluation into practice 515.8 How much will the evaluation cost, and whatresources are needed?At the planning stage of any project a budget should be drawn up toprovide the resources needed for both monitoring and evaluation.5.8.1 Budgeting for evaluationWhen drawing up a budget, try to estimate all the expenses whichmight be incurred:i. Financial costsThese wil l depend on the nature and size of the project orprogramme and the type of monitoring and evaluation envisaged.They will include: salaries of any people engaged for specific exercises daily expenses, travel costs (hire of vehicle, petrol) costs of meetings and w orkshops mater ials for product ion of reports and other methods offeedback.

If an organisation is planning an internal evaluation, the costs ofthis should be put into the budget. Costs might include the salaryand expenses of a consultant to come and work with the team.ii . Non-financial costsThere will be opportunity costs for project staff and participants,and others who may be asked to collaborate. People may be askedto spend t ime t a lk ing to ex te rna l eva lua to r s , o r a t t end ingworkshops, which will mean that other work will have to berescheduled.5.8.2 Matching scope to resourcesThe amount of money available will affect the scope and methodsof evaluation. Some more formal quantitative techniques, such as


53/96

52 A basic guide to evaluation for development workersfull-scale surveys, can be more expensive than some of the lessformal qualitative techniques carried out by project holders andparticipants themselves.Evaluations with a greater degree of participation of the peopleconcerned are likely to cost more in terms of time, and therefore ofopportun ity costs, at different stages of the evaluation p rocess.

Clarity of purpose will help in deciding what resources shouldbe used. For example, if evaluation is to provide information forinternal decision making a less detailed exercise will probably beadequate. If the evaluation is to improve accountability, then theresults should be detailed enough to provide this, and the cost ofthe evaluation will be correspondingly greater.

When drawing up a budget for a project, it can be helpful tohave an agreed sum allowed for evaluation. Some of the official aidagencies allocate 5 per cent of programme budget for evaluationactivity. For small projects, evaluation can be a relatively highpercentage of overall costs.

5.9 Reporting back and follow up: how to make sureevaluation information is usedDeciding at the planning stage who needs to be told about thefindings of the evaluation, and in what form, makes it more likelytha t an ev alu ati on w ill be useful a nd ha ve a po sitiv e effect.Different stakeholders will need different sorts of information, indifferent forms, for different uses.5.9.1 The uses of evaluation resultsResults can be used to: Improve organisation and managem ent, particularly throug henhanced communication between different levels of staff as aresult of the evaluation exercise. He lp in decision making , by indicating where action should betaken, where training or specialist help is needed, or whe re furtherresearch would be useful.


54/96

Putting evaluation into practice 53 Im prove p la nn ing , by p ro v id ing in fo rm at ion abou t pas tperformance, an d influence policy m aking. P rov ide in fo rmat ion fo r fund ra i s ing and advocacy , bydemonstrating how needs can be met as a result of developmentac t iv i t i e s , o r by showing how some majo r cons t ra in t s ondevelopm ent require action at national or international level. Imp rove public accountability, by dem onstrating that resourceshave been used effectively.5.9.2 How will findings be presented?The product of internal and more informal exercises that are part ofthe ongoing routine of a project team may simply be a record ofkey discussion points and actions agreed, why and by whom.

More formal evaluation studies should produce a report. Theformat and style of the report should be made clear to whoever isresponsible for writing it. (See terms of reference, section 6.3.2..) Itmay be necessary to present the report, or the summary, in adifferent form to the beneficiaries, perhaps by pictures or video tonon-literate groups.Resources may be needed for feedback workshops, seminars,p repara t ion o f r epo r t s and popu la r book le t s , t r ans la ted i fnecessary, as part of a strategy for sharing the findings of theevaluation w ith others wo rking in similar fields.5.9.3 Follow upThe evaluation should not be expected to come up with clear-cut,ready -made solutions. Evaluation can only provide the material fordecision making , the decisions have to be made by those manag ingthe project.All those involved in the project should meet to discuss theissues raised by the evaluation and report, and decide about thefollow-up measures, including a time frame for implementation. Itmay or may not be appropriate for staff of the funding NGO toattend such a meeting, or act as a facilitator, depending on whathas been their role in the evaluation.


55/96

54 A basic guide to evaluation for development workersBeyond the level of the project, the results can be used to buildup a picture of what does and does not seem to work in particular

areas of work. This can feed into debates on policy and overallprogramme planning. It is also important to identify if there isinformation that could be valuable for larger debates on publicpolicy, with major official funding agencies or government.5.9.4 Who owns evaluation and evaluation results?This relates to who is in charge of the evaluation process, and theway in which different people take part. If an outside agencycarries out an external evaluation with little or no dialogue ornegotiation with the project in question, it is likely that the projectparticipants will not 'own' the process or findings. (They may alsofeel resentment at being used for someone else's research.) This isespecially likely if there is no feedback of data to them. If theevaluation is intended to be truly participative, then a com mitmentto collective and individual ownership of the process and itsoutcomes is vital.Where there are several major parties involved, conflict can arisewhere there has been insufficient clarity as to the purpose of theexercise. This may cause confusion regarding the relative emphasisto be given to the process of the evaluation compared to theproduct. For example, where one objective tha t ha s been identifiedis 'learning', then everyone also needs to be clear who is to do thelearning, and in what way. Is the intention for the participants tolearn from the experience of being involved in the process ofevaluation, or are the funders anxious to learn, from a detailedreport , whether their money is being wisely used? Or is theevaluation designed to enable both kinds of learning to take place?The team leader needs to be capable of interpreting the learning forthe different parties.There are ethical questions to be considered in relation to theuse of information generated through evaluation. If a consultant iscontracted to do an evaluation, or facilitate an internal exercise,clear rules must be established about what they can and cannot dowith information gathered during the exercise. People givinginformation also need to feel sure that it will not be used against


56/96

Putting evaluation into practice 55them, by getting into the wrong hands. Trust is an importantingredient in all evaluation work.


57/96


58/96

Some practical considerations when planning an evaluation 576.2 How to make sure an evaluation w ill be successfulHere are some rem inders of points that have been m ade earlier: Be realistic abo ut eva lua tion , recognise th at it is a politicalprocess in which different views of development, hidden agendas,and uneq ual pow er relationships operate. Be clear abou t the pu rpo se of a par ticula r exercise and the keystakeholders involved. Ne gotia te prior ities , an d be clear abo ut objectives; a singleexercise cannot answer every question. Be clear about wh at evaluation can and cannot do. Do not use itfor the wrong purposes. Evaluation is not the same as decisionmaking. Plan evaluation activity as an integral part of project work , sothat projects and programmes are designed with evaluation inmind. Discussing and planning for evaluation at an early stageensures that people expect it to happen, and so feel less threatenedby it wh en it does happen . Be clear abo ut wha t pe op le's responsib ilities are in differentevaluation exercises, so that they know what is expected of them. Choose an appro ach and m ethods ap prop riate to the type ofwork being evaluated and the questions being asked. Involve peop le, particularly those who will use the information,from the outset, so they 'ow n' the process, and will make use of thefindings. Be prep ared to ada pt and refine plans so that evaluation can becarried out at an appropriate time. Encourage feelings of respect and trust among all involved.

6.3 Planning and managing an evaluationDetailed planning should start well in advance. Check that theplans for evaluation made at the assessment stage of the project are


59/96

58 A basic guide to evaluation for development workersstill valid; it may be that the passage of time and the developmentof the project, or the quality of monitoring, makes the originalplans redundant.A meeting of the stakeholders should be called by the personorganising the evaluation, to confirm the underlying principles ofthe original plan, and establish the operational details. If theevaluation is to be on a large scale, you may need to appoint a'steering committee'. Make sure that everybody feels confident thattheir interests w ill be faithfully represented by this com mittee.

The stakeholders need to reach agreement on the principalobjectives of the evaluation. There are likely to be some obviousobjectives, which everyone will agree about, but other objectivesmay be put forward which are only regarded as important by anindividual. It is not always possible to reach consensus, and thepower relations between the stakeholders may be such that oneparty may have more authority. For example, a major hinder mayinsist on external evaluation when there is a question of furtherfunding. Usually, however, there is a lot of scope for discussionabout specific evaluation objectives, and the methods to be used;and some bilateral and multi lateral funders are very open tosuggestions from funding NGOs about methodology.It may not be possible to dispel entirely the fears of someparticipants about a planned evaluation. The value of their effortsis to come under scrutiny, and they may themselves have a veryhigh personal investment in the work. For th is reason, anyevaluation that may affect funding decisions should happen wellbefore the end of the funding period, and a sufficient winding-dow n period should be allowed (this may be a year or more).In order to make sure that the evaluation is useful, it is vital toconsult the people w ho are mos t likely to be able to use the results,at an early stage. This can avoid a waste of time and effort incollecting information which is unnecessary. If people are askedbeforehand about the kind of information which they would findrelevant, they are much more likely to find the results of theevaluation of interest to them, and to make use of the findingsw hen plann ing for the future.Agreeing on prior i t ies can help to l imit the scope of theevaluation, and make it more manageable. If necessary, rankingmethods can be used to help people reach agreement about the


60/96

Some practical considerations when planning an evaluation 59purposes and the questions to be asked. Once everyone involved isclear about the main purposes of the evaluation, then it is easier tosettle questions of method and approach. It is always easier tonegotiate when the various parties are open about what they reallywant . For example, i f the evaluat ion is in tended to increaseaccountability, and, in addition, a major funder is planning to usethe results of the evaluation in coming to a decision on extendingthe funding, then th is lat ter considerat ion should be freelyacknow ledged, so that there are no false expectations.You should allow plenty of time for the planning of a majorevaluation, to make sure everyone involved, or at least everyoneon the steering committee, is able to take part in meetings anddecide on a final p lan.6.3.2 Drawing up the terms of reference (TOR)Many evaluations suffer from the over-wide scope of the exercise.The terms of reference (TOR) become a shopping list, rather than aguide to focus the work.The TOR set out the formal agreem ents about the evaluation, itsscope, purpose, and the methods to be used, and outline thespecific tasks of the evaluation team leader. Those managing theevaluation process are responsible for drawing up the TOR. Wheremore than one person or agency is involved, it should be agreedw ho has ultimate autho rity. It m ay take several attem pts before thefinal version of the TOR is agreed by everyone. It is important thatthe TOR incorporate a gender perspective.

Good TOR pav e the way for a good eva luation, acting as a pointof reference throughout. They should be drawn up well in advanceof the date of the evaluation, in order to allow adequate time forplanning, selecting and employing evaluators, sorting out thelogistics, and briefing everyone involved. The TOR should reflectboth the needs of staff and others involved to learn from theirexper ience , and the need o f the o rgan i sa t ions to improveperformance and accountability. Evaluators may help to draw upthe TOR and should be asked to review and comment on thembefore beginning the evaluation. A good evaluator is likely to raisequestions about the initial TOR.


61/96


62/96

Some practical considerations when planning an evaluation 61specific tasks and responsibilities, particularly any writing andmanagerial tasks.6.3.3 ConfidentialityThe TOR should state any conditions about the subsequent use ofinformation gained d uring the exercise, by all team members. Thisshould cover both general confidentiality and the own ership of thespecific report. The evaluation team has access to a great amount ofcon f iden t i a l in fo rmat ion abou t the fund ing agency , theimplementing NGO, and their personnel, men and women at thegrassroots. It is crucial that the terms on which this information isprovided are made clear to all concerned at the outset. Similarly,the evaluators need to have confidence that they can be candid intheir observations and recommendations, without fear that thesewill be circulated indiscreetly.

6.3.4 Recruitment and selection of evaluatorsChoosing evaluators with the appropriate skills and experience isimportant to the success of an evaluat ion. The purpose, themethods chosen, the type and size of the project or programme,and the budget available will determine whom to recruit. Youshould consider the following po ints:1 The rang e of skills and exp erience need ed . This includes theissue of communication between the evaluator and the subject ofevaluation e.g. if the evaluator cannot speak the language, goodinterpreters will also have to be hired, or if it is not acceptable formen to speak with women in a part icular society , i t wil l benecessary to have a wo men w ith the relevant skills on the team.2 The gender balance, and gender-awareness, of the team.3 The balance between evaluation skills, technical know ledge, andin-depth local knowledge. Remember that, while empathy andknowledge are important, bringing in people with a differentcultural perspective can be stimulating; for example, people withexperience of working within governments, as well as those froman NGO background.


63/96

62 A basic guide to evaluation for development workersAt least two to three months may be needed to recruit suitableevalua tors. Good consultants are likely to be booked u p for several

months in advance. Use the widest possible network of contacts toident ify candidates . Sources include other donors, academicinstitutions (local and international), partner organisations.Exper ience is ve ry im po r ta n t . Eva lu a to r s shou ld ha veexperience of similar evaluations, and an understanding of thekind of work or project involved, and at least some members of theteam should have a good knowledge of the country, and of thelocal situation.Once references have been checked and consultants selected, theTOR should be discussed with evaluators before a contract isagreed. Staff of funding NG Os m ust be sure to observe the term s ofthe recrui tment pol icies of their organisat ion. Contracts forindividual members of the team should specify the written reportsthat they will be expected to prepare; clarity about this aspect ofthe work will be helpful to the team leader.

6.3.5 Essential documentationEvaluators need to see all the relevant inter

Documents

A Basic Guide to Evaluation for Development Workers