Who Are the Question Makers - Undp Participatory Evaluation

Who are the question-makers?

Who Are the Question-makers?A Participatory Evaluation Handbook

OESP Handbook SeriesEditorial Board: Sharon Capeling-Alakija, Carlos Lopes, Abdenour Benboualiand Djibril DialloManaging Editor: Janet Donnelly

© OESP, 1997Office of Evaluation and Strategic PlanningUnited Nations Development Programme

One United Nations Plaza, New York, NY 10017

Foreword

Introduction

● Purpose of the Handbook

● Audience

● Organization of the Handbook

Part One:Participatory Evaluation: An Overview

● Evolution of the Participatory Approach

● Participatory Action Research

● Rapid Rural Appraisal and Participatory Learning and Action

http://www.undp.org/eo/documents/who.htm (1 of 4) [29/01/2002 18:59:07]


● Farming Systems Research

● Self-Evaluation and Beneficiary Assessments

Part Two:Participatory Evaluation

● Functions of Participatory Evaluation

● Differences Between Participatory and More Conventional Evaluations

● Rationale for a Participatory Approach to Evaluation

Part Three:Participation within UNDP

● Participation as a Philosophy

● Participation and UNDP Programming

Part Four:How to do a Participatory Evaluation

● Framework

● Practical Considerations

● Possible Reactions to the Participatory Evaluation Process

Part Five:Training Module ''Money and Mambas''



● Introduction

● Purpose

● Audience

● Format. ● Guide to the Facilitator

Money and Mambas or Listening to the People. Case Study: Participatory Evaluation of a Water and Sanitation Project

Questions for Group Work

Annexes:Annex I.

● Glossary

Annex II.

● A Sampling of Tools for a Participatory Evaluation

Annex III.

● Who's Doing What? Organizations Supporting Participation

Annex IV.

● Resource Persons, Groups and Institutions



Bibliography


Who are the question makers? - Foreword

Foreword

"Who Are the Question-makers?: A Participatory Evaluation Handbook" is the first in a new handbook series being launched by the UNDP Office of Evaluation and Strategic Planning (OESP). This series aims to provide development practitioners with tools, examples, exercises and case studies that will help in translating plans into actions and theory into practice.

This handbook evolved from work undertaken during the past seven years in UNDP, first by the Central Evaluation Office (CEO) and in the past two years by OESP. Interest in participatory evaluation bas deepened as UNDP has striven to improve interaction with and serviced delivered to end-users and beneficiaries, particularly at the grass-roots level. Work began in late 1989 with an occasional paper commissioned by CEO entitled "Participatory Evaluation: Questions and Issues" prepared by Kim Forss. This was followed by a research and pilot test phase during which three project evaluations were conducted along participatory lines.

While the original intention of undertaking these pilot exercises was to produce guidelines on participatory evaluation, as we gained more experience, we realized that participation in not a process that can be mandated from above or guided from the centre. In this realization, we in OESP travelled the same path as the authors of the World Bank Source Book on Participations who found that "the best way of learning about participation is to experience it directly. The second best way is by seeing what others have done in the name of participation, talking to them, and seeking their guidance".

http://www.undp.org/eo/documents/wfore.htm (1 of 3) [29/01/2002 18:59:12]


The present handbook has been designed to capture OESP's learning on participation and share it with you in the hope that you will be motivated to try the best method (doing something yourself) through experiencing the second best method (reading about what others have done)! To achieve this, the handbook is divided into two distinct sections: the first provides an overview of participatory evaluation while the second includes a self-contained training module consisting of a case study that documents an attempt at participatory evaluation, warts and all. This case study has been successfully used in training courses conducted for Junior Professional Officers during the past two years.

A few works of explanation will help you to understand why we chose the title "Who are the Question-makers?" As Michael Quinn Patton, one of the most eloquent and persuasive advocates for user-focused evaluation, puts it "Language matters. It simultaneously suggests possibilities and communicates boundaries." In this instance, the unfamiliar conjunction of "question maker" as distinct from the more usual "question asker" was chosen deliberately because it communicates a more active involvement of the stakeholder in the process of evaluation. A question-maker has more responsibility for seeking the answers than a more passive question-asker.

This handbook has been a collaborative effort by many people and we would like to take this opportunity to acknowledge their contributions: our thanks to Jennie Campos and Francoise Coupal, who helped to put the handbook together; to Kim Forss and Claus Rebien, who put the original case study together; and to the informal brainstorming and feedback group of Nurul Alam, Abdenour Benbouali,



Janet Donnelly, Naheed Haq, Peter Hazelwood, Rosein Herweijer, Mala Liyanage, Rema Pai Nanda, Kaarina Valtasaari, Samir Wanmali and Rob Work. Carlos Lopes provided valuable insight and acted as a sounding board throughout the preparation of the handbook. Hearty thanks are due to the JPOs who participated in the induction coursed held in New York from late 1994 through 1996. The detailed feedback we received from these training sessions was of particular value to us in preparing section two. We would also lie to acknowledge Barbara Brewka for her excellent editing work, enforcing rigour in the way we express ourselves, and Maureen Lynch for coordinating the design and layout stages of the publication. Finally, like all successful projects, this handbook had its champion. Chandi Kadirgamar's unstinting devotion to this initiative at every stage from conceptualization to publication is a clear manifestation of her commitment to fostering participatory practices in UNDP.

We hope this handbook proves worthy of its name and is frequently referred to, not only because it proves to be a useful resource but also because it is enjoyable reading. We see this as the first version of many and would therefore look forward to hearing your ideas and suggestions on how it can be improved. Your input in expanding the annex on resource persons, groups and institutions who have experience in participation would be particularly appreciated.

Sharon Capeling-Alakija

Director

Office of Evaluation and Strategic Planning


Who are the question-makers? - Introduction

Introduction:

Experience has shown that participation improves the quality, effectiveness and sustainability of development actions. By placing people at the centre of such actions, development efforts have a much greater potential to empower and to lead to ownership of the results.

The UNDP Office of Evaluation and Strategic Planning (OESP) has been assessing the value and role of participation as part of its broader effort to redefine the function and role of evaluation within the organization. During this process, OESP has had to address several key questions, including:

● How can evaluation be a tool for development? ● How can evaluation build local capacity and contribute to a learning

culture? ● How can evaluation contribute to the achievement of sustainable

human development (SHD)?

● What is the value added of more responsive evaluation methods and would such methods require a change in attitudes and skills by UNDP staff?

In its search for answers to these questions, OESP has been experimenting with initiatives that involve greater participation of programme stakeholders and beneficiaries. Commonly referred to as "participatory evaluations", these experiments challenge the traditional way in which development is viewed and carried out.

While to some people, the participatory approach may represent a radical departure from past practices, others see it as a logical step in the evolution of development thinking and methods. For example, UNDP policies to promote decentralization, national execution of programmes and partnerships have all sought to transfer ownership to its partners in programme countries. Further decentralization is taking place within developing countries. Grass-roots efforts, bottom-up approaches, initiatives that empower are all focusing attention on the poor and disenfranchised, whose opinions and participation are increasingly being sought.

http://www.undp.org/eo/documents/wintro.htm (1 of 4) [29/01/2002 18:59:23]


The growing interest in participatory evaluation parallels the growth of such concepts as empowerment, democratization, partnership and sustainability. Each of these concepts attempts in one way or another to give a greater say to the spectrum of voices in our programming countriesnot only to national governments but also to civil society, communities and municipalities, the poor and the disenfranchised who have been the object of development cooperation and whose voices have not been adequately heard.

Purpose of the Handbook

In view of the growing importance that is being attributed to participation, this handbook has been prepared to:

● provide UNDP staff with a better understanding of what is meant by a participatory approach to evaluation and how they can support the participatory evaluation process;

● help to introduce participatory evaluations into UNDP programming, thereby enabling the multitude of stakeholders that are central to UNDP development effortsthe poor, local communities, Governmentsto have a much stronger voice through development efforts that are more responsive to their needs and that contribute to capacity-building at the local and national levels;

● strengthen the learning and management culture of UNDP.

Audience

The principal users of the handbook, which has been designed primarily for UNDP staff, include:

● Resident Representatives and Deputy Resident Representatives, who oversee the country programmes and make critical decisions about the allocation of resources, which programmes and projects will be evaluated



and the approach to be used; ● National and International programme staff and Junior Professional

Officers (JPOs), who are directly responsible for the day-to-day management of projects and who appraise, review or evaluate development activities and need assistance in designing field missions and preparing terms of reference (TOR).

UNDP staff may also want to share this handbook with colleagues who are interested in applying participatory evaluation techniques to their projects. Thus, it will also be useful for Government counterparts, project leaders and consultants who need to have a better understanding of how a participatory evaluation works and how it fits into UNDP programming.

Organization of the Handbook

This volume provides the information needed, and helps to develop the sensitivity and skills required, to support evaluations that place greater emphasis on stakeholder participation in the evaluation process. It is divided into five parts.

Parts one to four, which present an overview of the participatory evaluation approach, include:

● a brief description of the evolution of the participatory approach; ● a comparison of participatory evaluation with more conventional

evaluation approaches; ● a discussion of the role of participation in UNDP; ● a description of the framework of a participatory evaluation and a

discussion of some of the practical issues involved in doing such an evaluation.

Part five consists of a stand-alone package developed around the case study MONEY AND MAMBAS. It describes an attempt at undertaking a participatory evaluation of a rural water supply and sanitation project and focuses on the practical aspects of applying participatory evaluation techniques:



● Pre-planning, including negotiation of the TOR, assessing the participatory evaluation context and identifying enabling and inhibiting factors surrounding that context;

● Collaborative planning with stakeholders; ● Data-gathering and analysis; ● Reflection and follow-up.

This case study is presented as a training module which can be the subject of a miniworkshop to introduce staff to the practice of participatory evaluation. We suggest that this exercise can be accomplished within 3 to 4 hours.

A glossary of basic terms, examples of some of the basic tools that can be used in participatory evaluations as well as lists of manuals and resource persons, groups and institutions are presented in the annexes.


Who are the questions makers? - Part One

Part One:Participatory Evaluation: An Overview

Evolution of the Participatory ApproachThe emergence of what has become known as the participatory evaluation approach reflects much wider experimentation in development that has been taking place in various parts of the world since the 1970s. It has primarily involved development practitioners and social researchers in a wide variety of fields, e.g., adult education, sociology, rural development, agriculture and applied research. Only now has it entered the policy-making spheres of large development agencies.

What is increasingly being called participatory development began for some with the critical analysis of society and the inequities it generates, leaving the poor voiceless and dominated. For others, participatory development is less ideological or philosophical: it started with the exploration of more responsive techniques and approaches at the grass-roots level, involving the poor, project stakeholders and beneficiaries. For those involved specifically with evaluation, there has been a growing dissatisfaction with conventional modes of assessment that claim to be scientifically neutral and unbiased yet have had very little impact on how development activities are carried out.

The following pioneers or schools of thought have contributed to the emerging field of participatory development and, more specifically, to participatory evaluation.

Participatory Action Research Participatory action research (PAR) has its origins in the work of social scientists from developing countries who have been experimenting with PAR over the past 20 years. Influenced by such authors as Paulo Freire, Orlando Fals-Borda and Mohammad Anisur Rahman, the "basic ideology of PAR is that self-conscious people, those who are currently poor and oppressed, will progressively transform their environment by their own praxis. In this process others may play a catalytic and supportive role but will not dominate" (Fals-Borda, 1991:13). Along similar lines, Paulo Freire, in his book Pedagogy of the Oppressed, outlines an educational philosophy that actively involves the poor in critically analysing their social situation,

http://www.undp.org/eo/documents/whop1.htm (1 of 3) [29/01/2002 18:59:47]


thus creating the potential for them to transform their environment. Once considered radical, the work of these authors is gaining increasing prominence and is credited with critically challenging mainstream thinking and influencing the development of participatory development.

Rapid Rural Appraisal and Participatory Learning and ActionRapid rural appraisal (RRA) first emerged in the late 1970s, spearheaded by Robert Chambers at the University of Sussex, England, in response to lengthy assessment methods used in development. RRA enables donors to seek information and insight quickly from local people about local conditions. Over time, RRA sought to be less extractive and more participatory in the collection of information by involving local people in data-gathering and analysis through the use of popular education methods, such as mapping; transect walks; scoring and ranking with seeds, stones or sticks; and institutional diagramming. As the emphasis shifted from collecting data quickly to the involvement of end-users and learning from the experience, RRA became known as participatory learning and action (PLA). PLA activities have been undertaken in over 130 countries by development practitioners, NGOs and donors.

Farming Systems Research Research in this field emerged in the 1970s, mainly in response to concerns about the skewed benefits of the Green Revolution. In contrast to research station experiments, which were difficult to replicate in the field, systems research supported farmer-managed trials in which rural people selected alternatives for experimentation and implementation. It recognized the breadth of knowledge farmers had of their own interrelated systems of production and livelihood and supported experiments conducted by the farmers.



Self-Evaluation and Beneficiary AssessmentsThe term "self-evaluation" is most often used to describe a process of permanent, internal evaluation involving staff at all levels or beneficiaries with a view to generating information that can inform decision-making. NGOs, such as World Neighbors, academics and donors have been experimenting with the concepts of self-evaluation and beneficiary assessment. As a result of its experiences with beneficiary assessments, the World Bank views them as essential to building programmes that are responsive and relevant to recipients of Bank loans, providing Bank managers with the tools to improve the quality of development operations.

As mentioned previously, all of these approaches and schools of thought have influenced the emerging field of participatory development.


Who are the question-makers? - Part Two

Participatory Evaluation

Participatory evaluation, a dimension of participatory development embodying many of the same concepts, involves the stakeholders and beneficiaries of a programme or project in the collective examination and assessment of that programme or project. It is people centred: project stakeholders and beneficiaries are the key actors of the evaluation process and not the mere objects of the evaluation.

Participatory evaluation is reflective, action-oriented and seeks to build capacity by:

● ?providing stakeholders and beneficiaries with the opportunity to reflect?on a project's progress and obstacles;

● ?generating knowledge that results in the application of lessons learned?and leads to corrective action and/or improvements;

● ?providing beneficiaries and stakeholders with the tools to transform their environment.

Functions of Participatory EvaluationParticipatory evaluation thus serves four key functions, some of which concern the stakeholders and beneficiaries while others relate to the funding agencies.

● It helps to build the capacity of stakeholders to reflect, analyse and take?action. While such analysis should occur throughout the life of a project,?it is never too late to involve project recipients in evaluations at mid-?term or even at the end of a project. UNDP staff may also witness their?own growth and enrichment through their involvement in the evaluation process.

● It contributes to the development of lessons learned that can lead to?corrective action or improvements by project recipients. When project?stakeholders are involved in analysing problems, constraints and?obstacles, they can often propose solutions. Their sense of ownership of?the process, of final recommendations and of action plans makes them?much more likely to introduce necessary changes.



● It provides feedback for lessons learned that can help programme staff?to improve programme implementation. A participatory evaluation not?only looks into the past but also guides projects into the future.

● It helps to ensure accountability to stake-holders, managers and donors?by furnishing information on the degree to which project objectives?have been met and how resources have been used. Answers to these?questions will help programme managers make critical decisions about?continuing or terminating a project's funding.



The focus on lessons learned is an essential dimension of participatory evaluations. Such evaluations should help to guide projects into the future by giving stakeholders the tools with which to take corrective action. In addition, lessons learned should provide donors with the insight and tools to improve programme delivery and management

Differences between Participatory and More Conventional EvaluationsParticipatory evaluations differ from more conventional evaluations in several critical ways. Figures 1 and 2 illustrate some of these differences.

As shown in Figure 1, conventional evaluations have been more donor focused and donor driven. The donor is the key client, providing financial support and defining the TOR for the evaluation. Participation of project stakeholders in the definition of the TOR is minimal. More often than not, the evaluation is carried out more to fulfil a management or accountability requirement than to respond to project needs. An outside expert or evaluator is hired to conduct the evaluation.

The evaluator collects the data, reviews the project or programme and prepares a report. In most cases, stakeholders or beneficiaries play a passive role, providing information but not participating in the evaluation itself. The process can be considered more linear, with little or no feedback to the project.

In a participatory evaluation, the role and purpose of the evaluation change dramatically. Such an evaluation places as much (if not more) emphasis on the process as on the final output, i.e., the report. The purpose of the evaluation is not only to fulfil a bureaucratic requirement but also to develop the capacity of stakeholders to assess their environment and take action.

Stakeholders and beneficiaries do more than provide information. They also decide on the TOR, conduct research, analyse findings and make recommendations. The evaluator in conventional evaluations becomes more of a facilitator in participatory evaluations, animating workshops, guiding the process at critical junctures and consolidating the final report, if necessary, based on the findings of the stakeholders. The process is much more circular, as shown in Figure 2.

Participatory evaluations also call into question the notions that only scientific inquiry provides valid information and that outside experts or those independent of the project or programme somehow hold the ultimate truth. Participatory evaluations recognize the wide range of knowledge, values and concerns of stakeholders and acknowledge that these should be the litmus test to assess and then guide a project'sperformance.

While the participatory approach to evaluation poses its own challenges, it has the capacity to empower



recipients. The active participation of stakeholders can result in new knowledge or a better understanding of their environment. It is this new knowledge and understanding that can enable them to make changes they themselves have discovered or advocated. Stakeholders feel a sense of ownership of the results which does not come from an outsider or a donor.

LEVELS OF END-USER PARTICIPATION IN EVALUATION

Dimensions of evaluation/Levels of participation

Low Medium High

Evaluation initiator Commissioned or obligatory evaluation typically part of programme development. Meets institutional needs. Evaluation done to, on or about people.

External evaluator invites end-users to assist in one or more evaluation task(s).

Evaluation in which end-users collaborate with external facilitator or among themselves to asses, review and critically reflect on strategies formulated for them.

Purpose Justify or continue funding. Ensure accountability. Levels of funding or sustained support.

Gain insights into development activity from end-users' perspective. Shift focus from institutional concerns to end-user needs and interests.

Promote self-sufficiency and sustainability by linking end-users to evaluation planning cycle. Develop relevant, effective programme decision-making based on end-user views, opinions, recommendations. Increase ownership in & responsibility for success-failure of development interventions.

Questions-maker(s) ? Agency heads, administrators, outside clientele, persons distances from evaluation site.

End-users with external evaluator at various stages of evaluation generally determined by the evaluator.

End-users, external facilitator, persons most affected by development intervention.



5 Method(s) Established research designs, statistical analyses, reliance on various quantitative methods. Product (findings) oriented (mathematical in nature). Dominated by math whiz kids.

Qualitative methods favored but also includes quantitative methods. Values a process focussed on open-ended inquiries. Uses methods that give voice to voiceless.

Relies on highly interactive qualitative methods but does not disregard quantitative tools. "The process is the product". Inventiveness and creativity encouraged to adapt the methods to the context being evaluated.

Evaluator's versus Facilitator's Role

Evaluator takes lead in designing evaluation. Formulates questions/survey forms with no input from those evaluated. Steers overcome by setting design.Assumes objective, neutral, distant stance.

Evaluator works collaboratively at various stages with end-users. Is partner in evaluation and imparts evaluation skills. Shares lead with end-users.

Evaluator becomes more of a facilitator. Facilitator acts as catalyst, confidante, collaborator. Takes lead from end-users. Has few if any pre-determined questions.

Impact/Outcome Reports, publications circulated in house. Findings rarely circulated among end-users. Findings loop into planning stage with little input from end-users.

Shared data-gathering but limited participation in data analysis. End-user views loop into planning stage. Increased understanding of end-user experiences.

End-user more capable of meaningful decision-making based on effective involvement in evaluation. Findings become property of end-users or community. Participation in analysis is critical.

The purpose, methods, role of the evaluator and impact of the evaluation will vary considerably depending on the type of evaluation and the level of participation of donors, stakeholders and beneficiaries, as shown in the following table. In evaluations with a high degree of participation by stakeholders and beneficiaries, for example, the stakeholders rather than the donors become the question-makers and the evaluations are driven by the stakeholders and recipients.

Rationale for a Participatory Approach to EvaluationAll too often conventional evaluation reports sit on shelves or desks and have little or no impact on project beneficiaries or development practice either in the field or at headquarters. This can be attributed in part to a lack of input or feedback from those whose lives are affected by a programme or project, who have their own perceptions of what they need and how things should be done, yet who have little or no opportunity to make their views known.



Participatory evaluations breathe life into more conventional evaluation approaches by involving project stakeholders in all aspects of the evaluation: designing the TOR, collecting and analysing data, formulating recommendations and making changes in the implementation of a project's activities. In addition, supplementing more formal methods of inquiry, such as standard questionnaires or one-on-one interviews, with nonformal techniques can yield richer information than the use of only formal methods.As a result of the active involvement of stakeholders in reflection, assessment and action, a sense of ownership is created, capacities are built, beneficiaries are empowered and lessons learned are applied both in the field and at the programme level, increasing effectiveness. There is growing evidence that sound, sustainable development requires their participation throughout the development process in project planning, decision-making, implementation and evaluation.


Who are the question-makers? - Part Three

PARTICIPATION WITHIN UNDP

Participation as a Philosophy It is not enough to advocate participation; one must also be participatory. Many have argued that to be truly participatory, it is first necessary to look inward at the attitudes, behaviours and practices that create the culture of an organization.

What does it mean to be participatory in the workplace, especially in an organization such as UNDP, which is characterized by a mosaic of cultures, attitudes and behaviours? Some staff may feel that their work environment is highly participatory while others experience attitudes and barriers that prevent their voices from being heard and taken seriously. Bureaucratic structures may impede participatory processes.

Being participatory involves more than using a particular technique or approach. It reflects an attitude towards human interaction and the way that individuals learn from one another that should permeate all levels of an institution from headquarters to the field and to the project level. It means exhibiting a willingness to share decision-making, power and perhaps even resources.

Empowering others often requires letting go of one's own power or at the very least sharing the power that one already has, a challenge to those institutions and groups unaccustomed to working in a collaborative or participatory manner. However, staff who have been involved in participatory exercises have often commented on feeling empowered and seeing creativity unleashed either individually or as a group as a result of that experience.

Creating a transparent environment is an important dimension of being participatory. Hidden agendas become open agendas where everyone has the opportunity to influence and shape events. While this may appear as idealistic to some, others view it as a need that it is now time to fulfil.

Participation and UNDP ProgrammingHow does a participatory approach fit into UNDP programming? As illustrated in the circular diagram, such programming is traditionally



composed of at least six stages:

● identification of project/programme concept or problem ● project/programme design ● appraisal ● implementation ● monitoring ● mid-term evaluation and/or end-of-project evaluation.

In addition, the Project Performance Evaluation Report (PPER) and the Tripartite Review (TPR) are undertaken on a regular basis.

Participatory techniques and evaluations can be used at different stages of a project: in the design stage, where baseline data and needs analysis are required; at mid-point, to review progress and reorient project activities; and at the end to assess the project's achievements. In a participatory approach, key variables include: who identifies the project concept or problem; who makes the decisions; and who conducts the evaluation. However, use of such an approach implies greater involvement of the stakeholders in all aspects of the project cycle.

Identification of the Project and Programme Concept or ProblemProjects and programmes are more likely to be successful if they respond to a real need identified by the local population, institution or government ministry. This means involving the key stakeholders in identifying their needs and assessing the most appropriate options for meeting those needs.

Project and Programme DesignTraditionally, projects and programmes are designed by an outside expert according to UNDP specifications. The role of the stakeholder is primarily to provide information that can then be incorporated into the project and programme. However, projects and programmes based on participatory approaches involve the stakeholders in fleshing out the design. Through workshops and focus groups, stakeholders participate in preparing the logical framework analysis (LFA) (an example of an LFA is contained in annex II) and the overall framework for the execution of the project and programme.



Project and Programme AppraisalAn appraisal, which is usually completed once a project and programme have been designed, can be performed by both the stakeholders in the field and UNDP staff.

Project and Programme ImplementationA project or programme with a high degree of participation will customarily involve stakeholders in its implementation. Participation may mean involving them in decision-making about such issues as the planning and organization of activities, the use of resources, and the delegation of roles and responsibilities.

Project or Programme Performance Evaluation ReportThe PPER, an instrument for monitoring progress, is usually prepared by project or programme management once a year, three months before a tripartite review. It records the activities and progress made in the production of outputs, describes the activities and results to be produced over the next 12 months, and includes any recommendations for action. Under a more participatory approach, the PPER should be part of an annual review undertaken by stakeholders. Ongoing and periodic reviews should be built into the project from the start. The PPER, which indicates differences between the original project design and what the project has produced, should mirror the findings and recommendations of the stakeholders.

Tripartite ReviewThe TPR is a formal, planned, periodic monitoring mechanism for joint decisions and implementation which is undertaken at least once a year. The UNDP country office customarily invites all those concerned to participate in the TPR and sets the agenda for the issues to be discussed. Under a more participatory approach, a TPR should be undertaken at the project site and involve stakeholders in joint decision-making and implementation. Project or programme stakeholders could play a greater role in the meeting, including its organization and setting of the agenda. Invitations to attend could be extended to representatives of NGOs, parastatals and the private sector.

In-depth EvaluationIn-depth evaluations may take place during a project or programme (ongoing), towards or at its end (terminal) or at least two years after it has been completed (ex-post). An experienced participatory evaluation facilitator



(see Practical Considerations) will, of course, facilitate the process, but stakeholders should be involved in defining the aims of the evaluation and the assessment indicators as well as in analysing the findings.


Who are the question-makers? - Part Four

HOW TO DO A PARTICIPATORY EVALUATION

For some time now, participatory evaluation practitioners have travelled various highways and byways in search of the best route to useful participatory evaluations. However, no single map, no universal set of directions or procedures has emerged to guide the travellers to that destination.

Effective participatory evaluation practitioners remain undaunted by this situation, recognizing that participatory evaluations are context-specific, rooted in the concerns, interests and problems of programme end-users. They know that the complex process of situating an evaluation in the end-users' immediate reality is what charts the route and determines the evaluation's purpose and direction. Flexibility is their watchword.

Depending on the particular context in which the participatory evaluation approach is applied, choices must be made about the degree to which end-users can realistically participate in the process. The levels will vary, as illustrated in the table on page 14. Using these levels as benchmarks for an evaluation can help to make it more participatory.

FrameworkThe following participatory evaluation framework can be incorporated into UNDP programming arrangements by programme staff and their collaborators, including government offices, NGOs and community members. It consists of four basic phases:

1. Pre-planning and preparation 2. Generating evaluation questions 3. Data-gathering and analysis 4. Reflection and action

Pre-planning and Preparation"In participatory evaluation, it is important not to belabor the issue of whether or not to try participatory evaluation. Resist waiting for the perfect time. Just take a stab at it! If you wait for the perfect conditions, you may never do it. All you can do is try and by trying you will learn from failure: Think small but find a problem that may serve five people in the world. Then celebrate for having tried to make the investigation more inclusive of people's voices."

Rajesh Tandon, Participatory Evaluation Conference, University of Massachusetts, Amherst, Massachusetts, USA, 1985

Typically, the early stage of a participatory evaluation is managed at the institutional



level far from the day-to-day lives of end-users. It should lay the foundation for a participatory evaluation that is rooted in stakeholder interests.

● In order to establish stakeholder interest in conducting a participatory evaluation, garner broad-based support by soliciting end-user input and collaboration.

● Since participatory evaluation strives for transparency, openly discuss the purpose, goals and objectives and the various supporting or competing agendas of the evaluation. Continually ask the basic question: "Who wants to know what for what purpose?"

● When stakeholder input is difficult to solicit, review secondary sources, such as programme documents, to gain an understanding of the context. Refer to qualitative and quantitative baseline data and consult with stakeholders either directly or indirectly.

● Address logistical matters, such as drafting and negotiating the TORs, identifying participatory evaluation facilitators and stakeholders, and other administrative matters.

Generating Evaluation QuestionsWhen asked what questions she intended to investigate once in the field, the participatory evaluation facilitator responded with a blank stare. "I have no recipe, no yellow brick road to follow. In participatory evaluation, I rarely know what questions will be asked; that is up to end-users. With them, we survey their situation and share our respective fears, anxieties, hopes and dreams for the participatory evaluation. Then we form questions worth asking."

J. Campos, UNIFEM Field Notes, La Paz, Bolivia

● Discuss and decide with end-users which data collection methods have a high probability of yielding data that are useful and relevant to both outsiders and insiders.

● Assess the current research skills of the persons involved in the participatory evaluation and provide training as needed.

● Determine whether or not different methods will be needed for collecting various types of data. Consider a mix of data-gathering techniques.

● Take into account the prevailing socio-cultural and political climate. Political instability or geographic distances, for instance, can have a bearing on the logistics of a participatory evaluation effort as well as on which data collection methods are feasible. Sensitivity to the socio-cultural milieu, indigenous language issues, gender issues and cultural diversity, especially regarding cultural minority groups, is key.

● Negotiate evaluation questions with stakeholders. Workshops for large groups or field visits to the end-users' workplace or home are appropriate venues for contact.

● Negotiate data collection techniques and provide training as needed. In this phase, the participatory evaluation facilitator or participatory evaluation team works



shoulder-to-shoulder with key actors.

Data-gathering and Analysis"It is better to have less perfect but more usable data and data that can more easily be shared than to have a massive amount of data that becomes the private and often confidential possession of a few expert-specialists."

(Maria-Thérése Feuerstein, 1988)

● Design appropriate venues for meeting with end-users and working with them in a participatory manner. Workshops for large groups in which a cross-section of representative end-user groups can work together are effective for dealing with evaluations that are multi-level or multi-faceted. For more face-to-face contact, field visits can satisfy the requirement of gathering data from individual sources or from small groups working as focus groups.

● Provide thorough instruction or training for participatory evaluation team members who work as data gatherers. Triangulation, a simple research tool in which evaluation members confer frequently and regularly to cross-check, verify and validate the process and data, is an effective strategy to use during the data-gathering phase.

Reflection and ActionBy entering another reality it is absolutely necessary for he or she who is going to in some way perform a very difficult exercise, an almost impossible exercise, that is to 'deknowledge-ize' ourselves. This means to forget the knowledge which we had before and to begin again. But now this time inside of the new reality or cultural frame of reference. This way the people can have more power than we do. Always, the best rule is to know that we do not know that new situation as do the people who live it. It is through our disempowerment that they are empowered a little.

Paulo Freire, 1985

The final phase of a participatory evaluation is characterized by the creation of solutions to end-user problems.

● Have the participatory evaluation group begin with the problems or evaluation questions that were originally defined and articulated by end-users. The goals of this activity are:

●

● to validate end-user experience by using it as the basis for future action plans rather than using outsider's plans;

● to motivate end-users to find solutions and act on them rather than avoid them; and

● to promote a sense of self-determination and sustainability through feelings of



empowerment.

Suggested Steps to Make Each Phase of an Evaluation Participatory

Pre-planning and Preparation

● Outline a conceptual framework based on participatory evaluation principles. ● Define parameters for the participatory evaluation (i.e., what can and cannot be

achieved). ● Assess constraints and resources or enabling and inhibiting factors. ● Identify the participatory evaluation facilitator, team members and stakeholders. ● Negotiate the purpose and objectives of the participatory evaluation with the key

actors.

Generating Evaluation Questions

● Facilitate participatory workshops in, or field visits to, stakeholder workplace or residence.

● Collectively identify the main focus of the evaluation.

Data-gathering and Analysis

● Provide necessary training in data-gathering methods. ● Gather data collectively. ● Analyse data collectively.

Reflection and Action

● Prioritize problems to be solved or questions to be answered. ● Coordinate resources for resolving problems identified during the evaluation. ● Take collective action.

Practical ConsiderationsResident Representatives, Deputy Resident Representatives, National Programme Officers and JPOs may be confronted with numerous questions prior to funding a participatory evaluation or in determining whether or not a participatory evaluation is appropriate.

For Which Type of Project/Programme is A Participatory Evaluation Relevant?Projects/programmes that have a clearly identified group of end-users and beneficiaries lend themselves to experimentation with this methodology.

When Should a Participatory Evaluation Be Done?



Participatory evaluations may take place during the course of a project (usually at its mid-point), towards or at the end, or a significant amount of time (e.g., two years) after a project has been completed.

Undertaking an evaluation at mid-point offers several advantages. It represents an opportunity to take stock of a project's progress to date, its achievements and any obstacles encountered. Lessons learned can be applied and corrective action can be taken if necessary. Since mid-term evaluations are forward-looking, they can provide stakeholders with the tools to take different courses of action.

Terminal evaluations, which usually look at the longer-term impact of a project or programme, tend to be done towards the end of a project whereas ex-post evaluations are undertaken at least two years after a project has been completed. Since all of the stakeholders may not be involved or associated with the project or programme after its completion, the level of their participation in ex-post evaluations may vary considerably.

How Long Does a Participatory Evaluation Take?The duration will vary with each project, depending on its complexity and the capacity and availability of stakeholders to participate in all aspects of the evaluation. It may be more important to view the evaluation as an activity done in phases rather than in one block of time. For example, the following phases or combination of phases could be part of the evaluation: defining the parameters of the evaluation; planning; collection of data; and analysis of data and recommendations. Usually about 10 person-weeks of consultant services (facilitator(s) should be calculated for a period lasting from three to six months.

Sufficient time should be allowed for pre-planning and setting the stage in the field with the various stakeholders. The commitment of all stakeholders is critical to a participatory evaluation and adequate time must be allocated to develop relationships of trust and an understanding of what participatory processes entail. This could be done in two phases. Phase one could include setting the stage and defining, with the stakeholders, the parameters of the evaluation. The second phase could involve the analysis of findings and proposed changes.

How Much Will a Participatory Evaluation Cost?As might be expected, evaluation costs will vary depending on the activity, e.g., the level of expertise that exists within the project (if local experts are unavailable, it will be necessary to seek external consultants), the amount of baseline data already collected, the availability of stakeholders to participate in the evaluation process. The participatory evaluation facilitation team may be composed of national and international consultants. About ten weeks of consultant services and travel should be estimated. It may be necessary to factor in more than one trip for the facilitator if the evaluation is to take place over an extended period.

In general, evaluation costs may include the following:



● consultant services for one or more participatory evaluation facilitators ● travel and per diem costs ● communication costs ● consultant servicesnationals ● travel and per diem expenses of Government representatives where the need arises ● other costs related to data-gathering ● translation costs for the final report in the local language.

What Are the Role and Key Skills of a Participatory Evaluation Facilitator?Hiring (done by UNDP) a knowledgeable participatory evaluation facilitator is critical to a successful participatory evaluation. Not all evaluators know about or are trained in participatory methods and techniques.

Without a careful screening of candidates, a participatory evaluation runs the risk of becoming conventional, with limited stakeholder participation.

Participatory evaluation facilitators usually have an academic background in the social sciences and typically are social science researchers or development practitioners. They should have field experience, experience as educators of adults or as informal trainers and have a reasonable grasp of qualitative methods, such as participatory rural appraisal (PRA) and group dynamics techniques. They must also have the capacity to:

● listen ● guide and facilitate discussions, helping the group to ask key questions ● encourage trust ● delegate tasks and responsibilities ● plan actions to help bring together the viewpoints of various stakeholders ● create an environment of sharing and reflection.

The facilitator must act as a catalyst or stimulator, managing the evaluation without being seen as directing it.



What If the Project Was Not Designed with Participation in Mind?Ideally, participation should be part of the project design from the beginning and embody



the spirit of collaboration and interaction between and among different stakeholders throughout the course of project or programme execution. However, not all projects or programmes have been designed with participation in mind. Nevertheless, it is never too late to build in greater consultation with all stakeholders through field visits and monitoring trips or during TPRs. Consulting before taking decisions is an important step towards achieving greater participation in decision-making, in implementation and in the benefits of development actions. The level and degree of stakeholder participation will depend on a number of factors:

● context of the project ● degree of willingness and commitment on the part of all stakeholders to

participate in a participatory evaluation exercise ● availability of baseline data ● availability of time and resources to enable stakeholders to collect data ● any external constraints that may impede stakeholder participation.

If the evaluation process is to be meaningful, then at the very least, stakeholders should participate in defining the parameters of the evaluation, analysing the findings and proposing solutions. Their involvement in the collection and analysis of data may depend primarily on the availability of time and resources. Ideally, the evaluation report should reflect the findings, concerns and recommendations of stakeholders. Where more conventional evaluations are carried out, external evaluators should share their findings with the stakeholders to ensure that recommendations are appropriate.

What Is the Role of the Resident Representative, Deputy Resident Representative, National Programme Officer and Junior Professional Officer?Participatory evaluations challenge conventional UNDP and technical cooperation practices. Where Resident Representatives or government ministries or institutions anticipate that an external evaluator will do everything, there may be some initial skepticism about embracing participatory methods and practices. As one of the key stakeholders in the MONEY AND MAMBAS case study remarked:

"I must confess that I was very skeptical about this participatory evaluation thing at first. But I have been surprised at how well it has worked, and I can see that my staff have benefitted. Their relations with the community will be better. We all have a better understanding of the project as well as other issues concerning water in the communities."

Resident Representatives can play a key role by being supportive of participatory evaluation processes in the face of such skepticism. Resident Representatives or Deputy Resident Representatives will also play an important role in determining whether or not resources will be allocated for a mid-term review or evaluation.

The National Programme Officer or Junior Professional Officer (JPO) plays an important



role in assessing the levels of understanding and commitment that exist concerning participatory evaluation and what such an evaluation would entail in terms of stakeholder participation. Often they will be responsible for defining the initial TOR with the participation of stakeholders. Part of the job of National Programme Officer, JPO, Resident Representative and Deputy Resident Representative will be to promote a better understanding of the benefits of participatory approaches, such as capacity-building, greater ownership of results and more effective programming.

How Should the Terms of Reference (TOR) Be Prepared?In conventional evaluations, it is customary for UNDP to detail quite extensively the parameters of the evaluation and the issues that the external evaluator will examine.

For participatory evaluations, it is critical that the TOR provide as much flexibility as possible.

Since the stakeholders play a key role in defining the parameters of the evaluation, the TOR should help to launch the evaluation process without any second guessing of the issues that the stakeholders will place on the table. Thus, consideration should be given to holding a number of workshops with the key stakeholders:

● a planning workshop where stakeholders can define the parameters of the evaluation

● a smaller workshop for data collection ● another workshop for the analysis of data and feedback.

The TOR for the evaluation should address issues such as:

● project context ● purpose of the evaluation ● proposed methodology ● preliminary identification of the stakeholders ● an evaluation strategy that outlines the various of the evaluation, such as a

planning workshop, data collection, analysis of data, and a feedback session ● duration of the participatory evaluation exercise as a whole and estimates of the

time required for each phase ● composition of the evaluation team and specification of the expertise required ● responsibilities of the consultants ● resources required.

The initial TOR should be prepared with the participation of key project or programme stakeholders. The specific issues that will be examined by the stakeholders, however, should be left for the first phase of the evaluation exercise once a participatory evaluation facilitator has been identified.



Possible Reactions to the Participatory Evaluation ProcessIt's so confusing! Some participatory evaluation efforts can appear unorganized, feel chaotic or take meandering routes in achieving their objectives. However, a participatory evaluation is an emergent process. It takes its lead from the concerns, issues and problems that appear during discussion, dialogue and interaction with participatory evaluation partners. Confusion and/or ambiguity occur early in the evaluation and must be anticipated since such an evaluation involves collaboration and negotiation to arrive at mutual agreements between and among persons who probably have never met or worked together before.

It's out of control ! Ambiguity in the participatory evaluation approach is further heightened when the outside participatory evaluation facilitator views the participatory evaluation effort as one that is "out of her/his control". In one sense, this is ideal since one of the primary goals of a participatory evaluation is to share control of the process in order to engender ownership. The participatory evaluation process readily becomes clear once outsiders become conversant with the life struggles, understandings and sensibilities of those who have been voiceless and excluded from the development process. To give others control, be "out of control"!

But is this really learning? At its most fundamental level, the participatory evaluation approach is investigative and educational. It is about systematically involving the least powerful, least visible and least assertive actors in evaluating development efforts devised on their behalf. Drawing on their experiences and views exposes the true nature of the problems as they see them and engages them in a process of creating and articulating viable solutions to those problems.

The evaluator assumes the role of facilitator, who works to promote learning moments. The approach supports end-users in their efforts to confront, analyse and find solutions. Involving end-users means patiently waiting for their questions to emerge, but the wait can reap long-lasting benefits. Tapping their knowledge and using it to determine future plans can promote sustainability. Such evaluations can be empowering for both the end-users and the evaluation facilitators.

The poignant words of a Mayan woman from Guatemala who was involved in a United Nations-sponsored participatory evaluation demonstrate the value of participation:

"I didn' t think my simple words would be important. I didn't think your big bosses would accept them. But I know that without my words you would not understand our reality. You have made me think that what I think in my head is worth the time. You have made me remember what I had forgotten to think about."


Who are the question-makers? - Part Five

TRAINING MODULE

INTRODUCTIONThe case study described in this training module is based on UNDP's effort to conduct a participatory evaluation in an actual country setting. Participatory evaluation, an innovative approach to evaluation, can contribute to sustainable development by involving key stakeholders in assessing programmes and projects from their perspective. Involving stakeholders at all levels of evaluation can lead to more comprehensive assessments of development and can effectively draw upon beneficiary views and opinions to redirect development planning.

PurposeThe module is designed to:

● help participants become knowledgeable about and conversant with participatory evaluation as a beneficiary-centred evaluation approach;

● provide participants with a forum for discussing the strengths and weaknesses of an actual participatory evaluation effort and how participatory evaluation can complement conventional evaluation practice;

● provide participants with a forum in which to discuss basic participatory evaluation tools and the requirements for applying a participatory evaluation approach.

AudienceThis training module targets UNDP personnel, especially Assistant Resident Representatives, junior-level personnel and programme officers. Other government or non-governmental groups or agencies involved in UNDP programme and project planning, implementation, and monitoring and evaluation are also included.

FormatThe case study format used in MONEY AND MAMBAS actively involves trainees in discussions and analysis of participatory evaluation theory and practice. Working through the case study helps trainees to understand events that supported and worked against the participatory evaluation exercise. All factual details and names of individuals involved in the case study have been



changed to avoid distraction from the learning objectives of the training module.

Guide to the Facilitator

Pre-workshop Activities

● It is recommended that the facilitator become familiar with the details of MONEY AND MAMBAS. Further information is available from OESP, DC-1, 21st floor, One UN Plaza, New York, NY 10017, USA.

● Photocopy and distribute the case study on pages 37-45. Advise participants to read the case study prior to the training session. While it is possible to have workshop participants read the case study as part of small group discussions during the session, their discussions, analyses and recommendations can be better informed by prior reading.

Workshop ActivitiesThe module, which has been designed to help groups to analyse the pros and cons of participatory evaluation during a single training session, is organized for a three-to-four-hour time frame but can be adapted to meet specific needs. Below is a suggested schedule with approximate times for each activity.

Introduction (30 - 45 minutes):

● Welcome and introduction by facilitator. ● Facilitator provides a general description of the case study and

explains that small group discussions will address questions related to the various parts of the participatory evaluation case study.

Small groups (30 - 45 minutes):

● Facilitator divides the group into small groupsno more than 5 persons per group.

● Facilitator instructs groups to discuss questions related to parts 1 and 2 of the case study ("Questions for Group Work" are included following the text of the case study).



Break (12 minutes):

● Short break if needed.

Plenary (45 minutes):

● In plenary, groups report their responses to questions relating to parts 1 and 2.

Small groups (30 - 45 minutes):

● Facilitator instructs groups to discuss questions pertaining to part 3.

Plenary (30 - 45 minutes):

● Groups report their responses to the questions pertaining to part 3.

● Facilitator directs the groups' attention to the "Additional Questions".

● Participants discuss how they would design and implement a participatory evaluation.

Money and Mambas1 or Listening to the People

A case study of a participatory evaluation of a water and sanitation projectThis case study is based on an actual participatory evaluation effort by UNDP in Eland. It is presented in three main parts:

1. Pre-planning and preparation 2. Getting started: generating evaluation questions 3. Data-gathering, data analysis and action.



PART 1.

PREPLANNING AND PREPARATIONIn the late 1980s, UNDP decided to conduct a number of evaluations with greater participation by local community and stakeholder groups. After an extensive review of possible projects for evaluation, the former UNDP Central Evaluation Officenow reconfigured as the Office of Evaluation and Strategic Planning (OESP)settled on a water and sanitation project in Eland.

The project chosen was called "Rural Water Supply and Sanitation". Its purpose was to supply water to homesteads in rural areas, to provide training in health and hygiene and to construct latrines.

The project appeared to have the right characteristics: during its design, there had been an effort to involve the people and when it was submitted to the programme approval committee in New York, the presenters had emphasized that it was aimed at the poor in rural areas and that it was built on community participation. Project work had started in 1990 and was scheduled for completion at the end of 1995. In addition, an evaluation was planned for mid-1994. Hence, the timing and subject appeared to be right.



In January, 1994, OESP in New York approached the Resident Representative of the UNDP office in Eland to ask if a participatory evaluation approach could be used. If so, OESP would pay the full costs of the evaluation from its funds instead of using the funds allocated on line 16 of the project budget. The Resident Representative consulted the authorities in Eland and their positive response made a participatory evaluation agreeable to all partners.

PART 2.

GETTING STARTED: GENERATING EVALUATION QUESTIONSOESP initiates a participatory evaluation of UNDP-Eland's community-based water and sanitation project. All start-up procedures begin. Preparatory work includes a desk review by Claus and Karl of all project documents and telephone conferences between Claus and Karl and OESP in New York. Contractual agreements with the two international consultants are confirmed by OESP. Additional members of the participatory evaluation facilitation team are recruited (Lane and Didi), with the inclusion of a female sociologist (Didi) almost an afterthought.

Standard UNDP procedures require clearly defined terms of reference (TOR), including precise definition of evaluation questions and data-gathering methods. In a participatory evaluation, pre-determined evaluation questions are often considered irrelevant to the immediate needs and concerns of programme beneficiaries. Participatory evaluation asks one important, fundamental question: Whose questions are being asked in a particular evaluation? Therefore, individuals attempting a participatory evaluation are challenged starting with the TOR. In MONEY AND MAMBAS, drafting the TOR presents the first major obstacle.The TOR dilemma quickly surfaced:

Bruce North's thoughts:Headquarters has come up with this experimental project. I've been briefed on this by Mendisa, our National Programme Officer. Our project is somewhat on track. Our goals of supplying water to rural home steads, providing health and hygiene training, and constructing latrines have been met. The Rural Water and Sanitation Department has put in pipes and filtration systems to community standpipes. Our expansion to 22,000 water users is within reach, but one disturbing trend remains: pit latrine construction (critical to sound water and sanitation) is far below target



because villagers refuse to build latrines according to government specifications. Their resistance jeopardizes the project. The original project design was a package in which the provision of water supply systems linked to rural sanitation through ventilated pit latrines is conditional upon community construction of ventilated pit latrines. We had villager buy-in at the start, but particip

One snag has already emerged. The TOR seem to be a problem. It seems that Claus and Karl (alleged participatory evaluation "specialists") insist that stakeholders determine the questions to be asked in a participatory evaluation. A novel and noble ideabut too time-consuming. I wonder about this participation fetish. Haven't we hired them to do the evaluation?

Claus and Karl in Sweden:Our first major obstacle came with the TOR. In participatory evaluation, pre-determined questions are typically irrelevant to stakeholders; they are questions "outside" of project beneficiaries' immediate needs. Who knows better than water users what works or doesn't? TORs for participatory evaluation should include end-users as question-makers, not just question-answerers. In participatory evaluation there is always "the chicken or the egg" dilemma: whose questions are more important, ours or theirs?

UNDPEland agrees with consultants that a question-generating process should be part of the TOR.

Because so much of a participatory evaluation is context-specific and cannot be pre-determined, participatory evaluation facilitators must bring a jack-of-all-trades tool kit with them when they arrive. Claus, the first consultant to arrive in Eland, is an experienced participatory evaluator but unaware of the surprises awaiting him.

March 1994. Claus's introductory meeting with the UNDP Resident Representative and programme officer was cordial yet distant. The tone of the meeting seemed non-committal and the meeting ended abruptly. His introductory meeting with a key Eland actor, Richard Kosi, was even more perplexing. Claus recorded the following thoughts:

Claus's thoughts:I am concerned, to say the least. UNDP does not seem to really know much



about the participatory evaluation nor do they seem interested. Frankly, the apparent lack of genuine buy-in expressed so far tells me they are not interested because this "was not their baby".

My meeting with Richard revealed that no one has been invited to Workshop I, which begins in three days. Richard seems to have an exclusive invitation list which includes certain beneficiaries and excludes others. He has excluded his mid-level and field staff. What will it take to convince Richard that all stakeholders need to be represented? Four months in the planning and I feel like I'm talking to myself. I wonder if anybody in the country office or the Rural Water and Sanitation Department took the time to read the two-page briefing note on participatory evaluation and the TOR detailing the evaluation exercise that were sent ahead of my arrival. I have strong doubts.

Richard's thoughts:This participation idea seems good. The way I see it, these Claus and Karl fellows are participatory evaluation experts being paid to carry out the entire evaluation. They are responsible for all the main aspects and for the final recommendations and writing up the report. My staff participates by helping. Claus wants all stakeholders to be involved from the start. How unrealistic this man is. Surely, only trained evaluators should collect the data. Don't these fellows get paid enough to handle everything?

Claus's thoughts:As if all of these setbacks are not enough, the Resident Representative informed me that neither he nor his deputy will attend Workshop I, but he will send his programme officer. UNDP's noncommittal stance and Richard's misunderstanding of participatory evaluation put the effort into question. Most disturbing are the differences Richard and I have regarding stakeholder involvement. How could Karl and I have so wrongly assumed that everyone would jump on our bandwagon? Maybe we should have spent more time sensitizing people to the value of participatory evaluation?

Despite the initial confusion, Workshop 1 started promptly a few days after Claus's arrival. The usual official welcomes and introductions were led by Richard Kosi. The 24 participants in attendance included 16 women and 1 Community Water Committee leader, representing programme beneficiaries; one Ministry of Health representative; one UNDP representative; three participatory evaluation facilitation team members; and the senior engineer



of the Rural Water and Sanitation Department and his deputy. The group focused quickly on the purpose of the evaluation and generated the first important list of evaluation concerns.

Examples of Problems and Questions Raised by the Stakeholders

● What should be done when handpumps do not function properly? ● What should be done with government workers (e.g., soldiers and

teachers) who use the water but do not pay? ● What should be done with the people who allow their cattle to drink at

the community water taps? ● The elders must be advised not to send young children to fetch water

to avoid their contaminating clean water with dirty hands when they collect it.

● What should be done if the community wants development and advancement but the rural health motivators do not meet?

● Chiefs do not understand their role in participation in the project. ● Women's concerns are not taken seriously by community leaders. ● Women are the promoters of development in communities and

homesteads. It would be better if they were represented at the Chief's inner, highly contentious committee.

● Imported foreign ideas can be a problem (with particular reference to pit latrines in this case).

● Dirty containers are used to transport water from a clean source.

The issues brought the participatory evaluation into sharp focus from the beneficiaries' point of view. The women's group was particularly animated and articulate. They more than anyone else grasped the participatory evaluation idea. The Rural Water and Sanitation Department's Senior Water Engineer and his staff did not participate in the lively discussions of the small groups. They did an excellent job of holding themselves aloof from the exchange of views.

Claus's thoughts:Given the aloofness of the Rural Water and Sanitation Department chief officer and staffers, it appears that the process is either of no interest to them or it may appear threatening. Could I have been wrong about everyone's buy-in for the participatory evaluation? Do they fear that it may probe sensitive management issues that could put them in a bad light? Why aren't water



officials interacting with the actual water usersthe women?

Claus, Lane and Didi led the group through a discussion about important participatory evaluation steps, including data-gathering techniques, recording data and basic data analysis. At the day's end, Richard mentioned another problem. He announced that tomorrow's workshop session must end early to accommodate participants residing in the countryside. The rushed schedule, although a considerate logistical decision, could potentially sabotage the participatory goal of stakeholder involvement in planning the participatory evaluation.

Claus, at the end of Day 1:Day 1 has finally ended. My thoughts are scattered. The first few days were full of problems, but this is to be expected. The skepticism shown by the Rural Water and Sanitation Department officials doesn't help. There is a spirit lacking here... a core something missing. Many of Kosi's staff, some of the primary project stakeholders, were absent or were assigned to field duty by him. So much for stakeholder participation. It seems like a half-hearted effort. With Karl in Sweden and Didi not expected until tomorrow, even our facilitation team worked at half strength. What on earth might we expect for the following phases? Only time will tell.

Surprisingly, Day 2 progressed smoothly and quickly. The discussion in small groups generated constructive recommendations regarding data-gathering methods. Questionnaires and a work plan were developed based on questions that stakeholders had identified regarding issues to be addressed by the evaluation. The group decided to tailor questionnaires to address the concerns of particular groups and to distribute them to women homesteaders, Water Committee officers, community water-minders, and Rural Water and Sanitation Department field staff, all of whom would be responsible for gathering the data. Because of time constraints, Claus ended up having to prepare the questionnaire.

Claus to himself:A new understanding and interest in participatory evaluation seemed to emerge today. Almost everyone seemed more comfortable with the concept and with each other. Perhaps the evaluation sketch that Lane and I enacted yesterday contributed to this new attitude. Everyone had the chance to comment on what should be evaluated. There is less skepticism and more



enthusiasm in the air. Tomorrow, I leave Eland. Karl and I will return four weeks after data have been collected. It is all in their hands now.

PART 3.

DATA-GATHERING, DATA ANALYSIS AND ACTIONAprilMay 1994. Four weeks elapsed between Workshops 1 and 2.

In the interim, Lane and Didi conducted the data-gathering phase. All data were collected by Lane and Didi, who did not include other stakeholders although their inclusion had been agreed upon in Workshop 1. It seemed that villagers who represented communities in Workshop 1 and drew up the list of questions reverted to the traditional role of question-answerers, not question-makers, during the data-gathering phase instead of collecting information and asking questions that had been identified during the planning workshop.

When Karl and Claus returned to Eland to review and analyse data, they learned that only 22 water schemes had been visited, which meant that only 200 beneficiaries had been interviewed. This figure was well below what had been planned. Group interviews, and not the individual interviews that had been planned originally, had been conducted.

Karl to himself:When I first saw the data, I strongly doubted its quality: not enough water schemes had been included; many interview reports lacked valuable responses; and the group format may have compromised the reliability. It was evident that Lane and Didi did not have sufficient background in working with empirical data and, worse, Claus and I had failed to give them sufficient long-distance support. Do we go ahead anyway?

The four-week separation had also promoted another unexpected event. Richard unilaterally decided to reconfigure the workshop participant list. He invited more senior-level policy-makers, including Government Ministry representatives, UNDP and Rural Water and Sanitation Department officials. Principal secretaries of the Ministries of Finance and Economic Planning, Agriculture, Labour and Health were the new faces that replaced all but four of the 16 rural village women water users. In addition, a few Chiefs or their representatives had been invited.



Karl to Claus:So much for participatory decision-making. The decisions that were originally made and agreed upon by the group have been changed in our absence. Is there some hidden agenda here that we are not part of? Whose participatory evaluation is this anyway? Is this what is meant by "sharing control" in participatory evaluation? Who wants to control what for what purpose? What about the women? Will it be men who speak for the women again?

Despite the changes, Workshop 2 progressed well: 30 people attended, including four beneficiaries, and OESP's evaluation officer from New York attended the final session. The group collectively reviewed the data and quickly categorized the main findings into socio-cultural, technical and organizational issues. The group review process allowed workshop participants to agree with or argue the data. The presence of Chiefs representing numerous traditional community groups was particularly welcomed.

Participants sped through the final steps and organized plenary presentations, which focused on problem analysis and recommendations for future action. The following comments help to convey how some issues were resolved by the group.

Rural Water and Sanitation Department and Ministry of Health staff:Our group questioned why villagers refuse to follow our rules of building pit latrines. Our designs are excellent and have worked in other places. What's their problem?

End-users:We object! We object to that conclusion! The Ministry of Health design is no good for us! We want latrines with walls that allow for air flow and let in the light. In the Ministry of Health design, the pit is far too deep and the walls collapse because of imprecise site and soil specifications. Worst of all, the latrines are dark and wetperfect housing and nesting ground for the venomous mamba snake! Let's see you share your latrine with a mamba!

Richard, Senior Rural Water and Sanitation Department Officer:



You think the mamba snake is the worst problem? Not so! The problem has nothing to do with mamba snakes but has everything to do with not being able to retain good staff. The Rural Water and Sanitation Department is not a government ministry with an official budget. The Ministry of Finance classifies us as an externally funded project dependent on donor offerings. Our economic uncertainty undermines any long-range planning and does little to boost staff morale. In short, we cannot attract or retain the best technicians, much less technicians with a sensitivity to rural villagers. Now, that is the problem.

Clearly, end-users saw the threat and fear of mamba snakes in the pit latrine as the major problem while the Rural Water and Sanitation Department saw money and personnel as the main issues. The discussion continued:

Traditional Chiefs:All problems would be solved if the Rural Water and Sanitation Department would work more closely with the traditional power structure through which we exercise authority. All coordination and entry into community affairs should go through us. This is strictly a problem of what happens when you bypass traditional authority.

Various voices:No, we reject that idea. We object! Yes, we do need closer coordination between Rural Water and Sanitation Department field activities and the community groups. Training would be a good place to begin, and maybe the Chiefs should be the first to be trained. That's right. The Chiefs need training!

Rural Water and Sanitation Department field staff:Many Chiefs and their Councils do not understand modern development needs. Chiefs need education, too, but they are difficult to persuade to come. If they don't come when invited, they send representatives instead.

Chiefs:Chiefs could be encouraged to attend training. Just offer incentives for our time.

Various voices:We protest, protest, protest...!



End-users:Why do Chiefs need incentives? Isn't being a chief incentive enough? Why should Chiefs be paid to take part in training that they need to do their job? Others don't get any incentives. We don't get incentives, yet we give of our limited time to attend Water Committee meetings and even these workshops.

Chiefs:If the Rural Water and Sanitation Department wants to reach the Chiefs and work with the community through us, it needs to learn to work through channels it does not normally work through. We would be more likely to come if we were invited to training via the King's offices, our traditional power structure. The money is not the issue.

Workshop participants spoke frankly and openly and challenged each other on important issues. The participatory evaluation facilitators had hoped that the final session would result in conclusions and recommendations for future action. This was not accomplished. Facilitators found that the exercise had generated more data and information than was expected and the final session did not allow for a thorough discussion of all issues. Yet four clear action points were identified by the group: (a) the need for user designed latrine construction; (b) improved community training for water committees; (c) improved Rural Water and Sanitation Department-community coordination; and (d) improved relations with traditional power structures.

After the workshop, plans for the facilitation team's remaining days in Eland were made. The team produced a draft of the final report and circulated it to the Rural Water and Sanitation Department and UNDP for initial comments. Commitments were made by both offices to collaborate on the translation of the report's summary and its circulation to the participants in both workshops.

Most revealing to Karl and Claus were the lessons learned by their Eland counterparts:

Richard to Karl and Claus:I must confess that I was initially very skeptical about participatory evaluation. I was surprised at how well it worked and I saw that my staff



benefited. I believe their interaction with the communities they serve will improve. We all have a better understanding of the project and a new appreciation for the value of soliciting stakeholder input.

Lane:What a tremendously satisfying experience! The participatory evaluation often seemed like a logistical nightmare, but it was a revelation to me. Even after 15 years working on water issues, I was struck by the articulate precision with which women end-users pinpointed critical water issues that I had overlooked.

My favourite problem was the one asking that Water Committee members not drink beer before their meetings.

Didi:I was amazed at how much I learned about my country. I travelled to rural areas to see women in their natural surroundings. I was most amazed to find that the subject of water provided the entry point for discussion of community problems. Despite the Rural Water and Sanitation Department's many accomplishments, the crucial problem was simple: water containers had too small an opening, which precluded cleaning and encouraged algae growth. We learned that sanitation problems need to focus on water collectors. While visiting the homesteads, I often saw that kids, the chief water collectors, had the dirtiest little hands!

The eight-month participatory evaluation process had finally ended. Long after the final days, questions lingered. Had the process been effective? What would its long-term impact be? Could the costs be justified? Most important, what would happen in the coming months in Eland?

Money and Mambas or Listening to the People

Questions for Group WorkA. Questions about Parts I and 2: Stakeholder Issues

1. Identify the major stakeholders.



2. What steps were taken to include or exclude various stakeholders? 3. What principal roles did they play? 4. What conclusions can you draw about the stakeholders' roles as

question-makers?

PREPARATION ISSUES

1. Analyse the preparatory work for the participatory evaluation. a. What right and wrong actions were taken? b. What would you have done differently? c. What difference(s) might your decisions have made? d. Discuss the TOR "snag".

B. Questions about Part 3: Data-Gathering Issues

1. Discuss and analyse the strengths and weaknesses of the data-gathering phase.

2. Discuss the final phase: a. reporting and writing evaluation findings b. content and language of the final report c. use of findings.

PROCESS ISSUES

1. Discuss the overall management of the participatory evaluation. What can you say about the various elements, including:

a. TOR b. workshop timing c. selection of participants d. scheduling of international consultants e. group dynamics f. overall organization?

ACTION ISSUES

1. In your view, what actions might the UNDP country office take after its involvement in the participatory evaluation?

2. What can be surmised about future actions of the various groups



represented at the workshops, including 3. What possible inter/intra-agency changes might result?

C. Additional Questions: Application Issues

1. What is involved in the TOR for a participatory evaluation? 2. Describe the type of consultant you would hire for a participatoryn

evaluation. 3. How would you ensure that stakeholders are effectively involved in 4. What support training might be needed to undertake a participatory

evaluation? 5. The case study presents a non-traditional approach to evaluation,

which was designed to complement, not replace, traditional evaluations. How might a similar effort be useful to your country office?

6. The evaluation described in MONEY AND MAMBAS was not driven externally by UNDP consultants but rather by stakeholders. How might such an attempt work in your country context?

7. Contrast the case study with the UNDP basic Programme and Projects Manual (PPM). How could participatory evaluations be worked into the PPM cycle?


Who are the question-makers? - Annex 1

Annex I.

Glossary

Baseline data a set of conditions existing at the outset of a programme or project. Results can be measured or assessed against such data. In participatory development, it is important that stakeholders participate in identifying the key sources of information and the indicators required for measuring performance.

Beneficiary assessment an approach to information-gathering that places the emphasis on the perceptions of the principal actors. Qualitative research methods are stressed, including direct and unobtrusive observation methods and interviewing, including semi-structured and open-conversation interviews.

Case-study analysis a learning tool used in training or group settings to stimulate the processes of dialogue, discussion and analysis. Typically based on real-life incidents, case studies illustrate the range of possible enabling and inhibiting factors in a given development activity. Case-study group discussions can increase understanding of broader issues involved in programme and project development.

Donor the funding organization, whose role in participatory evaluation is to mobilize resources which can increase beneficiary involvement in the evaluation and ensure more relevant feedback for the purposes of long-term planning. Donors in participatory evaluation exercises listen more and dictate less.

Emergent knowledge data that emerge informally as part of an evaluation encounter and not as a response to pre-designed evaluation questions. Identifying emergent knowledge leads to inductive analysis, which is critical in participatory evaluation. According to Michael Quinn Patton, an evaluation specialist, inductive analysis means that the patterns, themes, and categories of analysis emerge out of the data rather than being imposed prior to data collection and analysis.

http://www.undp.org/eo/documents/whoa1.htm (1 of 3) [29/01/2002 19:02:24]


Evaluator in participatory evaluations, the stakeholders or programme end-users and/or beneficiaries, who are involved in all stages of the evaluation process, from defining the terms of reference and collecting and analysing data to making recommendations and taking corrective action or making improvements.

Facilitator an outside expert in participatory evaluation methods, who has the capacity to listen; help the group to ask key questions; guide and facilitate discussions; encourage trust; delegate tasks and responsibilities; create an environment of sharing and reflection; and plan actions to help bring together the viewpoints of the various stakeholders.

Focus group small discussion group that concentrates on a specific topic. A group facilitator assists in focusing the discussion on strategies for defining solutions to particular problems. Used in evaluation as a means of starting a discussion, identifying needs and clarifying key points.

Participatory evaluation the collective examination and assessment of a programme or project by the stakeholders and beneficiaries. Participatory evaluations are reflective, action-oriented and seek to build capacity.

Qualitative methods methods that minimize the use of numerical analysis, such as interviews, observation, testimonials and various PRA methods to elicit information from stakeholders. Information obtained in this way can help to illuminate data and numbers.

Random sampling a selection made without method or conscious choice.

Secondary sources sources such as periodic progress reports, annual reports, memos, sectoral studies and baseline data. They serve as background and foundation material and resources for an evaluation.

Stakeholder an actor that has a vested interest in a given project, activity, or issue. Stakeholders may include groups affected by development actions, such as the poor, women, workers, farmers or the community at large, as well as other actors that can affect the outcome of a project, i.e., government officials, institutions, project personnel or the local government. In participatory evaluations, stakeholders assume an increased role in the



evaluation process as question-makers, evaluation planners, data gatherers and problem solvers.

Stratified sampling a selection that ensures representation of a cross-section of a community according to such characteristics as age, gender, social class or race.

Triangulation a process of cross-checking and cross-validating by talking with and referring to various sources.



Annex II.

A SAMPLING OF TOOLS FOR PARTICIPATORY EVALUATIONSOver the years, many tools and manuals have emerged that have contributed to the growing recognition of participatory evaluation as an evolution in, and a valuable alternative to, conventional evaluation methods. While the stages in undertaking a participatory evaluation can be clearly delineated, there is no one model, recipe or tool for such an evaluation. This flexibility has allowed the approach to grow and evolve.

The tools used depend, of course, on the nature of the project, the context and the stakeholders. Since each project setting is unique, different tools will be required, based on the particular political, cultural, economic and social characteristics of the project. Such tools can be used independently or in combination.

It should also be noted that the use of participatory evaluation methods should not preclude the use of quantitative methods. In fact, depending on the initiative, it might be beneficial to combine various methods and approaches. It is the responsibility of the experienced participatory evaluation facilitator to determine the most appropriate tools, depending on the context and nature of the evaluation.

Some of the most common tools are described in this section. For further information on how to apply them, a list of key manuals is also provided.

Beneficiary assessmentA beneficiary assessment involves the participation of beneficiaries in evaluating a planned or ongoing development activity and builds on the experience of participant observation. Assessing the value of an activity as it is perceived by its principal users, this tool seeks to provide a context for quantitative data by letting beneficiaries' voices, values and beliefs be expressed. Methods include direct observation, conversational interviews, and participant observation, which involves the protracted residence of an outsider in a community for a period ranging from several weeks to two or three months. These methods should be used by an experienced observer and inquirer.

Notes from a Participant ObserverAfter more than two years of existence, a fishing cooperative in the state of Rio Grande do Norte had attracted only about 10% of the fishermen for whom it was intended. To attempt to determine why, World Bank project officers and local management agreed to try out the participant observer evaluation method (which has since become known as beneficiary assessment), using host-country observers. Project officials selected a young man in his twenties who had recently received a university degree. He was to live in two fishing communities for several weeks each and spend an additional few weeks in and around the central cooperative in the state capital of Natal. After almost three months in these areas living with and talking to fishermen, the participant observer made the following observations about the role of local fish buyers ("intermediaries") in the project:

"The fishermen are less exploited if they deal with the cooperative, yet none of them are conscious of this, owing to the anticooperative campaign carried out by the intermediaries (many of whom are also fishermen). The majority of the active (cooperative) members are not individuals who are conscious of the advantages of cooperativism but fishermen who do not get along well with the intermediaries. On the other hand, many nonmembers give preference to the intermediaries in order to maintain ties of family or friendship.

There exists misinformation about the actual prices (offered by the cooperative for fish) such that none



of the nonmember fishermen can say exactly what this price is. The notion that the fishermen have... is that the price is lower than what the intermediary pays, whereas the price of the cooperative is 20% higher than the price of intermediaries. This error is daily reinforced by the intermediaries. The anticooperative mentality is such that the fishermen do not believe that the cooperative is paying a higher price when they are informed that this is indeed the case".

As a result of the information from the participant observation, the local manager of this Bank-supported fishing cooperative instituted a comprehensive new promotional campaign to educate the fishermen about cooperativism, took steps to replace a cooperative administrator seen by the fishermen as cold and uncommunicative, and redistributed profits to the fishermen, thus redressing one of their major grievances.

Source: Lawrence Salmen, Listen to the People, 1987:93-94.

Focus group A focus group brings together a representative group of 10 to 15 people, who are asked a series of questions related to the task at hand. A facilitator guides discussion. Focus groups, which draw from local experience and traditions and provide local insight, are useful in project design and in assessing the impact of a project on a given set of stakeholders. While focus groups are commonplace among North American advertising agencies, they are being used increasingly in the field to validate project designs or help to assess project performance.

Seasonal calendar This calendar uses visual representation to help to identify events that occur seasonally. It is useful for a wide range of projects and activities (planting, harvesting, identifying seasonal health risks, seasonal transmission of HIV). It can also highlight such constraints as drinking water availability, drainage blocks, labour availability, food intake, etc. A seasonal calendar can be created on the ground using stones, sticks or leaves or on paper.

Logical framework analysis (LFA) The framework presents, in a nutshell, the core activities and outcomes of the project as well as the key indicators for monitoring and measuring its results. Customarily, this exercise is undertaken by the person responsible for the project design with some type of input from the field. However, it is possible for stakeholders to be the authors of the LFA by holding special workshops. Once the objectives have been agreed upon, stakeholders can work at defining the respective activities, results and performance indicators. Not only do stakeholders provide valuable input into the project design, but they also have a much better understanding of the project.

LFA is used by many donors as a planning tool to summarize the activities, results and verifiable indicators related to each objective of a project. For project evaluations, the LFA will usually be a key source of information referred to by the stakeholders since it contains a summary of the purpose of the project and its expected results. The verifiable indicators will also help stakeholders to measure the degree to which project results have been achieved. It is important to review the LFA with project stakeholders to ensure that the indicators are still valid.

LOGICAL FRAMEWORK ANALYSIS

Specific Objectives

Activities Results Verfiiable Indicators



To promote the psycho social and physical development of marginalized childrentwo to seven years of age

1. The technical team revises the child development model from various per spectives: psycho-motor, cognitive and communication skills, nutrition and health.

2. Social workers, caregivers and parents are trained in childhood development.

3. Caregivers prepare balanced, nutritious meals.

4. The growth and development of children are monitored on a regular basis.

1. The daycare centers adopt childcare models and manuals prepare on technical and administrative aspects of childhood development and daycare centres, respectively

2. Training is provided on a quarterly basis to stakeholders.

3. Caregivers prepare varied menus for children.

4. 100 per cent of the daycare centres measure and weigh the children on a regular basis.

1. Manuals are prepared, distributed and used by the daycare centres.

2. 100 per cent of the children are monitored on a regular basis by the caregiver.

3. There are improvements in the children's basis caloric intake.

4. Per cent of children who meet standard psycho social and physical development for their age group

5. Per cent of social workers, caregivers and parents who apply training in childhood development.

Reduce employment

Two skills training cources: appliance repair and telephone installation

120 tranees

Within three years, 100 men and women workers will become qualified in two skill areas- appliance repair and telephone installation- and will find jobs in these fields. For the majority, these are expected to be new, pemanant jobs. The training is expected to permit growth of these sectors.

1. Training of the level required by prospective employers.

2. Number of men and women placed in jobs for which they were trained, within one year of training, and then three years later.

3. Number of new



jobs created in these sectors.

Participants of marginalized populations and graa-roots organizations in political decisions that affect them.

1. Leadership training

2. Assistance with strategic planning and action plans.

3. Letter-writting campaigns and other acts of solidarity when supported organizations are harasses.

1. Within two years, supported organizations seeking land reform in country X are able to anylise current cordinates and develope action plans. They consult with like-minded groups.

2. Within three years, supported organizations take on advacacy roles.

3. Within five years, supported organizations are assemblies where represented in political decisions are taken.

1. Quality of analysis of the country's political, economic ans social conditions.

2. Quality of action plans.

3. Degree of participation in consultations on land redorm.

4. Number of lobbying activities and responses. Media coverage.

5. Extent of representation in political assemblies.

Semi-structured interview A semi-structured interview, which is less formal than a structured interview, allows for conversation and the reciprocal transmission of information. Preparation usually involves outlining the broad areas of inquiry, leaving specific questions to be formulated during the interview itself.

In her book, Participatory Program Evaluation, Judi Aubel highlights a series of questions that were part of an interview guide for community health nurses in Gambia and Sierra Leone. As she points out, the questions should be sequenced with the easier questions coming first and less intimate questions coming before more personal ones. The questions are open-ended and seek to collect in-depth information on people's attitudes, opinions and knowledge. This allows the interviewer the time to gain the confidence of the person being interviewed. The questions should also be kept simple.



The following are some of the questions that were used in an interview guide for Community Health Nurses:

1. What was your role in the Nutrition Education Pilot Campaign (NEPC)? 2. What was the role of the Mothers' Committee in the program? 3. To what extent did they assume that role? 4. Did you receive your fuel subsidy? 5. Was the fuel given adequate for carrying out your NEPC activities? 6. What was your role in monitoring the NEPC activities? 7. Were you trained on how to monitor the NEPC activities? 8. What information did you collect in the monitoring? 9. How frequently did you monitor the activities?

10. Did you encounter any obstacles in monitoring the activities? 11. What did you do with the information provided?

Source: Judi Aubel, Participatory Program Evaluation: A Manual for Involving Program Stakeholdersin the Evaluation Process, CatholicRelief Services, 1993:38.

Social mapping This tool, which can be used at various stages of a project, involves community members in drawing maps of community structures, institutions, associations and resources on the floor, ground or paper. Mapping can provide insight into the interactions or lack thereof within the community, the resources that are available and access to them.

Social Mapping: The Importance of Having a Good Cross-Section of ParticipantsThe diagram shown below shows the importance of ensuring a good cross section of participants in a social mapping exercise and different gender interpretations of one's community.

When men were asked to map their village, they showed their village as a network of roads and services used by them. They showed the official residences of the chief and chairman, the cotton trees where the different clans met. As for the location of the school and hospital its proximity to the village did not seem to be of concern. It was suggested that wells be located near each of the village clans.

When women were asked to map their village, they saw it through their own lenses. Women pointed to the areas where they collected water and fuel and worked in the fields. Their attention was focused more on the village per se than the outlying regions. The women specified specific houses inhabited by village leaders. The women suggested that the hospital and school be located near the village with the well at the center of the community for the greatest benefit of all.

Social mapping can highlight different perceptions of one's social environment. What is important to one group may be less important to another. For that reason, it is critical that stakeholders are well represented and come from a cross-section of the community.

Testimonial A testimonial records a person's thoughts, feelings and experiences in the first person narrative style. It is a way of learning about a project or its impact through the voices of participants and stakeholders. Testimonials can help to reveal the degree of empowerment, the way in which income is used, how decisions are made or issues tackled. They can also help to corroborate other sources of data and information and provide a more personal insight into a project's achievements. Usually testimonials are taped and played back to the participant.



Sample Testimonial:Ms. Mosammat Jainab Bibi, the Manager of the Shahapur Bittaheen Women's Cooperative Society in the district of Jamalpur, Bangladesh, joined the Cooperative Society in 1984 as a Manager. She is involved in paddy-husking and poultry-rearing activities. She received training on Members Education, cow rearing and poultry. She is also attending the Manager's Training regularly.

"I joined the Society in 1984. Mr. Tara, the local upazila official had distributed 15 wheat feeding cards among 15 vulnerable female villagers. One day he told to mobilize another 15 women to form a society. We did it and he helped us to form a BRDB society. We deposited Tk.1 per week as savings. We were not united then. We did not know each other. When the other 15 women joined us we held a weekly meeting. We continued it and Mr. Tara would also attend. We generated a little fund and Mr. Tara and we deposited it at BRDB office. With our consent he formed a BRDB society for us. We deposited Tk.1 or 2 as savings in 1984. We did everything by ourselves like raising savings, depositing them at the bank, issuing verity vouchers, taking receipts from bank etc. We registered our society on 30.3.85. It is nearly 7 or 8 years that we have been running our society....

We take a loan every year and husk paddy which provided us with some profit. We spend a little of that for education of our children. Previously, the Railway School was completely reluctant to admit our children. The directors of BRDB asked us once: "How many are you?" We answered we were 46. They replied that means at least 46 children and advised us to go and admit our children in the Railway school and gave us hope that they would help us. We went there, the teachers were in panic. We asked them: "Why do you not want to admit our children-because we are poor? Since we have no clean clothes? Why do you admit rich children?" Then the teachers agreed to admit our children.

They told us to pay Tk.10 for each boy or girl as an admission fee. We had protested earlier but realized very soon that we had to pay Tk.10 because it was compulsory for everyone. They gave us 7 days to collect the money. We collected the money and admitted our children in the school. That's how we overcame the problem...We cultivate fish collectively.. We have no pond but we requested one old man to provide us with his pond for fish culture. "We will cultivate fingerlings or young fish in your pond. We will sell the fish after 2-3 months regularly and the rest will be yours", we said. We took the pond under this condition and we earned Tk.880 in 2 months. We have been cultivating fish for 5-6 years and earn Tk.500-600 each 2-3 months. We maintain the pond and take care of it, catch the fish and sell them. We do not get the help of any men. We have utilized our training fully."

The above excerpts of a dialogue were part of a broader evaluation exercise of a training component of a Bangladeshi project. The dialogue method was used to complement and enrich the quantified data obtained through the interview questionnaire, case studies and file review.

Source: Yusuf Kassam, 1995:6-7.

A Transect Walk Through Mbusyani, Kenya

Soil Loose, deep red soil Sandy soils and small patches of red soils

Shallow sandy soils rocky in most parts



Water About 1/4 of households have shallow well area also has three dams & one spring

A river infested with bilharzia, two poorly maintained dams

Water in Kilindiloni river salty. River Kathana has bilharzia. Roof catchment in progress

Vegetation All natural vegetation cleared to give way to settlement

High proportion of natural vegetation mainly acacia lantana, canola grasses

Natural vegetation consisting of acacia shrubs and grass

Social-Economic Indicators

1/2 of household heads in wage employment majority have magati roofs brick or stone walls

1/2 of household- corrugated iron roofs, 1/2 grass thatched, brick walls

Mainly grass thatched houses

Food Crops Maize, beans, pigeon peas, bananas

Maize, beans, a lot of sore pigeon millets, fruits, bananas

Maize, beans, peas, bananas, fruits

Cash Crops Coffee Coffee Coffee

Achievements (Last 5 Years)

Soil conservation, tree planting, water development - wells, roof catchment

Soil conservation, water development, dams

Some soil and water conservation

Forestry/Agro Forestry Widespread agroforestry with grevillea, eucalyptus, mangoes, and paw paws

Minimal tree planting but mangoes and paw paws planted

Very little tree planting

Resources Management

A lot terracing embankments reinforced with multi-purpose grasses

A lot of bench terracing

Limited soil conservation

Problems Inadequate water, education and health facilities, famines and lack of dip facilities

Water, famine, inadequate education and health facilities

Water, Transport and Food

Opportunities Rehabilitation 3 dams, one spring. External assistance-tools, market

Water development-dam, well, roof catchment. Government assistance

Water development-dams,well, roof catchment. External assistance



Source: National Environment Secretariat. Participatory Rual Appraisal Handbook: Conducting PRAs in Kenya,1991:21

Transect walk A transect is usually a straight cut through the community. A transect walk involves walking through a community with the local people. It seeks to cover all major ecological, production and socially stratified zones of that community and usually includes observation, asking questions, pointing and discussing what is being seenzones, land, vegetation, local markets, community service centres, schoolsand mapping the areas.

Venn diagram A Venn diagram, of usually circular areas, can be used to look at relationships within institutions or relationships between the community and other organizations. It illustrates different participant perceptions of access to resources or of social restrictions, for example. Circles of various sizes are cut out of paper and given to participants, who are then asked to allocate the circles to different institutions, groups or departments. The larger the circle the more important it is. The circles may overlap, showing the degree of contact between institutions or groups.

A Potato Project in Pakistan:

Participants from headquarters and the regions were asked to identify the different institutions, associations and target groups related to their project. Participants were asked to select circles of different sizes to represent the importance of an institution or group. It was noted that the headquarters staff located in Islamabad mapped a wide range of institutions with which they maintained contact: donors, regional research agencies, private businesses. Provincial staff closest to headquarters had less knowledge of international linkages yet knew of many linkages, such as the research agencies, farmers and different private-sector groups. Those most remote showed even fewer linkages and less interaction with other institutions.

In barrack, one of the villages, the following indicators were used to distinguish levels of wealth within the village:

Pile/Category Household No. Indicators



I 43, 42, 38, 36, 34, 28, 41, 35 •mostly landowners

•some houses are made of permanent materials

•own more livestock (cows, carabaos, pigs and chickens)

•mostly receiving remittances from children working in Manila or abroad

II 39, 37, 27, 22 20, 19, 17, 33 21,18

• mostly either tenants or tenants and at the same time andowners of small land parcels

• mostly own a number of livestocks (cows, carabaos, pigs and chickens)

• mostly receiving remittances from children working in Manila

III 40, 31, 26, 25, 23, 16, 7, 4, 2, 30, 24, 15, 1

• tenents of small land parcels

• hired labourers

• majority don't have caraboas and other livestock

• old folks dependent upon children's support

IV 32, 29, 13, 12, 10, 9, 3, 5, 6, 8, 11, 14

• majority are not cultivating any land parcels

• no caraboas (as draft animal)

• dependent mostly on any of the following sources of income:

-fishing (small scale)

-tuber gathering (small scale)

-gathering and selling



firewood/charcoal

-hired labour/maids

Source: F.T. Banlina Ly Tung, "Farm Experiences of Wealth Ranking in the Phillippines: Different Farmers Have Different Needs" in "Special Issues on the Application of Wealth Ranking", RRA Notes, no. 15, International Institute for Environment and Development, May 1992: 48-50.

Wealth ranking Wealth ranking uses the perceptions of villagers or the local population to rank households within a village according to wealth. The local population is involved in listing the criteria to identify the poor and those who are better off. What emerges is a set of criteria that in the eyes of the local population indicates what constitutes poverty or wealth. The results can and are many times at odds with conventional methods of classifying people solely according to income. For example, in a small village in India, when villagers listed 30 criteria for identifying the poor, land ownership was only one of the variables. Villagers viewed a widow who had land but could not cultivate it as poor whereas Indian planners did not consider her to be poor.

Wealth Ranking in the Philippines:The Farm and Resource Management Institute of the Philippines tested wealth ranking in three villages. The Institute obtained a list of households from the Barangay captain, the political head of the village. Cards were prepared for each family (see diagram to the right). Five informants for each village were identified and asked to sort the cards into three to five piles, indicating the different wealth groups.

Zielorientierte Projektplanung (ZOPP) (objectives-oriented project planning) ZOPP is a set of procedures and instruments that seek to integrate stakeholder concerns into project planning. Introduced by Gesellschaft für Technische Zusammenarbeit (GTZ) (German Agency for Technical Cooperation) in 1983, it involves a number of stages, including participation analysis, problem analysis, defining the problem tree, and an analysis of objectives, all of which is undertaken over a one-week period and at different stages of the project cycle, from project design and management to evaluation. ZOPP involves participation analysis, which seeks to integrate the interests and expectation of persons and groups significant to the project.

ZOPP in Action:The following diagram shows ZOPP in action. Pieces of paper in various sizes and colours have been prepared in advance. The different shapes represent different ideas, priorities or themes which can then be reorganized easily on a blackboard. Participants will use a few words to describe their idea on the piece of paper. ZOPP can be used in evaluation to brainstorm over the key areas of evaluation, indicators of assessment, and roles and responsibilities for data-gathering.

Best of the Best: Manuals for Doing Participatory Evaluations

1. Participatory Learning and Action - A Trainer's Guide Authors: Jules Pretty, Irene Gujit, Ian Scoones and John Thompson International Institute for Environment and Development (IIED), London, 1995. This manual, which offers a step-by-step approach to participatory learning and action methodologies, is an excellent resource for trainers and practitioners interested in participatory



development. 2. Participatory Program Evaluation: A Manual for Involving Program Stakeholders in the Evaluation Process

Author: Judi Aubel Catholic Relief Services, Senegal, December 1993. Available through PACT, New York. Basing her work on experiences in Africa, Asia and Latin America, the author details the various steps involved in participatory evaluation and draws examples from field experience.

3. Partners in Evaluation: Evaluating Development and Community Programmes with Participants Author: Marie-Thérése Feuerstein MacMillan Education Ltd., London, 1986. This pioneering work offers a thorough step-by-step approach to participatory evaluation. Excellent graphics and drawings accompany the text. A must-read for participatory evaluation facilitators and others interested in participatory evaluation.

4. Participatory Evaluation: Tools for Managing Change in Water and Sanitation Author: Deepa Narayan World Bank Technical Paper no. 207, The World Bank, Washington, D.C., 1993. This paper provides policy-makers, managers and planning and evaluation staff with information about participatory processes and indicators that can be used to involve stakeholders in programme evaluation.

5. Poverty and Livelihoods: Whose Reality Counts? Author: Robert Chambers Division of Public Affairs, United Nations, Development Programme, New York, 1994. This policy paper, commissioned by UNDP for the World Summit for Social Development (Copenhagen, March 1995), presents a new paradigm for assessing the realities of poor people and deciding what needs to be done.



Annex III.

WHO'S DOING WHAT? ORGANIZATIONS SUPPORTING PARTICIPATIONVarious donors and organizations have been experimenting with the concept of participation for a number of years. For some, participation is not simply a technique or approach to be applied at different stages of the project cycle; it is also a philosophy about how development is approached and implemented.

This section seeks to summarize key organizational thinking on the subject of participatory development and evaluation. It should be noted that for most donors, participatory evaluation is relatively new and challenges conventional modes of evaluation with its emphasis on outside external experts. There is much work to be done before participatory evaluation methods and approaches are mainstreamed into the operational processes of organizations. More time will also be needed to change attitudes, approaches and institutional cultures.

Canadian International Development Agency (CIDA)Support for participatory development has been concentrated primarily at the project level. The creation of umbrella mechanisms or microrealization projects has permitted the funding of numerous discreet activities at the grass-roots level that promote and foster participation. Workshops have also been launched to try to introduce staff to PRA techniques and approaches. The agency is currently seeking to address participation at the policy and programme levels in a more consistent manner. CIDA is also exploring, in cooperation with the World Bank Inter-Agency Group on Participation, the institutional changes that are required to mainstream participatory approaches within donor organizations.

Food and Agriculture Organization (FAO)In 1992, FAO put forward a Plan of Action on People's Participation with the aim of ensuring the "active participation of people in the achievement of sustainable development... Without participation, rural development initiatives are unlikely to be sustainable in the long run and rural inequities are unlikely to be addressed" (FAO: 1992:5). FAO has established a People's Participation Service within the Division of Women and Participation. This service, established in 1992, has built on the work pioneered in the mid-



1970s, initially through the Rural Organization Action Programme and Small Farmer Development Programmes and in the 1980s through the People's Participation Programme, which aimed at promoting self-managed and self-reliant groups at the community level.

Gesellschaft für Technische Zusammenarbeit (GTZ).Since the mid-1980s, GTZ has spearheaded a number of efforts aimed at increasing participation. The ZOPP method was introduced into GTZ's project cycle management approach. This method seeks stakeholder participation in the planning and monitoring processes of projects and is now used with other methods and tools, such as PRA. Regional Learning Groups on Participation have been created with a view toward gaining a better understanding of what is happening at the local level and how the organization can be adapted to promote participation at the project and societal levels. Numerous projects have been initiated that have adopted bottom-up approaches aimed at initiating ownership and the self-organization capacity of the community.

International Fund for Agricultural Development (IFAD)IFAD has been concerned with participation of the rural poor over the past ten years. People's participation is viewed both as a means and an end to poverty alleviation. Apart from believing that "investing in the production potential of the poor can bring high returns", IFAD views participation as a catalyst for self-supporting and sustainable development. IFAD has invested over $2.6 billion in self-help, participatory development projects targeted at small-holder farmers, the landless, rural women, fishermen, nomadic herdsmen and agro-pastoralists

(Lineberry, 1989).

International Institute for Environment and Development (IIED)Located in England, IIED is a policy research institute linking environmental concerns with the development needs of resource-poor people in the South and with other global environment and development priorities. The Institute has been instrumental in developing and spreading participatory methodologies and approaches. Since 1988, it has put out a series of user-friendly RRA notes, which are now called PLA Notes-Notes on Participatory Learning and Action. Development practitioners from around the world contribute to the notes by sharing their experiences, conceptual reflections



and methodological innovations.

Overseas Development Administration (ODA)For ODA of England, participation is seen as a central element in achieving effective, sustainable development. ODA is concerned with how to involve other key secondary and primary stakeholders in the monitoring and evaluation process and developing systems and procedures for more systematic assessment of impact. The agency has recently published a number of guides that look at measuring participation, doing a stakeholder analysis, and enhancing stakeholder participation in aid activities.

Society for Participatory Research in Asia (PRIA)PRIA is an NGO based in India that promotes people-centred development, participatory research and participatory development. As an educational support institution, PRIA offers training in participatory development and participatory methodology to grass-roots organizations and to personnel from bilateral and multilateral organizations and from government and semi-government institutions. PRIA conducts participatory training of trainers and participatory evaluations for grass-roots organizations.

United Nations Development Programme (UNDP)UNDP has defined grass-roots participation as "a process whose objective is to enable people to initiate action for self-reliant development and acquire

the ability to influence and manage change within their society". The promotion of participatory approaches and empowerment of people are key intervention points for UNDP activities. Establishing close partnerships with NGOs, community-based organizations (CBOs) and other civil society groups is an essential element of external support. This builds on earlier efforts to promote regional and global programmes that seek to promote participation at the grass-roots level.

United Nations Children's Fund (UNICEF)UNICEF advocates a people-centered, poverty alleviation approach to environment and development. Participation is seen as key to empowering people to take control of their lives and to act to meet their own needs.

United States Agency for International Development (USAID)Participation is a key dimension of the Agency's efforts to redesign



government. A systemic approach is currently being used in redesigning systems and operations so that they are more responsive to USAID customersthe poor. Participation is seen as essential for moving from macro-management, top-down approaches to those that empower and enable. USAID holds a series of informal, monthly meetings on participatory development for its staff. Speakers, who come from both inside and outside the agency, are invited to share their experiences. A Senior Advisor specifically promotes the integration of participation into Agency activities and acts as a focal point at USAID.

World BankThe World Bank defines participation "as a process through which stakeholders influence and share control over development initiatives, and the decisions and resources which affect them". A Learning Group on Participatory Development, established four years ago with representation from all the Bank's departments, has led the Bank to adopt an action plan that advocates greater stakeholder participation in all aspects of the Bank's operations. Participatory development is increasingly being looked at as essential for achieving greater efficiency and sustainability in Bank operations. The recent World Bank Participation Sourcebook provides an overview of the Bank's experience as well as methods and tools that enable the poor to participate. The Bank has also piloted several beneficiary assessment and stakeholder evaluations and supported participatory approaches in Bank operational activities.

At an international level, the World Bank is spearheading an inter-agency group on participation composed of six sub-groups: institutional changes and mainstreaming of participatory approaches; project preparation and implementation; training and learning; monitoring and evaluation; information dissemination; and country-level capacity-building.

World NeighborsWorld Neighbors is a small development agency with an overall budget of $3.2 million supporting programmes in approximately 22 countries that seek to "strengthen the capacity of marginalized communities to meet their basic needs". World Neighbors has been interested in participatory development and evaluation since the 1980s and published its own handbook on self-evaluation. More recently, World Neighbors has been concerned with strengthening its ability to learn from its experience, particularly in terms of



"strengthening community capacity" to meet basic needs, and to use this experience to influence peer agencies and policy-makers. World Neighbors has been applying PRA techniques, such as social mapping, wealth ranking, time-line, household composition and identification of women of reproductive age, to its development work.



Annex IV.

RESOURCE PERSONS, GROUPS AND INSTITUTIONSThe following individuals, groups and institutions provide training, research and publications related to participatory evaluation as it is discussed in this handbook.

Ms. Ariane Berthoin AntalDirector of Ashnage International Institute for Organizational ChangeFrench-Geneva Campus741 66 ARCHAMPS, FRANCETEL: (33-50) 31-5600FAX: (33-50) 31-5606

Oguz BaburogluSchool of Business AdministrationBilkent UniversityP. 0. Box 806572 MaltepeBilkent, Ankara, TURKEY 06533TEL: (90-31) 2-266-4164FAX: (90-31) 2-266-4985E-mail: [email protected]

Mr. Tim BakerUniversity of Taxmaniac/o 85 Berkley StreetHawthorn, Victoria, AUSTRALIA 31 22TEL: (61-3) 98192504FAX: (61-3) 98182504

Dr. Rémy-Claude BeaulierCIDA - Social Policy BranchPlace du Centre200 Promenade du PortageQuébec, Canada KI A OGTEL: (1-819) 953-6376FAX: (1-819) 953-6356



E-mail: [email protected]

Prof. L. Dave BrownInstitute for Development Research210 Lincoln StreetBoston, Massachusetts 02111 USATEL: (1-617) 422-0422FAX: (1-617) 422-0494E-mail: econet:[email protected]

Mr. Federico ButeraInstituto RSO srl Ricerche sui Sistemi Org Via Leopardi 11-201 23 Milano, ITALY

Dr. Juanita CamposAdjunct Faculty & Research ScientistBureau for Applied Research in Anthropology (BARA)Anthro Bldg. 317, AUniversity of ArizonaTucson, Arizona 85721 USATEL/FAX: (520) 743-0622 (home)TEL: (1-520) 622-5549 (BARA)FAX: (1-520) 622-5449 (BARA)

Ms. Alice Carloni, Rural SociologistInvestment Center DivisionTechnical Cooperation DepartmentFood and Agriculture Organization (FAO)Viale delle Terme di Caracalla1-001 00 Rome, ITALYTEL: (39-6) 522-55799FAX: (39-6) 522-54657E-mail: [email protected]

Hillel EfratKibutz Industries AssociationP. O. Box 400012, Tel-Aviv 61400



8 Shderot Shaul Hamelech, ISRAELTEL: (972-3) 6955413/4/5FAX: (972-3) 6951464

Dr. David EvansCenter for International Education285 Hills House SouthUniversity of MassachusettsAmherst, MA 01003 USATEL: (1-413) 545-0465FAX: (1-413) 545-1263E-mail: [email protected]

Dr. Rosalind EybenPrincipal Social Development AdvisorOverseas Development Administration94 Victoria StreetLondon SW1 E5DH, EnglandUNITED KINGDOMTEL: (44-1 71) 917-0566FAX: (44-1 71) 917-0197E-mail: [email protected]

Prof. Orlando Fals-BordaInstituto de Estudios PolíticosUniversidad NacionalBogotá, COLOMBIATEL: (1-571) 3681 579FAX: (1-517) 3687 471

Dr. Marie-Thérése Feuerstein49 Hornton StreetLondon W8 7NT, ENGLAND

Mr. Phil Glaser, DirectorRadford, Glaser & Associates22 David StreetPietermaritzburg 3201, SOUTH AFRICA



TEL: (0331) 428185FAX: (0331) 425482

Mr. Alfonso GonzalezGrupo de EstudiosAllende 7, ColoniaApartado Postal 76Mexico City, MEXICO

Mr. Davyd J. GreenwoodJohn S. Knight Professor of International StudiesCenter for International StudiesCornell University170 Uris Hall Tower Rd.Ithaca, New York 14853-7601, USATEL: (1-607) 255-6370FAX: (1-607) 254-5000

Mr. Peter Gubbels, CoordinatorWorld Neighbors Organizational & Institutional DevelopmentVoisins Mondiaux, OI BP 1315Ouagadougou OI BURKINA FASOTEL: (226) 34.55.60FAX: (226) 34.15.92E-mail: [email protected]

Mr. Khalid El HariziSenior Evaluation OfficerOffice of Evaluation & StudiesInternational Fund for Agricultural Development (IFAD)107 Via del Serafico00142 ROME, ITALYTEL: (39-6) 545-92059FAX: (39-6) 519-1702

Mr. Michael I. HarrisonDept. of Sociology & AnthropologyBar Ilan University



52900 Ramat Gan, ISRAELTEL: (972-3) 531 8333FAX: (972-3) 535 1825E-mail: bitnetf42166@barilan

Mr. Thomas KubySr. Planning OfficerGTZ Dag-Hammarskjold-Weg 1-2Postfach 51 80D-6236 Eschborn bei Frankfurt/Main, GERMANYTEL: (49-61) 967-91741FAX: (49-61) 967-96109E-mail: [email protected]

Prof. Yvonna LincolnDept. of Educational AdministrationTexas A & M University222 Mt. Harrington Education CenterCollege Station, TX 77843 USATEL: (1-409) 845-2716FAX: (1-409) 862-4347E-mail: [email protected]

Dr. Erma ManoncourtProgramme Communication/Social MobilizationUNICEF3 United Nations Plaza D-H-40FNew York, NY 10017 USATEL: (1-212) 702-7245FAX: (1-212) 702-7154E-mail: [email protected]

Dr. Ineke Meulenberg-BuskensCentre for Science DevelopmentHuman Science Research CouncilPrivate Bag 41Pretoria 0001, REPUBLIC OF SOUTH AFRICATEL: (27-12) 202-2604FAX: (27-12) 202-2421



E-mail: [email protected]

Mr. Timothy MurphyEuropean Bank for Reconstruction and DevelopmentOne Exchange SquareLondon EC@A 2EH, UNITED KINGDOMTEL: (44-171) 338-6020FAX: (44-171) 338-6848E-mail: [email protected].

Mr. Peter ReasonCenter for Action Research in Professional PracticeUniversity of BathSchool of ManagementBath BA2 7AY, UNITED KINGDOMTEL: (011-44) 225-826792FAX: (011-44) 225-826473

Mr. James Sessions, DirectorHighlander Research and Education Center1959 Highlander WayNew Market, Tennessee 37820 USATEL: (1-615) 933-3443FAX: (1-615) 933-3424

Mr. William StaubSr. Social Development SpecialistSocial Development DivisionAsian Development BankManila, PHILIPPINESTEL: (63-2) 632-6756FAX: (63-2) 741-7961E-mail: [email protected]

Prof. Marja-Liisa SwantzUnited Nations UniversityWorld Institute for DevelopmentEconomics Research



Katajanokanlaituri 6B00160 Helsinki, FINLANDTEL: (358-9) 615-9911FAX: (358-9) 693-8548E-mail: [email protected]

Dr. Rajesh TandonSociety for Participatory Research in Asia42, Tughlakabad Institutional AreaNew Delhi 110062, INDIA

Mr. Francisco Vio GrossiSecretary General - CEEALDiagonal Oriente1604, Casilla 6257Santiago, CHILE 22

Ms. Sonam YangchenProgramme Advisor NGO ProgrammeSustainable Development and Poverty Elimination Division Bureau for Policy and Programme Support UNDPOne UN Plaza, DCI Rm. 2058New York, NY I0017 USATEL: (1-212) 906-6029FAX: (1-212) 906-5857E-mail: [email protected]


Who are the question-makers? - Bibliography

BIBLIOGRAPHY

References Mentioned in the Text

Aubel, Judi. Participatory Program Evaluation: A Manual for Involving Program Stakeholders in the Evaluation Process. Catholic Relief Services, Senegal, 1993.

Banlina, F. T. and Ly Tung. "Farm Experiences of Wealth Ranking in the Philippines: Different Farmers Have Different Needs". International Institute for Environment and Development. "Special Issues on Application of WealthRanking". RRA Notes, no. 15, May 1992, pp. 48-51.

Campos, Juanita Diane. Towards Participatory Evaluation: An Inquiry into Post-Training Experiences of Guatemalan Community Development Workers. Dissertation submitted to the University of Massachusetts, School of Education, May 1990.

Chambers, Robert. "Participatory Rural Appraisal (PRA): Analysis of Experience", in World Development, vol. 22, no. 9, 1994:1253-1268.

Coupal, Françoise. "Participatory Project Design: Its Implications for Evaluation. A Case Study from El Salvador". Paper presented at the 1995 Evaluation Conference, 1-5 November, Vancouver, British Columbia.

Dinámica Integral Ltda. El Metaplan-Metodología Eficiente y Eficaz para Lograr Objetivos con Participación Grupal, Colombia.

Fals-Borda, Orlando and Mohammad Anisur Rahman (eds.). Action and Knowledge: Breaking the Monopoly with Participatory Action Research. London, Intermediate Technology Publications, 1991.

Farrington, John and Adrienne Martin. "Farmer Participation in Agricultural Research: A Review of Concepts and Practices", Agricultural Administration Occasional Paper no. 9, London, 1988.

http://www.undp.org/eo/documents/whobib.htm (1 of 16) [29/01/2002 19:03:13]


Fernandez, Maria. Participatory-Action Research and the Farming Systems Approach with Highland Peasants. Colombia, Department of Sociology, University of Missouri.

Feuerstein, Marie-Thérése. Partners in Evaluation: Evaluating Development and Community Programmes with Partners. London, MacMillan, 1986.

Freedman, Jim. Participatory EvaluationsMaking Projects Work. Technical Paper no. TP94/2. Calgary, Canada, Division of International Development, International Centre, University of Calgary, 1994.

Freire, Paulo. Pedagogy of the Oppressed. New York, Continuum Publishing Company, 1993.

GTZ. ZOPP: An Introduction to the Method. Frankfurt, Germany, March 1988.

Guba, Ego and Yvonna Lincoln. Fourth Generation Evaluation. England, Age Publications, 1990.

International Institute for Environment and Development. "Special Issues on Application of Wealth Ranking". RRA Notes, no. 15, May 1992.

Kassam, Yusuf. "The Combined Use of Participatory and Survey Methodology in Evaluating Development Projects. A Case Study of a Rural Development Project in Bangladesh". September 1995. Paper presented at the 1995 Evaluation Conference, 1-5 November, Vancouver, British Columbia.

Marsden, David, Peter Oakley and Brian Pratt. Measuring the Process: Guidelines for Evaluating Social Development. Oxford, 1994.

Narayan, Deepa. Participatory Evaluation Tools for Managing Change in Water and Sanitation. Washington, DC, World Bank Technical Paper no. 207, 1993.

National Environment Secretariat, Government of Kenya; Clark



University; Egerton University; The Centre for International Development and Environment of the World Resources Institute. Participatory Rural Appraisal Handbook: Conducting PRAs in Kenya. Natural Resources Management Support Series no. 1, 1991.

Overseas Development Administration. Note on Enhancing Stakeholder Participation in Aid Activities. England, Social Development Department, ODA, April 1995.

Pretty, Jules, Irene Gujit, Ian Scoones and John Thompson. Participatory Learning and Action: A Trainer's Guide. London, International Institute for Environment and Development, 1995.

Rahman, Mohammad Anisur. People's Self-Development. London, Zed Books, 1993.

Salmen, Lawrence F. Beneficiary AssessmentAn Approach Described. Environment Department Papers no. 23, Washington, DC, World Bank, July 1995.

UNDP. Building a New UNDP. New York, Office of Evaluation and Strategic Planning, 1995.

_____. "Report of the UNDP Capacity Development Building Retreat". New York, Office of Evaluation and Strategic Planning, 1995.

_____. Strategic Management in UNDP. New York, Office of Evaluation and Strategic Planning, 1995.

_____. UNDP Evaluation Findings in 1994. New York, Office of Evaluation and Strategic Planning, 1995.

_____. UNDP Evaluation Findings in 1995. New York, Office of Evaluation and Strategic Planning, 1996.

Uphoff, Norman. "Monitoring and Evaluating Popular Participation in World Bank-Assisted Projects" in B. Bhatnagar and A. C. Williams, Participatory Development and the World Bank: Potential Directions for



Change, World Bank Discussion Paper no. 183. Washington, DC, World Bank, 1992.

Participatory Evaluation

Anayanwn, C. N. "The Technique of Participatory Research in Community Development", Community Development Journal, January 1988, vol. 23:11-15.

Armstrong, John and Michael Key. "Evaluation, Change and Community Work", Community Development Journal, October 1979, vol. 14:210-223.

Campos, Juanita Diane. Towards Participatory Evaluation: An Inquiry into Post-Training Experiences of Guatemalan Community Development Workers. Dissertation submitted to the University of Massachusetts, School of Education, May 1990.

Chambers, Robert. "Participatory Rural Appraisal (PRA): Analysis of Experience" in World Development, vol. 22, no. 9, pp. 1253-1268, 1994.

_____. "Shortcut Methods for Gathering Social Information for Rural Development Projects" in M. Cernea, Putting People First. Oxford, Oxford University Press.

Clark, Noreen and James McCaffery. Demystifying Evaluation. World Education, Boston, Massachusetts, 1979.

Coupal, Françoise. "Participatory Project Design: Its Implications for Evaluation. A Case Study from El Salvador". Paper presented at the 1995 Evaluation Conference, 1-5 November, Vancouver, British Columbia.

Cuthbert, Marlene. "Evaluation Encounters in Third World Settings" in M. Q. Patton (ed.), Culture and Evaluation, San Francisco, Jossey-Boss, pp. 29-35.

DAC Expert Group on Aid Evaluation. Evaluation and Participation. An Approach Paper. Development Studies Unit, Department of Social



Anthropology, Stockholm University, September 1995.

Derricourt, Nick and Peter Oakley (eds.). "Participation in Evaluation", Special Issue, Community Development Journal, vol. 23:1, 1988.

FAO. Community ForestryParticipatory Assessment, Monitoring and Evaluation. Rome, FAO, 1989.

Fernandes, Walter and Rajesh Tandon. Participatory Research and EvaluationExperiments in Research as a Process of Liberation. New Delhi, Indian Social Institute, 1981.

Fetterman, David (ed.). Qualitative Approaches to Evaluation in Education. New York, Praeger Press, 1988.

Feuerstein, Marie-Thérése. "Community Participation in Evaluation: Problems andPotentials.", International Nursing Review, 1980, vol. 27 (6), 187-190.

_____. "Evaluation by the People". International Nursing Review, 1978, vol. 25 (5), 146-153.

_____. "Finding the Methods to Fit the People: Training for Participatory Evaluation", Community Development Journal, 1988, vol. 23: 16-25.

_____. Participatory Evaluation: by, with and for the People. IDEA: Reading Rural Development Communications Bulletin. London: University of Reading Agricultural and Extension and Rural Development Centre, April 1982, vol. 14: 18-23.

_____. Partners in Evaluation: Evaluating Development and Community Programmes with Participants. London, MacMillan Education Ltd., 1986.

Forss, Kim. Report from the Participatory Evaluation of the Royal Government of Eland/UNDP Project ELA/90/003Rural Water Supply and Sanitation. 6 September 1994, UNDP.



_____. Participatory Evaluation: Questions and Issues. A Report Commissioned by the Central Evaluation Office, Occasional Paper no. 1. New York, UNDP, October 1989.

Forss, Kim and Claus Rebien. "Evaluation with (and without) Participation. A Case Study of the Evaluation of the UNDP project URT/89/003-Strengthening the Ministry of WaterTowards Achieving IWSSD Objectives". Report. New York, UNDP, 1992.

Freedman, Jim. Participatory EvaluationsMaking Projects Work. Technical Paper no. TP94/2. Calgary, Canada, Division of International Development, International Centre, University of Calgary, 1994.

Gariba, Sulley. "Participatory Impact Assessment as a Tool for Change: Lessons from Recent Experience in Poverty Alleviation Projects in Africa". Paper presented at 1995 Evaluation Conference, 1-5 November, Vancouver, British Columbia.

Ghai, D. and D. Westendorff (eds.). Monitoring Social Progress in the 1990s: Data Constraints, Concerns and Priorities. UNRISD, Avebury, Aldershot, 1994.

Guba, Egon and Yvonna Lincoln. "Do Inquiry Paradigms Imply Inquiry Methodologies?" in David Fetterman (ed.). Qualitative Approaches to Evaluation in Education: The Silent Revolution. New York, Praeger Press, 1988, 89-115.

_____. Fourth Generation Evaluation. England, Age Publications, 1990.

Hall, Budd, Alan Etherington and Ted Jackson. "Evaluation, Participation and Community Health Care: Critique and Lessons". Paper presented at the American Health Association, Toronto, Canada, November 1979.

Jackson, Edward. "Participatory Impact Assessment for Poverty Alleviation Opportunities for Communities and Development Agencies". Paper presented at the 1995 Evaluation Conference, 1-5 November, Vancouver, British Columbia.



Jackson, E. T. and Kassam, Yusuf. Better Knowledge, Better Results: Participatory Evaluation in Development Cooperation, 1996.

Kassam, Yusuf. "The Combined Use of Participatory and Survey Methodology in Evaluating Development Projects. A Case Study of a Rural Development Project in Bangladesh". September 1995. Paper presented at the 1995 Evaluation Conference, 1-5 November, Vancouver, British Columbia.

Kensey, David. Evaluation in Nonformal Education. The Need for Practitioner Evaluation. Amherst, Massachusetts, Centre for International Education, University of Massachusetts, 1981.

_____. "Participatory Evaluation in Adult and Nonformal Education". Adult Education, vol. 31:155-168.

Krishnamurthy, Lakshmi. "Exploding Myths: The Participants Evaluate" in W. Fernandes and R. Tandon (eds.). Participatory Research and Evaluation: Experiments in Research as a Process of Liberation. New Delhi, Indian

Social Institute, pp. 151-165.

Krueger, R. A. Focus Groups: A Practical Guide for Applied Research. Newbury Park, Sage Publications, 1988.

Lackey, Alvin, Mack Peterson and Jeff Pine. "Participatory Evaluation: A Tool for Community Development Practitioners". Journal of the Community Development Society, 12 pp., 1981.

Marsden, David, Peter Oakley and Brian Pratt. Measuring the Process: Guidelines for Evaluating Social Development. Oxford, 1994.

Merrifield, Juliet. "Putting the Scientists in Their Place: Participatory Research in Environmental and Occupational Health". Highlander Center Working Paper, New Market, Tennessee.

Merschrod, Kris. "Participation in Program Evaluation at the Regional Level



in Honduras". Rural Development Participation Review, vol. 11:18-22, 1980.

Narayan, Deepa. Participatory Evaluation Tools for Managing Change in Water and Sanitation. Washington, D.C., World Bank, Technical Paper no. 207, 1993.

Oakley, Peter. "Conceptual Problems of the Monitoring and Evaluation of Qualitative Objectives of Rural Development". Community Development Journal, vol. 23:3-11, January 1988.

_____. "Evaluating Social Development: How Much or How Good?". IDEA: Reading Rural Development Communications Bulletin. London, University of Reading Agricultural and Extension and Rural Development Centre, vol. 14:12-17, 1982.

Pagaduan, Maureen and Elmer M. Ferrer. "Working as EqualsTowards a Community Based Evaluation System." Community Development Journal, vol. 18:146-159, 1983.

Participatory Research Network. Report on lnternational Forum on Participatory Evaluation. New Delhi, India, International Council for Adult Education, 1988. 82 pp.

Pfohl, Jacob. Participatory Evaluation. A Users' Guide. New York, PACT Publications, 1986.

Quinn Patton, Michael. Alternative Evaluation Research Paradigm. Frank Forks, North Dakota, University of North Dakota, February 1965.

_____. Culture and Evaluation. San Francisco, Jossey-Bass, 1985.

_____. "Developmental Evaluation". Evaluation Practice, vol. 15, no.3, pp. 311-319, 1994.

_____. How to Use Qualitative Methods in Evaluation. Newbury Park, Sage Publications, 1987.



_____. Qualitative Evaluation Methods. Beverly Hills, Sage Publications, 1980.

_____. UtilizedFocus Evaluation. Beverly Hills, Sage Publications, 1978.

Robinson, Sheila and Philip Cox. Process Evaluation in Nepal: Tracking Capacity-Building in Health Development. Dialogue on Development, Technical Paper no. TP95/1, Division of International Development, International Centre, The University of Calgary, Canada, 1995.

Rugh, Jim. Self-Evaluation Ideas for Participatory Evaluation of Rural Community Development Projects. Oklahoma, World Neighbors, 1992.

Salmen, Lawrence F. Beneficiary Assessment - An Approach Described. Environment Department Papers no. 23, Washington, D.C., World Bank, July 1995.

_____. Listen to the People - ParticipantObserver Evaluation of Development Projects. Washington, D.C., World Bank, 1987.

Srinivasan, Viji. "The Methodology of Participatory Evaluation" in W. Fernandes and R. Tandon (eds.), Participatory Research and Evaluation: Experiments in Research as a Process of Liberation. New Delhi, Indian Social Sciences Institute, 1981.

Stecker, Carl. Participatory Evaluation: A Review of the Literature and Its Implications for Participatory Community Development, Comprehensive Paper One, University of Massachusetts, School of Education, April 1993.

Swiss Directorate for Development Cooperation and Humanitarian AID (SDC). Mirror, Mirror on the Wall... Self-Evaluation in Development Cooperation. Bern, Evaluation Service, August 1991.

Tandon, Rajesh and Walter Fernandes. Participatory Evaluation: Issues and Concerns. New Delhi, Society for Participatory Research In India, 1988.

_____. Participatory Research and Evaluation. New Delhi, Indian Social Institute, 1981.



_____. Participatory Research and Evaluation: Experiments in Research as a Process of Liberation. New Delhi, Indian Social Institute, 1982.

_____. Report of International Forum on Participatory Evaluation, 1-5 March 1988, New Delhi. New Delhi, 1988.

Uphoff, Norman. "A Field Methodology for Participatory Self-Evaluation". Community Development Journal, vol. 26, no. 4, 1991.

_____. Monitoring and Evaluating Popular Participation in World Bank"Assisted Projects" in B. Bhatnagar and A. C. Williams, Participatory Development and the World Bank: Potential Directions for Change, World Bank Discussion Paper no. 183. Washington, DC, World Bank, 1992.

_____. "Participatory Evaluation of Rural Development Projects", The International Fund for Agricultural Development, Monitoring and Evaluation Division, Economic and Planning Department, 1991.

World Bank. Evaluation and Development, Proceedings of the 1994 World Bank Conference. World Bank Operations Evaluation Department.

Yacoob, May and Tom Cook. "Evaluating Community Participation". WASH Working Paper, no. 98, Rosslyn, Virginia, USA, 1991.

Participatory Development

Australian International Development Assistance Bureau. "Social Analysis and Community ParticipationGuidelines and Activity Cycle Checklist". Appraisal, Evaluation and Sectoral Studies Branch, 1991.

Beaulieu, Rémy-Claude and Violetta Manoukian. "Participatory Development: A Brief Review of CIDA's Experience and Potential". Policy Branch, CIDA, February 1994.

Cernea, Michael. Putting People First. Sociological Variables in Rural Development. World Bank, 1991.



Chambers, Robert. "Paradigm Shifts and the Practice of Participatory Research and Development, IDS Working Paper #2, England, April 1994.

_____. "Participatory Rural Appraisal (PRA): Analysis of Experience" in World Development, vol. 22, no.9, pp. 1253-1268, 1994.

_____. "Participatory Rural Appraisal (PRA): Challenges, Potentials and Paradigm" in World Development, vol. 22, no. 12, pp. 1437-1454, 1994.

_____. Poverty and Livelihoods: Whose Reality Counts?, Institute of Development Studies, University of Sussex, Development Paper 342, England, 1995.

_____. Rural Appraisal: Rapid Relaxed and Participatory. Development Paper #311, Institute of Development Studies, University of Sussex, October 1992.

_____. "The Origins and Practice of Participatory Rural Appraisal" in World Development, vol. 22, no. 7, pp. 953-969, 1994.

Cohen, John and Norman Uphoff. Rural Development Participation: Concepts and Measures for Project Design, Implementation and Evaluation. Rural Development Monograph no.2, Rural Development Committee, Centre for International Studies, Cornell University, New York, January 1977.

Donnelly-Roark, Paula. "Re-inventing Bureaucracy for Sustainable Development -Donor Organizations and Participatory Development". New York, UNDP, November 1993.

Fals-Borda, Orlando and Mohammad Anisur Rahman (eds.). Action and Knowledge Breaking the Monopoly with Participatory Action Research. London, Intermediate Technology Publications, 1991.

Glade, William and Charles Reilly. Inquiry at the Grassroots: An Inter-American Foundation Fellowship Reader. Virginia, 1993.

GTZ. ZOPP: an Introduction to the Method. Frankfurt, Germany, March



1988.

Freire, Paulo. Pedagogy of the Oppressed. The Continuum Publishing Company, New York, 1993.

International Institute for Environment and Development (IIED). PLA NotesNotes on Participatory Learning and Action, IIED, Sustainable Agriculture Programme, London. Distributed regularly three times a year.

_____. RRA Notes. Special Issues on Applications of Wealth Ranking, Sustainable Agriculture Programme, United Kingdom, May 1992.

_____. RRA Notes. Special Semi-Special Issue on Participatory Approaches to HIV/AIDS Programmes, no. 23, June 1995.

Kumar, Krishna. Rapid Appraisal Methods. Washington, DC, International Bank for Reconstruction and Development, 1993.

Lineberry, William P. Assessing Participatory Development: Rhetoric Versus Reality. Westview Press, 1989.

Mabey, Christopher and Paul Iles. Managing Learning. Routledge, London, 1994.

Meister, Albert. La participation pour le développement. Paris, Les Editions ouvriéres, 1977.

Mosse, David. "Authority, Gender and Knowledge: Theoretical Reflections on the Practice of the Participatory Rural Appraisal". Network Paper no. 44, Agricultural Administration (Research and Extension) Network, Overseas Development Institute, London, December 1993.

Narayan, Deepa. The Contribution of People's Participation: Evidence from 121 Rural Water Supply Projects. Environmentally Sustainable Development Occasional Paper Series no. I, Washington, DC, World Bank, 1995.

Oakley, Peter et al. Projects with People: The Practice of Participation in



Rural

Development. Geneva, ILO.

Overseas Development Administration. Guidance Note on How to Do Stakeholder Analysis of Aid Projects and Programmes. England, Social Development Department, ODA, July 1995.

_____. Guidance Note on Indicators for Measuring and Assessing Primary Stakeholder Participation. England, Social Development Department, ODA, July 1995.

_____. Note on Enhancing Stakeholder Participation in Aid Activities. England, Social Development Department, ODA, April 1995.

Rahman, Mohammad Anisur. People's Self-Development. London, Zed Books, 1993.

Schneider, Hartmut and Marie-Héléne Libercier (eds.). Participatory Development from Advocacy to Action. Paris, OECD, 1995.

Scoones, Ian. "Investigating Difference: Applications of Wealth Ranking and Household Survey Approaches among Farming Households in Southern Zimbabwe". Development and Change, vol. 26 (1995), pp. 67-88.

Smillie, Ian. "NGO Learning, Evaluation and Results: Life in a Three-Ring Circus". Draft commissioned by the OECD Development Centre, Paris, for a conference in June/July 1996.

Tripp, Robert. Farmer Participation in Agricultural Research: New Directions or Old Problems. Institute of Development Studies, University of Sussex, Discussion Paper no. 256, February 1989.

UNDP. Grassroots Participation: Defining New Realities and Operationalizing New Strategies. UNDP Discussion Paper.

Uphoff, Norman. "Participation's Place in Rural Development: Seeking Clarity through Specificity". World Development, vol. 8, pp. 213-235.



Uphoff, Norman, M. L. Wickramasinghe and C. M. Wijayaratna. "Optimum Participation in Irrigation Management: Issues and Evidence from Sri Lanka". Human Organization, vol. 49, no. 1, 1990.

World Bank. The World Bank and Participation. Washington, DC, World Bank Operations Policy Department, September 1994.

Participatory Development and Evaluation manuals

Aaker, Jerry and Jennifer Shumaker. Looking Back and Looking Forward: A Participatory Approach to Evaluation. New York, Heifer Foundation, Private Agencies Collaborating Together/PACT, 1994.

American Council of Voluntary Agencies for Foreign Service. Evaluation Sourcebook. New York, 1983.

Aubel, Judi. Guidelines for Planning and Conducting Studies Using the Group Interview Technique. Geneva, International Labour 0rganization, 1994.

_____. Participatory Program Evaluation: A Manual for Involving Progra Stakeholders in the Evaluation Process. Senegal, Catholic Relief Services, December 1993.

Case, Roland, Mary Andrews and Walter Werner. How Can We Do It? An Evaluation Training Package for Development Educators. New York, InterAction American Council for Voluntary International Action, 1988.

CEDPA. Training Trainers for Development. Washington, DC, Centre for Development and Population Activities, 1995.

Davis-Case, D'Arcy. The Community Toolbox: The Idea, Methods and Tools for Participatory Assessment, Monitoring and Evaluation in Community Forestry. Rome, Community Forestry Unit, FAO, 1990.

_____. Participatory Assessment, Monitoring and Evaluation: A Field Manual. Rome, Community Forestry Unit, FAO, 1989.



FAO. Guidelines for Participatory Nutrition Projects. Rome, FAO, 1993.

_____. Participatory Monitoring and Evaluation. Handbook for Training Field Workers. Bangkok, Regional Office for Asia and the Pacific, February 1988.

_____. The Community's Toolbox: The Idea, Methods and Tools for Participatory Assessment, Monitoring and Evaluation in Community Forestry. Community Forestry Field Manual 2, Rome, 1990.

Gajanayake, Stanley. Community Empowerment: A Participatory Training Manual on Community Project Development. Office of International Training and Consultation, Northern Illinois University, PACT Publications, New York, 1993.

Gosling, Louisa and Mike Edwards. Toolkits. A Practical Guide to Assessment, Monitoring, Review and Evaluation. Development Manual 5. London, Save the Children, 1995.

International Institute for Environment and Development. RRA Notes,PLA Notes. Sustainable Agriculture Programme, London. Published quarterly.

Magnani, David P. (ed). Building Organizational Effectiveness Through

Participation and Teamwork: A Training Manual. New York, PACT Publications, 1992.

Narayan, Deepa and Lyri Srinivasan. Participatory Development Took Kit. Washington, DC, World Bank, 1994.

National Environment Secretariat, Government of Kenya; Clark University; Egerton University; The Centre for International Development and Environment of the World Resources Institute. Participatory Rural Appraisal Handbook: Conducting PRAs in Kenya. Natural Resources Management Support Series no. 1, 1991.

Pretty, Jules. Irene Gujit, Ian Scoones and John Thompson. Participatory



Learning and Action: A Trainer's Guide. London, International Institute for Environment and Development, 1995.

Srinivasan, Lyra. Options for Educators. A Monograph for Decision-Makers on Alternative Participatory Strategies. New York, PACT Publications, 1992.

_____. Tools for Community Participation. A Manual for Training Trainers in Participatory Techniques. Prowess/UNDP Technical Series Involving Women in Water and Sanitation, 1993.

Audio-Visual Material

PROWESS Video, Audio Plus Video, New Jersey, 1995.

World Vision Australia. The PRA Report: Walking in Their Shoes. Research and Policy Unit, Melbourne, Australia.

Salmen, Lawrence. Listen to the People, 1987.


Documents

Who Are the Question Makers - Undp Participatory Evaluation