Emergency & Rehabilitation Coordination Unit. 2 An organization is comprised of its projects Understanding how the project cycle works is the first step

Emergency & Rehabilitation Coordination Unit

2 An organization is comprised of its projects Understanding how the project cycle works is the first step to understanding how monitoring and evaluation fits into the day-to-day operations of the organization

Why learn the Project Cycle? An organization is comprised of its projects Understanding how the project cycle works is the first step to understanding how monitoring and evaluation fits into the day-to-day operations of the organization 3

Need IdentifiedProposal DevelopmentDonor ApprovalRecruitment/ProcurementImplementationMid-Term EvaluationContinued ImplementationFinal Evaluation 4

Need Identified: A need can be identified in many ways: Through previous projects Through assessments Through calls from Donors Through calls for assistance from Government Etc... 5

Proposal Development: Numerous individuals are involved in designing a project. Operations, Field Staff, M&E Staff, Reporting, etc. The design of a proposal varies depending on the donor, but there are some common elements: Background Information Beneficiary Criteria Implementation Plan Risks Monitoring and Evaluation Log Frame Budget 6

Proposal Development Continued: Background Information should be linked to the identified need and should also explain why an organization is qualified to implement the project. Beneficiary Criteria defines the type of beneficiaries the project will target (specifics vary, but vulnerable is a general criteria) Implementation Plan outlines the activities that will take place and where Risks allows the organization to identify possible obstacles that may arise 7

Proposal Development Continued: Monitoring and Evaluation in terms of the proposal, is how an organization provides accountability to the donor. The Log Frame outlines what the project seeks to achieve and how those achievements will be measured Budget 8

Donor Approval: Once the Donor has approved and signed the proposal it becomes a binding contract. The proposal is a legal document that states, on the one side, the donor will provide money to the organization, and on the other, that the organization will achieve certain results using the money in the manner outlined in the budget. 9

Recruitment & Procurement: After the funds are released (and for some organizations, before) recruitment of staff and procurement of inputs starts. This stage is primarily the responsibility of logistics and HR with assistance from Operations 10

Implementation: General activities. This varies greatly program to program. In the case of FAO, implementation is largely covered by implementing partners which requires Letters of Agreement (LOAs) which are also legally binding documents outlining how the Service Provider (SP) will use the funds/inputs provided. In the case of FAO, though the addition of SPs is slightly more complicated, activities are usually pretty straight forward. 11

Mid-Term and Final Evaluations Donors generally require at least one interim report, and sometimes more. They also require a final evaluation report. These reports should cover how the project is fairing against the indicators outlined in the log frame. They must also note any actions that deviate from the original proposal. 12

There are two primary reasons for the monitoring and evaluation of projects 1) To ensure the project is progressing in the best way possible. 2) To measure whether the project is reaching (and has reached) the goals outlined in the proposal.

One of the most common misconceptions about M&E is that it is designed to show the bad in a project. Many organizations do not report accurate data for fear that it will reflect badly on them.

The goal of M&E is to determine how to improve a project, rather than to police it. All project designs, to some extent, are based on assumptions and though these may be rooted in experience, they can sometimes be mistaken. It is important to identify where changes need to be made Though problems may arise, they do not necessarily reflect mistakes on the part of an organization or its staff

By providing accountability and transparency organizations show themselves to be honest Donors want to see that organizations are implementing the projects they proposed. Identifying and rectifying issues that arise in the field shows that an organization puts the implementation of the project (and thus the beneficiaries) first.

UNDP states that Monitoring and Evaluation should be Results Oriented and provide assessments of the Relevance, Progress and Success of a project

Need IdentifiedProposal DevelopmentDonor ApprovalRecruitment/ProcurementImplementationMid-Term EvaluationContinued ImplementationFinal Evaluation As a technical team, M&E can assist in designing and implementing Needs Assessments M&E should always have the final say on Indicators M&E is responsible for providing the data for the Mid-Term Evaluation Monitoring Continued Monitoring Data collected for evaluation is used to describe the effects and outputs on an intervention

There are two main ways to collect data during a project: 1) Basic implementation facts These are generally recorded by all organizations and include x number of inputs distributed to x number of beneficiaries 2) Perceptions of beneficiaries and stakeholders Often both qualitative and quantitative data is collected to determine if the perceptions of the project are those predicted The important component for monitoring and evaluation is number 2

Qualitative Deals with descriptions and observations Cannot be measured e.g. what is your favorite food? Quantitative Deals with numbers Can be measured e.g. how many times a day do you eat? Sample Size A portion that is used to represent a whole In very simple terms, a piece of cake can determine what the whole cake should taste like Based on scientific methodology

There are 3 standard methodologies for collecting data from project stakeholders. 1) Key Informant Interviews Held with a single person and often open ended Usually qualitative in nature 2) Focus Group Discussions Held with a group, normally between 6 and 12 people Also usually qualitative 3) Surveys Cover a wide range of people Usually more quantitative, though they often seek to quantify qualitative data through the use of multiple choice questions Often based on a sample size.

Since emergency projects take place in a short amount of time and often cover a large population, qualitative data is not usually possible to collect, and takes too long to analyze Surveys are carefully designed based on project goals so that they can be analyzed and eventually determine how a project progressed using qualitative data in a way that can be quantified Due to the large number of beneficiaries a sample size of the beneficiary population is usually used.

Database vs. Analysis There is a significant difference in the needs of a database system and an analysis system Database: The primary concern for a database is how to store a large amount of data in a small space Analysis Analyzing information requires that the data itself be 1) Clean = Spelling and formatting are uniform throughout the database and other sections have been checked and reworded so they can be compared 2) Simple to read 3) In cases where the data is to be merged/compared with other datasets, the field headings and formatting needs to be exactly the same.

Reporting Results M&E Reports are quite different to general reports. The goal of an M&E report is to state results in a factual manner. Though there may be some need to explain some results, conjecture should be avoided wherever possible M&E uses information that can be verified though a technical process

There are 4 standard methodologies for collecting data: 1) Desk Review Review existing information 2) Key Informant Interviews Usually held with a single person and often open ended Usually qualitative in nature 3) Focus Group Discussions Held with a group, normally between 6 and 12 people Also usually qualitative 4) Surveys Cover a wide range of people Usually more quantitative, though they often seek to quantify qualitative data through the use of multiple choice questions Often based on a sample size.

A desk review can be an incredibly useful source of information The first step of a desk review is to determine what data may already exist on the population group being targeted in the study (e.g. beneficiary population) This data can serve as a baseline or may be useful for helping to design questions and determine what type of data collection tool to use

Usually a One-on-One Open Ended Interview Qualitative. Allows the interviewer to ask additional questions as the conversation progresses Should try to keep it at one-on-one to help the person being interviewed feel comfortable (and thus more likely to share) Should try to respect the position of the person being interviewed E.g. if the person being interviewed is high level government, then it should be a very high level staff member conducting the interview

How to record information As there is likely only a general list of questions to ask, and the primary benefit is the ability to ask questions in an ad hoc manner, it is important to ensure there are detailed notes from such an interview. Ideally the interview should be recorded but you must have the permission of the person being interviewed in order to record them. Alternately take limited notes during the interview (if you write the whole time you may not hear important information, and it is awkward for the person speaking.) Be sure to write detailed notes up right after the interview is complete

Different Meanings of FGDs In academic terms, a focus group discussion (FGD) is qualitative and involves a group of people discussing a given topic. (As opposed to answering a multiple choice question which does not involve the same discussion). However, the terminology is now widely used to describe group surveys, some of which may have pre-set answers. For the purpose of this discussion, FGDs refers to the academic definition

Why Conduct an FGD? FGDs are used, primarily when you require general opinions, and are useful for helping confirm or reject assumptions. E.g. To understand the different roles and responsibilities of men vs. women and how theyve changed over a long period amongst a group of IDPs, you could hold FGDs with Men and Women.

How to Conduct an FGD FGDs should have a facilitator and a note taker. Ideally they should involve activities that illicit responses. The most important thing to remember with FGDs is that the participants should be doing the talking. It is very important to try not to LEAD their answers

Difficulties Encountered Conducting FGDs

First Questions: What information do we need to know? What ways can we collect this information? Do we need a baseline to compare? What type of data collection tool should we use? (Open ended qualitative or closed quantitative) Designing the Questions In quantitative surveys, it is just as important to design the answers as the questions Questions should be kept simple and in a format that is easy to use in the field and the language and terms used should be local where possible For surveys that are often reused, they should be constantly examined and improved

Remember: Always keep the reason for asking the questions in mind. Keep it Simple Try to keep is Short

Important Component of Targeted Interventions Donors want to know why an intervention is needed. Detailed assessments help to illustrate why. Needs assessments take many different forms, but can normally fall into Rapid or Detailed (though there are many in between)

Emergencies In an Emergency it is important to collect information as quickly as possible. 72 Hours 1 to 2 Weeks 1 Month

A coordinated assessment is always complicated. Important to ensure everyone agrees before moving on to the next step

Steps: 1. Determine what the goals of the assessment are (what the final report should be focused on) 2. Create a list of general questions the assessment seeks to answer 3. Determine the tools to be used and assign the general questions to respective tools 4. Design the tools 5. Collect the Data 6. Clean and Analyze the Data 7. Produce the Report

These should follow a similar process to the Coordinated Assessments

Important Note: Needs assessments determine where a project will take place and what it should focus on They should not be the sole responsibility of M&E or they may compromise the independence of the M&E department

Most important part of the proposal for M&E Determines what the organization is contractually (legally) obligated to report on. Within the log frame there are overall goals and activity specific goals Overall Goal What the project seeks to achieve as a whole Specific Goals What each activity will achieve

The indicators listed under each activity must be reported on They need to be MEASURABLE They need to be SIMPLE They need to be easily UNDERSTOOD and not open to misinterpretation

Look at the Sample Log Frame provided. Please take 10 minutes to go through it and identify indicators that you think are Good or Bad

Using the same log frame, please work together to identify alternative indicators for each activity. Be sure to think about how they will be measured.

Designing an M&E Plan Examine the Indicators Determine what information you must collect Identify any additional information you should collect Determine the tools you will need to use Design the tools Execute the Plan

Using the indicators you just agreed on, come up with a monitoring and evaluation plan for the project. Beneficiary Criteria is vulnerable individuals affected by floods

Documents

Emergency & Rehabilitation Coordination Unit. 2 An organization is comprised of its projects Understanding how the project cycle works is the first step