EMERGENCY OPERATIONS AND REHABILITATION DIVISION (TCE)

M&E Toolkit

Transboundary Animal Diseases (TADS)

VERSON 1

8 November, 2006

by Andy Lyons, Consultant

REGIONAL INTERAGENCY COORDINATION SUPPORT OFFICE (RIACSO)

JOHANNESBURG, SOUTH AFRICA

ii

iii

THE FAO M&E ACTIVITY TOOLKIT SERIES This monitoring and evaluation (M&E) toolkit is one of a series that has been developed by the FAO Regional Interagency Coordination Support Office (RIACSO) in Johannesburg, South Africa. The goal of these toolkits is to strengthen and harmonize monitoring and evaluation of supported emergency activities throughout the southern Africa region. To achieve this goal, these toolkits provide a core set of indicators and templates that enable common data collection and the aggregation of results across multiple project sites within the region. The audience for this M&E toolkit is anyone who is involved in control of TADS or implementing TADs projects. This includes veterinary officers, FAO staff at country and regional levels, National Project Coordinators, and consultants. These toolkits are a work in progress. To be successful they must address regional cross-cutting information needs on project effectiveness, while also accommodating a diverse set of working conditions, operational resources, and information requirements of numerous national and local stakeholders. This is not an easy task, hence feedback from users is essential. Please channel all feedback through your local FAO Emergency Coordination Unit (ECU). Comments should be sent to the Regional Information Officer at RIACSO, Phil Fong, at phillip.fong@fao.org.

How to use this toolkit 1. Review the core output and outcome indicators (page 3)

2. Review the sample data collection forms (page 13)

3. Develop a workplan for data collection, processing, and analysis that will work for your specific context (page 12)

4. Implement the workplan.

Development of this toolkit has been made possible by the generous support of The Government of the Republic of South Africa under OSRO/RAF/510/SAF.

iv

v

TABLE OF CONTENTS The FAO M&E Activity Toolkit Series .................................................................................................. iii Introduction................................................................................................................................................1

Overview of Transboundary Animal Diseases......................................................................................1 FAO’s Support for Fighting Transboundary Animal Diseases .............................................................1

FAO's Regional Monitoring Strategy ........................................................................................................2 Basic vs. Preferred Standards for M&E ................................................................................................3

Monitoring TADS......................................................................................................................................3 TADS Core Indicators...........................................................................................................................3 Recording the Location of Surveillance and Vaccinations ...................................................................5

Nuts and Bolts of Data Collection for TADs M&E...................................................................................6 Overview of Information Flow..............................................................................................................6 Data Collection Tasks ...........................................................................................................................7

TADs Training ..................................................................................................................................7 Public Awareness Activities .............................................................................................................8 Herd, Abattoir and Market Inspections.............................................................................................8 Vaccinations......................................................................................................................................8 Outbreaks ..........................................................................................................................................9 Terminal Evaluation..........................................................................................................................9 Post-project Evaluation ...................................................................................................................10

TADs M&E Database Software ..............................................................................................................11 Making a Workplan for TADs M&E......................................................................................................12 References................................................................................................................................................12 Annex I – Sample Data Collection Forms ...............................................................................................13

Adapting the forms..............................................................................................................................13 Vaccination Log ..................................................................................................................................14 Surveillance & Testing Activity Log ..................................................................................................15 Template for the M&E Section of Quarterly Reports .........................................................................16

Annex II – TADs M&E Database............................................................................................................18 Annex III – Data format for sharing indicator data .................................................................................19

Why Standards for Data Formats are Needed .....................................................................................19 Table Definitions for Output Indicators ..............................................................................................19 Table Definitions for Outcome Indicators...........................................................................................21 Other Summaries.................................................................................................................................21 Summaries for Making Maps ..............................................................................................................22

Annex IV – Framework for the Assessment of Outbreak Responses......................................................23 TABLE OF TABLES Table 1. TADS core indicators ..................................................................................................................4 Table 2. Data collection tasks in TADs ....................................................................................................7 Table 3. Output indicators #1: training....................................................................................................19 Table 4. Output indicators #2: public awareness activities......................................................................20 Table 5. Output indicators #3: surveillance activities..............................................................................20 Table 6. Output indicators #4: vaccination activities...............................................................................21 Table 7. Outcome indicators #1: percent vaccinations deemed effective................................................21

vi

1

INTRODUCTION Overview of Transboundary Animal Diseases Livestock are an important element of many household economies throughout southern Africa. Livestock provide an important source of protein, draught power, and offer households a mechanism to build savings that can be drawn upon during periods of economic stress (i.e., walking bank account). Livestock are also important culturally in many areas, serving as a medium of exchange through which social networks are built and maintained. Because of their importance at the household level, many poverty alleviation programmes target livestock as a means of improving nutrition, increasing income, and building household assets. It is estimated that 60% of the population within the SADC region depend on livestock. At a national level, livestock is an important sector of agricultural production, accounting for 20–40% of agricultural GDP within SADC countries. Much of this production is traded regionally and beyond, providing an important source of revenue for government. Livestock also feature in many national HIV/AIDS prevention and mitigation strategies. Small-stock in particular, such as poultry, sheep and goats, offer an important way of reducing vulnerability to HIV/AIDS. All SADC member states have ambitious plans for the livestock sector as a major factor in both poverty reduction and food security strategies. Since 2001, southern Africa has experienced severe and persistent drought that has reduced areas of grazing in many countries, leading to wildlife-livestock competition for forage and watering points. The increased frequency of contact has facilitated transmission of foot-and-mouth disease (FMD) viruses from buffaloes to livestock. At the same time, government veterinary services in many SADC countries are inadequate due to chronic shortages of budget resources, insufficient supply of skilled manpower, the HIV/AIDS pandemic, and in some areas political tension or civil strife. This combination of factors has resulted in numerous new epidemics of FMD and Contagious Bovine Pleuropneumonia (CBPP), to the point where some epidemics have become endemic in the region. These outbreaks have decimated livestock populations in several areas, caused bans on livestock movement both domestically and across borders, and in general threatened the animal health status of the whole region. FAO’s Support for Fighting Transboundary Animal Diseases Perhaps the greatest challenges in preventing and containing livestock disease outbreaks lie along national borders. Many of the borders within SADC are highly porous, distant from national veterinary centres where testing facilities tend to be located, and are focal points for trade. Added to these challenges is the need to coordinate outbreak responses in veterinary departments in two or more countries, whose strategies and institutional capacities may be very different. For these reasons, FAO, with the financial support of the South African government under project OSRO/RAF/511/SAF, has prioritized transboundary regions for its effectors to reduce animal diseases. The main focus of the TADs project is supporting livestock and veterinary departments in strengthening surveillance systems and information flows. This will enable vet departments to detect and respond to outbreaks in a more timely and coordinated manner.

2

There are four main types of activities supported under the TADs project1:

• capacity building: training and education, regional workshops, development of country monitoring programmes

• public awareness: development and dissemination of information products • surveillance: physical and sero-surveillance inspections of animals in herds, abattoirs, and

markets; mobile veterinary laboratories; construction of surveillance pens; a pilot surveillance project using digital pen technology2

• strengthening preparedness for outbreak responses: procurement of vaccinations, developing mechanisms for reporting and coordinating outbreak responses

FAO’s support for reducing transboundary animal diseases builds upon on-going activities by national livestock departments. Animal inspections, vaccinations, and public awareness campaigns are tried-and-true methods for preventing and containing disease outbreaks. The TADs project provides resources to strengthen these established activities, as well as explore a few innovative strategies to address some outstanding gaps, particularly in the areas of information flow, mobile testing, and educational materials for public outreach. FAO'S REGIONAL MONITORING STRATEGY Monitoring and evaluation (M&E) in general serves three purposes:

• accountability – M&E allows donors, partners, and beneficiaries to see whether a project is being managed responsibly, is on course, and achieving the desired impacts.

• implementation – M&E facilitates implementation of an activity. For example, to plan vaccination campaigns one needs to know where previous vaccinations have taken place, numbers of animals vaccinated, efficacy rate, etc.

• improving project design – M&E allows project staff to make improvements in both the design and implementation of projects, for example targeting, the sequencing and timing of tasks, training needs, complementary services, etc.

FAO Emergency Operations Service (TCEO) and other humanitarian relief agencies are trying to strengthen M&E for all of the above reasons. To strengthen activity monitoring in southern Africa, FAO TCEO is using a three-pronged approach3:

• a conceptual framework outlining a regional M&E system based on the information needs and operational resources of FAO and its core partners

• a set of core outcome indicators which cut across multiple activities and project sites, to demonstrate aggregated impact at national and regional levels

• a series of M&E toolkits like this one providing guidelines and tools for monitoring different types of activities

1 see OSRO/RAF/511/SAF project document for details 2 digital pen technology allows data written on a paper form to be electronically transferred to a server through cellular phone networks or the internet, thereby eliminating the time-consuming task of manual data entry 3 see Lyons (2006) A Monitoring and Evaluation Framework for FAO Emergency and Rehabilitation Activities in Southern Africa. FAO RIACSO, Johannesburg, South Africa.

3

Why all this new fuss over M&E?

What's wrong with the way we've been reporting?

M&E in emergency interventions is frequently a challenge due to time constraints, limited staff resources, and the short time frame of most projects. M&E is therefore often delayed until the project is nearly over, or focuses only on the immediate outputs of an activity. A recent review of terminal reports has shown that most project reporting has not met the basic information needs of donors and project planners, who have a strong interest in project impact, lessons learned, and the linkages between activities. Without evidence to demonstrate that activities are producing results, the credibility of the entire programme is at risk. This toolkit represents one mechanism to fill in some of these gaps by providing guidelines and templates for a more systematic approach toward data collection and analysis.

Basic vs. Preferred Standards for M&E One of the challenges of trying to harmonize monitoring in multiple project sites across multiple countries is that the operational context and resources for M&E vary tremendously. To accommodate the varying levels of staff and funding, and not over-burden offices that are poorly resourced, the M&E guidelines that follow are classified as Basic and Preferred . The basic level of monitoring covers all of the key information needs, including the core outcome indicators, using a minimum amount of time and staff resources. The preferred standards also capture all of the core information needs, but go slightly beyond to get a better understanding of an activity's impact relative to the baseline conditions, a more thorough understanding of cause and effect, and an assessment of the longer-term impact and sustainability issues. All FAO ECUs and their implementing partners are expected to implement a monitoring system which at a minimum meets the Basic M&E standards, and as many of the Preferred standards as resources permit. Some ECUs may also wish to borrow from the preferred standards to meet reporting obligations under funding arrangements from other donors or joint-programming agreements. MONITORING TADS TADS Core Indicators To strengthen monitoring of FAO supported TAD activities, Emergency Coordinators from southern Africa developed a set of core indicators during an M&E workshop in February 2006 (Table 1). The proposed indicators capture both short-term improvements that are expected as a direct result of the supported TADs activities, as well as the ultimate goals or outcomes which focus on improving the timeliness and coordination of information on animal disease. In addition, several longer-term and/or complex processes were identified for inclusion in a terminal evaluation (page 9).

4

The difference between output and outcome indicators

Monitoring is often divided into looking at the process of conducting an activity, and its ultimate impact. Process is measured by describing the immediate outputs, such as the number of vaccinations given or people trained. Impact is evaluated by defining and measuring the medium and long-term outcomes of an activity, such as faster and more effective responses to disease outbreaks. In order to claim that an activity made a difference, we need evidence of both the process and the final outcomes. Hence most M&E systems simultaneously keep track of both process and impact by defining separate indicators for outputs and outcomes.

Table 1. TADS core indicators

Indicator Interpretation Output Indicators

1. Number of people trained, by type of training and gender

Training is the main focus of the capacity building component, and the number of people trained is an important process indicator that will show which groups are being reached and which topics are being addressed.

2. Number of communication or information products developed and distributed, by type of product (e.g., posters, leaflets, radio and TV programmes)

Public education in local languages is important for preventing outbreaks, this indicator will measure the mechanisms developed to increase public awareness.

3. Number of herds inspected Herd inspections are one of the key strategies used to detect outbreaks before they spread.

4. Number of sero-surveillance tests

Sero-surveillance is used to detect or confirm suspected occurrences of livestock disease.

5. Number carcasses inspected at abattoirs, slaughter slabs, and village slaughter sites

Carcass inspection is being supported because it is an efficient way to detect disease over a large area and reduce transmission at slaughter sites.

Basic

6. Number of animals vaccinated, by type of animal and vaccination

Animal vaccinations are one of the core control strategies and can prevent disease outbreaks from starting or spreading.

Outcome Indicators 7. Number of outbreak/ surveillance reports collected and disseminated

The immediate goal of much of the training and support is better and faster information sharing on surveillance and outbreaks. This indicator will capture whether these activities are resulting in better reporting, including written, oral and electronic formats.

Basic

8. Turn-around time for data processing and analysis

Time is critical in responding to disease outbreaks. This indicator will reflect the degree to which the supported mechanisms are reducing the amount of time for information to be shared within and between livestock departments.

5

Indicator Interpretation Preferred 9. Percent vaccinations

deemed effective Vaccinations will only reduce disease spread if they’re biologically effective. This indicator is needed to validate whether or not the vaccination activities are producing the desired outcomes. Vaccination effectiveness can be measured through laboratory testing or herd protection (immunity).

10. Timeliness and effectiveness of responses to disease outbreaks

Although we don’t hope to see any outbreaks, if one should occur then timeliness and effectiveness of the response is the ultimate measure of project impact. This is a composite indicator that requires a mix of qualitative and quantitative data (see Annex IV, page 23).

Recording the Location of Surveillance and Vaccinations The physical processes in disease transmission produce strong spatial patterns in outbreaks. Prevention and containment strategies must therefore also follow a spatial strategy to ensure that all affected areas are treated, and diseases don’t “slip through the cracks” to new areas. Hence many of the core TADs indicators on surveillance and vaccinations should be disaggregated by location, in addition to other types of disaggregation4. To capture the spatial element and allow indicator data to be displayed on maps, the Preferred M&E standard encourages all records of surveillance and vaccination activities to include a location reference, using either GPS coordinates or the official name of known places such as villages, towns, or administrative areas. The TADs M&E database (see page 11) can streamline the process of visualizing spatial patterns by providing tables that can be used by GIS software to make maps.

What about impacts at the household level? Why isn’t there a core indicator for that?

Although the ultimate goals of TADs include increased income, nutrition, and food security at the household level, a core indicator for household level food security has not been adopted for two reasons. First, the operational focus of TADs is appropriately on government livestock departments, who have the official responsibility and greatest resources for providing animal health services. Any impact the TADs project might have at the household level would only be indirect, through veterinary services, making it difficult to understand or interpret any observed changes in household food security, whether positive or negative. Secondly, household food security is a difficult characteristic to measure, and to do it properly requires a significant amount of training and fieldwork. This doesn’t mean that household level food security is not important, only that it be looked at as an evaluation question, when additional resources for fieldwork are often available. The guidelines for a terminal evaluation of TADs (see page 9) include questions on how the project has affected individual households.

4 see Paskin (1999) for details

6

NUTS AND BOLTS OF DATA COLLECTION FOR TADS M&E Overview of Information Flow Unlike many other emergency projects that are implemented at a local level by NGOs, TADs activities are implemented at the national and regional level by livestock department and FAO staff. Thus there are fewer “players” in data collection and analysis than for example garden or irrigation projects. A second difference is that TADs activities are less likely to follow a seasonal cycle but are more continuous and concurrent. Training, vaccinations, surveillance, and public awareness activities can all take place simultaneously, at any time of the year. Each country in the TADs project has a National Project Coordinator who is the focal point for information collection and dissemination. Veterinary officers conducting vaccinations, surveillance, and public awareness events need to keep the Project Coordinator informed about progress made, numbers of animals inspected, etc. Some sample templates for reporting activities are presented in Annex I (page 13), or existing forms may be used. Other activities, such as workshops and public awareness events, may be implemented at national or regional levels. A project monitoring database has been developed to help compile data at national and regional levels to facilitate the production of summaries and maps (see page 11). This general flow of information is illustrated in Figure 1 below. Figure 1. TADs Information Flow from the Field to Regional Level

NationalCoordinator

livestock inspections& vaccinations

RIACSORegional Livestock

Coordinator

reports & presentationsmaps

regional summaries& feedback

updates fordonors & partners

activity reports

Country 2

Country 3

national summaries

soft copies of summary datasurveillance

results via digitalpen website

workshops

public awareness events

lab reports on vaccination effectiveness

7

At each level, information will be used for different purposes. Some offices will use M&E data to prepare monthly progress updates. Others will need it for preparing mid-term reviews and terminal reports, and still others will use the data for more in-depth assessments or lessons learned briefs. The key to an effective vertically integrated information system is to know what needs to be shared and in what format. In general, higher management levels need only summaries of information collected at field/laboratory levels and detailed information should always and readily be available to them upon request. Annex III (page 19) describes the format for sharing the core output and outcome indicators for TADs. Data Collection Tasks Table 2 below describes the main activities in a TADs project and the accompanying data collection tasks. The sections that follow describe these tasks in greater detail. This table can serve as a guide for a data collection workplan, but is incomplete because it doesn't specify individual personnel who will be assigned for each task, geographic areas, transport requirements, amount of time needed, budget requirements, etc. (see page 12). Table 2. Data collection tasks in TADs

Data Collection Activity Basic Preferred (in addition to basic)

TADs training • attendance records – number and gender of participants, their organizations, topics covered

• entry and exit knowledge assessments

public awareness activities • the number and types of activities • area of coverage • focus group discussions on the

effectiveness of public awareness activities

inspections – herd and abattoir

• dates, locations, number of animals tested, (see sample recording template, page 15)

• record locations using GPS • digital pen project – summarize the

number of observations, turn-around time, actions taken

vaccinations • dates, locations, number of animals vaccinated (see sample template page 14)

• blood samples from vaccinated animals are sent for testing for effectiveness

outbreaks • conduct an assessment of the timeliness and effectiveness of the response (see Annex IV, page 23)

terminal evaluation • interviews with stakeholders • additional focus groups discussions • an external consultant is contracted

to facilitate the evaluation TADs Training Training on TADs topics is one of the main activities under capacity building, and is organized at both the national and regional level. For M&E purposes, the Basic standard calls for recording the number of participants, their gender, the organizations they come from, and the topics covered (see sample reporting template, page 16). This type of information is easy to collect from workshop reports, and will give a good picture of the total number of people trained per topic. In addition, the Preferred M&E standard encourages some assessment of the impact of training. Entry and exit exams are useful for measuring the knowledge gained during a training, which is a good proxy for short-term impact. Other methods can be used to assess the medium-term (i.e., six months after the training) and long-term impacts. Such assessments are extremely useful for

8

reviewing not only the impact of training, but the relevance of the curriculum, sustainability, and effectiveness of instructional materials. Public Awareness Activities Public awareness activities include development of brochures, posters, radio and TV programmes, and presentations made at public events. The Basic M&E standard calls for recording the type and number of information products disseminated, and where they were distributed (see quarterly reporting template page 16). The Preferred standard encourages additional probing into the effectiveness of public awareness activities, by asking questions such as what proportion of people were exposed to the material, whether they understand the messages, and whether behavior changed. A variety of methods can be used to assess education campaigns, including focus group discussions and randomized surveys. A prerequisite for evaluating communication campaigns, which is often lacking, is clearly articulating the communication goals and objectives. Herd, Abattoir and Market Inspections A major thrust of the TADs project is better disease surveillance through more frequent inspections of herds, abattoirs, and markets. For M&E purposes, the most important information to collect is the number of inspections, dates, locations and analysis of overall situation. This information should be captured by the veterinary officers using a template similar to the Surveillance & Testing Activity Log found in Annex I (page 15). The Preferred M&E standard encourages the use of GPS coordinates when recording locations that have not already been mapped. To capture the assistance provided by external organizations, including FAO, the Surveillance & Testing Activity Log includes a column called ‘External support’. In this column, officers at either the field or provincial level can write 1) how the surveillance activity was supported (e.g., through training, procurement of equipment or vaccines, transport), and 2) by whom. This will enable the Veterinary Department to report back to their partners how the different forms of assistance were utilized. A summary of the surveillance activity records should be forwarded to the National Coordinator on a monthly basis so a national summary can be prepared and any required actions initiated. The TADs M&E database has a data entry form to facilitate producing summaries of surveillance activities at national and regional levels. Livestock surveillance recorded using digital pens are automatically captured and entered into a central database, allowing national and regional coordinators to view summaries of surveillance activity in real time. Hence to prevent duplication in reporting, inspections that are made using digital pen technology should not be included in any summaries sent to the National Coordinator. Vaccinations The Basic M&E standard requires information on the number of animals vaccinated, by type of animal and vaccination. This data should be captured by the veterinary officers using their current forms or a template similar to the Vaccination Log presented in Annex I (page 14). A summary of vaccination activities conducted should be forwarded to the National Coordinator on a monthly or quarterly basis so a national summary can be prepared. The TADs M&E database has a data entry form to facilitate producing summaries of vaccination activities at national and regional levels.

9

The Preferred M&E standard also recommends conducting some laboratory tests on blood samples of vaccinated animals to determine the effectiveness of vaccinations. This assessment is essential to establish whether vaccinations are working, so that any problems with the quality of the vaccines, cold chain, or administration of the vaccines can be diagnosed and corrected. Lab results should be forwarded to the National Coordinators and Chief Veterinary Officers and summarized on the national quarterly report (page 16). Outbreaks If an outbreak should occur, the Preferred M&E standard encourages an assessment of the response for outcome indicator #10: Timeliness and effectiveness of responses to disease outbreaks. There are many aspects of ‘timeliness’ and ‘effectiveness’, so the assessment requires a case study approach as opposed to assigning a single number. An assessment framework is provided in Annex IV (page 23), which can be used to guide the documentation of an outbreak. Assessments do not have to be long and detailed, and can be conducted in-house or jointly with other livestock departments. Copies of outbreak assessments should be forwarded to the Regional Livestock Officer through the National Project Coordinator. Terminal Evaluation Terminal evaluations are often a contractual obligation, but should really be seen as an opportunity to share the results achieved with partners, review experiences, document lessons learned, and make recommendations for future projects. If funds are available, an outside consultant Preferred can be helpful by bringing a wider set of evaluation skills and a broader perspective on TADs. However even when funds are tight, an internal evaluation is highly worthwhile for packaging, interpreting, and communicating the results of the M&E system. The elements of a final evaluation often include:

• analysis and interpretation of project M&E data • interviews with major stakeholders • focus group discussions with beneficiaries • assessment of the broader context

Most terminal evaluations include focus group discussions and interviews with major stakeholders, such as livestock department staff, veterinarians, livestock owners, community leaders, government officials, and NGOs. Terminal evaluations also provide an opportunity to ask questions that ongoing M&E systems can’t because 1) the time frame needed to see change is longer than the project reporting cycle, 2) complexity of measurement, and 3) other factors are known to affect the observed outcomes. Questions in a terminal evaluation are usually outlined in a Terms of Reference, and should be based on the issues that emerged during the course of the project. It is generally preferable to highlight three to five key issues for a project evaluation, rather than ask every question in the book and risk getting an evaluation that is extremely general or based on scanty evidence.

10

Sample Evaluation Questions for a Terminal Evaluation of TADs Projects

Some typical issues addressed in terminal evaluations for TADs include:

• Summarize the main achievements of the TADs project (output and outcome indicators)

• Which departments or organizations were targeted for capacity building? Describe any observed changes in the human and technical capacity of selected groups.

• What impact did project activities have on the speed and efficiency of reporting surveillance results to decision makers?

• Describe any changes in the timeliness and effectiveness of responding to outbreaks.

• How did implementing partners and other stakeholders perceive the TADs project? Describe any barriers to implementation.

• Describe any change in government support for controlling TADs, for example through larger budgetary allocation, policy, allocation of resources on workplans, or other forms of support.

• How have the TADs activities contributed to livelihoods of livestock owners, for example through fewer losses of income and draft power from mortality, morbidity, or quarantine? What will be needed to sustain any gains in livelihood, and what additional assistance could be needed to maintain the progress made?

• Describe the goals and implementation of public awareness activities. What monitoring strategy was used to measure the impact of public awareness activities, and what were the results?

• If technology trials were part of the project design (e.g., digital pen technology), discuss the results of the trial. Summarize any technical reviews that were conducted and whether expanding adoption of the technology should be considered.

• Describe any significant patterns changes that took place in the broader context, such as climatic patterns, civil unrest, market disruption, etc., and how these changes might have affected the success of the TADs project.

• Describe issues concerning sustainability of the TADs project. Is there any way the project could have, or should be, implemented differently to make the achievements more sustainable?

• Present recommendations for future TADs projects Post-project Evaluation Output and outcome indicators do a reasonably good job in tracking the short and medium term impacts of TADs projects, however other anticipated results will only emerge a year or more after the project is over. Post-project evaluations are rare in emergency programmes because there is usually no requirement nor funding to conduct them. Nevertheless, the Preferred M&E standard recommends a post-project evaluation because there is no other way to determine whether the capacity building activities made a difference, and whether the surveillance and communication systems developed were sustainable on their own or dependent on continuous donor subsidies.

11

A post-project evaluation is similar in approach to a terminal evaluation, with particular emphasis on changes in the broader context, unanticipated outcomes, creation of dependency, and sustainability of project impact. The best way to prepare for a post-project evaluation is to ensure that the terminal evaluation is well-documented, and data are archived in an appropriate format for re-analysis. TADS M&E DATABASE SOFTWARE The accompanying CD (Annex II, page 18) contains a customized database application which has been designed to accompany the data collection templates presented in Annex I (page 13). The TADs M&E database, which is in Microsoft Access format, features a custom menu system with user-friendly data entry forms for each of the main data collection forms. The database will allow you to:

• enter M&E data on multiple computers and then merge the records into a single master database

• open ready-made tabular summaries (queries) and charts for analysis of the latest data entered

• generate a printed report of output and outcome indicators • export some or all of your raw data to Excel for further analysis and presentation • export indicator data in a standard format so you can send it electronically to another project

office (see Annex III, page 19) • import updates sent by the regional office with fixes and enhancements • supply the tabular information needed to make maps of your M&E data using an external

GIS program such as Dynamic Atlas or ArcView. The TADs M&E database will not interpret your data for you, nor write a report, but the statistical summaries and charts produced will greatly facilitate the process of analyzing impacts and lessons learned. For additional information on installing and using the TADs M&E database, please see the users manual or select 'Help' from the main menu.

Reporting Database Problems and Desired Enhancements

Users of the TADs M&E database are requested to report problems or confusing error messages so that an update can be prepared and sent out. Problems that affect one user are likely causing trouble for other users as well. In addition, please forward any requests for additional summaries (tabular queries or charts) for data analysis. For example, if you would like to see a summary showing the number of CBPP vaccinations given by district, but don’t see an appropriate query on the menu system, please forward a description of the desired summary so a query can be designed and an update sent to you and other users via email. (In the interim, you could export the raw data to Excel and create the analysis there). In this way, the database will gradually become much more useful over time. All feedback on the database should be forwarded through the national Emergency Coordination Unit to the Regional Information Officer at RIACSO (Phil Fong <phillip.fong@fao.org>).

12

MAKING A WORKPLAN FOR TADS M&E Indicators and data collection templates won't collect any data by themselves or produce any analyses. For that you need an M&E workplan which operationalizes the details of monitoring. Each National Coordinator needs to take the lead on developing a workplan for TADs M&E in light of the local resources, information needs, and operational constraints. An M&E workplan specifies:

• what data is going to be collected, when and by who • how, when, and where paper forms will be entered into the database • who is going to do the analysis and write-up • estimated costs associated with the above tasks

This toolkit is not an M&E workplan, but the tools and recommendations presented provide many of the core elements of an M&E workplan. To produce an M&E workplan, the following additional steps are recommended:

1. Read through this toolkit to get an overview of what is needed and ideas for your specific context.

2. In addition to the information needs at the regional level, which have already been defined, make a list of the most important national and local level information needs and questions that you would like to address through M&E.

3. Look at the list of Basic and Preferred indicators on page 3, and decide which ones will be adopted for your country.

4. Look at the sample data collection forms in Annex I, and decide which ones will be used and whether they need to be modified. If existing forms already capture the same information, it is probably better to continue using them.

5. Develop a workplan which outlines all of the needed M&E tasks, including the sampling methodology, field work, data entry, analysis, and writing. The workplan should specify the dates and staff involved in each task.

6. Discuss the M&E workplan and its implementation with implementing partners, revising it as needed.

7. Integrate M&E responsibilities into job descriptions, individual workplans, and Letters of Agreement.

REFERENCES Geering, W.A., Roeder, P.L., and Obi, T.U. 1999. Manual on the preparation of national animal disease emergency preparedness plans. FAO, Rome. [http://www.fao.org/docrep/004/x2096e/X2096E00.htm] Paskin, R. 1999. Manual on infectious disease surveillance. FAO, Rome. [http://www.fao.org/DOCREP/004/X3331E/X3331E02.htm] For additional information on TADs, see the website for the Emergency Prevention System (EMPRES) for Transboundary Animal and Plant Pests and Diseases: http://www.fao.org/empres

13

ANNEX I – SAMPLE DATA COLLECTION FORMS The sample data collection forms in this annex include:

1. Vaccination Log 2. Surveillance & Testing Activity Log 3. Template for the M&E Section of Quarterly Reports

Adapting the forms These sample data collection templates have been designed to be as generic as possible, however some additional adaptation will undoubtedly still be needed to suit the local context. (To modify the forms, soft copies in MS Word format can be found on the M&E database CD, see Annex II). Minor design changes that won't affect data entry can be made fairly easily, such as the addition of a logo, signature lines, labels, explanatory text, etc. Adding new fields, columns or sections is also possible, however this could also require modification of the accompanying database if the additional information needs to be captured. If new fields, columns, or questions are needed, it is recommended that you consult with the RIACSO office and other ECUs using the TADs M&E database to see if other users would also like to adopt the new fields.

We already use TAD Info or another monitoring system for TADs activities, why do we need to start using these new forms?

The short answer is you don't, provided that your current monitoring tools can produce the required numeric summaries for the core outcome indicators outlined in Annex III. TAD Info and other databases developed by veterinary departments in the region are usually designed to capture epidemiological information, but may also capture some of the data required needed to generate the core output and outcome indicators for TADS (see 3). The data collection forms in this toolkit are provided as sample templates that can be adopted when nothing comparable is currently available. These templates have been designed to be as generic as possible, but can be adapted at the country level to capture additional information as needed. Another advantage of adopting the sample templates is that the accompanying database software can be used to simplify data entry and analysis.

14

Vaccination Log Instructions. This template can be used by veterinary officers to maintain a record of vaccinations given that have been supported by FAO TADS or other projects. This template does not need to be used if there is already another reporting form that captures the same information.

This template can be photocopied or copied into a notebook. A copy of this summary should be forwarded to the TADs National Project Coordinator on a monthly or quarterly basis.

Livestock office: Location:

Reporting period: From: To:

Vaccination Activities During the Reporting Period

Location(s) GPS 3 Date(s) type1 name

Farming system2 south east

Species 4 Num animals

Vaccinated against 5

Outbreak response 6

External support 7

Comments

1 type of location: farm, village, crush pen, dip tank, abattoir, other (specify) 2 farming system: traditional, small holding, commercial 3 GPS: if known and not previously recorded

4 species: cattle, pigs, sheep, goats, poultry, equidae, dogs, cats, wild animals

5 vaccinated against: FMD, CBPP, rinderpest, lumpy skin, rift valley fever, blue tongue, swine fever, African swine fever, avian influenza, Newcastle, anthrax, rabies, sheep scab, other 6 response to an outbreak: yes or no 7 external support provided: training, equipment, vaccines, transport, other, (specify donor for each)

Prepared by: Date:

15

Surveillance & Testing Activity Log Instructions. This template can be used to record surveillance activity, including herd inspections, abattoir inspections, and other inspections. This template does not need to be used if there is already another reporting form being used that captures the same information.

This template can be photocopied or copied into a notebook. A copy of this summary should be forwarded to the TADs National Project Coordinator on a monthly or quarterly basis. This form is not needed to record inspections that have been made with digital pens.

Livestock office: Location:

Reporting period: From: To:

Surveillance Activities During the Reporting Period

Location GPS 3 Date type 1 name

Farming system 2 south east

Species 4 Nature of diagnosis 5

Num inspected

Disease 6 Outbreak response

External support 7

Comments

1 type of location: farm, village, crush pen, dip tank, abattoir, other (specify) 2 farming system: traditional, small holding, commercial 3 GPS coordinates: if known and not previously recorded 4 species: cattle, pigs, sheep, goats, poultry, equidae, dogs, cats, wild animals

5 nature of diagnosis: suspicion, clinical, post-mortem

6 disease: FMD, CBPP, rinderpest, lumpy skin, rift valley fever, blue tongue, swine fever, African swine fever, avian influenza, Newcastle, anthrax, rabies, sheep scab, other 7 external support provided: training, equipment, reactants, other (specify donor for each)

Prepared by: Date:

16

Template for the M&E Section of Quarterly Reports Instructions. This template is intended to serve as an outline for the M&E section of quarterly reports from countries receiving FAO support for TADs activities. This report is usually prepared by the national Project Coordinator or National Information Officer. This template only outlines the M&E section of quarterly reports, other sections (e.g., budget tracking, workplan) may still be needed but are not described here. This template represents the minimum content for M&E, additional data or findings can and should be included if available. The tables in this report can be generated manually or using the TADs M&E database. Country: Region:

Reporting period: From: To:

Surveillance Activities During this Period

Farming system

Location type

Species Nature of diagnosis

Num inspected

Disease Outbreak or routine

External support

Comments

Vaccination Activities Summary

Farming system

Location type

Species Num animals

Vaccinated against

Outbreak response

External support

Comments

Vaccination evaluation If any sero-surveillance or post vaccination herd inspection were conducted during the reporting period to measure vaccination effectiveness, please summarize the test results, especially the percent deemed effective. Training Activities During this Period

Participants Venue Topic(s) organization(s) males females

External support

Comments

If any entry/exit tests or other assessments were made during the training, please describe the results here.

17

Public Education and Awareness Activities Summary

Information Product(s)1 Quantity distributed

District(s) External support

Comments

1 e.g., brochures, posters, radio and TV programmes, presentations, drama

Please describe any outcomes, lesson learned, problems encountered, or other experiences with public awareness activities. Outbreaks If any disease outbreaks were encountered during the reporting period, please include an assessment of the timeliness and effectiveness of the response (see Annex IV)

18

ANNEX II – TADS M&E DATABASE The CD below contains a Microsoft Access database that has been designed to accompany the data collection forms and required outputs described above. To use the database, simply copy the “TADS M&E” folder to your hard drive and then open the database file with MS Access version 2000 or later 5. See page 11 for more information about the features of the database. The CD also includes soft copies of this toolkit and the data collection forms in Annex I. These templates can be modified to suit local requirements as described above.

TAPE CD ENVELOPE

HERE

5 If you get an error message when trying to open the database, saying that the database is read-only, first make sure you’re opening the copy of the database you put on the hard drive, and not the copy on the CD. If this doesn't solve the problem, close Access, select the database file in Explorer, choose Properties from the File menu, and then uncheck the ‘read only’ checkbox. The database should now be able to open without the error message.

19

ANNEX III – DATA FORMAT FOR SHARING INDICATOR DATA Why Standards for Data Formats are Needed One of the primary reasons for harmonizing how activities are monitored is to facilitate aggregation of results across project sites and over time. Without the ability to aggregate M&E results, it is difficult for donors and project managers to determine whether a national or regional programme is a good investment. Merging M&E results is conceptually simple, but requires M&E data to be captured in a uniform manner and formatted in the same way to allow data from multiple sites and years to be merged together. This toolkit and the accompanying TADs M&E database have been designed to streamline this process. The table below shows the required format for the core output and outcome indicators that have been agreed upon for TADs (see page 3). Although most of these indicators are defined as a single number (e.g., number of vaccinations), many require some type of breakdown (e.g., by type of vaccination). Furthermore, because indicator data will be merged across multiple project sites in multiple countries and over multiple time periods, additional columns are needed to indicate where and when the data originated from. Hence indicator data should be reported in the tabular formats outlined below, not as single numbers. The accompanying TADs M&E database is equipped to generate the tabular summaries below in a format than can be electronically sent to other offices and easily merged. Any National Coordinator that chooses to use its own system for data processing must be able to generate the tables below in a common file format, such as Excel, Access, DBF, or CSV. The regional office may be able to provide technical support in this effort if needed. Table Definitions for Output Indicators

Table 3. Output indicators #1: training Column Label Value

CountryID one of the values from tlkp_Countries ProjectSiteID one of the values from tlkp_ProjectSites (e.g., the name of a project site or border

region, or “national” if the record represents all project sites in the country) SeasonID one of the values from tlkp_Seasons, e.g., "2005 off season" = 2; "2005/06 main

season" = 3; "2006 off-season" = 4; "2006/07 main season" = 5, etc. ProjectID one of the values from tlkp_Projects (e.g., 1= OSRO/RAF/511/SAF) TrainingTopicID one of the values from tlkp_TrainingTopics, (e.g., 1=identifying TADs,

2=communication procedures, 3=digital pen technology, etc.) NumMalesTrained number of males trained NumFemalesTrained number of females trained ExternalSupportID one of the values from tlkp_ExternalSupport (e.g., 1=FAO training, 2=FAO

equipment, 3=FAO vaccines, 4=FAO transport, etc.) 6 Note: this table will give an accurate representation of the number of people trained per topic, but it will not be possible to add these together to get an accurate total of the number trained in all topics, because duplication has not been eliminated (people who attend more than one training). It is therefore important to not aggregate the total number of people trained when reporting.

6 If indicator data is shared in a relational database format (e.g., Access), ExternalSupportID should be saved in a separate details tables with a many-to-one relationship linked on the primary key (because many activities will be supported by more than one form of assistance and/or more than one donor). If formatted in a spreadsheet or tab-delimited text file, ExternalSupportID should be listed as a comma or pipe | delimited list.

20

Table 4. Output indicators #2: public awareness activities Column Label Value

CountryID one of the values from tlkp_Countries ProjectSiteID one of the values from tlkp_ProjectSites (e.g., the name of a project site or border

region, or “national” if the record represents all project sites in the country) SeasonID one of the values from tlkp_Seasons, e.g., "2005 off season" = 2; "2005/06 main

season" = 3; "2006 off-season" = 4; "2006/07 main season" = 5, etc. ProjectID one of the values from tlkp_Projects (e.g., 1= OSRO/RAF/511/SAF) InfoProdTypeID one of the values from tlkp_InfoProdTypes (e.g., 1=brochure, 2=poster, 3=radio

programme, 4=drama, etc.) NumDistributed number distributed ExternalSupportID one of the values from tlkp_ExternalSupport (e.g., 1=FAO training, 2=FAO

equipment, 3=FAO vaccines, 4=FAO transport, etc.) (see footnote page 19) DistrictID the ID number(s) for the 3rd level of administrative areas (in most countries this is

the district), taken from the table tbl_GeoCodes.7 Table 5. Output indicators #3: surveillance activities

Column Label Value CountryID one of the values from tlkp_Countries ProjectSiteID one of the values from tlkp_ProjectSites (e.g., the name of a project site or border

region, or “national” if the record represents all project sites in the country) SeasonID one of the values from tlkp_Seasons, e.g., "2005 off season" = 2; "2005/06 main

season" = 3; "2006 off-season" = 4; "2006/07 main season" = 5, etc. ProjectID one of the values from tlkp_Projects (e.g., 1= OSRO/RAF/511/SAF) Date date LocationTypeID one of the values from tlkp_LocationTypes (e.g., 1=farm, 2=village, 3=crush pen,

4=dip tank, 5=abattoir, 6=other) LocationName location name FarmingSystemID one of the values from tlkp_FarmingSystems (e.g., 1=traditional, 2=small holding,

3=commercial) GPS_X the X GPS coordinate for the surveillance site, if known, in decimal degrees

(positive values for eastern hemisphere) GPS_Y the Y GPS coordinate for the surveillance site, if known, in decimal degrees

(negative values for southern hemisphere) SpeciesID one of the values from tlkp_Species, (e.g., 1=cattle, 2=pigs, 3=sheep, 4=goats,

5=poultry, 6=equidae, 7=dogs, 8=cats, 9=wild animals, etc.) NatureDiagnosisID one of the values from tlkp_NatureDiagnoses (e.g., 1=suspicion, 2=clinical,

3=post-mortem) NumInspected number of inspected animals DiseaseID one of the values from tlkp_Diseases (e.g., 1=FMD, 2=CBPP, etc.) OutbreakResponse true if the surveillance was done in response to an outbreak, otherwise false ExternalSupportID one of the values from tlkp_ExternalSupport (e.g., 1=FAO training, 2=FAO

equipment, 3=FAO vaccines, 4=FAO transport, etc.) (see footnote page 19)

7 If indicator data is shared in a relational database format (e.g., Access), DistrictID should be saved in a separate details tables with a many-to-one relationship linked on the primary key (because many public awareness campaigns will encompass multiple districts). If formatted in a spreadsheet or tab-delimited text file, District ID numbers should be listed as a comma or pipe | delimited list.

21

Table 6. Output indicators #4: vaccination activities Column Label Value

CountryID one of the values from tlkp_Countries ProjectSiteID one of the values from tlkp_ProjectSites (e.g., the name of a project site or border

region, or “national” if the record represents all project sites in the country) SeasonID one of the values from tlkp_Seasons, e.g., "2005 off season" = 2; "2005/06 main

season" = 3; "2006 off-season" = 4; "2006/07 main season" = 5, etc. ProjectID one of the values from tlkp_Projects (e.g., 1= OSRO/RAF/511/SAF) Date date LocationTypeID one of the values from tlkp_LocationTypes (e.g., 1=village, 2=town, 3=district) LocationName location name GPS_X the X GPS coordinate for the surveillance site, if known, in decimal degrees

(positive values for eastern hemisphere) GPS_Y the Y GPS coordinate for the surveillance site, if known, in decimal degrees

(negative values for southern hemisphere) SpeciesID one of the values from tlkp_Species, (e.g., 1=cattle, 2=pigs, 3=sheep, 4=goats,

5=poultry, 6=equidae, 7=dogs, 8=cats, 9=wild animals, etc.) NumAnimalsVaccinated number of animals vaccinated DiseaseID one of the values from tlkp_Diseases (e.g., 1=FMD, 2=CBPP, etc.) OutbreakResponse true if the vaccination was in response to an outbreak, otherwise false ExternalSupportID one of the values from tlkp_ExternalSupport (e.g., 1=FAO training, 2=FAO

equipment, 3=FAO vaccines, 4=FAO transport, etc.). (see footnote page 19) Table Definitions for Outcome Indicators

Table 7. Outcome indicators #1: percent vaccinations deemed effective Column Label Value

CountryID one of the values from tlkp_Countries ProjectSiteID one of the values from tlkp_ProjectSites (e.g., the name of a project site or border

region, or “national” if the record represents all project sites in the country) SeasonID one of the values from tlkp_Seasons, e.g., "2005 off season" = 2; "2005/06 main

season" = 3; "2006 off-season" = 4; "2006/07 main season" = 5, etc. ProjectID one of the values from tlkp_Projects (e.g., 1= OSRO/RAF/511/SAF) SpeciesID one of the values from tlkp_Species, (e.g., 1=cow, 2=chicken, 3=sheep, etc.) DiseaseID one of the values from tlkp_Diseases (e.g., 1=FMD, 2=CBPP, etc.) NumEffective_SampSize the total number of vaccines tested NumEffective the number of tested vaccinations deemed effective

The other outcome indicators can be derived from output tables 1-4, with the exception of #8 (Turn-around time for data processing and analysis) which will be assessed based on interviews and analysis of digital pen data, and #10 (Timeliness and effectiveness of responses to disease outbreaks) which will be assessed based on case studies of outbreaks (see Annex IV). Other Summaries The five tables described above capture the core indicators that have been defined for TADs, however, by no means exhaust the types of analyses that can be generated from the TADs M&E data. There are many more summaries that can be generated at the national or regional level to help derive lessons learned and make improvements in the design and implementation of TADs projects. Some of the additional analyses include summaries of:

22

• a profile of surveillance activities in terms of passive vs. active strategies • correlations between vaccinations and surveillance results • spatial patterns in vaccination effectiveness • spatial correlations between public awareness activities and surveillance results

Summaries such as the ones above will be available in the TADs M&E database and can be used when preparing reports, however only the five summaries of output and outcome indicators need to be forwarded to the RIACSO office. The TADs M&E database is a work in progress and although it will have an initial selection of summaries available, user feedback will be needed to develop additional ones. Summaries for Making Maps In addition to numerical summaries, the TADs M&E database can be used as a source of data for making maps with GIS software. For example, maps could show the location of vaccination sites or inspections, where the color or size of the symbols represents a variable such as the number of animals vaccinated, etc. To make such maps, the first step is to construct a query in Access that includes either the latitude longitude coordinates of each point, or the name of a place that corresponds to a feature on a GIS layer. The second step is to link the GIS software with the query. Most common GIS programs, including Dynamic Atlas, ArcView, and ArcGIS, have the ability to link to tabular data stored in Access.

Creating new summaries for the TADs M&E database

Summaries like the ones described above are generated by designing a query object in MS Access. A query in Access can be thought of as a “window” on your data. Queries can be saved and each time you open the query, it presents the latest data. National Coordinators and project staff are in the best position to know what type of analyses or summaries would be most useful for analyzing a TADs project, and so are encouraged to think about how the M&E data can be analyzed to answer different questions. New queries can be added to the TADs M&E database by skilled users, or by sending a request to the RIACSO office. Alternately, the raw data can be exported to Excel or a statistics program and analyzed there. Access is a very flexible program and National Coordinators and their staff are encouraged to obtain an instructional book, and/or attend a training, on using MS Access for data analysis.

23

ANNEX IV – FRAMEWORK FOR THE ASSESSMENT OF OUTBREAK RESPONSES Introduction One of the core indicators for TADs is “Timeliness and effectiveness of responses to disease outbreaks” (page 3). This indicator can not be measured with a single number, but requires a case study approach to discuss the different aspects of “timeliness” and “effectiveness”. This framework presents an outline for conducting an assessment of the response to an outbreak. All outbreaks should be documented even if they are very small or quickly contained. If resources allow, an external person can be contracted to conduct the assessment, but if not even a short 2-3 page assessment can be useful. Outbreaks that cross borders can be combined into one assessment. The sections presented in this framework are only suggestions for documenting an outbreak, they can be modified or enhanced as needed. Section 1. Timeline Present a timeline of the outbreak and the response, including:

• when and how it was first discovered (list specific dates and places) • description of management responses • how and when surveillance information was shared with stakeholders, including headquarters,

other livestock offices, livestock owners, police and customs, neighboring countries, etc. • results of additional surveillance activity & detections throughout the course of the outbreak • when the outbreak was declared contained

Section 2. Assessment of Surveillance Information Sharing Discuss what went well and what didn’t in terms of surveillance and disseminating information on the outbreak. How big was the outbreak when it was discovered? Which stakeholders were informed quickly, and which took a long time? What caused the delays in surveillance and information sharing, if any? If there were any delays is informing sharing, how did this affect the timeliness of response? (e.g., was there a delay to get authorization to respond, reach consensus on a response strategy, or allocate resources for responding). Section 3. Assessment of the Response Discuss the strengths and weakness of the management responses. Were the selected responses appropriate for the type of outbreak? Were the responses effective in containing the outbreak? How did the outbreak change after the response, in terms of its growth or spread? If the outbreak occurred in more than one district or country, was the response coordinated? Section 4. Perception of Stakeholders How was the detection of the outbreak and the response perceived by stakeholders, including livestock owners, veterinary department staff, and community leaders, and colleagues in neighboring countries? When did each group of stakeholders learn about the epidemic and through which means? Did they understand and cooperate with the responses taken? Had they been exposed to any of the awareness campaigns on TADs? Section 5. Conclusions and Recommendations Discuss the overall effectiveness and timeliness of the response to the outbreak, and whether this outbreak was dealt with faster and more effectively than previous outbreaks, just the same, or worse. Discuss any contribution made by the activities supported by FAO (e.g., surveillance systems, communication systems, training). Make any recommendations for future responses.