Embed Size (px)
Citation preview
MINISTRY OF EDUCATION AND TRAINING TONGA
MONITORING AND EVALUATION STRATEGY
Participatory M&E
INTRODUCTION
by
Vilimaka Foliaki
Monitoring and Evaluation Advisor
Tonga Education Support Program, Phase 2
JANUARY 26, 2016
Table of Contents
Background to this M&E Strategy ............................................................................................................1
The M&E need in the Tonga Ministry of Education and Training .................................................................. 1
The current status of organizational M&E ..................................................................................................... 2
What is M&E? ........................................................................................................................................2
Why M&E? ..................................................................................................................................................... 2
Two main parts of M&E ................................................................................................................................. 2
Monitoring ................................................................................................................................................. 2
Evaluation .................................................................................................................................................. 3
This M&E Strategy ..................................................................................................................................4
Part of an ongoing M&E Capacity building .................................................................................................... 4
A guideline to diverse M&E activities ............................................................................................................. 4
Objectives of this M&E Strategy .................................................................................................................... 5
Scope of this M&E Strategy............................................................................................................................ 5
Utilizing the M&E Strategy ......................................................................................................................6
Focus on The Logical Framework ................................................................................................................... 6
Problem Analysis ............................................................................................................................................ 7
Stakeholder management .............................................................................................................................. 7
SMART Objectives .......................................................................................................................................... 7
Indicator matrix and M&E Plan ..................................................................................................................... 7
M&E reporting ............................................................................................................................................... 7
Appropriate M&E System for the MET ....................................................................................................8
Traditional M&E ............................................................................................................................................. 8
Participatory M&E ......................................................................................................................................... 8
Four (4) Critical Phases of M&E: Ensure a functional M&E System ............................................................... 9
Phase 1: Planning for M&E ........................................................................................................................ 9
Phase 2: Data collection ............................................................................................................................. 9
Phase 3: Make data usable ...................................................................................................................... 10
Phase 4: Use M&E data for decision making ........................................................................................... 10
References ........................................................................................................................................... 11
1
Background to this M&E Strategy
The M&E need in the Tonga Ministry of Education and Training The current Corporate Plan (CP 2015 – 2018) of the Tonga Ministry of Education and Training (MET)
provides the organization’s strategic directions for pursuing broad national outcomes which are outlined in
the Tonga Strategic Development Framework II: A more progressive Tonga: enhancing our inheritance
2015-2025 (TSDF II). Through the CP 2015-2018, MET aims to deliver its own organizational outcomes for
three years with specific annual targets. The Key Performance Indicators (KPIs) specific to these
organizational outcomes, and clearly stated in CP 2015-2018, are required to be monitored and evaluated
on a regular basis.
In the Foreword of CP 2015-2018, the Honorable Minister of Education, Samiuela ‘Akilisi Pohiva provides an
insight to the organization’s need for monitoring and evaluation (M&E):
“Accurate and valid information, therefore, is essential to determine strategic policy directions and in the
identification of specific interventions in the Corporate Plan. No plan, however, is perfect, and it is expected
that in the process of implementation, monitoring and evaluation, some adjustments would be necessary
over the 3-year period to ensure that the overall direction is maintained.” (CP 2015 – 2018, p. 3).
Further, the Introduction of CP 2015-2018 also re-emphasizes the importance of M&E to the organization’s
attainment of its outputs, outcomes and targets:
“The Corporate Plan aims to deliver the seven outputs with specific annual targets over three years,
measured against specific KPIs. These will be reviewed, monitored and evaluated on a quarterly basis to
provide sound assessment on the performance of the Ministry, thus ensuring that the outcomes are achieved
to standard…” (CP 2015-2018, pp. 6-7).
The design of this M&E Strategy is informed not just by direct experiences with M&E activities in the MET but also by a
series of existing guiding documents, in particular the MET Corporate Plan 2015-2018 and the Tonga Strategic
Development Framework 2015 – 2025, as shown in Figure 1 below.
Figure 1: This M&E Strategy is informed by both organizational and national guiding documents.
2
The current status of organizational M&E Unfortunately, however, the M&E system in MET is very weak, fragmented, inconsistent, and almost non-
functional. Most importantly, MET lacks the necessary staff with the required M&E knowledge and skills.
Apart from the M&E personnel who are on a short-term contract to work with externally-funded
development programs in MET (such as TESP2 and PEARL), there are no permanent MET staff specifically
for conducting M&E for the organization.
As experienced by the M&E Advisor for TESP2 and author of this document, many MET sections don’t have
an M&E plan or a Logical Framework. There is evidence which suggests that there have been some effort in
the past to develop M&E in some sections of MET, such as the Quality Assurance division, Curriculum
Development, and Human Resources division, but efforts to further develop, support, or expand these
efforts within these sections or to other key areas of the organization may have been unsuccessful.
MET is dependent on externally-funded programs such as TESP2 and PEARL for the attainment of most of
its organizational outputs and outcomes. Unfortunately, however, the current model which MET utilizes for
managing the key relevant activities (especially data collection and management) needs to be re-designed
so that it is more transparent and collaborative to facilitate mutual (MET and development program) and
timely access to quality data.
The above issues mean that MET is facing a lot of challenges in trying to achieve its wishes described in its
current Corporate Plan CP 2015-2018. Ongoing professional development on M&E needs to be conducted
to promote a culture of M&E within the organization. This M&E Strategy is part of a wider effort to build
the M&E capacity of the MET.
What is M&E? Monitoring and evaluation (M&E) are vital activities for any organization that aims for continuous progress
in its performance. The overall process of M&E provides people in an organization a systematic way of
finding out about what works (or doesn’t work) and why.
M&E involves not just the process of planning for and collection of data but also the analysis of collected
data, in order to determine whether progress is being made towards pre-specified goals and objectives.
Why M&E? The main purpose of M&E is two-fold:
1. To measure the quality of performance; and
2. To provide a basis for accountability (to key stakeholders such as development partners,
Minister and CEO, etc.) in the use of resources.
Two main parts of M&E Monitoring and evaluation (M&E), as the name suggests, has two main parts: i) Monitoring and ii)
Evaluation. These are described in detail below.
Monitoring This refers to the ongoing process of gathering data to inform us about our progress at any point in time.
Monitoring is often referred to as process evaluation because it focuses on the implementation process (or
how activities are carried out). In a teaching and learning context, for example, monitoring includes all the
formative assessment activities; evaluation refers to the summative assessment activities.
3
Other examples of monitoring exercises include regular assessment of the relevance of activities and
checking on progress towards objectives.
Compared to evaluation, monitoring activities are more regular. Monitoring provides regular feedback and
track changes over time. The information from monitoring processes are used for management control –
such as identifying and addressing problems with implementations thus improving the chances of attaining
broader goals and objectives (which are measured later during evaluation).
Evaluation This refers to measurements or making value judgments of the worth of an initiative. Evaluations are less
frequent than monitoring activities. This is always carried out at the end of an initiative. However, mid-term
evaluations and other evaluations are also carried out depending on the information needs of relevant
stakeholders.
An example of evaluation is the calculation of the Gender Parity Index of primary school enrolment at the
end of a 3-year project to find out whether the participation of girls in primary education has improved
since the beginning of the project.
M&E is integral to effective management at all levels of an organization. The current MET Corporate Plan,
as we’ve seen earlier, has emphasized that all organizational outcomes are required to be monitored and
evaluated.
M&E is needed at all levels of MET as an organization. From the classroom to the principal’s office to the
CEO’s office, an M&E system will help guide practice and ensure that key activities are carefully monitored
and that objectives are achieved and evaluated.
4
This M&E Strategy
Part of an ongoing M&E Capacity building This M&E Strategy is part of a wider and ongoing effort to improve the M&E capacity of MET. Apart from
conducting M&E activities specifically for TESP2, the author of this document had also conducted a series of
M&E professional development workshops for different sections and groups of staff of MET. This M&E
strategy is informed by experiences from these activities.
Besides this Introduction unit, this M&E Strategy contains six (6) other units with each unit covering
relevant information about an important M&E process or theme. These processes or themes were
identified, from the author’s experiences (described above), as pressing and urgent for the overall M&E
needs of MET. The M&E units are:
1. Introduction (this document)
2. The Logical Framework
3. Problem Analysis
4. Stakeholder Analysis and Management
5. Indicator matrix and M&E Plan
6. SMART Objectives
7. M&E Reporting
There are other M&E themes which can be later incorporated into this M&E Strategy. Other important
M&E themes such as Baseline Study, Data Analysis, M&E Budget, etc. can be developed and added later to
this strategy. The above units, however, represent the M&E areas that need to be appreciated now to set
the foundation for further development in the M&E capacity of the organization.
A guideline to diverse M&E activities This Strategy provides a guideline for carrying out M&E activities required for results-oriented reporting of
development initiatives (externally funded or otherwise) such as plans, policies, regulations, projects and
programs, and all result-oriented activities that are administered in MET.
The M&E needs of MET is very diverse, which is a reflection of its size (as the biggest organization in Tonga)
and the great diversity of activities that take place at different levels, sections and units of the organization.
MET caters for four (4) different levels of education: early childhood education, primary education,
secondary education, and tertiary education, and offering educational opportunities for both formal and
TVET. Besides the usual programs and activities of MET, there are also externally funded projects and
programs (such as TESP2 and PEARL amongst others) with each having a different set of M&E
requirements.
Due to the great diversity in the M&E needs of different MET levels, sections, and initiatives, this M&E
Strategy provides a general guideline for M&E by way of a seven (7)-unit series (listed above).
5
Objectives of this M&E Strategy This M&E Strategy acts as the MET’s guide toward effective utilization of resources and realization of
organizational goals and objectives. It is a flexible guide to M&E. It does not provide a one-size-fits-all M&E
plan of action.
With MET’s growing reliance on development partners for the funding of its operational activities, this
strategy also provides a pathway for the organization towards greater accountability, reflective practice,
continual learning and progress.
To achieve these aims, this strategy puts forward three objectives:
1. To establish a culture of monitoring and evaluation in the MET;
2. To support the development of a robust foundation for monitoring and evaluation of activities,
plans, policies, projects, programs and even sub divisions of the organization and the organization
as a whole; and
3. To assist in establishing a functional and sustainable M&E system in the MET.
Scope of this M&E Strategy This M&E Strategy is specifically designed for the Tonga Ministry of Education and Training (MET) as an
organization. It is developed based on direct experience with M&E in the MET and a thorough
understanding of not just education in Tonga but also the socio-cultural milieu within which development
activities in Tonga occur.
Specifically, this strategy should be used throughout the MET management cycle in:
1. Assessing any activity that fits within the scope of MET work plans (corporate, strategic, etc.);
2. Preparing project and program designs to ensure they are systematically developed and logical;
3. Implementing approved projects and programs;
4. Reviewing project and program designs; and
5. Monitoring and evaluation of projects, programs, policies, plans, units, sections, etc.
6
Utilizing the M&E Strategy
Focus on The Logical Framework This M&E Strategy suggests that all M&E work in the MET to begin with the creation of a Logical
Framework as it is a prerequisite to all other M&E requirements (as seen in the radial diagram Figure 2).
In M&E work, the Logical Framework not only ensures that the overall design is logical but also helps to
identify the relevant indicators and their sources, relevant activities, as well as the key assumptions and
risks. The Logical Framework provides the big picture awareness of all the key M&E elements or themes
(which are covered in the other units and described in the surrounding circles – of the radial diagram Figure
2 below).
Figure 2: The Logical Framework is priority and central to understanding the rest of the units. The rest of the units are extensions of the Logical Framework and therefore they provide feedback and support to the Logical Framework.
7
The rest of the M&E Strategy units can be studied and implemented in any order, after the Logical
Framework has been created. However, the selection of what unit (or processes) to use thereafter will
depend on the M&E need and prior experiences (with M&E) of the user staff or section. It is important to
note that all of the M&E Strategy units and the M&E processes outlined in them must be carried out in
order for an M&E system to work.
Note that the M&E Strategy units (represented by the surrounding circles) are just extensions of the Logical
Framework. Each of these units is introduced briefly below:
Problem Analysis Having a thorough understanding of a problem is a precondition to designing a solution that works. This
unit describes The Problem Tree as a tool anyone can use to thoroughly understand a core problem and its
root causes. Such awareness is vital for designing multiple interventions that effectively address the
diversity of root causes. Please refer to this unit if you feel that a relevant initiative may not be effective or
your activities and interventions are not really solving the problems they are supposed to solve.
Stakeholder management Stakeholders are very important as they are the ones who provide us with key resources, data and
information. This unit describes Mendelow’s Power-Interest Grid as a tool we can use to understand our
stakeholders and to effectively manage our relationships with them. This unit contains some very
important management tools such as a template for a Stakeholder register and template for a Stakeholder
management strategy.
SMART Objectives This unit focuses our attention on our objectives and the central role that they play in M&E. It provides a
list of criteria for writing objectives to ensure they can be easily monitored and evaluated. The unit also
provides an Objective checklist which can be used by anyone to improve the quality of their objectives.
Please refer to this unit if you feel that relevant goals and objectives need to be better understood or
improved.
Indicator matrix and M&E Plan This unit discusses indicators, in general, and Educational indicators in particular. It also describes what an
Indicator matrix is and how it can be used as a M&E Plan. This unit also contains an Educational Indicator
register, which lists 30 common educational indicators together with descriptions of how each indicator is
calculated.
M&E reporting This unit discusses why reporting is important for a M&E system to function. It also discusses two major
types of M&E reporting: internal reporting and external reporting, plus templates for these types of
reporting. This unit further clarifies the difference between two groups of stakeholders: internal
stakeholders and external stakeholders.
8
Appropriate M&E System for the MET
Traditional M&E Traditionally, M&E has been the work of external experts who come in to monitor or measure progress
toward outcomes. In these M&E systems, the management (represented by Level X in Figure 3 below) of
the organization is responsible for identifying the development needs of the organization. The
management is also responsible for the design of projects and programs that are implemented at other
levels of the organization (represented by Level Y in Figure 3 below).
The management then commissions M&E experts who are external to the organization to carry out the
monitoring of progress and evaluation of achievements (see Figure 3). During the M&E processes, the
activities of individuals in Level Y as well as their processes are being carefully and ‘objectively’ monitored,
and their achievements measured.
One major disadvantage of traditional M&E systems is that they inadvertently disempower the most
important stakeholders, the direct beneficiaries (e.g. schools, staff, students), by treating them as objects
of research. In situations such as these, it is difficult for these stakeholders to develop a sense of ownership
of the development activities, the processes of evaluation and the results that come out of it.
Participatory M&E In Participatory M&E (sometimes abbreviated as PM&E), the direct beneficiaries of development initiatives
(Level Y) are actively engaged not just in the identification of development needs, the planning and
designing of projects but also in carrying out the monitoring and evaluation activities, as shown in Figure 4.
A significant advantage of participatory M&E over traditional M&E, is its inclusive approach and the
promotion of ownership of development initiatives which, in turn, ensures benefits are long term. Without
local ownership, change is likely to be short-term and limited (Communication for Social Change
Figure 3: Traditional M&E – External M&E 'experts' are commissioned by management to carry out the monitoring and evaluation of initiatives that are being implemented at lower levels of the organization.
Figure 4: PM&E – Beneficiaries (Level Y) of development are actively engaged in monitoring and evaluation. The management (Level X) facilitates and offers support.
9
Incorportaed, 2005). In addition, PM&E is more inclusive and more responsive to the needs and aspirations
of direct beneficiaries and those most directly affected. This M&E Strategy is designed for a Participatory
M&E System.
The MET is a very big and complex organization with many important development activities happening in
its sections (such as educational levels and subdivisions). Further, all of these development activities have
far reaching effects throughout the nation. These activities demand that the MET has a functional and well-
established M&E system to track the progress of these activities and to ensure they are successful.
Four (4) Critical Phases of M&E: Ensure a functional M&E System
M&E can be divided into four (4) phases shown in Figure 5.
All the M&E processes covered in the M&E Strategy units take place in the four phases above. These four
phases are discussed in detail below.
Phase 1: Planning for M&E This includes the creation of the Logical Framework, Problem analysis, Indicator matrix and M&E Plan,
Stakeholder Analysis, etc. These specific processes, as discussed in the M&E Strategy units, take place
during the planning phase of M&E. A M&E system will not work if the required planning activities are not
carried out.
Further, since the suggested M&E system is a Participatory M&E one, all the above-mentioned planning
activities should be carried out collaboratively between key stakeholders.
Phase 2: Data collection Both monitoring and evaluation need data and they will not work if data are not collected. So it is
important to design relevant data collection instruments and use them to collect data according to planned
schedules. A M&E system will not function if data are not collected either for monitoring or evaluation.
Figure 5: The four M&E phases.
10
Phase 3: Make data usable A M&E system will not work too if the collected data are not analyzed and transformed into meaningful
information and reported to key stakeholders – both internal and external. It’s important as well to note
that the staff and sections of MET who are directly involved with development initiatives are also our
stakeholders, and therefore they should be informed about the results of M&E analysis.
Phase 4: Use M&E data for decision making A M&E system will not work if stakeholders are not using M&E data and information to make decisions. A
M&E system is considered useful or functional if stakeholders are using the information obtained through
M&E to make important decisions. M&E reports – both monitoring reports and evaluation reports should
be made available to relevant stakeholders, in particular the MET staff and sections who are directly
involved with development initiatives, and they should be encouraged to make important work decisions
based on these information.
11
References Catholic Relief Society (CRS). (2008). Monitoring and Evaluation Planning. Monitoring and Evaluation Short
Cuts: Guidelines on Monitoring and Evaluation Planning.
Communication for Social Change Incorportaed. (2005). Who measures change? An introduction to
participatory Monitoring and Evaluation of communication for social change. Retrieved from CFSC
Home: http://www.communicationforsocialchange.org/
Folley, E., Mishook, J., Thompson, J., Kubiak, M., Supovitz, J., & Rhude-Faust, M. K. (2008, January). Beyond
Test Scores: leading Indicators for Education. Retrieved from Annesberg Institute for School
Reform: http://annenberginstitute.org/publication/beyond-test-scores-leading-indicators-
education
Government of Tonga. (March 2015). Tonga Srategic Development Framework II: A more progressive Tonga
- enhancing our inheritance 2015 - 2025. Nuku'alofa: Government of Tonga.
Kaagan, S., & Smith, M. S. (1985). Indicators of Educational Quality. Educational Leadership.
Lewis Grant Associates. (n.d). Logical Framework Approach. Retrieved from LGA - We help you achieve your
objectives: http://lgausa.com/top_page.htm
Practical Concepts Incorporated (PCI). (1979). The Logical Framework: A manager's guide to a scientific
approach to design and evaluation. Retrieved from USAID: From the Americal People: Development
Experience Clearinghouse: https://dec.usaid.gov/dec/home/Default.aspx
Wikipedia. (2015). Project Management. Retrieved from Wikipedia:
https://en.wikipedia.org/?title=Project_management
