1
A Monitoring and Evaluation
Framework for the ETDP SETA
Prepared by: Research Chair – Evaluation
UKZN
October 2016
2
TABLE OF CONTENTS
1. Executive Summary ………………………………………………………………………. 3
2. The Purpose of M & E ……………………………………………………………………. 5
3. Scope of M & E………………………………………………………………………………. 6
4. ETDP SETA Approach to M & E…………………………………………………… …. 6
5. Levels of M & E………………………………………………………………………………. 9
6. Key Indicators and Priority Targets ………………………………………………… 11
7. Evaluations …………………………………………………………………………………… 12
8. Data Collection ……………………………………………………………………………… 14
9. Reporting ………………………………………………………………………………………. 14
10. Annexures ……………………………………………………………………………………….16
10.1 The Results Framework of the ETDP SETA’s strategic outcome-
oriented goals 2015/16 – 2019/20 …………………………………………… 16
10.2 The Proposed M & E Plan and Institutionalization Strategy ………..20
3
Executive summary
The M & E Framework proposed here sets out the minimum monitoring and
evaluation requirements that will enable collection and provision of indicators of
progress, or lack thereof, and effective review of the ETDP SETA’s programmes. It will
guide the overall monitoring and evaluation functions of the SETA during
implementation of its current Five Year Strategic Plan 2015/16 to 2019/20.
The M & E activities will essentially be focused on the performance of six programmes
that have been formulated by the ETDP SETA to respond to the national priorities of
seeking that there should be quality basic education (outcome 6.1) and a skilled and
capable workforce to support an inclusive growth path (outcome 6.5) that the SETA
aims to contribute to. The proposed participatory M & E system is underscored by the
belief that effective implementation of M & E strategies requires the buy-in and a sense
of ownership by those who will be practising it. For the strategies to succeed, M & E
should be institutionalized to involve people and their capacities, processes and
systems.
In essence the M & E activities proposed in this framework will take place at two
distinct but closely connected dimensions: one level will focus on the outputs, which
are the specific products and services that emerge from processing inputs through
programme activities, whilst the other focuses on the outcomes and impacts which are
the changes that the ETDP SETA aims to achieve through the delivery of its
programmes. As the custodian of and resource for M & E in the ETDP SETA, the M &
E Unit will continue to conduct monthly and quarterly performance monitoring and
conduct field monitoring visits to verify the information gathered. Working closely
with the Research Chairs that have been established, the M & E Unit in collaboration
with the Research and Skills Planning Unit and the Programme Managers will
continue to develop and/or upgrade existing monitoring tools to ensure their fit-for-
purpose with current programmes and plans that will be monitored and evaluated.
Risks that have been identified per each of the six programmes should be monitored
simultaneously with the activities, outputs and outcomes by the programme
staff/managers and the M & E Unit and these should be reflected upon in the quarterly
reports that will be produced.
A results framework gives an indication of what will be monitored and evaluated at the
programme level and it includes indicators, baselines, targets and means of
verification that will be tracked. Special care should be taken though to ensure that
demand for objective verification does not lead to exclusive focus on quantitative
4
indicators at the expense of more qualitative indicators which may be harder to verify
but may better capture the essence of the changes that are taking place.
Evaluations will not be one-time events; instead they will be exercises involving
assessments of differing scope and depth carried out at several points in time in
response to evolving needs for evaluative knowledge and learning. A key requirement
for all evaluations will be that they should be linked to the identified outcomes as
opposed to only implementation or immediate outputs. It is also expected that
evaluations will not only answer questions generated by the monitoring process but
will importantly also draw heavily on data generated through monitoring which
include baseline data, information on the programmes’ implementation processes,
and measurement of progress towards the planned results through indicators. Several
criteria are put forward for consideration when decisions on what to evaluate are
made, and these include factors such as: use, purpose and timing of an evaluation,
resources invested, the likelihood of future initiatives in the same are or programme,
anticipated problems, need for lessons learnt, and alignment and harmonization to
national, regional/provincial and the ETDP SETA’a corporate priorities.
The data collected in monitoring and evaluation processes will need to provide the
information that is necessary to measure the indicators. The M & E team must identify
how to collect the information necessary to measure the performance indicators,
whether it will be through existing data collection systems or whether new systems will
be developed. Currently all structures of the ETDP SETA have some M & E functions
and therefore report on M & E.
Finally an M & E Plan is presented together with an institutionalization strategy as an
integral component of the M & E Framework. The institutionalization strategy
proposes that a gradual approach to phase in participatory approaches to M & E in the
ETDP SETA be adopted in order to prevent disruptions and instability that may result
when new methods and systems are abruptly introduced. The proposed participatory
methods will be introduced as part of the institutionalization of M & E process in the
capacity building training that is scheduled to start in 2017.
5
1. The Purpose of M & E
Monitoring is conventionally defined as a continuing function that aims primarily to
provide the management and main stakeholders of an ongoing intervention with early
indicators of progress, or lack thereof, in the achievement of results. Evaluation, on
the other hand, is a selective exercise that attempts to systematically and objectively
assess progress towards achievement of results in an intervention. It involves
assessments of differing scope and depth carried out at several points in time in
response to evolving needs for evaluative knowledge and learning during the effort to
achieve results. The overall purpose of monitoring and evaluation (M & E) is the
measurement and assessment of performance in order to more effectively manage the
outcomes and outputs known as development results. Performance is defined as
progress towards and achievement of results. Information from regular and planned
monitoring and evaluation activities is required for an effective and timely decision
making. Monitoring provides real-time information on ongoing programme or project
implementation required by management, whilst evaluation provides more in-depth
assessments. Whereas questions that are answered by evaluations are usually
generated by the monitoring process, evaluations on the other hand often draw heavily
on data generated through monitoring, including baseline data, information on the
programme or project implementation process, and measurement of progress towards
the planned results through indicators.
The ETDP SETA is a state agency charged with promoting and facilitating skills
development in the education, training and development sector in order to raise the
skills levels of employees and achieve a healthy balance between supply of and demand
for skills in the country’s labour market. In the context of its general mandate of
promoting skills development in the country, the ETDP SETA has developed a sector
skills plan (SSP) that is annually updated to guide implementation of its Five Year
Strategic Plan (2015/16 to 2019/20) which is the overarching programmatic
framework through which it aims to achieve its goals in the coming five years. Short to
medium term annual performance plans ( APPs) are simultaneously developed to
provide detailed annual targets and plans that will be implemented cumulatively to
build up to the goals that will be achieved in five years. To ensure that the intended
results in its interventions are achieved as planned and corrective actions are taken
timeously in the delivery of its programmes, the ETDP SETA has set up monitoring
and evaluation (M & E) systems and processes that are fully integrated in the planning
and implementation of its programmes. Alongside its Skills Planning and Research,
and the M & E Units, six Research Chairs (assisted by 11 Research Interns) have been
set up with selected higher education institutions (HEIs) to beef up its research
capacity to ensure that sound evidence-based policy decisions guide its endeavours.
This M & E Framework sets out the proposed minimum monitoring and evaluation
requirements to enable collection and provision of indicators of progress, or lack
thereof, and effective review of the ETDP SETA’s programmes. The framework will
guide the overall monitoring and evaluation functions of the ETDP SETA including its
6
current Five Year Strategic Plan 2015/16 to 2019/20 together with the short to
medium term subsidiary plans such as the current Annual Performance Plans
(2016/17), the Service Delivery / Implementation Plan, and the units’ operational
plans which incorporate individual employees’ work plans. The core purpose of the M
& E system1 proposed here, therefore, is to timeously provide information needed for
impact-oriented project management that not only ensures compliance with the
programmes’ strategies and approaches, but also improves responsiveness, efficiency
and effectiveness by providing constant feedback from programmes’ implementation.
2. Scope of M & E
This framework is primarily aimed at guiding the M & E activities that will be carried
out by the ETDP SETA in the coming five years. M & E activities will essentially be
focused on the performance of six programmes that have been formulated to enable
the ETDP SETA’s delivery on its mandate. Alongside monitoring and evaluating
activities, processes and systems carried out to achieve the goals and objectives of the
six programmes, the fit between policies, structures and processes will be
simultaneously assessed / reviewed periodically to ensure that the ETDP SETA’s goals
and objectives are achieved. On one hand activities, outputs and immediate outcomes
will be monitored and reported internally (i.e. by staff and managers within the
organization) to track implementation progress, whilst, on the other hand, periodic
and systematic reviews and evaluations will be commissioned2 to assess progress and
help improve performance by helping the organization to extract, from past and
ongoing activities, relevant information that can subsequently be used as the basis for
programmatic fine-tuning, reorientation and planning.
3. The ETDP SETA’s Approach to M & E
The approach of the ETDP SETA to the monitoring and evaluation functions is
informed by the need to establish a sustainable system that has the buy-in of all staff
in the organization where the system will be recognized as the critical yardstick or
instrument to measure the effectiveness of the organization’s service delivery as well
1 Monitoring as used throughout this document refers to ongoing activity that focuses on progress in terms of activities and it tracks mainly the use of inputs (activities), outputs, and also tracks (intermediate) outcomes. In contrast and complementing monitoring, evaluation refers to an assessment of a programme / project’s progress over a longer period of time and it takes place at specific moments during the lifespan of a project (see The PPD Handbook: A Tool kit for Business Environment Reformers:2005). 2 Although some internal evaluations are useful and recommended to strengthen the capacity of organizations, the international best practice of using both internal and externally conducted evaluations will be adopted since commissioning evaluations to external service providers carries an added advantage of minimizing subjective bias and leads to credibility which often results in better utilization.
7
as its units’ and individuals’ performance. The management of the ETDP SETA has
adopted the logic model that links inputs (what resources will be used to do the work),
to activities (what actions or processes will be done to produce results), to outputs
(what goods or services will be produced or delivered), to outcomes (what medium-
term results and /or change for specific beneficiaries will be achieved as a logical
consequence of delivering specific outputs), and finally to impacts (the long term
developmental results at societal level that achievement of specific outcomes will
contribute to) as the M & E framework that will guide implementation of its M & E
strategy. Table 1 below summarises the key questions that will be posed per each level
of the objectives hierarchy in this logic model:
Table 1: Information needs in the objective hierarchy that will be
sought by M & E
Level in objective
hierarchy
What to Monitor and Evaluate
Activities What needs to be done to produce the intended / planned outputs? Have
planned activities been completed on time and within budget? What unplanned
activities took place?
Outputs What direct tangible products or services has the programme delivered as a
result of activities? What are the things that different stakeholders must provide?
Key outcomes What changes have occurred as a result of the outputs and to what extent are
these likely to contribute towards the programme purpose and desired impact?
Where do we want to be in five years? What are the things that must be in place
first before we can achieve our goals and have an impact?
Purpose Has the programme and/or the ETDP SETA achieved the changes for which it
can realistically be held accountable?
Goal (Impact) What is the programme’s and ETDP SETA’s overall goal? To what extent has the
project contributed towards its longer-term goals? Why and why not? What
unanticipated positive or negative consequences did the project have? Why did
they arise?
(Adapted from: International Fund for Agricultural Development (2002): A Guide for Project M & E; www.ifad.org).
In order to ensure that a sustainable M & E system that has the buy-in of staff is
effectively utilized by the whole organization to track progress and take decisions that
will lead to timely corrective actions where necessary, a participatory approach to M &
E where both managers and staff keep track of their operations and continuously learn
from what they are doing, is proposed. The participatory M & E system that is
proposed here is undergirded by the belief that effective implementation of M & E
strategies is centred around effective institutionalization which comprises people and
their capacities, processes and systems (see The M & E Plan and Institutionalization
Strategy in Annexures). Hence this M & E Framework should be correctly viewed as a
work-in-progress since in the participatory M & E approach it is requisite to involve
8
people who are involved in collecting and using the data in the design and formulation
of the M & E system3.
The M & E system will provide regular reports on the progress of programmes in order
to support managers and implementers (including staff, technical service providers
and skills development facilitators) to ensure that interventions lead to desired results
and adequately address the needs of the targeted beneficiaries. In addition to the
conventional principles commonly used as parameters for M & E practice, specific
principles that will be used to guide implementation of the M & E strategy will include
the following:
Emphasis of the M & E system should be on benefits and rewards (carrots) rather than sanctions (sticks) since project managers and staff should see the value of M & E in order to appreciate and commit themselves to it.
Baselines – or alternatives to these – will be specially considered in setting up the M & E system as they make it possible to see change.
The M & E system should always ensure that it includes information which helps answer the five core evaluation questions: relevance, effectiveness, efficiency, impact and sustainability, and which corresponds to each of the levels of the objectives hierarchy.
Enough operational information is included to know if optimal use of resources is being made and operations are of good quality. To be able to manage for impact, an M & E system needs to track progress in relation to targets.
M & E should seek information that helps not only to check targets, but to also explain progress. Only by knowing why something is happening or why not, can we have a basis for deciding what corrective action is needed.
The M & E system should also seek out unintended positive and negative impacts in order to enable taking any corrective action that might be necessary.
The M & E system should try to only collect information that is actively used and avoid the common M & E trap of gathering too much data of limited quality and not analyzing it. Less data may lead to more useful information.
Information needs and indicators should be reviewed as the programmes evolve in the implementation of ETDP SETA’s mandate to include new information needs and delete those that are no longer relevant.
M & E integrity and trustworthiness should be underpinned by evidence-based information, methodological soundness and the expertise and professionalism of the evaluation teams.
3 This aspect (of involving both managers and staff involved in the operations in the design and formulation of the M & E system) will be incorporated in the training workshops of staff and managers that are scheduled to be carried out starting in September 2015 and will be facilitated by the Research Chair on Evaluations.
9
4. Levels of Monitoring and Evaluation
The ETDP SETA is focused on delivering on two of the fourteen key priorities of the
National Development Plan, and these are: 6.1: Quality Basic Education and 6.5: A
skilled and capable workforce to support an inclusive growth path. To respond to
these outcomes the ETDP SETA has set for itself the following strategic goals that
must be achieved by March 2020:
Strategic Outcome Oriented Goal 1: Research focused on improving sector
skills planning, employability and productivity for economic growth;
Strategic Outcome Oriented Goal 2: Highly skilled, transformed and
productive personnel for improved teaching and learning in schools, TVET
colleges and higher education institutions
Strategic Outcome Oriented Goal 3: People centred and service oriented
public workforce for improved service delivery within the post school
education and training
Strategic Outcome Oriented Goal 4: A seamless post school education and
training system that addresses access and progression into high level skills
and work placement for career and occupational development
Strategic Outcome Oriented Goal 5: Enhance capacity in non-schooling
constituencies such as non-governmental organisations (NGOs), library and
archival services, political parties and trade unions through skills
development
Strategic Outcome Oriented Goal 6: Strengthen the ETDP SETA corporate
governance, leadership and operational effectiveness and efficiency
In essence monitoring and evaluation proposed in this framework will take place at
two distinct but closely connected dimensions / levels: one level focuses on the
outputs, which are the specific products and services that emerge from processing
inputs through programme activities. The other dimension focuses on the outcomes
which are the changes that the ETDP SETA aims to achieve through delivery of its
programmes’ outputs. The main focus of monitoring (which is largely a function
carried out internally by managers and staff) will thus be on tracking progress in the
delivery of outputs and the immediate outcomes whilst evaluations (both internal and
external) will focus mainly on outcomes and impacts and also include process
evaluations to assess the extent to which programmes are operating in the way/s
intended.
The M & E Unit will continue to conduct simultaneously monthly and quarterly
performance monitoring for each programme. The M & E Unit report will be based on
unit reports and the reports from field monitoring visits which it conducts periodically
to verify the information gathered. Monitoring tools to track programme performance
will continue to be developed by the M & E Unit in collaboration with the Research
10
Chairs that have been established with the higher education institutions (HEIs).
Development of monitoring tools will naturally entail upgrading and updating of
existing tools to ensure that there is purpose fit with the current programmes and
plans that will be monitored and evaluated.
4.1 Annual Performance Plan
Although the participatory M & E of this framework promotes that everyone in the
organization keeps track of the operational and strategic aspects of their work (as
monitoring is a daily and spontaneous activity), the responsibility for accounting on
the planned activities and outputs as per the annual performance plan targets still lies
with the managers as monitoring programme performance is part of the management
function. The main thrust of M & E will be on the monthly, quarterly and annual
monitoring performance of the six programmes in terms of the planned outputs and
outcomes that are in the APP. In terms of programmes, programme managers
continue to be responsible for performance monitoring. Through the use of indicators,
baselines and targets they will monitor performance, keep track of the changes and
progress attained, gather and report information together with the staff through their
monthly and quarterly reports. Annual reports will thereafter assess whether the
outputs and outcomes attained in the year’s performance take the organization any
closer to achieving the five year targets outlined in the Five Year Strategic Plan.
4.2 The Five Year Strategic Plan
The monthly and quarterly performance monitoring outlined above for tracking the
Annual Performance Plan culminate in an annual assessment which measures the
year’s performance to determine if such performance takes the organization any closer
to attaining the programme goals intended to be achieved in five years. A mid-term
review will be held in year 3 of implementation to assess overall performance and take
stock of progress or lack thereof which should guide the organization to taking
informed decisions that may lead to corrective measures and adjustments where
necessary. It is therefore apparent that performance monitoring of both the Annual
Performance Plan and the Five Year Strategic Plan will take place concurrently during
implementation. In addition to these plans, the ETDP SETA Service Delivery /
Implementation Plan which has been developed will be monitored and reviewed as
well which should logically entail the evaluation of units’ operational plans and
individual employee plans as part of the organization’ Performance Management and
Development System (PMDS). Since the latter fall within the domain of programme
or project management, the ETDP SETA managers will report on these aspects as part
of their conventional management functions. The focus of the M & E Unit at this
(strategic) level will be to ascertain to what extent has the ETDP SETA’s Five Year
Strategy contributed to the realization of the National Development Plan outcomes.
This will entail examining the extent to which the various goods and services produced
by the SETA in that period have had an impact on the priority outcomes. It is
11
anticipated that at the end of the five year period of implementation major outcome
and impact evaluations will be conducted by the Research Chair on Evaluations not
only to determine the extent to which intended outcomes and impacts were achieved
but also to generate lessons to inform future planning.
4.3 Unintended outcomes or effects and risks
Since it is important to note that sometimes well intentioned actions may lead to
negative results, it will be necessary to devote time to thinking through various risks
and possible unintended effects or outcomes. Several risks and risk mitigation
strategies have been identified per each of the six programmes that are currently being
pursued by the ETDP SETA and these are encapsulated in the Five Year Strategic Plan.
These risks will be monitored simultaneously with the activities, outputs and outcomes
by the staff, programme managers and the M & E Unit and will be reflected upon
especially in the quarterly reports that will be produced. Similarly, implementation of
programmes may lead to unintended results or consequences. In reality these are
another form of risk which will happen and may lead to undesirable results. All
members of the organization are urged to note, discuss and document any possible
unintended results or consequences.
5. Key Indicators and Priority Targets
Indicators are signposts of change along the path to development. They describe the way to
track intended results and are critical for monitoring and evaluation (UNDP Handbook on
Planning, Monitoring and Evaluating for Development Results; p. 61)
Performance indicators are units of information that, when measured over time, will
document change. When supported with sound data collection, analysis and reporting,
indicators enable managers to track progress, demonstrate results, and take corrective
action to improve service delivery. Good quality results, i.e. well formulated impacts,
outcomes, outputs, activities and indicators of progress, are crucial for proper
monitoring and evaluation. Monitoring and evaluation will be challenging, making it
difficult for staff and managers to know how well plans are progressing and when to
take corrective actions if results are unclear and indicators are absent or poorly
formulated. Similarly without baseline data it is very difficult to measure change over
time and set targets or to monitor and evaluate. Hence indicators and baseline data
are considered to lie at the heart of the monitoring and evaluation system. It is in this
context that the importance of both indicators and targets in the monitoring and
evaluation encapsulated in this framework should be understood.
Both the APP and the Five Year Strategic Plan have identified key indicators that will
be monitored to track performance and achievement of progress or lack thereof in the
programmes’ implementation. To track performance in these measures, performance
measurements, and performance standards a key question that will be posed is: How
12
will we know if we are on track to achieve what we have planned? With regard to the
accompanying means of verification key questions that will be posed are: What precise
information do we need to measure our performance? How will we obtain this
information?
Table 2 (in Annexures below) presents a results framework that gives an indication at
the programme level of what will be monitored and evaluated. It includes indicators,
baselines, targets and means of verification that will be tracked. Both output / process
indicators (that measure the degree to which activities are being implemented) and
outcome / impact indicators (that measure changes resulting from the interventions)
will be closely monitored in monitoring and evaluation. M & E should take special care
that the demand for objective verification does not lead to exclusive focus on
quantitative indicators at the expense of more qualitative indicators that are harder to
verify but may better capture the essence of the changes taking place that are sought.
6. Evaluations
Evaluations of the ETDP SETA’s interventions will be carried out to assess,
systematically and objectively, progress towards attainment of the identified outcomes
that the SETA is seeking to achieve. These will not be one-time events, instead they
will be exercises involving assessments of differing scope and depth carried out at
several points in time (during and after implementation of the five year strategy) in
response to evolving needs for evaluative knowledge and learning in the efforts to
achieve the identified outcomes. A key requirement for all evaluations, including
process evaluations that assess relevance, performance and other criteria, is that all
evaluations need to be linked to the identified outcomes as opposed to only
implementation or immediate outputs. As briefly alluded to earlier in the document,
it is expected that evaluations will not only answer questions generated by the
monitoring process but will importantly also draw heavily on data generated through
monitoring, including baseline data, information on the programme/s’
implementation process, and measurement of progress towards the planned results
through indicators. The ETDP SETA has conventionally focused on commissioning
process, outcome and impact evaluations to ascertain that its interventions are leading
to the desired results and outcomes. It is envisaged that the M & E Unit within ETDP
SETA will develop and/or update an evaluation plan (if there is an existing one) which
will become an integral component of the M & E framework and will be compiled from
an aggregate of individual programmes’ evaluation plans which will have been
developed by the programmes’ managers and implementers.
In this framework we propose that, amongst the criteria used to commission
evaluations, the following be considered as well in deciding what to evaluate:
Uses, purpose and timing of an evaluation – Evaluations should be proposed
only when the commissioning units in the organization are clear at the onset
13
about why the evaluation should be conducted (purpose), what the information
needs are (demand for information), who will use the information, and how the
information will be used. The intended use determines the timing of an
evaluation, its methodological framework, and level and nature of stakeholder
participation, whilst the timing of an evaluation should be directly linked to its
purpose and use. To ensure its relevance and effective use of evaluation
findings, the evaluation should be made available in a timely manner so that
decisions can be made informed by evaluative evidence.
Resources invested – An area or programme in which the ETDP SETA has
invested significant resources may be subject to an evaluation as there are
usually greater accountability requirements concerning such a programme or
area.
The likelihood of future initiatives in the same area or programme –
Recommendations to guide future work often emanate from evaluations. As it
is an evaluation which enables programme units to take stock of whether
outputs have contributed to desired outcomes, it will therefore be necessary
when selecting an intervention to be evaluated to look for one in an area or
programme that is likely to continue to be supported by the ETDP SETA.
Anticipated problems – Evaluations can help prevent problems and provide an
independent outlook on existing problems. When selecting an outcome for
evaluation, it is therefore prudent to look for those with problems or where
complications are likely to arise as a result of an outcome being in a sensitive
area possible with other stakeholders or partners.
Need for lessons learnt – A consideration of what kinds of lessons are needed
to guide further implementation is critical when evaluations are to be
commissioned.
Alignment and harmonization to national, regional/provincial and the ETDP
SETA’s corporate priorities – Planned evaluations should, on one hand, be
aligned with national and regional/ provincial development priorities and, on
the other hand, should be aligned with the ETDP SETA’s corporate priorities
and be harmonized with national evaluations. Opportunities for joint
evaluations with other government departments and agencies should be
actively pursued to minimise costs and make effective use of existing capacities.
In determining the timing of evaluations, the ETDP SETA should consider
various decision-making points that exist (in the government departments and
agencies whose work affect the SETA) such as budget decision making,
development framework or strategy setting, and existing review processes for
development programmes and projects. For instance, if the national
government is undertaking an evaluation of the NDP outcomes that the ETDP
SETA is contributing to, ETDP-managed evaluations should enhance
complementarities and minimize duplicated efforts.
14
7. Data collection methods and sources
The data collected will need to provide the information that is necessary to measure
indicators in both monitoring and evaluation processes. The types of data collected
will, to a large extent, be determined by the feasibility of collecting such data given the
constraints of time and resources. The primary data that will be collected from the
intended ETDP SETA beneficiaries, service providers, employers and other
stakeholders through direct observations and other methods such as interviews and
focus group discussions, will be complemented (and not replaced) by secondary data
where literature, reports and analyses will be extensively studied to deepen and
strengthen understanding of particular issues and phenomena being monitored and
evaluated at any given time. During the data collection process, the M&E Unit should
assess the strategic information needs and the existing information systems and
capabilities to address those needs to determine what is feasible and what is not. The
M & E team must identify how to collect the information necessary to measure the
performance indicators, whether it will be through existing data collection systems or
whether new systems will be developed. In addition, it must determine how
information will be recorded and reported, including identifying any tools or forms
that will be needed.
8. Reporting including M & E roles and responsibilities
As a public entity the ETDP SETA is subject to the government departments’ and
public entities’ reporting procedures which, among others, include monthly, quarterly
and annual reporting. In these processes the importance of performance information
is increasingly being recognized as part of public sector monitoring and evaluation,
playing a significant role in the allocation of budgets and in the monitoring of service
delivery and value for money. Monthly reports that are presented by line managers
and unit heads to the Broad Management Team (BMT) meeting that is chaired by the
CEO capture the information aimed at addressing operational performance, while the
ETDP SETA’s quarterly performance reports provide information or progress report
on the implementation of the SETA’s Annual Performance Plan. These quarterly
reports embody procedures that have been set according to the National Treasury
Regulations that are aimed at facilitating performance monitoring, evaluation and the
taking of decision on corrective action. The ETDP SETA publishes its Annual Report
which are also tabled in Parliament and are made available to the ETDP SETA
stakeholders, constituencies and the general public. In terms of M & E the information
that is reported annually forms the basis for the annual performance review and
planning for the following year and the MTEF period. This annual review and planning
culminates in the Annual Performance Plan. In the meantime the ETDP SETA is also
required to report on financial and programme performance to external statutory
bodies such as the Auditor General (on quarterly basis), the National Treasury (on
15
quarterly basis), Portfolio Committee on Higher Education & Training (on quarterly
basis and ad-hoc reporting upon request), and the Department of Higher Education &
Training (quarterly and annual performance reporting).
8.1 The ETDP SETA’s M & E roles and responsibilities
Currently all structures within the ETDP SETA have some M & E functions and
responsibilities. The Accounting Authority monitors and evaluates the effectiveness
and efficiency with which the SETA utilizes public resources in its activities. It also
utilizes M & E findings to provide strategic and governance oversight and direction for
the SETA as a whole to ensure that: the desired outcomes and impact are achieved. In
relation specifically to M & E the Chief Executive Officer (CEO) has the responsibility
to ensure that the ETDP SETA maintains appropriate and valid performance
information systems; management and employees comply with M & E prescripts;
prompt action is taken in response to M & E findings; that unit/ provincial resources
are allocated to conducting M & E; M & E information is communicated to the relevant
audience; and that appropriate capacity exists for monitoring and evaluation and the
M & E system is operating effectively. Senior Managers are similarly charged with
ensuring, among others, that: an M & E Plan is developed for their areas of
responsibility and M & E functions are included in the performance agreements of
their staff. Line Managers/ Provincial Managers are similarly required to ensure that
information is available within the stipulated time and format to the M & E Manager
with the means of verification, whilst members of staff are expected to capture the
required data and ensure that the means of verification submitted are relevant to the
indicators.
The M & E Unit, on the other hand, is regarded as the custodian of the M & E
Framework and works to ensure its effectiveness through providing M & E services to
the ETDP SETA. The Unit acts as an M & E resource to the organization by: providing
advice and insight on good practices and support for effective use of M & E as a
performance management tool; coordinating, promoting and disseminating M & E
findings and information to support service delivery improvement and report on
follow-up action; assessing the M & E capacity within the ETDP SETA; and
coordinating training to build the requisite skills internally and to create a culture of
self-assessment among all staff.
16
ANNEXURES
Annexure One: The Results Framework
Table 2: The results framework of the ETDP SETA’s strategic outcome-oriented goals
2015/16 – 2019/20
Strategic
outcome-
oriented
goal
Planned
results /
outputs
Indicators Baseline Target Means of
Verification
Programme
1: Research
focused on
improving
sector skills
planning,
employability
and
productivity
A 5-year sector
skills plan (SSP)
An approved 5
year SSP
A 5-year SSP
has been
developed by
the ETDP
SETA and is
updated
annually
A complete
and
approved
SSP
submitted to
DHET and
Parliament
on due dates
Workplace
Skills Plans;
DHET’s
datasets (e.g.
EMIS, HEMIS
and TVETMIS)
and
commissioned
empirical
research
Programme
2: Highly
skilled,
transformed
and productive
personnel for
improved
teaching and
learning in
schools, TVET
colleges and
higher
education
institutions
25 000 primary
and secondary
school teachers
trained in subject
content,
methodology and
ICT skills in order
to improve their
competency
levels;
5000 teachers in
full service and
LSEN schools
enrolled for
braille and sign
language
facilitation
Number of
teachers
trained in
subject
content,
methodology
and ICT skills
Number of
teachers in full
service and
LSEN schools
enrolled for
braille and
sign language
facilitation
5680
teachers have
been trained
on subject
content
knowledge;
461 on ICT
learnerships.
394 teachers
in full service
schools
trained on
special
needs.
25000
primary and
secondary
school
teachers in
public
schools
5000
teachers in
full service
and LSEN
schools
Letters of
awards and
attendance
registers
Letters of
awards/
learners’
documents
including
agreements
and MIS
reports
17
5000 learners
assisted with
funding to study
towards Grade R
or Foundation-
phase teacher
qualification
Number of
learners
assisted with
funding to
study towards
Grade R or
Foundation-
phase teacher
qualification
278 have
been trained
for Grade R
or
Foundation-
phase
teacher
qualification
5000 or
more
learners
assisted with
funding to
study
towards
Grade R or
Foundation-
phase
teacher
qualification
Letters of
awards/
learners
documents
including
agreements
and MIS
reports
2000 TVET
college lecturers
and managers
have completed
relevant
programmes to
enhance their
capacity
Number of
TVET college
lecturers and
managers who
have enrolled
and completed
relevant
programmes
to enhance
their capacity
720 lecturers
have been
trained in
PIVOTAL
programmes
and 475 have
been exposed
to industry
workplaces
2000 or
more of
identified
TVET
college
lecturers and
managers
Letters of
awards/
learner s’
documents
including
agreements
and MIS
reports
500 HEI
lecturers trained
to gain
appropriate
competencies in
order to improve
student
performance
25 HEIs receive
financial support
for staff
development
Number of
academics in
research and
HE
institutions
enrolled at
Masters or
PhD levels
Number of
HEIs receiving
financial
support for
staff
development
15 in 2013/14
500 or more
25 HEIs
Letters of
awards/
learner s’
documents
including
agreements
and MIS
reports
HEMIS and
individual HEI
sources on
skills
development
Programme
3: People
centred and
service
oriented public
4000
unemployed
youth trained and
placed in public
Number of
unemployed
youth trained
and placed as
administrators
1000
estimated in
2014/15
4000 Signed
internship
agreements
and letters of
awards
18
workforce for
improved
service delivery
within the
post-school
education and
training
schools as
administrators
2000
unemployed
youth trained and
placed as library
assistants
Number of
unemployed
youth trained
and placed as
librarians
--------- 2000 Letters of
awards
confirming
placements
5000 national,
provincial and
district officials
are trained to
increase the
administrative
capacity of
critical office-
based personnel
Number of
national,
provincial and
district
officials
enrolled in
relevant
learning
programmes
---------- 5000 Letters of
awards/
learner s’
documents
including
agreements
and MIS
reports
Programme
4: A seamless
post-school
education and
training
system that
addresses
access and
progression
into high level
skills and work
placement
8000 youths
placed in
workplace
experience
programmes
(graduates and
undergraduates)*
Number of
youths placed
in workplace
experience
programmes
718
estimated for
2014/15
8000
Signed
workplace
agreements
and letters of
awards
Programme
5: Enhanced
capacity in
non-schooling
constituencies
such as non-
governmental
organizations
(NGOs);
library and
archival
services;
political
parties and
trade unions
through skills
development.
3 Trade union
federations are
supported *
Number of
trade union
federations
supported
-------
3 trade
union
federations
Letters of
awards and
signed
agreements
19
Programme
6: Strengthen
the ETDP
SETA
corporate
governance,
leadership and
operational
effectiveness
and efficiency
Train ETDP
SETA employees
through skills
programme*
Number of
ETDP SETA
employees
trained
through skills
programmes
-------
All ETDP
SETA
employees
are provided
with skills to
fill the gaps
identified in
their PMDS
HR Unit,
learner
agreements
= only one planned result / output is shown amongst several outputs planned. Comprehensive
indicators per each of the updated programmes are in the proposed M & E Matrix in Annexure
Two.
20
ANNEXURE TWO
A tentative4 M & E Plan and Institutionalization Strategy
2.1. The M & E Plan
2.1.1 Introduction to the M & E Plan
The M & E Plan sets out the proposed minimum monitoring and evaluation
requirements to enable collection and provision of indicators of progress, or
lack thereof, and effective review of the ETDP SETA’s programmes. The Plan
outlines the overall monitoring and evaluation functions of the ETDP SETA
including its current Five Year Strategic Plan 2015/16 to 2019/20 together with
the short to medium term subsidiary plans such as the current Annual
Performance Plans (2015/16), the Service Delivery / Implementation Plan, and
the units’ operational plans which incorporate individual employees’ work
plans.
The standards to which the M&E plan should be held include the following:
M & E must be useful and serve the practical and strategic information
needs of the intended users for decision making purposes (from
assessing program performance to allocating resources, etc.) Intended
users may include those at the highest central levels making decisions
about national programs to those making decisions to improve programs
at the district level.
The M&E plan should be realistic and practical. To the extent possible,
it should make the best use of existing data collection systems. If new
data collections systems or studies are involved, resources (cost and
technical capacity) must carefully be considered. Data collection, special
studies included in the M&E plan must abide by regulations and
protocols that consider the rights of those involved in and affected by
M&E activities.
4 The M & E Plan is designated tentative at this stage because it is still going to be work-shopped with staff and managers during the scheduled capacity building training workshops and the subsequent M & E institutionalization within the ETDP SETA to ensure that the information and results sought are guided by the Plan and are consistent with their expectations. The M & E Unit will need to consult especially with programmes’ managers to ensure that the indicators in the M & E Plan reflect actual data being collected and validate whether the information is the most useful for them in order to make decisions for the improvement of their programmes.
21
Finally, the M&E plan should provide technically accurate and useful
information for decision making and program improvement.
2.1.2 Capacity and Resources for M&E
The ETDP SETA currently has an M&E unit with staff trained in M&E methods.
The Unit is responsible for leading the development and coordination of the
M&E plan. This existing capacity for M & E in the ETDP SETA is complemented
by the Research and Skills Planning Unit which coordinates research and
planning with regard to skills. The ETDP SETA has also beefed up its internal
capacity through establishment of several research chairs one of whom – the
Research Chair on Evaluations – directly contributes to the SETA’s monitoring
and evaluation needs through several evaluation studies that it has conducted
and is currently coordinating M & E institutionalization within the SETA itself.
Above all, programme managers and implementers are charged with M & E
related work in their operations as they are responsible for developing and
implementation of monitoring in their line duties. Programme managers are
expected to analyze programmes’ performance and map out trends and
patterns of efficiency in the organization, and also use evaluative knowledge for
learning, accountability and service delivery improvements.
2.1.3 Problem statement
South Africa is experiencing skills shortages in critical sectors of the economy
and the education and training sector is no exception as it has been plagued
with inefficiencies in the performance of the system. The twelve constituencies
that are serviced by the ETDP SETA in the education and training sector are all
experiencing different skills development needs.
2.1.4 Translating Programme Goals and Objectives into M&E
Frameworks
In responding to the skills shortage in the education and training sector, the
ETDP SETA has set itself the following six strategic goals which can be
objectively measured:
Strategic Outcome Oriented Goal 1: Research focused on improving sector
skills planning, employability and productivity for economic growth;
22
Strategic Outcome Oriented Goal 2: Highly skilled, transformed and
productive personnel for improved teaching and learning in schools, TVET
colleges and higher education institutions
Strategic Outcome Oriented Goal 3: People centred and service oriented
public workforce for improved service delivery within the post school
education and training
Strategic Outcome Oriented Goal 4: A seamless post school education and
training system that addresses access and progression into high level skills
and work placement for career and occupational development
Strategic Outcome Oriented Goal 5: Enhance capacity in non-schooling
constituencies such as non-governmental organisations (NGOs), library and
archival services, political parties and trade unions through skills
development
Strategic Outcome Oriented Goal 6: Strengthen the ETDP SETA corporate
governance, leadership and operational effectiveness and efficiency
For each of these goals several corresponding objectives have been developed,
the pursuit of which will be measured through indicators (see Annexure 1: The
Results Framework above). These indicators will importantly provide answers
to two key questions: “What do we want to know at the end of the programme?”
and “What do we expect to change by the end of the programme?” (See attached
M & E Plan that incorporates the M & E Matrix with comprehensive indicators
that will be used to measure attainment of set objectives and expected results).
23
2.1.5 The Monitoring Plan
The following template captures the essence of what will be captured, how,
when and by whom in the monitoring plan
INDICATOR BASELINE
What is the
current value?
TARGET
What is the
target value?
DATA
SOURCE
How will it
be
measured?
FREQUENCY
How often will it
be measured?
RESPONSIBLE
Who will measure it?
REPORTING
Where will it be
reported?
Goal:
Outcomes:
Outputs:
Activities
Inputs
2.1.6 Information dissemination and use
This section of the Plan will detail how the information gathered will be stored,
disseminated and used. This will help ensure that findings from M&E efforts
are not wasted because they are not shared.
The various users of this information should be clearly defined, and the reports
should be written with specific audiences in mind.
Dissemination channels can include written reports, press releases and stories
in the mass media, and speaking events.
2.1.7 Implementation and mechanism for update
Here a mechanism for reviewing and updating the M&E plan will be included.
This is because changes in the programmes can and will affect the original plans
for both monitoring and evaluation.
2.2. The M & E Institutionalization Strategy
As briefly alluded to above, it is the belief of the authors of this document that effective
implementation of M & E strategies should be centred around effective
24
institutionalization which comprises people and their capacities, processes and
systems. In line with this we recognize the need for all staff of the ETDP SETA to have
a sense of ownership of the M & E system. Whilst we advocate for a participatory
approach to ensure that everybody in the organization buys in in the M & E, we are
aware of the fact that a haphazard introduction of such a system without preparing the
people affected will likely result in disruptions and cause instability which will be very
costly to the ETDP SETA and derail the organization from achieving its targets and
objectives.
In line with this reality, we therefore propose a gradual approach where participatory
M & E will be phased in hand-in-hand with the scheduled capacity building/ training
on M & E that will commence shortly with operational staff and managers. It is
envisaged that the M & E Plan (a component of the M & E Framework proposed above)
will be reviewed and validated with managers and staff in the training workshops. This
is particularly important in view of the fact that the views of people involved in actual
data collection for M & E are critical in the design and formulation of the data
collection instruments. Aspects such as frequency, feasibility, and availability as well
as responsibility for data collection methods and tools are particularly pertinent to
validate through a participatory approach.
Owing to the importance and urgency of institutionalizing the participatory approach
to M & E in the ETDP SETA, we therefore propose that the training sessions (four in
all) be aligned to this process which, ideally, should be done during this early years of
implementation of the Five Year Strategic Plan. This will importantly enable the
validation process which is critical in the efforts to institutionalize M & E and will
accompany several processes to realize this goal. Such validation will also inform
planning for the coming years of implementation.
References
ETDP SETA (2011) Monitoring and Evaluation Framework 2011/12 to 2015/16;
Higher Education and Training, Pretoria
___________ (2015) Annual Performance Plan 2015/16; Higher Education and
Training, Pretoria
____________ (2015) Five Year Strategic Plan 2015/16 to 2019/20; Higher
Education and Training, Pretoria
_____________ (2015) The ETDP SETA Performance Monitoring, Evaluation and
Reporting Policy; Higher Education and Training, Pretoria
UNDP (2009) Performance Monitoring and Evaluation Handbook; New York
25
UNISDR (2009) Monitoring and Evaluation Framework; Bangkok