Upload
vodat
View
218
Download
3
Embed Size (px)
Citation preview
Piloting Monitoring and Evaluation
Jean Luc MARCELLIN
IVETA 03 December 2013
Outline
• What is piloting
• What is monitoring
• What is evaluation
• How do they relate to each other
• M&E before the implementation of a new training
program (piloting phase)
• M&E during the implementation of a new training
program (piloting phase) or during the normal
course of a regular training program
• Observing and tracking carefully the
operation of a system or a process, in
real-time o Driver of a car on the road
o Air plane pilot
o Training manager
o Director of studies
o Monitoring is the process of systematically collecting data
(the key elements of program performance, usually inputs
and outputs) on a regular basis
o To monitor is to check on how the activities are progressing. It is systematic and purposeful observation
What is monitoring?
What is Piloting?
• This is the operation in which the pilot
decides and acts in real time using the
information provided by monitoring
indicators (dashboard / scoreboards).
What is Evaluation? • Assessment of the change in targeted
results (objective judgment) and make decisions for the future, during or after completing the project
• Evaluation should provide a clear picture of the extent to which the intended objectives of the activities of the piloting process have been realized. “Have we achieved our objective?” is the question to be asked
How do they relate to each other?
Evaluation (judgement, appraisal)
are closely
related
Integration
Interdependence
Piloting (to make
decision and act
accordingly)
Monitoring (gathering data,
information system)
Interdependence • Monitoring a training program = to observe (monitoring)
specific indicators in real time.
• Piloting = Based on the observations made through
monitoring, decide in real time what should be done to
achieve the goals, objectives and meet the defined standards
• Given the expected standards and objectives for the program: establish a process for collecting and processing information to monitor states, developments and gaps (real time).
• The information gathered is translated into indicators and feed scoreboards for monitoring the program.
• Depending on the evolution of these indicators, analyzes are made and decisions taken in real time (piloting).
M&E at different levels • In TVET, monitoring, piloting and
evaluation are related to training
programs, but also to the entire TVET
system at four different levels: 1. for each program (criteria and indicators),
2. for each program at national level (ATVET),
3. local (the VTPC),
4. for all of the VT system.
The first three levels feed the fourth.
Inter relations at the different levels
Training program X Local monitoring
and evaluation Indicators 1
Training program X National monitoring and evaluation
Compilation of Indicators 1 + national specifics indicators 2
Monitoring and evaluation ot the VET Systeme at
the local level (VTPC)
Indicators 1 +
Specifics indicators 3
Monitoring and evaluation of the TVET system at the natioonal level
Compilation of Indicators 1+2+3
+
Specific national Indicators 4
Piloting Monitoring and Evaluation at the local
level
Part 2
M&E at the local level • Within the VTPC, monitoring and
evaluation is done: o for each specific training program by program
team members and training manager in
cooperation with high level management
o At the intuitional level (specific indicators for
the VTPCs at the local level)
M&E at the local level What do we want to know?
• Does the VTPC have all the resources it needs to
deliver the training in term of human resources,
equipment, classrooms, workshops, learning and
teaching resources, and also alternative resources
(Public Private Partnership) in compliance with the
curriculum requirements?
• Is the training delivered in conformity with the training
program standards (chronology, numbers of
competencies, number of modules, number of hours per
modules theory /practice, competency assessment,
teaching and learning strategies) and in conformity with
Competency Based Training pedagogical principles and
practices?
M&E at the local level What do we want to know?
• What is the performance of training in terms of
student’s success rate, drop outs?
• What is the satisfaction of the stakeholders
mainly the students, the trainers, the employers
and the students parents?
Always keep in mind the qualitative aspects in terms of the object of the training, the motivation of the trainers,
motivation and learning experience of the students, aspect of time and resources management, the application of
appropriate learning and teaching strategies, governance, transparency, and the quality of the competency assessment
in cooperation with the industry.
We have to distinguish 2 types of situations:
• The monitoring and evaluation for the implementation
of a new training program (the first time this training
program is delivered)
• The monitoring and evaluation of a regular ongoing
training program
M&E at the local level (VTPC) • Concerning the training programs, what actions
should we follow, what must be monitored in order to
react in real time (piloting)?
• What are the indicators and what scoreboard do we
need?
• We need also to clearly define
• who should perform the monitoring and
evaluation,
• what are the data and information needed?
• Finally, for each actor, what is his own real time
of reference (the week, the month, the semester
the full cycle, or many cycle)?
• This will lead to the determination of the frequency of
the observation of the indicators for each actor,
involved.
M&E at the local level (VTPC)
The monitoring and evaluation for the implementation of a
new training program (the first time this training program is
delivered)
• What do we need to monitor during the preparation
phase of the delivery of the new training program?
• What do we need to monitor during the delivery of the
new training program?
For detailed answers to these questions we have designed
the implementation guide tool kit (11 actions) and the
monitoring and evaluation hand book.
M&E at the local level (VTPC)
The implementation of a new training program requires
that the following activities to be done
Action plan for PPP development
Diagnostic of the training center,
feasibility study, decision to go forward
or no, setup of the implementation
team. (real need?, and readiness to
face challenges by leadership team)
Training need analysis for teachers,
management and staff
Elaboration of an detailed implementation action plan
Resource allocation:: instructors, classrooms, labs and workshops,
training equipment, consumables, instructional material, budget,
administrative staff support, external support (PPP)
Carry out the training and coaching
Communication and marketing plan
Administrative organization
of the training, collection of
data for scoreboards,
Pedagogical organization of the training
(registration process, planning tools, assessments,
instructional support, students groups management, pedagogical coordination…).
M&E at the local level (VTPC)
According to the 4 axes of the M&E guidelines and hand book
(resources, conformity, performance, satisfaction), below are
examples of indicators
• The respect of the activities scheduled in the detailed
implementation action plan (Who ? Training manager,
When? weekly)
• The deliverables mentioned in the action plan (action 1 in
the piloting tool kit) (TM, W)
• The quality and conformity of the deliverable
• The training & coaching of the trainers, management and
staff according to the recommendation of the TNA report.
• The allocation of all the resources in compliance with the
training program requirements
What should be monitored during the preparation phase, before the
delivery of the new training program?
M&E at the local level (VTPC)
• Rate of implementation of the action plan based on the
projected schedule (time and resources management).
• Checklist of deliverables with mention about their
quality and conformity
• Rate of realization of the training plan derived from the
TNA
• Number of the beneficiaries compared to projected
number
• % of satisfaction related to the evaluation of the
training received
What should be monitored during the delivery of the training program
M&E at the local level (VTPC)
By who? Director (D), Training Manager (TM), Teachers (T) , Stakeholder (SH), ATVET, Student (S) etc.
Frequency? Weekly (W), Daily (D), Monthly (M) Life cycle of the training program (LC) etc.
• In terms of resources
• % of training equipment, learning material,
instructional resources , etc. in comparison with the
requirements of the training program
• Real costs compared to the initial budget
• % of training spaces provided (school premises,
classroom, workshops, laboratories, on the job
training facilities)
• Number of trainers allocated compared to the number
of trainers required
• Number of students enrolled compared to the
maximum number per group
• Availability of Internet access (bandwidth and speed)
What should be monitored during the delivery of the training program?
M&E at the local level (VTPC)
• Access to work places (student palcement PPP)
• Efficiency of the communication and marketing plan
• Number of visitors enquiring about the new training
program / # of phones calls
• % of increase of PPP conventions with private
sectors
• % evolution of the grants, gifts in kinds, equipment
loans etc.
• % of increase of teachers participation and training
in the workplace
• % of increase of workers from private sector
coming to give training in the VTPC
What should be monitored during the delivery of the training program?
M&E at the local level (VTPC)
• Use the tools provided in the piloting tool kit
At the beginning of the project
• Establish a gant diagram based on the implementation
chronogram provided (implementation tool kit)
• Establish a list of deliverable with the name of the
person responsible, the due date, the conformity and
the quality specifications requirements check list for
each deliverable
• For deliverables examples refer to the implementation
tool kit and the monitoring and evaluation hand book
Recommended best practices (1)
M&E at the local level (VTPC)
• Plan weekly meetings to gather all data required to
feed the indicators
• Plan monthly meeting to elaborate the scoreboards
based on the indicators selected.
• Plan monthly management meetings with the director
to take decision and act based on the scoreboards
• Update the work plan and calculate the ratio of
realization (respect of dates of deliverables should
comply to quality and conformity requirements)
• Automatize all these tasks by using an information
management system
Recommended best practices (2)
M&E at the local level (VTPC)
Questions ?
Thank you
Very much