Upload
samandar-mahmodi
View
221
Download
0
Embed Size (px)
Citation preview
8/3/2019 5 Day M&E Presentation Compiled
1/87
1
Address: Shahri Naw, between Hanzala Masjid and Blossom Hospital, Street #01, House # 50, Kabul, AFG
A Leader and Pace-setter in Capacity Development by Offering High QualityTrainings & Consultancy Services
8/3/2019 5 Day M&E Presentation Compiled
2/87
2
Monitoring & Evaluationof Projects
Part 09
by: Samandar MahmodiM&E, SPSS and Survey Design Specialist
8/3/2019 5 Day M&E Presentation Compiled
3/87
3
What is Monitoring & EvaluationThe collection of data and information inprograms and projects, in combination
with its analysis and use.Monitoring and Evaluation arecomplimentary, but distinct functions
8/3/2019 5 Day M&E Presentation Compiled
4/87
4
What is Monitoring?
Monitoring is the regular and continues collectionof data and information on specific indicators to:
Demonstrate to management and key
stakeholders the extent of progress, and showprogress in relation to what was planned
Assist in timely decision-making processes
Ensure accountability and Transparency
Provide the basis for evaluation and learning
identify implementation problems oropportunities
8/3/2019 5 Day M&E Presentation Compiled
5/87
5
What is Evaluation?
Assesses relevance, efficiency, effectiveness,impact and sustainability in relation to projectobjectives.
Supports strategic decision making processThe systematic and objective assessment of anon-going or completed project, program or policy,its design, implementation and results. The aim is
to determine the relevance and fulfillment ofobjectives, efficiency, effectiveness, impact andsustainability.
8/3/2019 5 Day M&E Presentation Compiled
6/87
6
Evaluation ContdEvaluation relies on data generatedthrough monitoring activities as well asinformation obtained from other sources(e.g., studies, research, in-depthinterviews, focus group discussions,surveys etc.). Evaluations are often (but
not always) conducted with theassistance of external evaluators. And itcould be Formativeor Summative
8/3/2019 5 Day M&E Presentation Compiled
7/87
8/3/2019 5 Day M&E Presentation Compiled
8/87
8
Why is M&E Important?Monitoring and evaluation helps program implementers to:
Make informed decisions regarding program operationsand service delivery based on objective evidence
Ensure the most effective and efficient use of resources
Objectively assess the extent to which the program ishaving or has had the desired impact, in what areas it iseffective, and where corrections need to be considered
Meet organizational reporting and other requirements, and
convince donors that their investments have beenworthwhile or that alternative approaches should beconsidered
8/3/2019 5 Day M&E Presentation Compiled
9/87
9
When do we Monitor and Evaluate?
We monitor to know whether, are we on the right path toachieve what we have planned or not
To find problems and weaknesses, solutions for problemsand improvements for weaknesses.
To find opportunities and strengths
M&E is an integral part of every project, from ()
M&E should be planned during the (.) of a project
M&E should be executed by all individuals and institutions
which have an interest in the projectMonitoring should normally be planned according tooperation/implementation plans
8/3/2019 5 Day M&E Presentation Compiled
10/87
10
Project Implementation and M&E
Look! This is a project, a projectinvolves several phases from startto finish.
1. You see a situation
2. Plan for a project to improve thesituation
3. Plan for improvements, selecttargets
4. Carry out the project
5. Produce outputs, outcomes andfinally impact
6. All these stages should bemonitored and evaluated whetherthey are as intended or not.
8/3/2019 5 Day M&E Presentation Compiled
11/87
11
Illustration of M&E in a Project
M&E will not disappear problem, or to curethem, and is not a solution, but they arevaluable tools. And can:
1. Help you identify problems and theircauses
2. Suggest possible solutions to problems
3. Push you to reflect on where you aregoing and how you are getting there
4. Provide you with information and insight
5. Encourage you to act on the information
and insight
6. Increase the likelihood that you will makea positive development difference
8/3/2019 5 Day M&E Presentation Compiled
12/87
12
Discussions
What did you learn so far?
Questions and Answers.
8/3/2019 5 Day M&E Presentation Compiled
13/87
13
Monitoring Vs. Evaluation
Monitoring Evaluation
What Inform.. Directly available info about implementation More detailed information; harder to get
Purpose Determines if activity proceeding asintended
To see whether objectives are being reached andLearning lessons for future
Timing Regularly during implementation Periodically and after completion
Sources Project Records Monitoring data, plus external information
Use of Results To improve quality of implementation, andadjust planning
To judge the impact, adjust objectives, decide aboutthe future of the project
Done by Project Staff Outsiders
Done for Management Funders and Planners
8/3/2019 5 Day M&E Presentation Compiled
14/87
14
Group Discussion
Distinguish Monitoring and Evaluation.
Your Practical Monitoring and Evaluationexperience in differentiating M&E.
Distinguish the followings: A Survey to measure overall targets you have set?
A trip to Balkh province to see how the
implementation of survey is going? A trip to Nangarhar to see how the project is being
implemented?
8/3/2019 5 Day M&E Presentation Compiled
15/87
8/3/2019 5 Day M&E Presentation Compiled
16/87
16
Different ways of doing an Evaluation
Rapid Participator Appraisal: Originally used in rural areas,while applicable in most communities. This is a qualitativeway of doing evaluations in a period of short time. It is usedas a starting point for understanding a local situation and is aquick, cheap and useful way to gather information. It involves
the use of secondary data review, direct observations, semi-structured interviews. It is flexible and interactive.
External Evaluation: Done by a carefully chosen outsider.
Internal Evaluation: Done by internal staff of the organization
of different organizational units.
Interactive Evaluation: This involves a very active interactionbetween outside evaluator and the organization beingevaluated.
8/3/2019 5 Day M&E Presentation Compiled
17/87
17
When do we need M&E results
during the Program Cycle?During situation analysis: identification ofoverall program focus, lessons learned frompast program implementation are studied and
taken into account in the program strategies;During program design: data on indicatorsproduced during the previous program cycleserve as baseline data for the new program
cycle. Indicator data also enable programdesigners to establish clear program targetswhich can be monitored and evaluated
8/3/2019 5 Day M&E Presentation Compiled
18/87
18
When do we need M&E results during the
Program Cycle? Contd
During program implementation: monitoringand evaluation ensures continuous tracking of
program progress and adjustment of programstrategies to achieve better results;
At program completion: in-depth evaluationof program effectiveness, impact and
sustainability ensures that the program wasenough effective and that lessons on goodstrategies and practices are available fordesigning the next program cycle.
8/3/2019 5 Day M&E Presentation Compiled
19/87
19
Types of Evaluation
Formative
Summative
8/3/2019 5 Day M&E Presentation Compiled
20/87
20
Formative EvaluationTo improve the design and performanceof an on-going program / project
Mid-term Evaluation or Mid-term Review
8/3/2019 5 Day M&E Presentation Compiled
21/87
21
Summative EvaluationTo make an overall judgment about theeffectiveness of a completed program,often to ensure accountability
Ex-post EvaluationImpact Assessment
8/3/2019 5 Day M&E Presentation Compiled
22/87
22
Management Information System (MIS)
An operational system that is capable ofcommunicating or reporting physical andfinancial information to relevant project
management staff, with a clear definition ofwho collects what information, how and when.
Or a database system used to collect project
data and information of different section, andpatterns.
8/3/2019 5 Day M&E Presentation Compiled
23/87
23
Management Information System (MIS)
Everything you monitor and evaluate should berecorded and kept in a structured system, both inhard and soft copies. And a rule of thumb in M&Esays that nothing should go in MIS system without
a proof and a hard copy in your files. Therefore,there is a very close link between MIS and M&E,that you should keep your M&E data in a wellfunctioning MIS system.
The hard copies or files of proof for what you have
reported, will be very useful in audit, or when justsomeone asks where did you get this information.
8/3/2019 5 Day M&E Presentation Compiled
24/87
24
MIS and M&E Relationship
Data
InformationImpact
EvidenceAction
Knowledge
Organize
analyze
Integrate (MIS)
Package
(MIS)
Package &
Communicate
to planner and
stakeholders
(MIS)
Influence
the plan
(Planners
)
Implement
the Plan
(Decision
Makers)
ManagementInformationSystem /
MonitoringAnd
Evaluation
8/3/2019 5 Day M&E Presentation Compiled
25/87
25
Areas where M&E is usedWe involve the combination of some the following
Informing
Planning and decision makingAssessing progress
Promoting Accountability
Learning
InfluencingPromoting partnership
8/3/2019 5 Day M&E Presentation Compiled
26/87
26
Informing, Planning and
Decision makingInforming: Translating Findings into usablecommunications
Planning and Decision making: based on M&E Data,making decisions, and modifying / formulating newplans
Or
Providing information about alternative concepts oroptional courses of actions
8/3/2019 5 Day M&E Presentation Compiled
27/87
27
Assessing progress and
promoting AccountabilityAssessing Progress: establishing whatprogress has been made, toward pre-
determined objectives, leading tomodifications where necessary
Promoting Accountability: explaining
how money will be / was spent,resources utilized, and the justification ofthat investment
8/3/2019 5 Day M&E Presentation Compiled
28/87
28
Learning and InfluencingLearning: understanding what hashappened in the past, to inform and
improve future approachesInfluencing: Generating evidence andarguments that the program influenced
target groups / reached the goals
8/3/2019 5 Day M&E Presentation Compiled
29/87
29
Promoting Partnership
Make the adjustment for the investmenton the project, and encourage the
partner (donor) for theExtension/expansion/Replication of theproject.
8/3/2019 5 Day M&E Presentation Compiled
30/87
30
What did you learn so far?
Questions and Answers!
8/3/2019 5 Day M&E Presentation Compiled
31/87
8/3/2019 5 Day M&E Presentation Compiled
32/87
32
Day Three
8/3/2019 5 Day M&E Presentation Compiled
33/87
33
Different Evaluation ApproachesApproach Major Purpose Typical Focus
QuestionsLikelyMethodology
Goal-based Assessing achievementsof goals and objectives
Were the goalsachieved? Efficiently?Were they the right
goals?
Comparing baselineand progress data
Decision-Making
Providing Information Is the projecteffective? Should itcontinue? How mightit be modified?
Assessing range ofactivities related toproject context,inputs, process andproducts.
Goal-free Assessing the full range ofproject effects, intendedand unintended.
What are all theoutcomes? Whatvalues do they have?
Judging projectworth, usingqualitative andquantitative methodsto come acrosspossible results.
8/3/2019 5 Day M&E Presentation Compiled
34/87
34
Types of Monitoring and Evaluation Data
Primary Data: This is the data which is collectedprimarily by your project through different ways andapproaches.
Secondary Data: This data is the analysis or use of data
from sources other than your organization. Baseline Data: Is base data for your indicators, which
could be primary or secondary.
Mid-Term Data: The primary data progress upon periodof time before end of the project.
End-line or Impact Data: The final progress of ourbaseline data for indicators, which is called the over allresult on the community you served.
8/3/2019 5 Day M&E Presentation Compiled
35/87
35
?What is Indicator?
8/3/2019 5 Day M&E Presentation Compiled
36/87
36
What is an IndicatorA tool to assess if we have achievedwhat we have set out to achieve
A monitoring tools to measure progresstoward objective
Can be used to measure different types
of factors (risk, inputs, process, output,outcome, impact)
Maybe qualitative or quantitative
8/3/2019 5 Day M&E Presentation Compiled
37/87
37
Why have indicators?Simplify details and complexity
Inform and guide decisions and actions
Can be effective operational andcommunication tools
Help define development as well as
measure it (M&E)
8/3/2019 5 Day M&E Presentation Compiled
38/87
38
?Who should define them?
8/3/2019 5 Day M&E Presentation Compiled
39/87
39
Who should define them?Policy makers, Implementers andbeneficiary groups
How?
Through Partnership and participation inidentifying, revising and using indicators
8/3/2019 5 Day M&E Presentation Compiled
40/87
40
What indicators are required?
Indicators are influential in guidingdecision making and action, so -
Measure what we value most
So you need to define indicators tomeasure different levels
No one indicator measure all aspectsof your project or development
8/3/2019 5 Day M&E Presentation Compiled
41/87
41
Levels of Indicators
Objectives Types of Indicators
Inputs Inputs: Different resources used to achieve precedingindicators
Activities Process: measures what happens during implementation.Completion or milestones from activity plan
Outputs Outputs: Immediate physical or financial outputs of theproject, number of teachers trained in new syllabus
Purpose Outcomes or effects: more specific to project activities(deal more with direct effect of project outputs onbeneficiaries)
Goal Impacts: Medium or long term developmental change, e.g.Proportion of girls completing secondary education,%increase
8/3/2019 5 Day M&E Presentation Compiled
42/87
42
So what do we monitor?
All Most Some Few
InputsOutputs
OutcomesImpacts
Monitoring
Process EvaluationEvaluation
Effectiveness Evaluation
Long-term effectsChanges in child
mortalitySocial norms
higher educationincreased
Intermediate effectsBehaviorchangeAttitudechange
Number ofstafftrainedCoverageof qualityservices
ResourcesStaffFundsMaterials
8/3/2019 5 Day M&E Presentation Compiled
43/87
8/3/2019 5 Day M&E Presentation Compiled
44/87
44
Indicators are either qualitative or
quantitative contdQualitative indicators, imply qualityassessment
Compliance withQuality of
Extent of
Level of
Note: it is possible to express quantitatively viaa scoring system (e.g. scoring system couldbe developed to track improvements inquality)
8/3/2019 5 Day M&E Presentation Compiled
45/87
45
Steps in developing indicators
1. Identify what is to be measured
2. Develop trail measures
3. Assess each trail indicator, usingagreed criteria
4. Select and organize the best
indicators for project and user group
8/3/2019 5 Day M&E Presentation Compiled
46/87
46
Question?What makes a good indicator...?
8/3/2019 5 Day M&E Presentation Compiled
47/87
47
What makes a good indicator?
Specific
Measurable
Available /Achievable
Relevant
Timely
SMART
8/3/2019 5 Day M&E Presentation Compiled
48/87
48
SMART
Specific: is the desired outcome clearlyspecified?
Measurable: can the objective be measure?Achievable: is the objective likely to beachieved within the time and budget?
Relevant: is the outcome relevant to theproject goal or main objective?
Timely: is the time specified to achieve theobjective?
8/3/2019 5 Day M&E Presentation Compiled
49/87
49
Answer the following questions:
1. What is a project?
2. What is M&E?
3. What are the different ways of doing evaluations?
4. What are the different evaluation Approaches?
5. What is an Indicator?
6. What is baseline data?
7. What is Impact?
8/3/2019 5 Day M&E Presentation Compiled
50/87
50
Thanks
See you in Next Session
8/3/2019 5 Day M&E Presentation Compiled
51/87
51
Day Four
8/3/2019 5 Day M&E Presentation Compiled
52/87
52
Monitoring tools
Which tools are used to gathermonitoring and evaluation data?
8/3/2019 5 Day M&E Presentation Compiled
53/87
53
Focus Group Discussion
A group of target groups are gathered, the topic isintroduced to them and they talk about it.
Focus Group Discussion is good because it is:
InexpensiveRapid Technique
Participants themselves are sources of data
Session typically last 1 to 2 hours
Share certain common characteristics (Sex, Age,Social background, etc)
Data is mostly accurate
8/3/2019 5 Day M&E Presentation Compiled
54/87
54
Used when?
Analysis + Planning
As part of review
Monitoring regular project lifetime
8/3/2019 5 Day M&E Presentation Compiled
55/87
55
The facilitator is expected toGuide each session
Allow the discussion to flow freely
Refocus the discussion as necessary
Intervene to bring out salience ifparticipant do not
8/3/2019 5 Day M&E Presentation Compiled
56/87
56
InterviewsThere are mostly four types of interviews
1. Structured Interviews: use a
questionnaire, fixed number ofquestions, pre-selected range ofanswers
2. Simi-structured Interviews: Do notuse questionnaire, topics andquestions are pre-determined, butmay arise more while interview.
8/3/2019 5 Day M&E Presentation Compiled
57/87
57
Interviews Contd
3. Unstructured interviews: Topics are pre-determined but questions are notdetermined, these interview are used to
collect general information, data from allaspect of programs or subjects.
4. Telephone Interviews: could bestructured, Simi-structured, or
unstructured, but by telephone, easy andcost-effective and are mostly veryaccurate.
8/3/2019 5 Day M&E Presentation Compiled
58/87
58
What contribute to a Good or
Bad interview?How do you judge the validity orreliability of information given at an
interview?Facts: Commonly agreed specific truth
Opinions: Persons or Groups views
Rumors: Information from an unknownsource
Be careful of above three things
8/3/2019 5 Day M&E Presentation Compiled
59/87
59
Some tips on conducting Interviews
Use same language as intervieweeObserve non verbal indicators, bodylanguage, facial expression
Use the 6 helpers (who, what, when, where,why, how)Formulate questions clearlyAvoid why questions too often
Avoid finishing their sentencesNote takingThank interviewees
8/3/2019 5 Day M&E Presentation Compiled
60/87
60
Types of questions
Closed-ended (record single response)
Open-ended (record single response)
Multiple Choice (MCQs) (single / more)
8/3/2019 5 Day M&E Presentation Compiled
61/87
61
Community Meetings
This involves a gathering of a large group ofbeneficiaries to whom questions, problems andsituations are put to inputs to help measuring
indicators. Community meetings are useful for getting a
broad response from many people on specificissues.
Community meetings are difficult to facilitate,and requires a very experienced facilitator.
8/3/2019 5 Day M&E Presentation Compiled
62/87
62
Taking Action
Monitoring and Evaluation have little value if theorganization or project does not act on the informationthat comes out of the analysis of data collected.
Once you have the finding, conclusions andrecommendations from your monitoring and evaluation
process, you need to:
Report to management and field staff Report to your stakeholders Learn from the overall process
Make effective decisions and how to more forward; and, ifnecessary
Modify your objectives, methods and approaches.
8/3/2019 5 Day M&E Presentation Compiled
63/87
63
Reporting
You need to report your findings, conclusionsand recommendations in different mechanism,to different stakeholders in different times in the
project cycle. Usually report to: Board Members
Management
Staff / Field staff
Beneficiaries Donors
Wider development community
8/3/2019 5 Day M&E Presentation Compiled
64/87
64
Problem Tree
Problem Identification, Setting objective
Situation
Analysis
Problem
Analysis
Objectives
Analysis
Alternative
Analysis
8/3/2019 5 Day M&E Presentation Compiled
65/87
65
Situation Analysis
The situation analysis should identify keyproblems/opportunities/Development
constraintsWhat the main problems are?
What are the opportunities to resolve?
What are the constraints to affectdevelopment?
8/3/2019 5 Day M&E Presentation Compiled
66/87
66
Situation AnalysisContd
Capabilities and Constraints
Social
Technical
Economic
Environmental
Political
8/3/2019 5 Day M&E Presentation Compiled
67/87
67
Problem Analysis
Explore causes and effects of problems
What and who are the main causes?
Why and when this happened?
How this happened?
Where this started?
What will happen next?
Who will ask us?
8/3/2019 5 Day M&E Presentation Compiled
68/87
68
Now develop a problem tree
Identify a focal problem (s)
Place the direct effect above
Place the causes below k
Extend as far as possible
Find solution for every analyzed
situation
Convert it to objectives tree
8/3/2019 5 Day M&E Presentation Compiled
69/87
69
Problem Tree
Focal Problem(Poor Reporting)
Will decreasefunding
Will fail theProject to closure
PoorManagement
No budget
Capacity buildingNeeded
Other staffRequired
8/3/2019 5 Day M&E Presentation Compiled
70/87
70
Identifying Core problem
What does a core problem means?
Is there solution for core problem?
Is this true? A core problem is the one with nosolution
No, management says, there is no problemwithout solution, but that is the existence of
negative state.
Why is there no WW3? Because ofmanagement and communication.
8/3/2019 5 Day M&E Presentation Compiled
71/87
71
Group work
Identify a core problem
Develop a problem tree
Convert the Problem tree toobjectives tree
8/3/2019 5 Day M&E Presentation Compiled
72/87
72
Questions and Answers
8/3/2019 5 Day M&E Presentation Compiled
73/87
73
Thank You
See you in next Session
8/3/2019 5 Day M&E Presentation Compiled
74/87
74
Day Five
8/3/2019 5 Day M&E Presentation Compiled
75/87
75
Introduction of a USAID ProjectPerformance Management/Monitoring
Plan (PMP)
8/3/2019 5 Day M&E Presentation Compiled
76/87
76
DAC Criteria
Development Assistance CommitteeDAC is a criteria of Evaluation developed in 2002 inSwitzerland, when evaluating programs and projects,its useful to consider DAC criteria.
RelevanceEffectiveness
Efficiency
Impact
Sustainability
8/3/2019 5 Day M&E Presentation Compiled
77/87
77
Relevance
Does the aid activity fit the policies of thedonor and the recipient (client)
Does it meet the priorities of the target groupAre activities consistent with the overall goaland objectives
8/3/2019 5 Day M&E Presentation Compiled
78/87
78
Effectiveness
A measure of the extent to which an aidactivity attains its objectives.Use the following questions to evaluate
effectiveness:
To what extent were the objectives achieved /are likely to be achieved?
What were/are the major factors influencingthe achievement or non-achievement of theobjectives?
8/3/2019 5 Day M&E Presentation Compiled
79/87
79
Efficiency
Efficiency measures the outputs -- qualitativeand quantitative -- in relation to the inputs.
It is an economic term which signifies that isthis uses the least costly resources possible inorder to achieve the desired results. Thisgenerally requires comparing alternative
approaches to achieving the same outputs, tosee whether the most efficient process hasbeen adopted.
8/3/2019 5 Day M&E Presentation Compiled
80/87
80
EfficiencyContdWhen evaluating the efficiency of a program ora project, it is useful to consider the followingquestions:
Were activities cost-efficient?
Were objectives achieved on time?
Was the program or project implemented in
the most efficient way compared toalternatives?
8/3/2019 5 Day M&E Presentation Compiled
81/87
81
Impact
Positive and negative effects of theintervention, direct or indirect, intended orunintended, use the following questions toevaluate Impact
What has happened as a result of theprogram or project?
What real difference has the activity made tothe beneficiaries?
How many people have been affected?
8/3/2019 5 Day M&E Presentation Compiled
82/87
82
Sustainability
Sustainability is concerned with measuringwhether the benefits of an activity are likely tocontinue after donor funding has been
withdrawn. When evaluating sustainability, usethe following questionsTo what extent did the benefits of a programor project continue after donor fundingceased?
What were the major factors which influencedthe achievement or non-achievement of of theprogram or project?
8/3/2019 5 Day M&E Presentation Compiled
83/87
83
What is a work plan and how to develop a
work plan?A work plan is a description of the necessary activities
set out in stages, with rough indication of the timingIn order to draw a good work plan, the implementers
should:
List all the tasks required to implement a project; Put the tasks in the order in which they will be
implemented; Show allocation of the responsibilities to the actors;
and Give the timing of each activity.Note: you can also allocate an estimated budget for
each of the activities in a work plan
8/3/2019 5 Day M&E Presentation Compiled
84/87
84
What is a work plan and how to develop awork plan?
The work plan is a guide to project implementation anda basis for project monitoring. It therefore helps to:
Finish the project in time;
Do the right things in the right order;
Identify who will be responsible for what activity
8/3/2019 5 Day M&E Presentation Compiled
85/87
85
A Common M&E system
A Monitoring and Evaluation system includes: Monitoring and Evaluation staffing and AoRs for every staff
Data collection
Filing system
Management Information System (Computerized)
Design of methodologies
Administering data collection instruments
Data Analysis
Reporting
Feedback to management and field staff Management of the performance monitoring plan (PMP)
Instruments and Methods for Data QualityAssurance/Assessment (DQA)
8/3/2019 5 Day M&E Presentation Compiled
86/87
86
So, Now
Lets close and let each other know how isyour Monitoring and Evaluation System?
I will let you know, how is my M&E system
8/3/2019 5 Day M&E Presentation Compiled
87/87
Thanks for the ParticipationSee you in next Training