37
Program Monitoring Program Monitoring and Evaluation and Evaluation Colorado State University,USA 14 September 2002 Milan K.Dinda Ph.D

M%20and%20 E Milan

Embed Size (px)

DESCRIPTION

 

Citation preview

Page 1: M%20and%20 E Milan

Program Monitoring and Program Monitoring and EvaluationEvaluation

Colorado State University,USA 14 September 2002

Milan K.Dinda Ph.D

Page 2: M%20and%20 E Milan

M I

L A

N -

CS

UM

I L

A N

- C

SU

Why the Need for M&E?

Page 3: M%20and%20 E Milan

MIL

AN

-CS

U

Why M&E? Why M&E?

Internal uses External uses Asses progress against targets Modify program design and

operation Identify staff needs Informed long range planning Make resource allocation

decisions Identify problems or

bottlenecks Develop a common

understanding of desiredoutcomes

Report to funding agency Respond to questions or

criticisms about the program Promote the program to

potential participants Seek additional and more

targeted funding Improve outreach, public

relations and marketing effort

Page 4: M%20and%20 E Milan

MIL

AN

-CS

U

Defining the Terms

Page 5: M%20and%20 E Milan

MIL

AN

-CS

UM

ILA

N-C

SU

What is Monitoring?What is Monitoring?

• A routine processroutine process used to determine the extent to which a program has been effectively implemented at different levels, in time and at what cost

• Basically an internal activity

• Carried out regularly, for example, monthly,quarterly, half-yearly or annually.

Page 6: M%20and%20 E Milan

MIL

AN

-CS

U

What is Evaluation?What is Evaluation?

• An episodic processepisodic process designed to determine the value or worth of a specific project or program.

• Could be both internal and external

• Links particular outputs or outcome directly to an intervention

Page 7: M%20and%20 E Milan

MIL

AN

-CS

U

M and E in a NutshellM and E in a Nutshell

Monitoring Monitoring EvaluationEvaluation

Routine, daily assessment of on-going activities and progress

Looks at what is being done

Tracks changes over time in inputs, processes and outputs

Episodic assessment of achievements

Examines what has been achieved

Measures outcomes/ effects and impact cased by interventions

Page 8: M%20and%20 E Milan

MIL

AN

-CS

U

Three Types of EvaluationThree Types of Evaluation

Process EvaluationProcess Evaluation

• Are activities actually implemented? (Content)

• Does the program reach its intended clientele? (Scope and Coverage)

• Are activities being implemented as planned with necessary inputs? (Timing)

Page 9: M%20and%20 E Milan

MIL

AN

-CS

U

Three Types of EvaluationThree Types of Evaluation

Outcome EvaluationOutcome Evaluation

• Examines short-term effects (based on program objectives)

• With HIV/AIDS interventions, immediate effects or outcomes are often relatedto behavior changes and underlying changes in knowledge, attitudes and beliefs (e.g., use of condom)

• Usually looks at “before and after situation”

Page 10: M%20and%20 E Milan

MIL

AN

-CS

UM

ILA

N-C

SU

Three Types of EvaluationThree Types of Evaluation

Impact EvaluationImpact Evaluation

• Assesses the longer-term results of a program • Takes a long time, are expensive and complex • Example: Reduction in the new infection of HIV/AIDS

Page 11: M%20and%20 E Milan

MIL

AN

-CS

U

Systems Approach to M&E

Page 12: M%20and%20 E Milan

MIL

AN

-CS

UM

ILA

N-C

SU

Systems Analysis to M&ESystems Analysis to M&E

Input OutputProcess Impact Outcome

Page 13: M%20and%20 E Milan

MIL

AN

-CS

UM

ILA

N-C

SU

Systems Analysis to M&ESystems Analysis to M&E

Inputs OutputProcess Impact Outcome

InputsInputs

• Resources invested in a program (supplies, people,

• 5 M’s: (Wo)Man, Machines, Money, Methods, Materials

Page 14: M%20and%20 E Milan

MIL

AN

-CS

UM

ILA

N-C

SU

Systems Analysis to M&ESystems Analysis to M&E

Inputs OutputProcess Impact Outcome

Process Process

• activities carried out to achieve objectives

• shows what is done and how well it is done, for example, how volunteers are recruited and trained to become peer counselors

Page 15: M%20and%20 E Milan

I C

O

M

P

I C

O

M

P

Systems Analysis to M&ESystems Analysis to M&E

Inputs OutputProcess Impact Outcome

Outputs Outputs

• immediate results of activities achieved (deliverables of the program)

• number of workshops conducted; MSMs reached by BCC materials; STI, VCT care; knowledge of HIV transmission

Page 16: M%20and%20 E Milan

MIL

AN

-CS

UM

ILA

N-C

SU

Systems Analysis to M&ESystems Analysis to M&E

Inputs OutputProcess Impact Outcome

Outcomes (Effects)Outcomes (Effects)

• changes observed among clients of the program

• changes in behavior or skills (e.g., increased usage of condoms among truck drivers; exchange of clean needles); reduction in STI trend; increase social support

Page 17: M%20and%20 E Milan

MIL

AN

-CS

UM

ILA

N-C

SU

Systems Analysis to M&ESystems Analysis to M&E

Inputs OutputProcess Impact Outcome

ImpactImpact

• changes in the longer term that occur in the community level as a result of a given program

• usually observed in community level (e.g., reduction in HIV/AIDS related mortality; decreased prevalence; coping capacity of the community)

Page 18: M%20and%20 E Milan

MIL

AN

-CS

UM

ILA

N-C

SU

Logical Framework and Systems Approach to M&E

Page 19: M%20and%20 E Milan

MIL

AN

-CS

UM

ILA

N-C

SU

What is a Logical Framework?What is a Logical Framework?

Management tool for strategic planning and project management

Systematically and logically sets out a program’s objectives, outputs and activities.

Page 20: M%20and%20 E Milan

MIL

AN

-CS

UM

ILA

N-C

SU

What is a Logical Framework?What is a Logical Framework?

Hierarchy of Aims

Goal (Impact):Goal (Impact): the ultimate end of the program to which the specific project will contribute (reduction of prevalence of HIV/AIDS among sex workers in Chiang Mai)

Purpose (Outcome/Effect):Purpose (Outcome/Effect): what the project is expected to achieve once it is completed within the allocated time (provision of counseling and clinic services to sex workers)

Outputs:Outputs: products or deliverables of activities (trained peer counselors; clinical services provided)

Activities (Inputs; Process):Activities (Inputs; Process): actions to be undertaken and the resources available to produce the outputs (upgrading of clinics)

Page 21: M%20and%20 E Milan

MIL

AN

-CS

UM

ILA

N-C

SU

Logframe and Systems Analysis Logframe and Systems Analysis

Page 22: M%20and%20 E Milan

MIL

AN

-CS

UM

ILA

N-C

SU

M & E Framework for HIV/AIDS Program

Page 23: M%20and%20 E Milan

MIL

AN

-CS

UM

ILA

N-C

SU

M&E Framework for HIV/AIDS ProgramM&E Framework for HIV/AIDS Program

OrganizationalResources

ExternalResources

TechnicalCompetence

ManagementProcesses

Multi-sectoralparticipation

Care andSupport

Counseling andPreventiveServices

IEC, BCC andAdvocacy

Socio-political,economic andcultural factors

Prevalence Statusof HIV/AIDS

Behavior andAttitude

Practices

Social/Community

Support, PolicyChanges

Inputs Process Outputs Outcomes Impact

Monitoring Evaluation

Page 24: M%20and%20 E Milan

MIL

AN

-CS

UM

ILA

N-C

SU

M & E Process

Page 25: M%20and%20 E Milan

MIL

AN

-CS

UM

ILA

N-C

SU

ProcessProcess

• Define the Purpose of M & E

• Define the Scope of M & E

• Develop Indicators

• Determine Data Sources and Methods

• Develop an Implementation Plan

• Disseminate and Utilize Results

Page 26: M%20and%20 E Milan

MIL

AN

-CS

UM

ILA

N-C

SU

Define the Purpose of M & EDefine the Purpose of M & E

Three W’sThree W’s What: What interventions to be monitored and evaluated?

Why: Is it a routine process or a short-term process for decision making? Is it to examine results at output, outcome or impact level?

Who: Who will use the results (e.g., NGO, program manager, CBD staff , community, donor)

Page 27: M%20and%20 E Milan

MIL

AN

-CS

UM

ILA

N-C

SU

Define the Scope of M & EDefine the Scope of M & E

Important: Review first your program goals and Important: Review first your program goals and objectivesobjectives

• Intervention Level (Coverage): Program level, community level

Causality Links: Cause and Effect of what?

Timetable: How long will the M & E be undertaken?

Page 28: M%20and%20 E Milan

MIL

AN

-CS

UM

ILA

N-C

SU

Develop Indicators Develop Indicators

• Marker of performance - to track progress in the performance of the HIV/AIDS interventions

• Operational measures - “if you can measure it you can manage it”

• Key Standards: QualityQuality (binary - yes/no measure, categorical); QuantityQuantity (rate, absolute number, average, proportion), CostCost and TimingTiming

Page 29: M%20and%20 E Milan

MIL

AN

-CS

UM

ILA

N-C

SU

Develop Indicators Develop Indicators

Characteristics of Good A Good Indicator Valid: It is applicable to the interventions being measured (percentage of condom usage)

Reliable: Bias-free and consistent

Specific: Identifies specific areas to be examined (users during the last 3 months in District 1)

Sensitive: Reflects changes in the situation (Increase in the number of condom users)

Operational: It is measurable or quantifiable (number, percentage)

Page 30: M%20and%20 E Milan

MIL

AN

-CS

UM

ILA

N-C

SU

Develop IndicatorsDevelop Indicators

Component IndicatorsInputs Number of personnel assigned to the program (Quantity)

Qualified personnel assigned (Quality) The total cost of program. Budget for counseling component (Cost) Timeline of the program (Timing)

Process Number of peer educators trained (Quantity) Number of peer leaders trained who passed the test (Quality) Cost per peer training (Cost) Period of training days (Timing)

Outputs Increase in the number of sex workers reached (Quantity) Percent of sex workers who understood HIV/AIDS risks (Quality) Added cost of increasing the numbers of peer educators (Cost) Frequency of follow-ups undertaken (Timing)

Outcomes Increase in condom use rate among sex workers (Quantity) Improvement in negotiation skills of sex workers (Quality) Costs associated with the increase in condom usage (Cost) Time required to achieve improvement in condom usage (Timing)

Impact Reduction in HIV/AIDS rate among sex workers (Quantity) Ability of sex workers to collectively negotiate for occupational safety

(Quality) Cost of achieving reduction in HIV/AIDS prevalence (Cost) Time required to achieve reduction in HIV/AIDS prevalence (Timing)

Page 31: M%20and%20 E Milan

MIL

AN

-CS

UM

ILA

N-C

SU

Determine Data Sources/Methods Determine Data Sources/Methods

• Program-based: Direct Interventions - routine program statistics, client surveys, FGDs, facility-based surveys, in-depth interviews, provider survey, program documents and reports, direct observations

• Community-based: Indirect Interventions - community-based surveys, FGDs, qualitative surveys, population census

• Context-based - parliamentary records, public libraries, media reports

Page 32: M%20and%20 E Milan

MIL

AN

-CS

UM

ILA

N-C

SU

Develop an Implementation Plan Develop an Implementation Plan

Tip: Review your Logframe (if you have)

• Who is doing what?

• When is it being done?

• How much time does it take?

• How much does it cost?

Page 33: M%20and%20 E Milan

MIL

AN

-CS

UM

ILA

N-C

SU

Disseminate and Utilize Results Disseminate and Utilize Results

• Utilize in Decision Making

• Discuss with program staff

• Advocacy

• Share to donors and other interested parties

• Share to stakeholders and communities

Page 34: M%20and%20 E Milan

MIL

AN

-CS

UM

ILA

N-C

SU

Principles of M & E

Page 35: M%20and%20 E Milan

MIL

AN

-CS

UM

ILA

N-C

SU

Principles (BLEND) Principles (BLEND)

• Bias-free • Logical and Clear

• Educational

• Non-threatening

• Diversity of views

Page 36: M%20and%20 E Milan

MIL

AN

-CS

UM

ILA

N-C

SU

Pitfalls to Avoid

Page 37: M%20and%20 E Milan

MIL

AN

-CS

UM

ILA

N-C

SU

Avoid Pitfalls Avoid Pitfalls

• Too scientific • Perfecting the design

• Ignoring Stories

• Probing only Success

• Getting Done With It