M%20and%20 E Milan

Program Monitoring and Program Monitoring and EvaluationEvaluation

Colorado State University,USA 14 September 2002

Milan K.Dinda Ph.D

M I

L A

N -

CS

UM

I L

A N

- C

SU

Why the Need for M&E?

MIL

AN

-CS

U

Why M&E? Why M&E?

Internal uses External uses Asses progress against targets Modify program design and

operation Identify staff needs Informed long range planning Make resource allocation

decisions Identify problems or

bottlenecks Develop a common

understanding of desiredoutcomes

Report to funding agency Respond to questions or

criticisms about the program Promote the program to

potential participants Seek additional and more

targeted funding Improve outreach, public

relations and marketing effort

MIL

AN

-CS

U

Defining the Terms

MIL

AN

-CS

UM

ILA

N-C

SU

What is Monitoring?What is Monitoring?

• A routine processroutine process used to determine the extent to which a program has been effectively implemented at different levels, in time and at what cost

• Basically an internal activity

• Carried out regularly, for example, monthly,quarterly, half-yearly or annually.

MIL

AN

-CS

U

What is Evaluation?What is Evaluation?

• An episodic processepisodic process designed to determine the value or worth of a specific project or program.

• Could be both internal and external

• Links particular outputs or outcome directly to an intervention

MIL

AN

-CS

U

M and E in a NutshellM and E in a Nutshell

Monitoring Monitoring EvaluationEvaluation

Routine, daily assessment of on-going activities and progress

Looks at what is being done

Tracks changes over time in inputs, processes and outputs

Episodic assessment of achievements

Examines what has been achieved

Measures outcomes/ effects and impact cased by interventions

MIL

AN

-CS

U

Three Types of EvaluationThree Types of Evaluation

Process EvaluationProcess Evaluation

• Are activities actually implemented? (Content)

• Does the program reach its intended clientele? (Scope and Coverage)

• Are activities being implemented as planned with necessary inputs? (Timing)

MIL

AN

-CS

U

Three Types of EvaluationThree Types of Evaluation

Outcome EvaluationOutcome Evaluation

• Examines short-term effects (based on program objectives)

• With HIV/AIDS interventions, immediate effects or outcomes are often relatedto behavior changes and underlying changes in knowledge, attitudes and beliefs (e.g., use of condom)

• Usually looks at “before and after situation”

MIL

AN

-CS

UM

ILA

N-C

SU

Three Types of EvaluationThree Types of Evaluation

Impact EvaluationImpact Evaluation

• Assesses the longer-term results of a program • Takes a long time, are expensive and complex • Example: Reduction in the new infection of HIV/AIDS

MIL

AN

-CS

U

Systems Approach to M&E

MIL

AN

-CS

UM

ILA

N-C

SU

Systems Analysis to M&ESystems Analysis to M&E

Input OutputProcess Impact Outcome

MIL

AN

-CS

UM

ILA

N-C

SU

Systems Analysis to M&ESystems Analysis to M&E

Inputs OutputProcess Impact Outcome

InputsInputs

• Resources invested in a program (supplies, people,

• 5 M’s: (Wo)Man, Machines, Money, Methods, Materials

MIL

AN

-CS

UM

ILA

N-C

SU

Systems Analysis to M&ESystems Analysis to M&E

Inputs OutputProcess Impact Outcome

Process Process

• activities carried out to achieve objectives

• shows what is done and how well it is done, for example, how volunteers are recruited and trained to become peer counselors

I C

O

M

P

I C

O

M

P

Systems Analysis to M&ESystems Analysis to M&E

Inputs OutputProcess Impact Outcome

Outputs Outputs

• immediate results of activities achieved (deliverables of the program)

• number of workshops conducted; MSMs reached by BCC materials; STI, VCT care; knowledge of HIV transmission

MIL

AN

-CS

UM

ILA

N-C

SU

Systems Analysis to M&ESystems Analysis to M&E

Inputs OutputProcess Impact Outcome

Outcomes (Effects)Outcomes (Effects)

• changes observed among clients of the program

• changes in behavior or skills (e.g., increased usage of condoms among truck drivers; exchange of clean needles); reduction in STI trend; increase social support

MIL

AN

-CS

UM

ILA

N-C

SU

Systems Analysis to M&ESystems Analysis to M&E

Inputs OutputProcess Impact Outcome

ImpactImpact

• changes in the longer term that occur in the community level as a result of a given program

• usually observed in community level (e.g., reduction in HIV/AIDS related mortality; decreased prevalence; coping capacity of the community)

MIL

AN

-CS

UM

ILA

N-C

SU

Logical Framework and Systems Approach to M&E

MIL

AN

-CS

UM

ILA

N-C

SU

What is a Logical Framework?What is a Logical Framework?

Management tool for strategic planning and project management

Systematically and logically sets out a program’s objectives, outputs and activities.

MIL

AN

-CS

UM

ILA

N-C

SU

What is a Logical Framework?What is a Logical Framework?

Hierarchy of Aims

Goal (Impact):Goal (Impact): the ultimate end of the program to which the specific project will contribute (reduction of prevalence of HIV/AIDS among sex workers in Chiang Mai)

Purpose (Outcome/Effect):Purpose (Outcome/Effect): what the project is expected to achieve once it is completed within the allocated time (provision of counseling and clinic services to sex workers)

Outputs:Outputs: products or deliverables of activities (trained peer counselors; clinical services provided)

Activities (Inputs; Process):Activities (Inputs; Process): actions to be undertaken and the resources available to produce the outputs (upgrading of clinics)

MIL

AN

-CS

UM

ILA

N-C

SU

Logframe and Systems Analysis Logframe and Systems Analysis

MIL

AN

-CS

UM

ILA

N-C

SU

M & E Framework for HIV/AIDS Program

MIL

AN

-CS

UM

ILA

N-C

SU

M&E Framework for HIV/AIDS ProgramM&E Framework for HIV/AIDS Program

OrganizationalResources

ExternalResources

TechnicalCompetence

ManagementProcesses

Multi-sectoralparticipation

Care andSupport

Counseling andPreventiveServices

IEC, BCC andAdvocacy

Socio-political,economic andcultural factors

Prevalence Statusof HIV/AIDS

Behavior andAttitude

Practices

Social/Community

Support, PolicyChanges

Inputs Process Outputs Outcomes Impact

Monitoring Evaluation

MIL

AN

-CS

UM

ILA

N-C

SU

M & E Process

MIL

AN

-CS

UM

ILA

N-C

SU

ProcessProcess

• Define the Purpose of M & E

• Define the Scope of M & E

• Develop Indicators

• Determine Data Sources and Methods

• Develop an Implementation Plan

• Disseminate and Utilize Results

MIL

AN

-CS

UM

ILA

N-C

SU

Define the Purpose of M & EDefine the Purpose of M & E

Three W’sThree W’s What: What interventions to be monitored and evaluated?

Why: Is it a routine process or a short-term process for decision making? Is it to examine results at output, outcome or impact level?

Who: Who will use the results (e.g., NGO, program manager, CBD staff , community, donor)

MIL

AN

-CS

UM

ILA

N-C

SU

Define the Scope of M & EDefine the Scope of M & E

Important: Review first your program goals and Important: Review first your program goals and objectivesobjectives

• Intervention Level (Coverage): Program level, community level

Causality Links: Cause and Effect of what?

Timetable: How long will the M & E be undertaken?

MIL

AN

-CS

UM

ILA

N-C

SU

Develop Indicators Develop Indicators

• Marker of performance - to track progress in the performance of the HIV/AIDS interventions

• Operational measures - “if you can measure it you can manage it”

• Key Standards: QualityQuality (binary - yes/no measure, categorical); QuantityQuantity (rate, absolute number, average, proportion), CostCost and TimingTiming

MIL

AN

-CS

UM

ILA

N-C

SU

Develop Indicators Develop Indicators

Characteristics of Good A Good Indicator Valid: It is applicable to the interventions being measured (percentage of condom usage)

Reliable: Bias-free and consistent

Specific: Identifies specific areas to be examined (users during the last 3 months in District 1)

Sensitive: Reflects changes in the situation (Increase in the number of condom users)

Operational: It is measurable or quantifiable (number, percentage)

MIL

AN

-CS

UM

ILA

N-C

SU

Develop IndicatorsDevelop Indicators

Component IndicatorsInputs Number of personnel assigned to the program (Quantity)

Qualified personnel assigned (Quality) The total cost of program. Budget for counseling component (Cost) Timeline of the program (Timing)

Process Number of peer educators trained (Quantity) Number of peer leaders trained who passed the test (Quality) Cost per peer training (Cost) Period of training days (Timing)

Outputs Increase in the number of sex workers reached (Quantity) Percent of sex workers who understood HIV/AIDS risks (Quality) Added cost of increasing the numbers of peer educators (Cost) Frequency of follow-ups undertaken (Timing)

Outcomes Increase in condom use rate among sex workers (Quantity) Improvement in negotiation skills of sex workers (Quality) Costs associated with the increase in condom usage (Cost) Time required to achieve improvement in condom usage (Timing)

Impact Reduction in HIV/AIDS rate among sex workers (Quantity) Ability of sex workers to collectively negotiate for occupational safety

(Quality) Cost of achieving reduction in HIV/AIDS prevalence (Cost) Time required to achieve reduction in HIV/AIDS prevalence (Timing)

MIL

AN

-CS

UM

ILA

N-C

SU

Determine Data Sources/Methods Determine Data Sources/Methods

• Program-based: Direct Interventions - routine program statistics, client surveys, FGDs, facility-based surveys, in-depth interviews, provider survey, program documents and reports, direct observations

• Community-based: Indirect Interventions - community-based surveys, FGDs, qualitative surveys, population census

• Context-based - parliamentary records, public libraries, media reports

MIL

AN

-CS

UM

ILA

N-C

SU

Develop an Implementation Plan Develop an Implementation Plan

Tip: Review your Logframe (if you have)

• Who is doing what?

• When is it being done?

• How much time does it take?

• How much does it cost?

MIL

AN

-CS

UM

ILA

N-C

SU

Disseminate and Utilize Results Disseminate and Utilize Results

• Utilize in Decision Making

• Discuss with program staff

• Advocacy

• Share to donors and other interested parties

• Share to stakeholders and communities

MIL

AN

-CS

UM

ILA

N-C

SU

Principles of M & E

MIL

AN

-CS

UM

ILA

N-C

SU

Principles (BLEND) Principles (BLEND)

• Bias-free • Logical and Clear

• Educational

• Non-threatening

• Diversity of views

MIL

AN

-CS

UM

ILA

N-C

SU

Pitfalls to Avoid

MIL

AN

-CS

UM

ILA

N-C

SU

Avoid Pitfalls Avoid Pitfalls

• Too scientific • Perfecting the design

• Ignoring Stories

• Probing only Success

• Getting Done With It