View
218
Download
2
Embed Size (px)
Citation preview
Monitoring and Evaluation:Frameworks
Learning Objectives
At the end of the session, participants will be able to: • Identify and differentiate between conceptual
frameworks, results and logical frameworks, and logic models
• Design goals and objectives for specific intervention programs
• Design frameworks for specific intervention programs• Discuss how frameworks are used for M&E planning
Module Outline
• Introduction to frameworks– Conceptual Frameworks– Goals and objectives– Results and Logical Frameworks– Logic models
• Exercises developing frameworks• Interface between frameworks and M&E planning• Work on group project
Why are Frameworks Useful?
Designing frameworks assist developing:• Clearly understood program/project goals and measurable,
long-term, short-term, and intermediate objectives• Clearly defined relationships between program/project
inputs, processes, outputs, and outcomes, and between program/project activities and the external context (environmental factors)
• Sound implementation and M&E plans
Types of Frameworks
• Conceptual frameworks• Results frameworks• Logical frameworks• Logic models
Conceptual Frameworks(a.k.a., research or theoretical frameworks)
Diagram that identifies and illustrates the relationships between all relevant systemic, organizational, individual, or other salient factors that may influenceprogram/project operation and the successful achievement of program or projectgoals.
M&E Purpose:• To show where program fits into wider context• To clarify assumptions about causal relationships• To show how program components will operate to influence outcomes• To guide identification of indicators• To guide impact analysis (causal pathways)
Similar frameworks:• Proximate Determinants
Conceptual Frameworks
Individual characteristics
Technicalinputs
Program supply Institutionalcapacity
Programsustainability
Healthstatus
HealthypracticesService
utilization
Conceptual Frameworks
Source: MEASURE Evaluation 2004 (draft)
National Tuberculosis (TB) Program
TBmortality
TB infectionExternal Factors• Demographic: (age, gender)• Socio-economic: (education, occupation)• Environmental: (war, weather)
Program Factors• Political commitment• Donor involvement• National TB program
Health Systems (DOTS)• Availability• Access• Utilization•Case Detection
TB Knowledge• Stigma• Treatable• Symptoms
Co-morbidity• HIV• Malnutrition• Alcoholism•Diabetes
TB Morbidity• Prevalence• Incidence• HIV co-infection• MDR-TB
Conceptual Frameworks
Source: UNICEF, State of the World’s Children, 1998
Immediate causes
Underlying causes at
household/ family level
Basic causes at societal
level
Outcomes
Insufficientaccess to food
Potential Resources: environment, technology, people
Quantity & quality of actual resources - human, economic and organizational – and the way they
are controlled
Inadequatematernal & child
care practices
Poor water/sanitation & inadequate health
services
Child malnutrition, death and disability
Inadequate Diseasedietary intake
Causes of malnutrition in society
Activity
• Get into groups of 4 or 5• Develop conceptual framework for utilization
of family planning/maternal and child health services– Identify factors that influence health service
utilization– Arrange factors into a conceptual framework
• After 30 minutes, a member of each group will share the framework with all participants
Goals and Objectives
• Goal: a broad statement of a desired, long-term outcome of the program
• Objectives: statements of desired, specific, realisticand measurable program results SMART
• Specific: identifies concrete events or actions that will take place• Measurable: quantifies the amount of resources, activity, or change
to be expended and achieved• Appropriate: logically relates to the overall problem statement and
desired effects of the program • Realistic: Provides a realistic dimension that can be achieved
with the available resources and plans for implementation• Time-based: specifies a time within which the objective
will be achieved
Source: GAP 2003
Formulating Program Objectives
To [action]
The [specify what knowledge, attitudes, skills, behaviors]
Among [specific population or segment]
From – to [from baseline to desired level]
or by [ x percent ]
or to [specific level]
By [time frame]
Goals and ObjectivesNational AIDS Committee (NAC) 2005 Strategic Plan• Goal: Prevent the spread of HIV epidemic and
minimise its impact on nation by 2009• Objective 1: Reduction of HIV prevalence in nation• Objective 2: Improved health & quality of life of
people infected & affected by HIV/AIDS
• Objective 3: Strengthened capacity of NAC & stakeholders to respond to the HIV/AIDS epidemic at all levels through improved research, M&E and improved management & coordination
Goals and Objectives: Activity
Source: UNICEF, State of the World’s Children, 1998
Immediate causes
Underlying causes at
household/ family level
Basic causes at societal
level
Outcomes
Insufficientaccess to food
Potential Resources: environment, technology, people
Quantity & quality of actual resources - human, economic and organizational – and the way they
are controlled
Inadequatematernal & child
care practices
Poor water/sanitation & inadequate health
services
Child malnutrition, death and disability
Inadequate Diseasedietary intake
Results Frameworks
Diagrams that identify steps or levels of results and illustrate the causal relationships linking all levels of a program’s objectives.
Purposes:• Provides a clarified focus on the causal relationships that
connect incremental achievement of results to the comprehensive program impact
• Clarifies project/program mechanics and factors’ relationships that suggest ways and means of objectively measuring the achievement of desired ends
Other terms used:• Strategic frameworks
Results Frameworks
Goal: Improved Health Statusand/or Decreased Fertility
Objective: Improved Use of Health/FP Servicesand/or Appropriate Practices
IR1: Access/Availability IR2: Quality IR3: Sustainability IR4: Demand
IR1.1:Commodities/Facilities
IR1.2: Equity
IR2.2: Provider Performance
IR2.3: Training/Supervision
IR2.4: Information System
IR3.1: Policy
IR3.2: Health Care Finance
IR3.3: Private Sector
IR4.1: Attitude
IR4.2: Knowledge
IR4.3: Community Support
Source: David Marsh, 1999
Results Frameworks
IR1: Increased availabilityof quality services
IR1.1: Services increased
IR1.2: Practitioners’ skillsand knowledge increased
IR1.3: Improved program management
SO1: Increase tuberculosis case detection rate to 70%
IR2: Increased demand for quality services
IR2.1: Customerknowledge of TB improved
Tuberculosis (TB) Control Programs
Source: MEASURE Evaluation, 2004 (draft)
IR2.2: Social support for TB practices increased
Results FrameworksDonor/USAID Reproductive Health Program
IR1 Strengthened sustainability of FP/RH Program
IR1.1 Improved policy environment for the provision of FP/RH services in the public and private sectors
IR1.2 Strengthened NGO advocacy for FP program
SO1: Increased Utilization of Family Planning/Reproductive Health Services
IR2 Expansion of high quality FP/RH services in the public and private sectors
IR2.1 Increased availability of postpartum and postabortion FP services
Source: USAID/Turkey Performance Monitoring Plan, 1998-2001
IR2.2 Increased accurate knowledgeof clients about modern methods
and FP services
IR2.3 Improved job performance of health providers, trainers, and administrators
SO 14: Human capacity and social resiliency increased
FIVE-YEAR GOAL: Foundation Established forReducing Famine Vulnerability, Hunger and Poverty
IR 14.1: Use of high impact health, family planning, and nutrition services, products, and practices increased
IR 14.2: HIV/AIDS prevalence reduced and mitigation of the impact of HIV/AIDS increased
IR 14.3: Use of quality primary education services enhanced
IR 14.1.1: Community support for high impact health interventions increased
IR 14.1.3: Quality of key health services improved
IR 14.1.2: Availability of key health services and products improved
IR 14.1.4: Health sector resources and systems improved
IR 14.2.3: Access to care and treatment for people living with HIV and AIDS increased
IR 14.2.4: Care and support for orphans and vulnerable children expanded
IR 14.2.2: Mother-to-child transmission of HIV reduced
IR 14.2.1: Reduced risk behavior
IR 14.3.3: Quality of primary education improved
IR 14.3.4: Equitable primary education services strengthened
IR 14.3.2: Planning, mgmt and monitoring and evaluation for delivery of primary education services strengthened
IR 14.3.1: Community participation in the management and delivery of primary education services strengthened
IR 14.2.5: A more supportive environment for responding to HIV/AIDS
Results Frameworks
Source: USAID/Ethiopia Integrated Strategic Plan, February, 2004 (revised draft)
Logical Frameworks
Present a standardized summary of the project and its logic. Purposes:• Summarizes what the project intends to do and how• Summarizes key assumptions• Summarizes outputs and outcomes that will be monitored
and evaluated
Other terms used:• Logframe matrix
Logical FrameworksProject Description Performance Indicators Means of
VerificationAssumptions
Goal: The broader development impact to which the project contributes - at a national and sectoral level.
Measures of the extent to which a sustainable contribution to the goal has been made. Used during evaluation.
Sources of information and methods used to collect and report it.
Purpose: The development outcome expected at the end of the project. All components will contribute to this
Conditions at the end of the project indicating that the Purpose has been achieved and that benefits are sustainable. Used for project completion and evaluation.
Sources of information and methods used to collect and report it.
Assumptions concerning the purpose/goal linkage.
Component Objectives: The expected outcome of producing each component's outputs.
Measures of the extent to which component objectives have been achieved and lead to sustainable benefits. Used during review and evaluation.
Sources of information and methods used to collect and report it.
Assumptions concerning the component objective/purpose linkage.
Outputs: The direct measurable results (goods and services) of the project which are largely under project management's control
Measures of the quantity and quality of outputs and the timing of their delivery. Used during monitoring and review.
Sources of information and methods used to collect and report it.
Assumptions concerning the output/component objective linkage.
Activities: The tasks carried out to implement the project and deliver the identified outputs.
Implementation/work program targets. Used during monitoring.
Sources of information and methods used to collect and report it.
Assumptions concerning the activity/output linkage.
Logical Frameworks
GOAL
A. Reduced burden of TB to reach global average levels
B. Further development of multi-drug resistant TB (MDR-TB) prevented
PERFORMANCE INDICATORS
A. Notification rate
B-1. Treatment outcome
B-2. Prevalence of Multi-drug Resistance in “new” and previously treated TB patients
MEANS OF VERIFICATION
A. Annual notification reports (surveillance)
B-1. Annual reports on outcome of treatment (cohort analysis)
B-2. Periodic reports on surveillance of anti-TB drug resistance
ASSUMPTIONS
- A dual HIV/TB epidemic causing increase in TB incidence does not occur
- Control of private practitioner and pharmaceutical sectors to prevent MDR
- Prevalence of resistance to second line anti-TB drugs low enough at the outset so as not to seriously compromise treatment success ratio
Taskforce on Communicable Disease Controlin the Region X: Tuberculosis
Logical Frameworks
PURPOSE
[Implementing] cost-effective measures for the prevention and control of TB operating within civil and penitentiary health services in the Task Force area
PERFORMANCEINDICATORS
1. Coverage of TB programmes in line with international recommendations.
2. Proportion of patients defaulting out of patients treated.
3. Proportion of previously treated cases among all cases.
4. Proportion of patients on ambulatory treatment out of all patients treated.
MEANS OF
VERIFICATION
1. Annual reports
2. Annual reports
3. National / local annual notification reports (surveillance)
4. Annual record reviews during site visits (consecutive series of patients)
ASSUMPTIONS
- Stable political situation, sustained political commitment and financing
- Sufficient numbers of competent health care personnel in the government sector
Taskforce on Communicable Disease Controlin the Barents and Baltic Sea Regions: Tuberculosis
Logical Frameworks
OUTPUTS
8. Measures to increase awareness of TB and its treatment among all members of the community developed and tested
PERFORMANCEINDICATORS 8.1. Number of
pamphlets / posters printed and distributed annually
8.2. Awareness of TB among target groups
MEANS OF
VERIFICATION
8.1. Material produced / distributed
8.2. KAP or other surveys (before/after)
ASSUMPTIONS
1. Relevant persons motivated to participate
2. Professional interest, sufficient financing
3. Target groups interested in (their) health and able to participate
ACTIVITIES
8.1 Identify groups at risk for TB
8.2 Develop advocacy material suitable for all target groups (not only risk groups)
8.3 Organize health education directed at all target groups
8.4 Involve the mass media
INPUTS
MEANS OF
VERIFICATION
Financial management reports
ASSUMPTIONS
Taskforce on Communicable Disease Controlin the Barents and Baltic Sea Regions: Tuberculosis(only one output and related activities shown here)
Activity
• Return to small groups from previous activity
• Develop goals and objectives for a program aimed at increasing utilization of FP/MCH health services
• Develop a results or logical framework for this program
• In 45 minutes, a member of each group will share the goals, objectives and framework
Logic ModelsDiagrams that identify and illustrate the linear relationships flowingfrom program inputs, processes, outputs, and outcomes. Inputs orresources affect Processes or activities which produce immediateresults or Outputs, ultimately leading to longer term or broaderresults, or Outcomes. Purposes:• Provides a streamlined interpretation of planned use of
resources and desired ends• Clarifies project/program assumptions about linear
relationships between key factors relevant to desired endsOther terms used:• M&E Frameworks, Logical Frameworks
Logic Models: Training
INPUT PROCESS OUTPUT OUTCOME IMPACT
Develop clinicaltraining
curriculum
Conducttraining events
Practitionerstrained in new
clinical techniques
Increase in clients served
by (newly) trained
providers
Declining morbidity
levels in target population
Problem Statement: HIV infection rates continue to rise, underscoring the importance for people toknow their serostatus, develop personalized risk-reduction strategies, and access care and treatment services.
Supervision and training for
VCT personnel
Risk behaviors decreased
Counseling and Testing Personnel*
Provide pre-test counseling
Refer HIV+ clients to appropriate
services (PMTCT, care and support,
treatment)
Financial Resources
VCT MIS**
HIV test kits
Referral system for
prevention & Tx services**
People know their HIV status
HIV+ people are referred to appropriate
services
Client records are
available
Clients (HIV+ and -) develop & adhere to personalized
HIV risk-reduction Increase in
care, prevention, and
treatment services for
HIV+, HIV-, and discordant
couples
HIV transmission rates
decreased
HIV incidence decreased
HIV morbidity
& mortality decreased
PROCESSES OUTPUTS OUTCOMES IMPACTSINPUTS
Service reports are produced
Health outcomes of
HIV + improve
VCT protocols,guidelines, and
training documents**
Infrastructure
Demand for Services
Provide HIV testing
Provide post-test counseling
Maintain patient records
Complete reporting
requirements
HIV+ Clients develop & adhere to
personalized HIV care,
support, and treatment plans
Program and services are improved
Continuity of care is
available
Logic ModelsVoluntary Counseling and Testing (VCT)
Results FrameworkTuberculosis (TB) Control Programs
IR1: Increased availabilityof quality services
IR1.1: Services increased
IR1.2: Practitioners’ skillsand knowledge increased
IR1.3: Improved program management
SO1: Increase tuberculosis case detection rate to 70%
IR2: Increased demand for quality services
IR2.1: Customerknowledge of TB improved
Source: MEASURE Evaluation, 2004 (draft)
IR2.2: Social support for TB practices increased
INPUT•Human and financial resources to develop and print educational brochure
PROCESS•Distribute brochure to health facilities
•Meet with physicians to promote distribution of brochure
OUTPUT•Brochure distributed to clients of facilities
OUTCOME•Increased customer knowledge of TB transmission and treatment
•Increased demand for quality TB services
Logic Model
IMPACT•Decreased TB infection, morbidity and mortality
Portion of model for tuberculosis control relating to increasing demand for quality services
Activity
• Return to small groups from previous activities
• Develop logic models for one of the activities of your program to increase utilization of FP/MCH health services
• After 20 minutes, a member of each group will share one of the logical frameworks
Frameworks for M&E Planning• Purposes:
– clarifying assumptions, goals, and interrelationships between factors relevant to the project or program
– defining objectives– selecting activities– defining levels of performance and desired results in terms
of planned activities and realistic, objective impacts• Monitoring and evaluation plans incorporate:
– program managers’ assumptions and objectives, in a given context
– a schematic design displaying the directional linkages between key program elements and/or planned results, and other relevant factors
Summary of Frameworks
Type of Framework
Brief Description Program Management
Basis for Monitoring and Evaluation
Conceptual Interaction of various factors
Determine which factors the program will influence
No. Can help to explain results
Results Logically linked program objectives
Shows the causal relationship between program objectives
Yes – at the objective level
Logical Logically linked program objectives, outputs, and activities
Shows the causal relationship between activities and objectives
Yes – at the output and objective level
Logic model Logically links inputs, processes, outputs, and outcomes,
Shows the causal relationship between inputs and the objectives
Yes – at all stages of the program from inputs to process to outputs to outcomes/ objectives
References• AusGuide. The Logical Framework Approach.
http://www.ausaid.gov.au/ausguide/ausguidelines/1-1-1.cfm• Bertrand, Jane T., Magnani, Robert J, and Rutenberg, Naomi, 1996.
Evaluating Family Planning Programs, with Adaptations for Reproductive Health, Chapel Hill, N.C.: The EVALUATION Project.
• Global AIDS Program. (2003) Monitoring and Evaluation Capacity Building for Program Improvement Field Guide, Version 1. U.S. Centers for Disease Control and Prevention, Atlanta, GA.
• Marsh, David. 1999. Results Frameworks & Performance Monitoring. A Refresher by David Marsh (ppt) http://www.childsurvival.com/tools/Marsh/sld001.htm
• Tsui, Amy. 1998. Frameworks (ppt). Presented at the Summer Institute, University of North Carolina, Chapel Hill.
• Tsui, Amy. 1999. Frameworks (ppt). Presented at the Summer Institute, University of North Carolina, Chapel Hill.
• UNICEF. 1998. State of the World’s Children.
Group Projects
• Form groups for your group projects
• For your project, develop– Goals and objectives– Conceptual framework– One other framework: results, logical or logic model