Pelletier program assessment_guide

The Program Assessment GuideStructuring Contextual Knowledge and Experience to

Improve the Design, Delivery and Effectiveness of Nutrition Interventions

David PelletierDivision of Nutritional Sciences

Cornell University

CORE Group Fall MeetingSeptember 13, 2010

Washington, DC

Acknowledgments

Collaborators on the MN Tool Project Allison Corsi, Lesli Hoey Cornell University Robin Houston, Global Nutrition Consultant

Funders and Facilitators of the MN Tool Project The World Bank Micronutrient Initiative Centers for Disease Control UNICEF GAIN A2Z (USAID and Gates Foundation)

Genesis of This Project

Micronutrient Forum (May 2009) • From Evidence to Large Scale Programs

The Innocenti Process (Sept 2008)• Country implementers, donors, researchers• Review of evidence for effectiveness at scale• Identification of overarching issues needing attention

Innocenti: Overarching Issues

1. Key stakeholders share common MN goals but lack the leadership to coordinate and for priority setting, advocacy and action

2. Stakeholder groups within the MN community do not communicate effectively with one another;

3. Stakeholders have misaligned and often competing priorities and approaches at both global and country levels.

4. The MN community has not adequately engaged with broader nutrition, health, or development initiatives;

5. The MN community has not harnessed the full potential of private sector resources, expertise, and delivery mechanisms

(continued)

Source: Micronutrient Programs: What Works and What Needs More Work? A Report of the 2008 Innocenti Process. Micronutrient Forum, Wash. DC.

Innocenti: Overarching Issues

6. Country teams lack guidance and authority to assess, analyze and take actions needed to fit local contexts

7. Weak program monitoring, evaluation and documentation hinders program effectiveness, learning, advocacy and accountability

8. Limited funding for programs9. Limited funding for implementation research hinders our

efforts to strengthen the design, management, implementation, evaluation, and financing of MN programs at scale

Relevance to the Current Interest in Nutrition-Ag Linkages

Many of the overarching issues are the same There is even greater difference in disciplinary lenses,

availability and standards of evidence, communication barriers, intervention complexity, implementation systems, etc. are greater

The narrative ‘we know what works’ has different evidence beneath it, but we have similar pressures and incentives to pretend we do know, implement at scale and show results quickly

Therefore, we have the same need to:• adapt interventions, evidence and experience to new contexts • question assumptions during intervention design• identify and address bottlenecks during implementation • assess and document progress and impacts

Locating Evidence, Knowledge and Experience Within the Policy Process

Identifying problems (problem definition)

Choosing solutions (policy formulation)

Detailed intervention design and planning (policy implementation)

Implementation

SurveysSurveillance

Reconnaissance, etc

EfficacyEffectiveness

Cost

Knowledge of and experience in the local

context *

Operations research, monitoring and

QA during implementationOutcomes, impacts and related factors

Evaluation

* Context includes epidemiology, implementation capacities, social acceptability, resources, competing priorities, policy preferences of actors, etc.

Program Assessment Guide (PAG)

1. Action Plan to Address Barriers & Enablers

2. Operations Research Agenda

3. Issues for Inclusion in M&E

4. Strategic Plan to Build Support, Capacity & Sustainability

Outputs

Participatory Procedures To Strengthen:

• The Systematic Integration Of Evidence, Contextual Knowledge & Experience

• Shared understanding, commitment, ownership, motivation & capacity to advance the micronutrient agenda

• Links with the broader nutrition and health agendas in the country

Processes

Laying the GroundworkModule 1: Clarifying the Problem and the Solution Module 2: Goals and Associated ValuesModule 3: Delivery SystemsModule 4: Hard to Reach PopulationsModule 5: People, Roles and Responsibilities

Program Assessment Guide

Building or Strengthening the Intervention Module 6: Needs, Inputs, Activities and System Changes Module 7: Action Planning Building Support Systems and the Enabling

Environment Module 8: Monitoring, Evaluating and Quality Improvement Module 9: Organizing, Leading and Managing

Program Assessment Guide

Some Distinctive Features of the PAG Approach

Link nutrition interventions to the broader health (and agriculture and other) agendas (Modules 1,2)

Delivery systems as social systems (Module 3) Five universal needs and local program theory (Module 6) Clear definition of roles and responsibilities (Modules 5,7)

and responsibility for follow-up (Module 9) Systematize and stress-test contextual knowledge and

experience (Module 6) Build capacity for intervention planning, assessment and

improvement

Creating Local Program Theory

A Common Example: Training Health Workers to Counsel Mothers

The Goal:TrainWorkers

ImprovedCounseling

ImprovedFeedingPractices

The Common Experience:

TrainWorkers

Little ChangeIn Counseling

Little ChangeIn FeedingPractices

Why?

Training ofHealth workersTo do nutrition counseling

ImprovedFeedingPractices for Young Children

“Input” “DesiredOutcome”

Building a Program Model: Specify Inputs and Desired Outcomes for Each Component

Nutrition Example: BCC IYCF Component


Caretakers ImproveFeedingPractices

Health workersInternalize theKnowledge & skills

Health workersConduct goodQuality counseling

CaretakersUnderstand & Internalize the Knowledge & skills

“Input” “DesiredOutcome”

Building a Program Model: Specify the Causal Pathway

“Causal Pathway”(often assumed and not made explicit)


Caretakers ImproveFeedingPractices

Health workersInternalize theKnowledge & skills

Health workersConduct goodQuality counseling

CaretakersUnderstand & Internalize the Knowledge & skills

“Input”“DesiredOutcome”

Building a Program Impact Model: Specify Inhibiting Factors

• Didactic training methods• Crowded curriculum

• Limited contact time with mothers• Materials not available• Disorganized counseling environments

• Poor comprehension• No hands-on experience

• Limited food diversity • No social support

“Inhibiting Factors”

“Causal Pathway”(often assumed and not made explicit)

• Outdated training materials• Rapid turnover in health workers• Trainers not well-trained

• Delivery Systems as Social Systems (Module 3)

• The Five Needs Tool (Module 5)1. A simpler, more practical way to build a program model2. Embraces but simplifies extensive social and behavioral

theory3. Considers all actors in the delivery system, not just

caretakers and households4. Reveals the inputs, activities and system changes needed for

all links in the delivery chain to function well

The PAG Approach to Creating Local Program Theory

• The Stress-Testing Tool (Module 5)1. Is used when considering input, activities or systems

changes to meet the five needs of various actors2. Reveals hidden assumptions and oversights3. Avoids group think in the planning process

Module 3: Delivery systems as social systems

A

National Regional Clinic Community

D HJ

Functionaries (barriers and enablers)

Household& Caregiver

Significant Others (barriers and enablers)

CB EI

GF

District

Five Universal Needs1. Awareness2. Knowledge, Information, Skill3. Motivation/ Commitment4. Resources5. Support from Others

The PAG Approach to Creating Local Program Theory

Village-levelHealth Volunteer

Role/Responsibility:

Provide mothers with information and education to encourage proper use of Gulazik

Awareness

Information, Knowledge,& Skills

Commitment/Motivation

Resources

Support from others

Inputs, Activities& System Changes

What is required in order for these needs to be met?

What does the village health volunteer need in order to be successful?





Module 6: What does each functionary and significant other NEED in order to fulfill their role/responsibility of the Sprinkles program (on a daily basis) and what inputs, activities and system changes are required to ensure their needs are met?

Systematizing and stress-testing contextual knowledge and experience (Module 6)

Proposers Challengers




Awareness



Resources

Support from others








Identifying Functionaries and Significant Othersin the Delivery and Support Systems

A

National Oblast Clinic Community

D HJ

FunctionariesCaregiver

Significant Others

CB EI

GF

Rayon

Systematizing and stress-testing contextual knowledge and experience (Module 6)

Proposers Challengers




Awareness



Resources

Support from others








Identifying Functionaries and Significant Othersin the Delivery and Support Systems

A

National Oblast Clinic Community

D HJ

FunctionariesCaregiver

Significant Others

CB EI

GF

Rayon

Actions to address barriers and enablers in the Action Plan

Questions for Operations Research

Issues to track in monitoring and QA

Outputs

Experiences from Kyrgyzstan and Bolivia

1. Sponsorship, Responsibility, Commitment, Follow-up2. Timing3. Preparation4. Participants5. Contextuality (selection and sequencing)6. Time constraints7. Facilitation

Before the workshop

During the workshop

Adaptation to Ag-Nutrition Projects

1. The initial focus is a multi-dimensional problem rather than a single solution

2. Use the UNICEF Conceptual Framework as a guide3. Consider food, health, care simultaneously - determinants and

solutions4. The primal need is to ensure that all three are addressed5. Conduct the five needs analysis on the “delivery systems” for

food and health and care interventions

Also,Consider how changes in one domain may affect the other

domains, e.g.,• Women’s time in ag versus time in child care• Women’s versus men’s control of income• Effect of crop choice on men’s versus women’s control• Effect of crop choice on men’s vs women’s labor in ag• Effect of crop choice on sales vs retention for own

consumption• Acceptability of foods for feeding young children• Efficacy of foods for meeting nutritional needs of young

children• Requirements for adopting and growing• Requirements for preparing, using and feeding• Requirements for sustaining over time (seasons and

years)• etc

VHWAgcommunity

household

district

region

national

Ag

Ag

Ag

Hlth

Hlth

Hlth

Simplified Schematic of Delivery Systems for Ag and Health

CareFood Health

Thank You

Five Needs in PAG Module 6 Behavioral Determinants *

1. Awareness of problem, consequences and ways to address it

Knowledge (1) Consequences (4)

2. Specific knowledge, information, skills, cues, reinforcement

Action planning (5), Skills (10), Memory, attention and decision processes (3),

3. Motivation, incentives, commitment, confidence for achieving the goal

Motivation (9), Emotion (2), Beliefs about self-efficacy (9)

4. Resources: time, food, money, materials, tools, equipment, etc

Environmental context and resources (1)

5.Support from others: relatives, friends, co-workers, supervisors, leaders, etc.

Social influences (2), Professional role and identity (1), Emotion (2)

* The determinants listed here are broad categories formed from an initial list of 128 theoretical constructs(21). Numbers in parentheses are the number of intervention techniques (out of the 35 considered in the analysis) judged by experts as likely to be effective in changing the determinant(26).

The Five Needs in Relation to the Broad Categories of Behavioral Determinants

Importance of an Explicit Program Model and Assumptions: Results of a Meta-Analysis

Outcomes for Assessing Effectiveness

TrainingMethod

Teacher’sKnowledge

Lectures and Discussion

10%

+ Demonstrations 30%

+ Practice and Feedback

60%

+ Coaching in the Classroom

95%

Fixsen, D.L., et al., Implementation research: A synthesis of the literature. 2005, National Implementation Research Network: Tampa, FL.



TrainingMethod


Teacher’s Skill


10% 5%

+ Demonstrations 30% 20%


60% 60%


95% 95%




TrainingMethod


Teacher’s Skill

Teacher’s Use of New Practices


10% 5% 0

+ Demonstrations 30% 20% 0


60% 60% 5


95% 95% 95%


Teachers’Knowledge

Teachers’Skill

Teachers’Practices

Lectures &Discussion &Demonstrations

Practice &Feedback

Coaching inClassroom

Simple (Faulty) Program Model

More Complete (Effective) Program Model

Teacher’sPractices


Teachers’Knowledge

Teachers’Skill

Teachers’Practices

Practice &Feedback

Coaching inClassroom

(An Even) More Complete Program Model


StudentLearning

• Disorganized classroom• No materials

• Inattentive (hungry)• Student absenteeism • Teacher absenteeism

• Teacher morale

3.3.2 Improved Nutritional Status

Prevent undernutrition through community-based programs that increase household resilience and are built on the proven effectiveness of community workers, who are often the best change agents to deliver nutrition education. These programs should be based on an analysis of the specific social and behavioral context in each location and promote nutrition messages aimed at improving key evidence-based practices including proper maternal nutrition, exclusive breastfeeding of infants under six months, appropriate introduction of complementary foods, and promotion of adequate water, sanitation, and hygiene practices.

» » Improve diet quality and diversity to prevent nutritional deficiencies and vulnerability to disease caused by low-quality diets. This can be achieved by strengthening agriculture and nutrition linkages through multiple approaches, such as increasing household production of nutritious foods, empowering women with access to income, and educating women about food utilization and nutrition as a means to improving the health and wellbeing of children in the household. We will invest in improving nutrition throughout the value chain, including research to improve the nutrient value of staple food sources and post-harvest preservation and processing techniques that increase the nutrient content of processed staple foods.

3.3.2 Improved Nutritional Status

» Improve delivery of nutrition services by health systems linked to community-based programs. Key nutrition services include community management of acute malnutrition (CMAM) and micronutrient supplementation. The CMAM approach ensures timely detection and referral of cases of severe acute undernutrition in the community to medical treatment by making services available at decentralized treatment points and through community outreach and mobilization. To save lives and reduce morbidity in the short- and medium-term, food-based approaches and prevention programs should be coupled with targeted micronutrient supplementation programs, including vitamin A supplementation for young children and iron folic acid supplementation for pregnant women. Our investments will include strengthening and integrating national and decentralized health and social systems responsible for planning, budgeting, and cost analysis of nutrition service delivery.

The preceding slides were presented at theCORE Group 2010 Fall Meeting

Washington, DC

To see similar presentations, please visit:www.coregroup.org/resources/meetingreports

http://www.coregroup.org/resources/meetingreports

Health & Medicine

Pelletier program assessment_guide